Independent Evaluation of headspace: the National Youth Mental Health Foundation

Evaluation plan

	
headspace: 
National Youth Mental Health Foundation,
The University of Melbourne

SPRC Report 20/08
Social Policy Research Centre
November 2008

For a full list of SPRC Publications see, www.sprc.unsw.edu.au or contact: Publications, SPRC, University of New South Wales, Sydney, NSW, 2052, Australia. Telephone: +61 (2) 9385 7802 Fax: +61 (2) 9385 7838 Email: sprc@unsw.edu.au

ISSN 1446 4179
ISBN 978-0-7334-2725-1

Submitted: July 2008
Published: November 2008

Research team
Social Policy Research Centre, University of New South Wales
Ilan Katz, Kristy Muir, Roger Patulny, Shannon McDermott, Sandra Gendera, Saul Flaxman, and David Abello

Advisors
Social Policy Research Centre, University of New South Wales
Dr Karen Fisher, Senior Research Fellow 
Dr Pooja Sawrikar, Research Associate 
University of New South Wales
Professor Mark Dadds 
University of Sydney and Westmead Hospital
Professor David Bennett
Applied Economics
Professor Peter Abelson 
Deakin University
Professor John Toumbourou

Authors
Kristy Muir, Shannon McDermott, Ilan Katz, Roger Patulny, Saul Flaxman, Sandra Gendera

Contact for follow up 
Dr Kristy Muir, Social Policy Research Centre: ph. 9385 7818; fax: 9385 7838; email: k.muir@unsw.edu.au 
Suggested Citation:
Kristy Muir, Shannon McDermott, Ilan Katz, Roger Patulny, Saul Flaxman, Sandra Gendera, Independent Evaluation of headspace: the National Youth Mental Health Foundation: Evaluation plan, report prepared for headspace: National Youth Mental Health Foundation, The University of Melbourne,  November 2008.

Contents
1	Executive Summary
1.1	Background
1.2	Evaluation objectives
1.3	Methods
1.4	Timeframes
2	Introduction
2.1	Background
2.2	Objectives
2.3	Evaluation of headspace
3	Evaluation Framework
4	Evaluation Hypothesis and Questions
Evaluation hypothesis
Key evaluation questions
5	Evaluation Methods
5.1	Rationale
5.2	Informants
5.3	Overview of methods
5.4	Policy, procedure and document analysis
5.5	Stakeholder surveys and interviews
5.6	Service co-ordination study
5.7	Young people study
5.8	Sustainability instrument
5.9	Economic evaluation
5.10	Meta-analysis/cross-strategy evaluation
6	Ethics
7	Timeframes and Deliverables for Evaluating Each Component
7.1	Key deliverables
8	References

Tables 
Table 1.1: Description of evaluation methods
Table 4.1: Evaluation questions, methods and data sources
Table 5.1: Methods by headspace components
Table 5.2: Policy and document analysis and evaluation objectives
Table 5.3: Interviews and/or surveys with stakeholders and evaluation objectives
Table 5.4: Number interviewed and surveyed for the in-depth young person study
Table 5.5: Young people and service delivery outcomes and evaluation objectives
Table 5.6: Outcomes for the cost-association analysis
Table 7.1: Evaluation components and timeframes for Wave 1 (2008)
Table 7.2: Evaluation components and timeframes for Wave 2 (2009)
Table 7.3: Key deliverables and timeframes

Abbreviations
AGPN		Australian General Practice Network
APS 		Australian Psychological Society
BMRI		Brain and Mind Research Institute
CA 		Community Awareness
CATI		Computer-Assisted Telephone Interview
CoE		Centre of Excellence
CYS		Communities of Youth Service
DOHA		(Australian Government) Department of Health and Ageing	
K-10		Kessler 10
MBS		Medicare Benefits Scheme
MHAGIC	Mental Health Assessment Generation and Information Collection
NO		National Office
PWI		Personal Wellbeing Index
SOFAS  	Social and Occupational Functioning Assessment Scale 
SPET		Service Provider Education and Training
YMHI		Youth Mental Health Initiative
YP		Young People
YSDF		Youth Services Development Fund

1	Executive Summary
1.1	Background
headspace and the University of Melbourne have commissioned the Social Policy Research Centre (SPRC) to evaluate headspace, the National Youth Mental Health Foundation, an initiative funded by the Australian Government Department of Health and Ageing (DOHA). This plan outlines the methodology for the evaluation of headspace.
headspace aims to promote improvements in the mental health, social well-being and economic participation of Australian young people aged 12-25 years. headspace has a particular focus on early identification and intervention for young people at risk of developing mental health problems, and those already showing early signs of mental health problems or associated drug and alcohol problems.
The headspace model involves the establishment of Communities of Youth Services (CYSs), which provide integrated and coordinated psychiatric, medical, allied health and vocational services in a primary care setting. CYSs provide young people at risk of mental health and related disorders with appropriate, youth-friendly support and treatment for mental health, drug and alcohol, and vocational problems.
The CYSs promote early help-seeking and advocate early intervention and the use of evidence-based treatment and care. They are supported by the headspace National Office (NO) and by the work of headspace’s other components – the Centre of Excellence (CoE), Community Awareness (CA), Service Provider Education and Training (SPET) programs, and the headspace Advisory Board.
1.2	Evaluation objectives
The independent evaluation of headspace (2008-2009) will examine the achievements, limitations and future directions of the program.
The main objectives of the evaluation are:
1.	to review the efficiency and effectiveness of headspace as an initiative, and of its individual components (headspace National Office, the Centre of Excellence, Community Awareness, and the Service Provider Education and Training programs);
2.	to assess the efficiency, effectiveness and performance of the CYSs:
o	in improving service integration and coordination;
o	in increasing rates of early detection and early intervention with young people experiencing mental ill-health;
o	in increasing the uptake of services by young people experiencing mental ill-health and associated substance-use disorders;
o	in increasing the use of evidence-based interventions for young people with mental ill-health and associated substance-use disorders;
o	in increasing the economic participation of young people with mental health and related issues, through a range of social recovery strategies; 
o	in establishing a culture of continuous evaluation and service improvement; and
o	in developing and maintaining sustainable business models;
3.	to evaluate the extent to which headspace as an organisation, and through each of its core programs and strategies, has influenced:
o	federal and state/territory government policy and resource commitments towards assisting young people with mental health and related issues;
o	community awareness of youth mental health issues and options for gaining assistance;
o	knowledge of evidence-based approaches to youth mental health and related issues by mental health workers and providers of academic training programs; and
o	the effectiveness and performance of each of the CYSs in meeting their objectives; 
and
4.	to contribute to the ongoing development of headspace and the evolution of the CYS models.

Evaluation hypothesis
The evaluation will assess the hypothesis:
That the headspace initiative has promoted and facilitated improvements in young people’s mental health, social well-being, and participation in education, training and employment, particularly through:
•	its financial and other support for a reformed approach to mental health services for young people which emphasises early intervention;
•	its engagement with young people and its promotion of information about youth mental health and related disorders, and about services available; and
•	its advocacy with all levels of government for reforms to the funding of youth mental health services.

Key evaluation questions
Based on the evaluation objectives and hypothesis, the key evaluation questions are:
•	What impact has headspace had on the mental health, social well-being and economic participation of the young people who access CYSs?
•	What impact has headspace had on community awareness, the youth service sector, and the government response to youth mental health in Australia?
•	How beneficial is headspace as an early intervention strategy for 12-25 year olds? 
o	For which young people is headspace most effective?
o	What aspects of the headspace model are most effective?
•	How effectively have headspace resources been used?
•	What lessons have been learnt on how to efficiently and effectively support young people with mental health and substance-related problems?
The evaluation will examine what works, why it works, under what circumstances and for whom (Pawson and Tilley, 1997).
1.3	Methods
A mixed method longitudinal approach will be used to meet the evaluation objectives, measure changes over time, and provide information that can contribute to program improvements. The evaluation will use qualitative and quantitative research to assess changes to processes and for people. Waves 1 and 2 of the evaluation will be conducted in 2008 and 2009 respectively.
Methods were designed using a program logic model (Cooksy at al., 2001). This model considers the aims and objectives of both the program and the evaluation; the key research questions; the context and focus of the research; the available sources of information; the availability and design of the instruments and data sources; the budget; and the timeframe. The methods are briefly described in Table 1.1.
1.4	Timeframes 
The timeframes for the evaluation have been separated into Wave 1 (to be conducted in 2008) and Wave 2 (to be conducted in 2009). The SPRC will provide quarterly progress reports to the headspace Evaluation Committee, with a major interim report to be delivered in November 2008, and a final report in 2009.
Table 1.1: Description of evaluation methods
Method	Description and explanation 	
Policy, procedure and document analysis	Policies and documents will be analysed in order to clarify the resources, processes and implementation within each of the headspace components. Government policies involving youth mental health and substance use will also be reviewed. The documentation will assist the evaluation to determine the type, nature and extent of the support, the services, and the information, training and communication strategies provided.	
Stakeholder interviews and surveys 
	There will be interviews with key stakeholders (including representatives from headspace NO, the Advisory Board, the CoE, CA, SPET programs, the CYSs, and federal and state/territory governments, as well as young people themselves and their families/carers, mental health service providers, and other service providers in CYS communities), in order to answer the evaluation questions, to track changes over time, and to help clarify why and how outcomes occur. Interviews and surveys will be conducted in both Waves of the evaluation.
Interviews will be either by phone (headspace component and government personnel) or in person (the stakeholders involved in the 10 CYS sites where in-depth evaluation is occurring, including the young people). Surveys will be completed on-line.	
Service 
Co-ordination Study
	A Service Co-ordination Study will be conducted to clarify the nature of the collaboration between services within CYS sites. This will examine the type, level and extent of co-ordination within CYSs, how co-ordination has been improved during the evaluation, and what conditions facilitate or hinder effective and efficient co-ordination. Information for this study will be collected by means of surveys distributed to the CYS staff and other service providers in Waves 1 and 2.	
Program/service delivery dataset
	Progress and other reports provided to headspace NO by each component of the initiative (where available) will be reviewed. These data sources will support the evaluation of each component by collecting the type, nature and extent of support and services, and the information, training and communication strategies provided.	
Young  people study 
•	MHAGIC dataset and young people outcome instruments
•	In-depth young people study
•	Secondary data
	The young people study will compare the experiences of and changes for young people (12-25 years) accessing headspace over time, with general population data on young people. For the population of young people accessing CYS sites, outcomes will be examined using data from the headspace dataset – the Mental Health Generation and Information Collection (MHAGIC). 
The main component of the study will be an in-depth analysis of a sample of 180 young people in ten CYS locations around Australia. Sites selected will represent a range of communities differing socio-economically, culturally and linguistically (including Aboriginal and Torres Strait Islander) and geographically (urban, regional and rural). This component of the evaluation will involve 100 young people in Wave 1, and 100 in Wave 2 (20 of whom will have been among those interviewed in Wave 1, in order to provide some longitudinal case studies). Where the young people consent, their families/carers will also be interviewed and surveyed. Both the young people and their families will be reimbursed with a $40 voucher each.
The in-depth interviews with the young people will focus on their attitudes to and experiences of seeking support through a CYS; their experience of referrals, service quality and service co-ordination; the appropriateness of the support they received, and the barriers and facilitators to taking up and engaging with services; their awareness of available supports; their perceptions of changes in mental health, substance use and well-being; and changes in community and economic participation.
Outcome instruments, such as the Kessler 10 (K10) and Social and Occupational Assessment Functioning Scale (SOFAS), will be used to determine changes in young people’s psychological distress and occupational, social and psychological functioning over time.
Comparisons will be made with the general population of young people using secondary population-based data for similarly placed young people who did not access a CYS. The comparisons will include measures of personal well-being, generalised health, contact with family members and friends, availability of people for support, how young people spend their time, perceptions about time spent alone, levels of generalised trust, and drug and alcohol use (the sources of this information are described in section 5.7, ‘Secondary data’). Data from the Medicare Benefits Scheme (MBS) will also be used to assess changes in early detection and early intervention among young people experiencing mental ill-health in the general population.	
Site observations	Researchers will conduct site observations of the processes occurring within each of the 10 CYS sites where the in-depth evaluations are occurring. This will assist in developing a thorough understanding of the factors that enable sites to maintain and strengthen their effectiveness, or alternatively to hinder it.	
Sustainability instrument
	The sustainability of CYSs will be examined by determining the extent to which certain factors are present, and by consulting with headspace stakeholders about the challenges and facilitators to attaining sustainability. This will be part of the surveys of CYS personnel, and of the interviews with CYS and other key stakeholders.	
Economic evaluation	The economic evaluation of headspace will compare the effectiveness of the various components of the program with the costs of achieving them. The main focus will be on the CYSs where improvements in young people’s mental health, social engagement, vocational functioning, and lessening of drug and alcohol usage will be compared to the costs of the program.	
Meta-analysis	The meta-analysis will assess the program as a whole. It will examine the structure of headspace and how it works, the contribution of each of the components to headspace as a program, and how the components add value to each other.	

2	Introduction
headspace and the University of Melbourne have commissioned the Social Policy Research Centre (SPRC) to evaluate headspace, the National Youth Mental Health Foundation, an initiative funded by the Australian Government Department of Health and Ageing (DOHA). 
This plan describes the evaluation plan and consists of the following sections:
•	Background;
•	Evaluation framework;
•	Evaluation hypothesis and research questions;
•	Methods and instruments;
•	Ethics; and
•	Timeframe and deliverables.
2.1	Background
Mental health disorders accounted for almost half the total disease burden among young people in 2004-05 (Australian Institute of Health and Welfare, 2007). Alcohol and drug use and misuse also tend to commence between the ages of 12-25 years. The age of initiation of use has decreased in recent decades (Degenhardt, Lynskey and Hall, 2000). This is an issue of concern, given that the lower age of initiation is a risk factor for subsequent harmful use and related problems, including mental health problems (Spooner and Hetherington, 2005). 
As reported by the Australian Health Ministers, mental health relates to an individual’s ability to cope and to well-being:
Mental health is a state of emotional and social wellbeing in which the individual can cope with the normal stresses of life and achieve his or her potential. It includes being able to work productively and contribute to community life. Mental health describes the capacity of individuals and groups to interact, inclusively and equitably, with one another and with their environment in ways that promote subjective wellbeing, and optimise opportunities for development and the use of mental abilities. (Australian Health Ministers, 2003: 5)
Mental illness relates to a condition that meets identified criteria for diagnosis, such as the Diagnostic and Statistical Manual of Mental Disorders (American Psychiatric Association, 2000). 
Despite research demonstrating that the first onset of mental disorders usually occurs in childhood or adolescence, treatment typically does not occur until some years later (Kessler et al., 2007; McGorry et al., 2007). From the perspective of young people, a number of barriers exist to seeking help with mental health issues. These include concerns about confidentiality, lack of knowledge of services, discomfort in disclosing health concerns, and inaccessibility and other characteristics of services (NSW Commission for Children and Young People, 2002; Booth et al., 2004). From the perspective of health service providers, barriers to service provision for young people include inadequate time, inflexibility, failure of government to fulfil its responsibilities, poor skills and lack of confidence in working with young people, and poor linkages with other relevant services (Kang et al., 2003; McGorry et al., 2007).
Young people experience critical transition points associated with increased risk and vulnerability. These transition points also represent opportunities for increasing resilience and the development of protective factors against risk. The pathways model emphasises the multi-factorial nature of vulnerability and risk, and the importance of intervening early at transition points, before mental health problems become entrenched (Bronfenbrenner, 1979).
Mental health is one of the Australian Government’s priority health areas. As part of the 2005-2006 ‘Promoting Better Mental Health’ Federal Budget initiative, the Australian Government provided funding of $69 million (to June 2009) to help young people with mental health problems. The cornerstone of this initiative is the establishment of the National Youth Mental Health Foundation: headspace.  The foundation will receive $54 million. 
2.2	Objectives
The broad mission of headspace is to promote improvements in the mental health, social well-being and economic participation of Australian young people aged 12-25 years. headspace has a particular focus on early identification and intervention for young people at risk of developing mental health problems, and for those already showing early signs or the associated drug and alcohol problems. A consortium of agencies is delivering headspace’s components:
i) Communities of Youth Services (CYSs)
ii) Centre of Excellence (CoE)
iii) Service Provider Education and Training Program (SPET)
iv) Community Awareness Program (CA)
headspace is guided by an Advisory Board with a range of expertise including mental health, general practice, drug and alcohol, business and policy expertise, as well as young people. The Advisory Board also has a number of subcommittees including the Evaluation Subcommittee which has responsibility for oversight of the Independent Evaluation of headspace. 
2.3	Evaluation of headspace
headspace is in the initial stages of setting up 30 sites around Australia to implement services. This evaluation is the first independent review of headspace. It is intended to examine the achievements, limitations and future directions of the program. The evaluation will be conducted over an 18-month period and completed in July 2009. 
The main objectives of the evaluation are:
1.	to review the efficiency and effectiveness of headspace as an initiative, and of its individual components (headspace National Office, the Centre of Excellence, Community Awareness, and the Service Provider Education and Training programs);
2.	to assess the efficiency, effectiveness and performance of the CYSs:
o	in improving service integration and coordination;
o	in increasing rates of early detection and early intervention with young people experiencing mental ill-health;
o	in increasing the uptake of services by young people experiencing mental ill-health and associated substance-use disorders;
o	in increasing the use of evidence-based interventions for young people with mental ill-health and associated substance-use disorders;
o	in increasing the economic participation of young people with mental health and related issues, through a range of social recovery strategies; 
o	in establishing a culture of continuous evaluation and service improvement; and
o	in developing and maintaining sustainable business models;
3.	to evaluate the extent to which headspace as an organisation, and through each of its core programs and strategies, has influenced:
o	federal and state/territory government policy and resource commitments towards assisting young people with mental health and related issues;
o	community awareness of youth mental health issues and options for gaining assistance;
o	knowledge of evidence-based approaches to youth mental health and related issues by mental health workers and providers of academic training programs; and
o	the effectiveness and performance of each of the CYSs in meeting their objectives; 
and
4.	to contribute to the ongoing development of headspace and the evolution of the CYS models.
The evaluation will be both formative – contributing to the development and refinement of policy and practice in relation to headspace – and summative – addressing the efficiency and effectiveness of headspace. It will involve all the interlocking components listed in 1. above, and the ways in which they interconnect.
The evaluation framework is designed to address the Statement of Requirement developed by headspace. 
3	Evaluation Framework 
The SPRC’s evaluation of the headspace initiative will use the systematic framework of the evaluation conceptual approach derived from Program Theory (Figure 3.1) (Bickman 1996). The evaluation will also incorporate a cost-effectiveness analysis and a meta-analysis, and will involve a formative process of ongoing feedback. 
The evaluation conceptual approach derived from Program Theory (Figure 3.1) distinguishes four linked stages in the process of human service delivery: inputs, process, outputs and outcomes. It is particularly valuable in attempting to understand the complex interaction of individuals, communities, service providers and government agencies over time. It helps draw attention to the ways in which the program is operationalised and implemented, how this impacts on the delivery of services and programs, and how the consequences of these are eventually expressed in terms of outcomes. Applying the approach to the evaluation of headspace draws attention not only to the outcomes of the strategy, but also to resourcing, participation, planning and implementation.
Figure 3.1 shows the links between the four stages of human service delivery, as set out in the evaluation conceptual approach of Program Theory, for the four interconnecting components of headspace (CoE, CA, SPET, and the CYSs) and the National Office. Although the methods for evaluating each of these components will differ, they all share some similar features. In particular, they all seek to measure the outcomes as set out in the evaluation objectives, and to relate those outcomes to the inputs – the resources allocated to the activities and their design and development – the processes – how activities are undertaken by the different strands of the evaluation – and the outputs – the number of different types of activities undertaken by various funded initiatives. In addition, the context also needs to be taken into account where appropriate. The SPRC evaluation will therefore go beyond the question of what works, and will consider why it works, under what circumstances and for whom (Pawson and Tilley, 1997).  

Figure 3.1: Evaluation conceptual approach

Within this framework a participatory methodology (Balcazar et al., 1998) will be adopted. This will involve stakeholders being consulted and engaged at each stage of the evaluation including design, collection and analysis. This method will give some ownership of the evaluation to stakeholders, and provide early evaluation data ‘feedback’ to the ongoing implementation and improvement of the program.
Meta-analysis/cross-strategy evaluation 
The evaluation will include meta-analysis or meta-evaluation. This approach recognises the ‘importance of accumulating knowledge gained through empirical study into summative statements that can serve as the foundation for future research and contribute to the evidence base’ (Wilson, 2000: 419). Although meta-analysis has traditionally been associated with quantitative research, it is gaining greater acceptance in the social sciences (Ashworth et al., 2004: 195). Meta-analysis allows the effects of differences in program design, processes and implementation to be assessed and the consequences of varying local circumstances and environment to be established. 
The meta-analysis in the headspace evaluation will triangulate the results from the qualitative and quantitative data sources from each component of the evaluation. It will bring together these components and draw conclusions about the program as a whole. The meta-analysis will analyse the links between the separate components of the headspace initiative to ascertain whether and how they add value to each other. It will also draw out the most important themes from the evaluation to provide a better understanding of the factors which underpin positive changes in young people’s mental health, and will therefore be able to comment on the overall logic model of headspace. This analysis will assist in building the evidence-base by identifying the key elements of successful program delivery but also highlighting areas of potentially useful further development and research. 
Cost-effectiveness evaluation
The evaluation will also include a cost-effectiveness component. This analysis will utilise quantitative data to provide clear information about the value added from the initiative. Cost-effectiveness analysis of headspace will help answer how effectively and efficiently headspace has achieved its objectives, by comparing the outcomes of the initiative with the costs. It does this by assessing how effectively resources have been utilised.  Whereas cost-benefit analysis requires dollar figures to be placed on all components of the analysis (costs and benefits); cost-effectiveness analysis allows the assessment of human benefits of the program (such as improved mental health and personal well-being), and is therefore more appropriate for the purposes of human service program evaluation. The aim is to compare the goals of the program with the outcomes of the program, and to decide whether the program is economical in terms of tangible benefits produced, given the amount of money spent (Schmaedick, 1993).
Formative evaluation
A formative evaluation approach will be used throughout the research project to ensure the lessons from the research are communicated back to the NO and to others responsible for the development and ongoing evolution of headspace. Factors that facilitate and hinder program success will be identified and these lessons communicated in an ongoing manner while also balancing the longitudinal research relationships with service and program providers. It is within this formative framework that the ongoing monitoring and evaluation processes will be established, developed and handed over at the end of the evaluation. 
4	Evaluation Hypothesis and Questions
Evaluation hypothesis
The headspace model involves the establishment of CYSs, which provide young people aged 12 to 25 years at risk of mental health and related disorders, with youth-friendly access to appropriate support, treatment and care from health, psychiatric and medical practitioners, as well as from mental health, drug and alcohol, and vocational assistance providers, all of whom work in an integrated and coordinated service delivery framework in a primary care setting. The CYSs promote early help-seeking, and provide early intervention and the use of evidence-based treatment and care.
The evaluation will assess the hypothesis:
That the headspace initiative has promoted and facilitated improvements in young people’s mental health, social well-being and participation in education, training and employment, particularly through:
•	its financial and other support for a reformed approach to mental health services for young people, which emphasises early intervention;
•	its engagement with young people and its promotion of youth mental health and related disorders, and of the availability of services; and
•	its advocacy to all levels of government in favour of reforms to the funding of youth mental health services.
Key evaluation questions
The key evaluation questions are: 
•	What impact has headspace had on the mental health, social well-being and economic participation of the young people who access CYSs?
•	What impact has headspace had on community awareness, the youth service sector, and the government response to youth mental health in Australia?
•	How beneficial is headspace as an early intervention strategy for 12-25-year-olds? 
o	For which young people is headspace most effective?
o	What aspects of the headspace model are most effective?
•	How effectively have headspace resources been used?
•	What lessons have been learnt about how to support young people with mental health and substance-related problems efficiently and effectively?
The evaluation will measure the effectiveness of headspace by examining what has worked, ‘for whom, under what circumstances and how’  (Hohmann, 1999: 87). This perspective moves beyond the usual stance in clinical mental health research, which examines whether a causal relationship exists between an intervention and an outcome. According to Illback et al. (1997), judgements of effectiveness are based on:
•	whether headspace is operating as planned;
•	the extent to which headspace has met its program goals (e.g. improving young people’s mental health, social well-being and economic participation); and 
•	whether headspace strategies are appropriately tailored to young people and meet the satisfaction of clients.
Efficiency refers to whether the program is providing the best results for the lowest costs (Royce et al., 2006: 275). The following table (Table 4.1) lists the key evaluation questions by methods and data sources.

Table 4.1: Evaluation questions, methods and data sources
Headspace objectives/ outcomes	Evaluation questions	Sources	Methods	
			Policy, document and procedure analysis	Stakeholder interviews and surveys	Service coordination survey	Program/ service delivery dataset	Young people outcome instruments	In-depth young people study	Secondary data	Site observation	Sustainability instrument	Cost-effectiveness analysis	Meta-analysis	
Young people 
Improved mental health, social well-being and economic participation of young people	What impact has headspace had on the mental health, social well-being and economic participation of the young people (12-25 years) who access CYSs? 	YP, Carers, CYSs, YP and carer participation		ü			ü	ü					ü	
	How beneficial is headspace as an early intervention strategy for 12-25 year olds? 
•	For which young people is headspace most effective?	CYSs, CA, CoE, SPET, Service Providers, NO, YP, carers, YP and carer participation	ü	ü			ü	ü		ü		ü	ü	
	To what extent have young people with mental health issues been assisted to participate in education, training and employment where necessary?	MHAGIC, Service providers, YP		ü	ü			ü						
Increased numbers of young people accessing services and seeking help 	How have service use and help-seeking by young people in CYS localities changed? Why do young people take up or not take up suggested/referred assistance? Has there been a change in demand for services?	CYSs, YP, carers, CA, Service Providers  		ü		ü	ü	ü	ü 
(MBS)					
Greater capacity for early identification of young people at risk of developing mental health issues	How and to what extent have young people at risk of developing mental health issues been identified in CYS localities?	CYSs, referrals (MHAGIC), service providers	ü	ü		ü	ü		ü
(Govt Health)					
	What lessons have been learnt about how to appropriately, efficiently and effectively support young people with mental health and substance-related problems at an early stage (strengths/gaps in the model)?	CYSs, NO, CA, CoE, SPET,  Consumer and Carer Participation, YP, carers											ü	
Community awareness
Greater community awareness of mental health problems	What is the number, type and focus of communication strategies that have been undertaken to raise awareness early of the importance of young people receiving/seeking help for mental health and drug and alcohol problems? 
To what extent have communication strategies at the local and national level encouraged young people to attend CYSs? How has headspace has promoted the needs of young people?
	CA, CYS, YP, NO, media monitors

	ü	ü				ü	ü
(BMRI CATI II)					
Increased national profile of headspace	To what extent is headspace recognised in the broader community?	Sample of public (CATI I & II), federal and state/territory governments, service providers, NO (media monitors) 
	ü	ü	ü				ü
(BMRI CATI I)					
Service reform
Increased youth-focused care in CYS regions 	How have the focus and delivery of youth mental health services changed in CYS localities? 	CYSs, CoE, service providers, SPET, referral sources (MHAGIC)	ü	ü	ü	ü				ü				
Increase in mental health service coordination in CYS regions	How and to what extent are services integrated, networked and coordinated in CYS localities?	CYSs, Service providers, referral sources (MHAGIC)	ü	ü	ü									
Increased evidence-based practice 	To what extent do CYS models and practices reflect evidence-based literature?	CYSs, CoE, SPET, MHAGIC	ü	ü		ü								
Increased availability of appropriate services for young people with mental health issues 	How appropriate are the services for young people within CYS sites? How appropriate is the assistance young people receive when they access CYSs? What does an episode of care look like for young people (phase, youth-specific)?	CYSs, YP, carers, service providers, Consumer and carer participators, referrals	ü	ü			ü	ü		ü				
	How efficient and effective have partnerships been in increasing capacity for early identification, appropriate responses to young people and additional service opportunities?	CYSs, service providers, YP, carers, SPET	ü	ü	ü			ü						
Increased quality of service	What impact have CYS models had on service capacity and quality?	CYSs, service providers, SPET, CoE, YP	ü	ü		ü		ü						
Government policies support early identification and early intervention for young people with mental health issues	What perceived impact has headspace had on government policy and/or resource commitments to youth mental health? How has headspace promoted the needs of young people to governments?
	Federal and state/territory government perception and policies; NO, media monitors	ü	ü										
CYSs are sustainable business models 	What are the risk and protective factors for CYSs sustainability and are CYSs addressing these? 	CYSs, NO	ü	ü							ü			
headspace provides ‘value for money’ for governments	How effectively have headspace resources been used?
	CYSs, CA, CoE, SPET, NO	ü	ü								ü		
	How have existing government funding streams been utilised within the CYSs?	MHAGIC	ü	ü										
Additional model issues	How and to what extent have headspace components supported CYSs to develop effective models, increase service capacity and community awareness, and provide integrated, coordinated and evidence-based service delivery?	CYSs, NO, CA, CoE, SPET, YP and Carer Participation.											ü	
	To what extent has youth and carer participation been developed and utlised locally and nationally? To what extent have these youth and carer participation models been effective?	National Youth Reference Group, NO, YP	ü	ü										
	What aspects of the model are most/least effective? How do the components add value to each other?	CYSs, NO, CA, CoE, SPET, YP and Carer Participation.											ü	
	What outputs and outcomes have been derived from the expenditure on headspace?	CYSs, YP, CA, CoE, SPET, NO, MHAGIC	ü	ü	ü		ü	ü	ü			ü		
	How has the management, governance and resourcing of headspace affected its roll-out, implementation and outcomes?	CYSs, NO, CA, CoE, SPET, YP and Carer Participation.											ü	
Abbreviations: BMRI - Brain and Mind Research Institute; CA - Community Awareness; CATI - Computer-Assisted Telephone Interview; CoE - Centre of Excellence; CYS - Communities of Youth Services, MHAGIC - Mental Health Assessment Generation and Information Collection; NO - National Office, SPET - Service Provider Education and Training; YP - Young People.	

5	Evaluation Methods
5.1	Rationale
A mixed method longitudinal approach (two Waves) will be used, and data will be triangulated to meet the evaluation objectives, measure changes over time, and provide information that can contribute to program improvements. This evaluation will use qualitative and quantitative research to assess changes to systems as well as individuals. Multiple sources of information will be used to assess program processes and impacts and to attribute causality (Kreger et al., 2007; Green, 2006). 
Methods were designed using program logic model (Cooksy et al., 2001) and incorporating a range of considerations. These include: the aims and objectives of the program, the objectives of the evaluation, the key research questions, the context and focus of the research, sources of information, purpose of data collection, instrument/measure/data source considerations, and methodological and budget and timeframe considerations.
The aims and objectives of headspace regarding community capacity, young people with mental health and related issues, service capacity, training and research and government policy were matched with the evaluation objectives to determine key research questions, evaluation tasks and the methods used. The methods were also selected to match the conceptual framework and to understand, track and describe changes/outcomes at an individual (young people and their families), program (components) and initiative level. 
Timeframe and budget were important determinants of the research design. For example, young people at school were not incorporated as a control group in the study because of the time it would take to include them, e.g. gaining ethics approval from education departments, engaging with a range of schools, recruiting a sample of comparative students, etc. 
5.2	Informants
Research informants include individuals working within and/or affected by each component of headspace. They include:
•	NO/Governance members (Advisory Board; Evaluation Advisory Subcommittee; other Advisory Board Subcommittees)
•	CoE personnel
•	CA program personnel
•	SPET program personnel
•	CYS program personnel
•	Young people 
•	Family/carers
•	Mental health service providers
•	Other service providers and key youth stakeholders (e.g. education providers, general health providers, youth workers, sporting clubs, juvenile justice workers, police)
•	Federal and state/territory government representatives

5.3	Overview of methods
The following measures are proposed to meet the evaluation objectives and answer the key research questions:
•	Policy, procedure and document analysis 
•	Stakeholder interviews and surveys (NO/Governance, CoE, CA, SPET, CYS, young people, family/carers, mental health service providers, other service providers, government)
•	Service co-ordination survey
•	Program/service delivery dataset
•	the headspace dataset, the Mental Health Assessment Generation and Information Collection (MHAGIC)
•	Young people outcomes instruments
•	In-depth young people study
•	Secondary data
•	Site observations
•	Sustainability instrument
•	Cost-effectiveness analysis
•	Meta-analysis

Table 5.1: Methods by headspace components
Method	Component(s)	
Policy, procedure and document analysis 	NO, CoE, CA, SPET, CYSs 	
Stakeholder interviews and surveys 	NO, CoE, CA, SPET, CYSs	
Program/service delivery dataset 	CoE, CA, SPET, CYSs	
Secondary data (BRMI, MBS, etc) 	CA, CYSs 	
Service co-ordination study 	CYSs 	
Young people outcome instruments	CYSs 	
In-depth young people study 	CYSs 	
Site observations 	CYSs 	
Sustainability instrument 	CYSs 	
Cost-effectiveness analysis 	headspace 	
Meta-analysis 	All components 	

5.4	Policy, procedure and document analysis
Policy analysis is an important part of understanding the processes and inputs within each of the headspace components. Policies and agreements within and between each of the components will be examined. Government policies involving youth mental health and substance use will also be reviewed. 
The documents and policies will be examined to understand the processes involved within and between headspace components (Table 5.2). The documents to be analysed will be finalised after consultation with headspace stakeholders, but might include:
•	policies and procedures used within the components
•	service agreements between components
•	progress reports
•	government policies on youth mental health and substance use
•	communication strategies
•	training documents
•	monitoring processes – reports, key performance indicators (KPIs)
This part of the evaluation will assist in reviewing and assessing the efficiency and effectiveness of the headspace NO, CoE, CA program, SPET and CYSs and meet a range of other objectives (see Table 5.2). Progress and other reports provided to headspace NO by each component of the initiative (where available) will be reviewed. Data on service delivery (outputs) will also be collected. These data sources will support the evaluation of each component by collecting the type, nature and extent of support, services, information, training and communication strategies provided. This section of the evaluation is dependent on each of the headspace components providing the relevant documents to SPRC. 
Table 5.2: Policy and document analysis and evaluation objectives
 Evaluation component and objective	
Policy analysis	
Procedure analysis	
Document analysis	

Governance/National Office	

Review and assess the efficiency and effectiveness of headspace NO	
ü	
ü	
ü	

Examine the governance of headspace	
ü	
ü	
ü	

Examine working relationships between NO and components	
ü	
ü	
ü	

Centre of Excellence (CoE)	

Review and assess the efficiency and effectiveness of CoE	
ü	
ü	
ü	

Examine CoE’s gathering, generating and dissemination of information regarding youth mental health and related issues; youth at risk; and effective models of support	

ü	

Examine use of documents and materials disseminated to components	

ü	

Service Provider Education and Training (SPET)	

Review and assess the efficiency and effectiveness of the SPET	
ü	
ü	
ü	

Review working relationship between SPET staff, CoE and other key education and health stakeholders to develop and deliver evidence-based education, training and practice support resources for various audiences.	
ü	
ü	
ü	

Review focus of training program material; recruitment and engagement of stakeholders in training; and knowledge gained by training participants 	
	
ü	
ü	

Community Awareness (CA) Program	

Review communication strategies to examine focus on awareness of youth mental health & related issues; importance of early help-seeking by young people at risk; increasing awareness of services for young people with mental health and substance misuse issue. 	
ü	
	
ü	

Communities of Youth Services (CYS)	

	

Review and assess the efficiency, effectiveness and performance of each CYS individually and collectively in meeting the initiative’s objectives	
ü	
ü	
ü	

Assess service integration and coordination	
ü	
ü	
ü	

Review community awareness of youth mental health & related issues	
ü	
	
ü	

Examine the CYSs’ knowledge and use of evidence-based interventions for young people with mental ill-health and associated substance use disorders	
ü	
ü	
ü	

Examine changes in the service capacity within CYS sites for responding to mental health and related issues through accessible, youth appropriate models	
ü	
ü	
ü	

Assess CYSs’ identification of young people at risk of developing mental health issues 	

ü	

headspace Initiative (cross strategy)	

Review federal and state/territory government policies and resource commitments to early identification of, and early intervention for, young people with mental health issues	
ü	

Review alignment of youth mental health policies across federal and state/territory governments	
ü	

Review the effectiveness and performance of each of the CYSs in meeting their objectives	
ü	

		

5.5	Stakeholder surveys and interviews
Table 5.3 outlines the surveys and/or interviews that will be conducted with key headspace stakeholders. Surveys will be designed to answer the evaluation objectives and to track changes in processes, inputs, outputs and outcomes over time. Semi-structured qualitative interviews will be conducted to understand why and how outcomes occur and/or processes are established. Stakeholders discussed in this section include representatives from all the headspace components, from mental health and other service providers, and from federal and state/territory governments. This section of the evaluation requires the headspace components to provide contact details for key stakeholders.
headspace personnel
Surveys and in-depth interviews will be conducted with management and staff representatives from each of the headspace components. Interviews with appropriate headspace personnel will be conducted either by telephone or face-to-face. Surveys will be administered via email. All personnel identified as appropriate will be asked to complete surveys at both evaluation Waves. Key staff members will also be interviewed. 
Interviews and surveys will focus on policies, plans, procedures, resources, implementation, agreements, types and amounts of support, and perceived outcomes. They will also examine monitoring processes, type, level and effectiveness of support provided to individual components of headspace by National Office, coordination between and within components, working relationships between and within components, engagement with government, consumers and families, and the establishment and implementation of the Collaborative Learning Network. Interviews and surveys with personnel from each component will address each of the evaluation objectives (see Table 5.3).
While many of the areas covered in the surveys and interviews will be common for all components, instruments will also be tailored for each component:
•	National Office
What resources and information are provided by headspace NO? How is headspace governed? How central is the role of the NO to the headspace initiative? What contract management and monitoring systems are in place? How are consumers and carers involved in the development, implementation and ongoing governance and development of headspace? 
•	
Centre of Excellence
What are the types and amounts of resources and information collated/developed and disseminated by the CoE? How is the CoE informing service provider education and training, community awareness and CYSs? How innovative is CoE? 
•	Service Provider Education and Training 
What are the types and amounts of educational resources, information and training provided by the SPET program? How are SPET programs developed? To what extent are evidence-based approaches incorporated? Who are training programs targeting? Who participates in the programs? 
•	Community Awareness
What types and amounts of communication strategies, resources and information are provided by the CA program? Who are the target audiences for communication strategies? How does the program support CYSs with local communication strategies? 
•	Communities of Youth Services
What types and amounts of support, resources, services and information are provided by CYSs? Do CYSs include a wide representation of key stakeholders? What roles do the lead agencies and local partnership organisations play in the delivery of services within the CYS model? How and to what extent are services integrated, networked and coordinated? How effective are service agreements? What factors facilitate strong relationships and partnerships? How efficient and effective have partnerships been in increasing capacity for early identification, appropriate responses to young people and additional service opportunities? What are the strengths/gaps in the CYS service assistance system? How has the service capacity within CYS sites changed as a result of headspace? How have CYSs changed the effectiveness and efficiency of mental health models of support and help provided to young people? How do CYSs utilise the support from NO, CoE, SPET and CA?
Service providers - mental health and other
Select service providers will be surveyed to assess the perceived impact of headspace on their knowledge and use of evidence-based interventions. Those closely involved in headspace will be surveyed/interviewed about their perceptions of the effectiveness and efficiency of the initiative, of each of the components, and of the Collaborative Learning Network; and the levels of service integration and co-ordination, the working relationship with the component(s), and the changes in service capacity.
This component of the evaluation will include conducting surveys with participants of the SPET program. A sample of participants will be assessed prior to and immediately after receiving training to determine changes in knowledge gains and practice and behaviour changes as a result of training and/or materials disseminated, including whether training has assisted service providers to help young people overcome barriers to meaningful participation in society.  
Government representatives
Representatives from each federal and state/territory government Health Department will be interviewed to examine the level of national, state and territory government engagement in headspace. These interviews will also ascertain the effectiveness of the promotion of headspace to health departments, the effect on policy and resource commitments to the early identification of and early intervention for young people with mental health and related issues, and the alignment of youth mental health policies across federal and state/territory governments.

Table 5.3: Interviews and/or surveys with stakeholders and evaluation objectives
	NO/
Gov	CoE	CA	SPE	CYSs	Young people	Family/ carers	Mental health service providers	Other service providers	Govt	
Governance/National Office (NO)											
Review and assess the efficiency and effectiveness of headspace NO	SI	SI	SI	SI	SI			SI			
Examine the roles played by NO in supporting headspace components	SI	SI	SI	SI	SI						
Examine the level of engagement of federal and state/territory governments and NGOs in supporting headspace objectives	SI									I	
Assess NO’s role in coordinating headspace activities 	SI	SI	SI	SI	SI						
Examine the engagement of consumers and carers in supporting the development of headspace, and in promoting the needs of young people with mental health issues and the work of the organisation generally	I					I	I				
Examine working relationships between NO and components	SI	SI	SI	SI	SI						
Assess the effectiveness of the Collaborative Learning Network	I				SI			S	S		
Centre of Excellence (CoE)											
Review and assess the efficiency and effectiveness of CoE	I	SI									
Examine the CoE’s gathering, generating and dissemination of information regarding youth mental health and related issues; youth at risk; and effective models of support		SI	SI	SI	SI	SI	SI				
Examine use of documents and materials disseminated to CYSs and for CA and SPET.		SI	SI	SI	SI						
Service Provider Education and Training (SPET)											
Review and assess the efficiency and effectiveness of the SPET	SI			SI				S	S		
Review working relationship between SPET staff, CoE and other key education and health stakeholders to develop and deliver evidence-based education, training and practice support resources for various audiences.		SI		SI					I		
Examine if training builds provider knowledge of evidence-based medical and psychosocial interventions appropriate for young people with mental health and related substance use disorders				I				S	S		
Review training material, recruitment and engagement of stakeholders and knowledge gained by training participants 				I				S	S		
Community Awareness Program (CA)											
Review and assess the efficiency and effectiveness of the CA Program	SI	SI	SI	SI	SI						
Review communication strategies to examine focus on awareness of youth mental health & related issues; importance of early help-seeking by young people at risk; increasing awareness of services for young people with mental health and substance misuse issue. 			SI			SI	SI				
Communities of Youth Services											
Review and assess the efficiency, effectiveness and performance of each CYS individually and collectively in meeting the initiative’s objectives	I				SI	SI	SI	SI	SI		
Assess service integration and coordination					SI			SI	SI		
Assess whether young people using CYS services have increased awareness of available mental health and well-being supports						SI					
Assess appropriateness of assistance young people receive in accessing services through CYSs					SI	SI					
Assess changes in mental health outcomes and substance use for young people who receive mental health and/or alcohol and other drug services through CYSs						SI					
Examine CYSs’ knowledge and use of evidence-based interventions for young people with mental ill-health and associated substance use disorders		SI			SI						
Examine changes in the service capacity within CYS sites for responding to mental health and related issues through accessible, youth appropriate models					SI			SI	SI		
Examine level of economic participation of young people with mental health and related issues through a range of social recovery strategies						SI	SI				
Examine changes to community connections for young people involved in CYSs						SI	SI				
Establish a culture of continuous evaluation and service improvement											
Assess the sustainability of the CYS models	I				SI						
headspace Initiative (cross strategy)											
Review federal and state/territory government policies and resource commitments to early identification of, and early intervention for, young people with mental health issues, & alignment between governments.										I	
Abbreviations: S – survey; I – interview.	

5.6	Service co-ordination study
A service co-ordination study will be conducted to understand the level of co-operation, co-ordination and collaboration between services within CYS sites. 
The aim of this component of the evaluation is to examine whether there is evidence that sites with higher levels of co-ordination between services produce better outcomes for young people in relation to their mental health, social well-being and participation in education, training and employment. This study aims to understand the type, level and extent of co-ordination within CYSs, how co-ordination has been improved during the evaluation, and what conditions facilitate or hinder effective and efficient co-ordination. 
The survey instrument will be based on elements of effective collaboration (Head, 2006) and will be adapted from instruments used in a number of similar evaluations. The survey will be administered at both Waves of the evaluation across all CYS sites. To decrease the burden on service providers and practitioners, this instrument will be incorporated into a broader survey about headspace. The longitudinal results from the survey will be analysed against results from other outcome components of the evaluation. This study relies on CYSs providing contact details of the services they work with in their communities. 
5.7	Young people study
We will assess outcomes for the population of young people accessing all CYS sites (through surveys, instruments and datasets), and conduct an in-depth analysis of a sample of young people within CYS sites (using additional instruments and qualitative methods). 
Young person headspace dataset (MHAGIC) 
The headspace dataset, the Mental Health Generation and Information Collection (MHAGIC) database, which is completed by CYS personnel for each young person who accesses a CYS site, will be used to measure a range of objectives (Table 5.5). This data will provide longitudinal information on the whole population of young people accessing CYS sites. These data will be analysed in a de-identified form, comparing young people individually (on entering a site, and either on exit or at the final Wave of data collection) and collectively, both within each CYS and between CYS sites in different geographical areas (where sample size allows). 
This data (capturing information on referrals and service use) will be used in collaboration with other sources to review community awareness of youth mental health and related issues. The demographics of young people assessed by CYSs and contained within the dataset will be used to find out whether these reflect the young people known to be at risk of mental health issues. The dataset will also be used to assess the uptake of services by young people within CYS sites, and the changes in mental health, substance use and economic participation for the young people who receive services.
Young people outcome instruments
A number of young people outcome instruments will be used in the evaluation (Table 5.5) to assess psychological distress, occupational social and psychological functioning, well-being and community connectedness. Some of these are already being used within MHAGIC, the headspace dataset.
Young people’s levels of psychological distress will be measured using the Kessler 10 (K10), a self-completion 10-item questionnaire rating psychological distress (based on restlessness, anxiety and depressive symptoms). Changes in young people’s levels of occupational, social and psychological functioning will be measured using the Social and Occupational Functioning Assessment Scale (SOFAS) score. The K10 and the SOFAS will be completed at CYS sites both on entry and on exit. Both of these instruments already form part of MHAGIC. 
These instruments will assist in assessing whether young people at risk of developing mental health issues are being identified within CYS sites, and whether the young people are experiencing mental ill-health and changes in mental health outcomes (at an individual and population level). They are useful instruments to track change between program entry and exit. 
The Personnel Wellbeing Index (PWI) will measure community connections and changes in well-being (overall and for numerous domains) of a sample of young people in ten CYS sites (see the in-depth young person study described below). Their well-being scores will be compared to population data (see below) (Cummins, 2005). 
The data from MHAGIC will be analysed to determine the overall average efficacy of the intervention across the 30 sites. Relationships within each site will be compared to develop an understanding of contextual factors. Statistical analysis will be used to understand any deviations between and within sites, and to establish whether and how differences between sites can be compared. Data from instruments such as the K10 and SOFAS will be compared to population data. The BRMI K10 data (if available) will be compared with the K10 data of young people attending the CYS sites. This will enable a comparison of the psychological distress levels of young people accessing headspace to the levels of a random sample of young people in the community.
Secondary data
Secondary data sources will be used to meet a number of research objectives (Table 5.5). The BMRI CATI-I and II survey data will be crucial to assessing headspace’s community awareness outcomes. 
Where available and comparable, other secondary data sources will be used to compare data for the general population of young people with the outcomes for the young people accessing headspace. For the 180 young people involved in the in-depth young people study (see below), the following areas will be explored and compared with data sources for the general population: Personal Wellbeing Index (PWI); generalised health questions, contact with family members and friends, and availability of people for support (source: ABS General Social Survey, 2006); how young people spend their time, perceptions about time spent alone, and levels of generalised trust (source: ABS Time Use Survey, 2006); and drug and alcohol use (source: AIHW National Drug Strategy Household Survey 2004). 
The Medicare Benefits Scheme (MBS) data will also be used to assess changes in early detection and intervention with the general population of young people experiencing mental ill-health, in order to compare them with the changes among the 180 young people in the in-depth study. This will involve examining changes in the number of items young people use – type of service (group therapy, psychological strategies and psychiatric consultations) by age, gender and location.
In-depth young person study 
An in-depth study of a sample of young people (n=180) within ten CYS sites will also be conducted. Sites selected will represent a range of communities differing socio-economically, culturally and linguistically (including Aboriginal and Torres Strait Islander) and geographically (urban, regional and rural). Site selection will be decided in collaboration with headspace NO. It will partly be determined by the timing of the site implementations and of the evaluation itself. 
In this part of the evaluation interviews and surveys will be conducted with 10 young people from each of the ten sites in each of Waves 1 and 2. This number of interviewees has been selected to ensure a contextual understanding of different sub-groups. 
The 100 interviews in Wave 2 will include 20 of the people interviewed in Wave 1 (Group A; Table 5.4). This will provide a sufficient number of longitudinal, in-depth case studies. The remaining 80 interviewees in Wave 2 will be a new group of young people recently engaged with their local CYS (Group B; Table 5.4). Group B is being consulted to understand how young people’s service experience with the CYS, might have changed since Group A was involved. This is important as the majority of Wave 1 young people (Group A) will no longer be accessing their local CYS site because the a period of intervention lasts approximately three months. 
To ensure longitudinal quantitative outcomes are available for a sizable group of young people involved in the in-depth study, the 80 young people from Group A in Wave 1 who are not being re-interviewed, will be asked to complete an on-line survey in Wave 2. This will provide longitudinal outcome data for 100 of the young people involved in the in-depth study (Table 5.4). 
Young people will be asked to volunteer to participate in this part of the research. They will be reimbursed for their time with a $40 voucher when interviewed and surveyed (Group A and B, Waves 1 and 2) and a $20 voucher when they are only surveyed (Group A, Wave 2). For both Waves, we will endeavour to recruit young people who have completed an ‘episode of care’ with headspace, that is, who have been referred, assessed, supported and exited, and who have therefore been connected with headspace over a period of time.
Table 5.4: Number interviewed and surveyed for the in-depth young person study
		Wave 1	Wave 2	
Group A 	Interview and survey	100	20	
	Survey only	-	80	
Group B	Interview and survey	-	80	
	Survey only	-	-	
Total YP consulted	100	180	

Where these young people consent, their family/carers will also be interviewed and surveyed. This in-depth study will significantly complement the quantitative data collection of the population of young people accessing CYSs. Where the young people consent to the use of identified MHAGIC data from the CYS sites, data from all sources (MHAGIC, and interviews and surveys with young person and with their family members and the service providers) will be triangulated. 
This in-depth study will provide a sound understanding of the experiences of a group of young people in relation to headspace, their service use, and their mental health and substance use. It will focus on young people’s attitudes to and experience with seeking support through CYSs; their experience with referrals, service quality and service co-ordination; the appropriateness of support and the barriers and facilitators to taking up and engaging with services; awareness of available supports; perceived changes in mental health, substance use and well-being; and changes in community and economic participation (Table 5.5 and Table 5.3).
Site observations
While conducting interviews in the 10 CYS sites, SPRC fieldworkers will conduct site observations regarding the processes occurring within each of the locations. This will complement the qualitative interviews and the policy, monitoring and document analysis. It is an important component of the research in regard to developing a thorough understanding of the factors that assist sites to maintain and strengthen, or alternatively hinder, their efficiency and effectiveness. The results can be used to inform future implementation of a more efficient and effective system of service delivery.

Table 5.5: Young people and service delivery outcomes and evaluation objectives
 	Secondary data	Young people outcome instruments*	Young person dataset	In-depth young person study	
Community Awareness Program				
Review the effectiveness of the Community Awareness Program	ü (BMRI CATI)				
Examine community awareness levels regarding the importance of early help-seeking by young people at risk; awareness of services for young people with mental health and substance misuse issue. 	ü (BMRI CATI)				
Early help-seeking by young people at risk; awareness of services for young people with mental health and substance misuse issue. 	ü (BMRI CATI)			ü	
Communities of Youth Services					
Review and assess the efficiency, effectiveness and performance of each CYS individually and collectively in meeting the initiative’s objectives			ü	ü	
Review community awareness of youth mental health & related issues			ü	ü	
Assess whether young people at risk of developing mental health issues are identified within CYS sites	ü (Govt Health)	ü(SOFAS, K10)	ü		
Assess young people’s awareness (those using CYS services) of available mental health and well-being supports			ü	ü	
Assess young people’s perceptions of service quality, integration and coordination 				ü	
Assess uptake of services by young people within CYS sites experiencing mental ill-health and associated substance use disorders	ü (MBS)	ü (stage of onset, K10)	ü	ü	
Assess changes in mental health outcomes and substance use for young people who receive mental health and/or alcohol and other drug services through CYSs		ü (K10, SOFAS, substance use; PWI)	ü	ü	
Examine level of economic participation of young people with mental health and related issues through a range of social recovery strategies			ü	ü	
Examine community connection changes for young people in CYSs	 	ü (PWI, ABS)	ü	ü	
*SOFAS - Social and Occupational Functioning Assessment Scale; K10 - Kessler 10; PWI - Personal Wellbeing Index; BMRI – Brain and Mind Research Institute; CATI – Computer Assisted Telephone Interviewing 	

5.8	Sustainability instrument
Managers within each CYS site will be requested to complete a sustainability instrument during Wave 2 of the fieldwork to assess the sustainability of the CYS models. Factors that encourage sustainability are known within the literature. The sustainability of CYSs will be assessed by determining the extent to which these factors are present. Some of these factors are: continuity of funding sources; stability and diversity of resources; commitment by the local agency network to service delivery; capacity building; ongoing service coordination; and staff development. This component will also include reviewing: the processes, procedures, agreements and policies in place to ensure CYS models are not reliant on individual people; the processes in place for models to adapt and adjust to change; and the risk and protective factors for CYSs and how CYSs plan for these. These questions will be incorporated into existing qualitative and quantitative instruments. Some questions will be asked in both Waves, while others will only be asked in Wave 2 of the evaluation.
5.9	Economic evaluation 
The economic evaluation of the headspace initiative will compare the effectiveness of the various components of the project with the costs of achieving them, using the most appropriate methods available. 
We will measure the quantum of incremental improvements in mental health, social engagement, reduction of drug and alcohol usage, and vocational functioning for young people using headspace and describe the costs of the program. This will provide a picture of the benefits and costs and assist in drawing conclusions about the value of the program.
It may also be possible to do a limited dollar valuation of some improvements, for example, vocational functioning or reduced drug and alcohol use. However, not all improvements can be incorporated into a dollar value (for example, improved social engagement). It will also be difficult to correlate outcomes, and to attribute outcomes to specific parts of the headspace program. During the study, SPRC may examine ways of dealing with these major issues. However at this stage, a limited set of cost-output and cost-outcome relations is the likely output from the study. These are discussed below.
Association between headspace costs and outcomes
Costs
The program logic model of headspace suggests that young people’s outcomes are based on the services provided by CYSs, whose service delivery is supported by the other components of headspace, CA, SPET, CoE and NO. Therefore the funding invested in the CYSs, CA, SPET, CoE and NO will be used to determine the set-up and recurrent cost of the model. 
The costs for CA, SPET, CoE and NO will largely include those incurred by DOHA and, where data is available, any other costs and funding sources.
Costs for the CYSs will be available through the individual CYS business plans and quarterly financial reports. Where the data is available, costs will also include:
•	Funding provided by headspace National from the Youth Services Development Fund (YSDF); 
•	Funding through the Youth Mental Health Initiative (YMHI) specifically provided to support the CYSs (i.e. funding of the YMHI allied health workers); and
•	Use of other available funding streams.

In addition, many CYSs have received additional in-kind support from their consortium members (e.g. use of premises, staffing, etc.) that is not a direct cost, as well as additional revenue (e.g. from community grants, co-payments from clients, etc.). This additional support will not be quantified against the DOHA funding. A preliminary overview of this additional support will be provided to assist a future in-depth economic evaluation of the total cost and effectiveness of the headspace model. These costs will be set out against the outcomes achieved (as identified in the effectiveness analysis). Establishment costs and ongoing costs will be presented separately as well as together.
It will not be possible to quantitatively determine the extent to which the money invested in each component of headspace (CoE, CA, SPET and NO) has contributed to young people’s outcomes. This is because there are no headspace sites without access to support for the various components, which could serve as control groups. Some insight into the value added by each of these components, however, will be obtained through other qualitative and quantitative methods conducted as part of the broader evaluation.
Outcomes
We will undertake a comparison of costs and effects. The effects will be based on comparing outcomes for participants over time and against national benchmarks. Where literature is available, we will also review the dollar benefits associated with the outcomes. The outcomes will include variables that assess the main objectives of headspace – improvements in young people’s mental health problems, social engagement, drug and alcohol problems and vocational participation. 
The key data sources for outcomes will be the SPRC Survey of Young People, the headspace administrative dataset (MHAGIC) and the BMRI CATI II survey (Table 5.6). A range of variables that fit within the four key outcome types will be analysed. The variables used will be those that show statistically significant improvement over time.  
Table 5.6: Outcomes for the cost-association analysis
Outcome Types	Variables	Sources	Comparison	
Mental health status	K10	MHAGIC, CATI II 	Longitudinal & compared to YP population in CYS sites	
	SOFAS	MHAGIC	Longitudinal	
	PWI	SPRC Survey	Longitudinal & compared to population data	
Social engagement	Frequency of contact	SPRC Survey, ABS GSS	Longitudinal & compared to population data	
	Favours	SPRC Survey, ABS GSS	Longitudinal & compared to population data	
	Support in crises	SPRC Survey, ABS GSS	Longitudinal & compared to population data	
Drug and alcohol usage	Frequency of drug & alcohol use	MHAGIC, SPRC Survey, AIHW N’l Drug Survey	Longitudinal & compared to population data	
Vocational functioning	Participation in education, training, employment, caring	SPRC Survey, MHAGIC, CATI II, ABS Census	Longitudinal & compared to population data & YP population in CYS sites	

Analysis
The association between the costs of headspace and the outcomes for young people will involve assessing the money spent on headspace against the improvements in outcomes for young people compared with the general population. The analysis will involve a number of steps:
1.	Outcome variables which show positive and significant change over time will be selected (data will be collected over time from young people participating in headspace from both the SPRC survey and MHAGIC);
2.	Outcomes derived from statistical modeling with appropriate controls will be converted into percentage changes to establish the extent of the effect change over time;
3.	A national benchmark from population data will be obtained for each outcome (national averages will be obtained from population-based data sources and/or the CATI II); 
4.	The gap between the outcome variables and national benchmarks will be compared to establish whether, and by how much, the gap has changed over time;
5.	 A brief review of the literature will be undertaken to determine, where available, the costs associated with poor outcome types, e.g. the costs of an acute psychiatric hospital admission. It is unlikely that associated costs will be found for all outcome variables, most particularly for social engagement outcomes.
This process will describe the cost of the intervention and the outcomes. It will be able to establish whether the intervention is associated with a significant decrease in the gap between headspace participants and the national average across the range of outcome types and whether headspace participants have experienced improved outcomes over time.  Finally, it will use existing literature to postulate the potential costs if these outcomes were negative. 
Cost-output analysis
A basic ‘cost-output’ analysis will also be undertaken to identify the number, range and type of outputs achieved by the CYSs, CoE, SPET, CA and the headspace NO for the funds invested by DOHA. Costs will be divided according to set-up and recurrent funding. Where data is available, these costs will be further disaggregated. This process will assist in demonstrating the efficiency or otherwise of headspace components for the funding invested.
Future economic evaluations of the headspace model
While it will not be possible to complete a comprehensive analysis of either cost-effectiveness or costs and benefits during the course of this current evaluation, SPRC will participate in general discussions about the future economic evaluation of the headspace model and provide advice on aspects such as other key data sources and possible comparators.
This would include working with a broader working party (consisting of headspace Evaluation Advisory Board members and others nominated by headspace NO):
•	to conduct an initial scoping of the impact of headspace on service providers not covered by headspace funding, e.g. State mental health services and hospitals, police and correctional services, Centrelink, or savings in terms of increased national productivity, etc.;
•	to participate in discussions to help identify data that would need to be collected now to inform a broader economic evaluation beyond mid-2009; and
•	to suggest methods for identifying regions, sites or sectors to compare with the headspace initiative.

In summary, SPRC will deliver an analysis that meets the minimum requirement of informing DOHA about the value added by the money invested in headspace in terms of outcomes and/or outputs, and contribute to the identification and development of a feasible long-term economic evaluation that includes a wider array of costs and benefits.
Traditional cost-effectiveness and cost benefit analysis
The two most common methods used to compare cost and effect are cost-effectiveness analysis (CEA) and cost-benefit analysis (CBA). These differ both in terms of their appropriateness for aggregating outcomes, and in terms of the eventual statistics produced, as can be seen from the following definitions from the Center for Substance Abuse Treatment, US Department of Health and Human Services:
Cost effectiveness studies are those which are attempting to analyze the relative efficiency of alternative approaches to improving health. These studies create “indices” which relate defined non-monetary “outcomes” to costs for these alternatives. Generally, only a single outcome measure can be accommodated.
Cost benefit studies differ from cost effectiveness studies only in that outcomes are measured using monetary indices. Cost benefit studies can include multiple and different types of outcomes that can be combined since they are each measured using monetary scales. Some outcomes that are examined in monetary terms in cost benefit studies include crime, victimization, criminal justice expenses, lost work due to illness, and receipt of social welfare benefits (Harwood et al., 2002).
There are problems with methods. A CBA cannot be conducted because of the impracticality of applying monetary estimates to many of the outcomes derived from the headspace evaluation. A CEA would seem to be more promising, because it does not require converting outcomes into monetary units. Instead, it simply compares the costs and outcomes of two or more alternative interventions/programs. However, the headspace evaluation would be unable to utilise this method either, because:
1.	there is no comparable program – the headspace model is a new initiative, which cannot be directly compared to any other health service (since headspace was implemented because of the dearth of early intervention mental health and drug and alcohol services for young people aged 12-25 years); and
2.	there is  no control group – resource and time limitations prevent a comparison of the young people participating in headspace with other young people experiencing similar problems who do not access headspace.
Given the inappropriateness of using either CEA or CBS approaches, SPRC will instead conduct the two-part economic evaluation described above.
5.10	Meta-analysis/cross-strategy evaluation
The meta-analysis will assess the program overall. It will examine the structure of headspace and how it works and the contribution of each of the components to headspace as a whole. Cross-strategy outcomes will be evident in relation to: community awareness of youth mental health issues and options for gaining assistance; the availability and appropriateness of services for young people with mental health issues; knowledge of evidence-based approaches to research into youth mental health related issues; and federal and state/territory government policies and resource commitments to early identification of and intervention with young people with mental illness.  
The meta-analysis will systemically review all the evaluation components and assess the extent to which the program as a whole meets its objectives (measurement and narrative). It will also establish how components add value to each other, identify key factors that facilitate or hinder successful program delivery, draw out factors which underpin both positive and negative change in young people’s mental health, comment on the overall model of headspace, and highlight areas for further program development/evolution and research. 
6	Ethics
Requirements for conducting evaluations involving young people (including ethical approval and consent)
The UNSW has a Code of Research Practice by which the SPRC abides. We also strive to maintain high standards of ethical practice and to respect confidentiality and privacy of research participants in all our research projects.
Ethics approval for this research will be obtained from UNSW Human Research Ethics Committee (HREC). The application will be submitted early during the evaluation process to ensure approval is granted prior to setting up the first stage of fieldwork.
Young people (12-25 years of age) will be invited to participate in the evaluation using an arms-length approach (via a letter of invitation). This letter will be provided to them via CYS personnel. Follow-up letters will be provided if necessary. The letters will contain statements telling the young people how information collected from them will be used, and about the measures taken by the researchers to ensure that their privacy and confidentiality will be maintained. At no stage will any of the participants be identified in the project or in the reporting of results. 
The SPRC will ensure that all participants give informed consent to participating in the research, that participation is voluntary and that participants are entitled to revoke their consent. Consent will be sought from all young people before they participate in the project. As recommended by NSW Commission for Children and Young people, young people will be engaged in the evaluation in a respectful manner (NSW Commission for Children and Young People, Undated).
For young people aged 12-16 years we will also require parental consent (The UNSW HREC requires that young people under 16 years of age gain parental/guardian consent in order to participate in research studies). We will also require consent from the guardian of any young person over 18 years of age who has a legal guardian. Once consent has been attained, suitable times and locations for the interviews and surveys will be arranged. 
Arrangements to conduct interviews will endeavour to ensure that service providers are available following the interviews should they be required. This will ensure that access to support services is available immediately, should distress be experienced by any interviewee. This proposal is cognisant of ethical issues involved in conducting research with vulnerable populations, and all intervention protocols will be adhered to where necessary and appropriate, consistent with the Children and Young Persons (Care and Protection) Act 1998. All researchers on the team will complete the working with children check.
If permission is granted, interviews will be recorded for accuracy and transcribed later. Recordings will be stored in a locked cabinet at the SPRC and in a secure electronic folder which will be accessible only to the research team. All data will remain confidential and will be destroyed after seven years. The participants will be provided with a $40 voucher when interviewed and surveyed, and a $20 voucher when they are only surveyed, to recognise their time and effort. 
The literacy and linguistic needs of participants from non-English speaking backgrounds will be accommodated through the provision of translators and interpreters as required. Where literacy is an issue, all forms can be delivered through sound recordings in English or in the appropriate community languages. Field workers from support organisations will be engaged when necessary. Alternatively, trusted persons or peers may be of assistance in some circumstances (but not as proxy respondents). An easy English version of the information statements and consent forms has been developed and used where appropriate. In addition, the researchers will be sensitive to participants’ needs and requirements relating to gender, cultural issues, disability and sexuality. The researchers have substantial experience conducting research with people with mental ill-health and/or disability.
7	Timeframes and Deliverables for Evaluating Each Component
The timeframes for each evaluation component have been separated into Wave 1 of the evaluation (to be conducted between April 2008 and the Interim Report, November 2008, Table 7.1) and Wave 2 (to be conducted primarily between January-June 2008, Table 7.2). 
Table 7.1: Evaluation components and timeframes for Wave 1 (2008)
Evaluation Components Wave 1	Tasks	Feb	Mar	Apr	May	Jun	Jul	Aug	Sept	Oct	
Project commencement	Implementation, 
ethics, planning	

	x	x	x							
	Evaluation plan	x	x	x							
	Finalise evaluation plan				x						
Policy, procedure and document analysis 	Data collection				x	x	x				
	Analysis					x	x	x			
	Report writing							x	x	x	
Stakeholder interviews & surveys	Finalise & pilot instruments		x	x	x						
	Recruit				x	x					
	Data collection				x	x	x				
	Analysis					x	x	x			
	Report writing						x	x	x	x	
Service coordination study	Finalise & pilot instruments				x						
	Recruit					x	x				
	Data collection					x	x				
	Analysis						x	x	x		
	Report writing								x	x	
Secondary data	Data collection				x	x					
	Data analysis					x	x	x			
	Report writing								x	x	
Young people outcome instruments	Data collection					x	x				
	Analysis							x	x		
	Report writing								x	x	
Meta-analysis	 								x	x	

Table 7.2: Evaluation components and timeframes for Wave 2 (2009)
Evaluation Components Wave 2	Tasks	Jan	Feb	Mar	Apr	May	Jun	
Policy, procedure and document analysis 	Data collection	 	x	x	 	 	 	
	Analysis			x	x			
	Report writing					x	x	
Stakeholder interviews & surveys	Review instruments	x						
	Recruit	x	x	x				
	Data collection		x	x				
	Analysis			x	x	x		
	Report writing				x	x	x	
Service coordination study	Review instruments		x					
	Recruitment list		x	x				
	Data collection			x	x			
	Analysis				x	x		
	Report writing					x	x	
Sustainability instrument	Finalise & pilot instruments		x					
	Data collection			x				
	Analysis				x			
	Report writing					x		
Secondary data	Data analysis		x	x	x			
	Report writing				x	x		
Young people outcome instruments	Data collection			x				
	Analysis				x			
	Report writing				x	x		
Cost effectiveness analysis			x	x	x	x		
Meta-analysis	 	 	 	 	x	x	x	
7.1	Key deliverables
As the Independent Evaluator, SPRC will provide quarterly interim reports to the headspace Evaluation Committee, with a major interim report delivered by November 2008, and a final report by June 2009. The key deliverables and corresponding timeframes as specified by headspace are in Table 7.3.
Table 7.3: Key deliverables and timeframes
Key deliverable/method	Timeframe	
Sign contract	April 2008	
Evaluation Plan 	Feb-May 2008	
Finalisation of evaluation framework	May 2008	
Pilot instruments	May-Jun 2008	
Stakeholder interviews and surveys	July-Sept 2008 	
Service Co-ordination Study	July - Sept 2008 	
Progress report 1	3 Sept 2008	
Secondary Data	Jun - Sept 2008 	
Young people outcome instruments	July - Sept 2008	
Analysis	Sept - Dec 2008	
Meta-analysis	Nov - Dec 2008 	
Draft major interim report	19 Nov 2008	
Major interim report	17 December 2008	
Policy, procedure and document analysis	Feb - Jun 2009	
Progress report 3	9 Feb 3009	
Stakeholder interviews and surveys	Jan - Mar 2009	
Service Co-ordination Study	Feb - Mar 2009	
Sustainability Instrument	Feb - Mar 2009	
Secondary Data	Feb - Mar 2009	
Progress report 4	1 April 2009	
Young people outcome instruments	Mar - Apr 2009	
Cost effectiveness analysis	Mar - Apr 2009	
Meta analysis	Apr - Jun 2009	
Draft Final Report	31 May 2009	
Final report	Mid June 2009	

8	References
American Psychiatric Association. (2000). Diagnostic and Statistical Manual of Mental Disorders. Fourth Edition Text Revision (DSM-IV-TR). Washington, DC: American Psychiatric Association.
Ashworth, K., Cebulla, A., Greenberg, D., and Walker, R. (2004). Meta-evaluation: Discovering what works best in welfare provision. Evaluation, 10 (2), 193-216.
Australian Health Ministers. (2003). National Mental Health Plan 2003–2008. Canberra: Australian Government.
Australian Institute of Health and Welfare. (2007). Young Australians: Their Health and Wellbeing 2007 (AIHW cat. no. PHE 87). Canberra: Author.
Balcazar, F.E., Keys, C., Kaplan, D. L., & Suarez-Balcazar, Y. (1998). Participatory action research and people with disabilities: Principles and challenges. Canadian Journal of Rehabilitation, 12, 105-112.
Bickman, L. (1996). The Application of Program Theory to the Evaluation of a Managed Mental Health Care System’, Evaluation and Program Planning, 19 (2): 111-119.
Booth, M. L., Bernard, D., Quine, S., Kang, M. S., Usherwood, T., Alperstein, G., et al. (2004). Access to health care among Australian adolescents young people's perspectives and their sociodemographic distribution. Journal of Adolescent Health, 34(1), 97-103.
Bronfenbrenner, U. (1979), The Ecology of Human Development, Cambridge Harvard University Press.
Cooksy, L. J., Gill, P., & Kelly, P. A. (2001). The program logic model as an integrative framework for a multimethod evaluation. Evaluation and Program Planning, 24(2), 119-128.
Cummins, R. (2005), Australian Unity Wellbeing Index, Survey 14, Report 14.0, Part B: Appended Tables ‘The Wellbeing of Australians – Personal Relationships’, Australian Centre on Quality of Life, Melbourne, http://www.deakin.edu.au/research/acqol/index_wellbeing/index.htm, accessed 21 July 2006.
Degenhardt, L., Lynskey, M., & Hall, W. (2000). Cohort trends in the age of initiation of drug use in Australia. Australian and New Zealand Journal of Public Health, 24(4), 421-426.
Green, L. W. (2006). Public health asks of systems science: To advance our evidence-based practice, can you help us get more practice-based evidence? American Journal of Public Health, 96(3), 406-409.
Harwood HJ, Malhotra D, Villarivera, C, Liu C, Chong U, Gilani, J (2002) ‘Cost effectiveness and cost benefit analysis of substance abuse treatment: an annotated bibliography,’ National Evaluation Data Services, Center for Substance Abuse Treatment, US Department of Health and Human Services.
Head, Brian (2006), Effective Collaboration, Australian Research Alliance for Children and Youth, Canberra.
Hohmann, A. A. (1999), 'A contextual model for clinical mental health effectiveness research', Mental Health Services Research, 1 (2),  83-91.
Illback, R. J., J. Kalafat and D. Sanders (1997). 'Evaluating integrated service programs', in R. J. Illback, C. T. Cobb & H. M. Joseph (eds.), Integrated Services for Children and Families: Opportunities for Psychological Practice (pp. 323-346), American Psychological Association, Washington D.C.
Kang, M., Bernard, D., Booth, M., Quine, S., Alperstein, G., Usherwood, T., et al. (2003). Access to primary health care for Australian young people: service provider perspectives. British Journal of General Practice, 53(497), 947-952.
Kessler, R. C., Amminger, G. P., Aguilar-Gaxiola, S., Alonso, J., Lee, S., & Ustun, T. B. (2007). Age of onset of mental disorders: a review of recent literature. Current Opinion in Psychiatry, 20(4), 359-364.
Kreger, M., Brindis, C. D., Manuel, D. M., & Sassoubre, L. (2007). Lessons learned in systems change initiatives: benchmarks and indicators. American Journal of Community Psychology, 39(3-4), 301-320.
McGorry, P. D., Purcell, R., Hickie, I. B., Jorm, A. F. (2007). Investing in youth mental health is a best buy, MJA, 197 (7 Suppl): S5-S7. 
NSW Commission for Children & Young People. (2002). Children and young people speak about getting help. Sydney: Author.
NSW Commission for Children & Young People. (Undated). Research and resources about participation. Sydney: Author.
Pawson, R., & Tilley, N. (1997). Realistic Evaluation. London: Sage.
Royce, D., B. Thyer, D. Padgett and T. Logan (2006), Program evaluation: an introduction (Fourth ed.), Thompson Learning Australia, Southbank.
Schmaedick, G.L. (1993), ‘Introduction: Do Nonprofit Organizations Need Cost-Effectiveness Analysis?, in Cost-Effectiveness in the Non-profit Sector: Methods and Examples from Leading Organizations, G.L.Schmaedick (ed), Quorum Books, Westport, CT
Spooner, C., & Hetherington, K. (2005). Social determinants of drug use (NDARC Tech Report No. 228). Sydney: National Drug and Alcohol Research Centre.
Wilson, D.B. (2000). Meta-analyses in alcohol and other drug abuse treatment research. Addiction, 95(suppl.3), 419-438.