'A Literature Review: Examining the Literature on How to Measure the 'Successful Outcomes': Quality, Effectiveness and Efficiency of Legal Assistance Services'
127 Pages Posted: 19 Feb 2014
Date Written: February 2012
This is a literature review for the Attorney-General’s Department. The brief was as follows.
This literature review will examine research, studies, reports, reviews and evaluation and other material both nationally and internationally around legal assistance service evaluations on the following:
This Literature Review has examined a significant number of research, reports, evaluations, reviews, academic writing, studies and submissions. Some of the key lessons that these documents reveal are detailed in this Executive Summary. Some conclusions and their basis are summarized in the section entitled, 'Conclusion – An Overview of the Findings of this Literature Review'.
Some documents were provided directly to the researcher and to the Attorney-General’s Department by the Legal Assistance Sector but have not been included in this Literature Review as they were outside its scope. However, many are useful and are discussed in this Literature Review.
This Literature Review highlights that legal assistance work is not only complex but that it is also complicated. Forty-seven international studies and ninety-one national studies were considered in the course of the conduct of this literature review.
The selection criteria for determining the 'usefulness' of the reports/reviews/evaluations/ studies was as follows:
•Written in the last decade.
•The Document/Study examines outcome, quality, effectiveness and efficiency or a combination of these considerations.
•The study sets out a clear question to be answered and the methodological approach was relevant to being able to answer the question asked.
•The method for extracting information or data was effective and relevant to the information sought.
•The questions asked of participants in the instruments used were relevant.
•The data collected was sufficiently clear in illiciting the information sought.
•Given the complicated and complex nature and diversity of the legal services and the clients served, the methodology was likely to reveal the reasons behind the responses or approach that the service adopted in terms of the considerations listed above.
•A number of approaches were taken to verify, complement and unpack the reasons for the answer and included a blend of quantitative and qualitative data rather than reliance on quantitative data or one method.
•The approach taken is relevant and of assistance in the context of the NPA and the Attorney-General’s ‘Strategic Framework on Access to Justice in the Federal Civil Justice System’ , the COAG Reform Agenda and to social inclusion and Indigenous disadvantage.
Significant difficulties are identified in much of the domestic and international literature in the measurement of outcome/results, quality, efficiency and effectiveness.
The literature domestically and internationally, identifies the lack of a common language with which to articulate results, the lack of a framework in which to capture them and the difficulties in being able to measure and prove success. Where such results based measurement exists it will often need to be descriptive, subjective and there is a risk that cannot be avoided, of its being anecdotal and vague.
Each program must be first understood to be able to inform how to identify and define outcomes and measure these and ensure adaptive learning and adaptive management processes rather than these being fixed and remote from the realities of practice.
Any approach must be able to adapt and incorporate changing realities and demands on the services that are being examined.
There is no one way which can make it easy to achieve a successful outcome. Good practice informed by good training, cultural awareness, sensitivity, adaptability and flexibility are key factors in effectively reaching and targeting vulnerable and disadvantaged groups. Legal assistance services operate at different levels. Within a legal assistance service different objectives and intentions can sit behind each program. Therefore, they cannot be measured as a 'lump' without first understanding the very nature, diverse ways of engaging that are required to target different client groups, complexity, layers and imperative and funding requirements that drive each of the many parts.
Keywords: measuring effectiveness, efficiency of legal services, access to justice, measuring impact of services, measuring quality legal services, indigenous legal services, community legal services, legal aid, family violence intervention services, legal assistance services
JEL Classification: D63, K49, D61, G38, H53
Suggested Citation: Suggested Citation