Indicator 16.6.2 Proportion of population satisfied with their last experience of public services

Indicator 16.6.2: Proportion of population satisfied with their last experience of public services

Target 16.6: Develop effective, accountable and transparent institutions at all levels

UNDP Oslo Governance Centre

Definition:

This indicator measures levels of public satisfaction with people’s last experience with public services, in the three service areas of healthcare, education and government services (i.e. services to obtain government-issued identification documents and services for the civil registration of life events such as births, marriages and deaths)^[3]. This is a survey-based indicator which emphasizes citizens’ experiences over general perceptions, with an eye on measuring the availability and quality of services as they were actually delivered to survey respondents.

Respondents are asked to reflect on their last experience with each service, and to provide a rating on five ‘attributes’, or service-specific standards, of healthcare, education and government services (such as access, affordability, quality of facilities, etc.). A final question asks respondents for their overall satisfaction level with each service.

It is recommended that survey results, at a minimum, be disaggregated by sex, income and place of residence (urban/rural, administrative regions). To the extent possible, all efforts should be made to also disaggregate results by disability status and by ‘nationally relevant population groups’.

A detailed questionnaire and implementation manual to produce the indicator is defined in the SDG 16 Survey Initiative[4]: The questions for 16.6.2 on healthcare, education and government services can be inserted into existing surveys, using these surveys’ additional batteries on demographics for subsequent disaggregation of results. This modular ‘add-on’ technique also allows for the cross-tabulation of satisfaction levels with other socioeconomic variables found in the larger survey, such as the health conditions of the respondent. This enables a more comprehensive analysis of disparities in the provision of services, and helps to pinpoint specific factors that influence satisfaction levels.

Concepts:

• Public services: As stated by the United Nations High Commissioner for Human Rights, “States are responsible for delivering a variety of services to their populations, including education, health and social welfare services. The provision of these services is essential to the protection of human rights such as the right to housing, health, education and food. The role of the public sector as service provider or regulator of the private provision of services is crucial for the realization of all human rights, particularly social and economic rights.”^[5]

While several definitions of ‘public services’ exist, they tend to have in common a focus on ‘common interest' and on ‘government responsibility’. For instance, the European Commission defines such services as “Services that public authorities of the Member States clarify as being of general interest and, therefore, subject to specific public service obligations.”^[6] Similarly, the African Charter on Values and Principles of Public Service and Administration (African Union, 2011) defines a public service as “Any service or public-interest activity that is under the authority of the government administration”.

• Public services ‘of general interest’: The methodology for SDG 16.6.2 carefully defines the scope of healthcare and education services to ensure that the focus is placed on services that are truly of general interest. In the case of healthcare services, for instance, preventive and primary healthcare services can be said to be truly ‘of general interest’: these services are relevant to everyone and they are most commonly found in both urban and rural areas. This might not be the case for hospitals that provide tertiary care, and as such hospital and specialist care is excluded from the questions on healthcare services. Likewise, in the case of education services, primary and lower secondary education services can be said to be truly ‘of general interest’, given their universality. University education, however, is excluded from the questions on education services.
• ‘Last experience’ of public services in the past 12 months: Indicator 16.6.2 focuses on respondents’ ‘last experience of public services’, and specifies a reference period of “the past 12 months” to avoid telescoping effects and to minimize memory bias effects. This means that only respondents who will have used healthcare, education and government services in the past 12 months will proceed to answer the survey questions.
• Service-specific standards – or ‘attributes’: The United Nations High Commissioner for Human Rights explains that “A human rights-based approach to public services is integral to the design, delivery, implementation and monitoring of all public service provision. Firstly, the normative human rights framework provides an important legal yardstick for measuring how well public service is designed and delivered and whether the benefits reach rights-holders”^[7]. For instance, the Committee on Economic, Social and Cultural Rights specifies that “The availability, accessibility, acceptability and quality of health-related services should be facilitated and controlled by States. This duty extends to a variety of health-related services ranging from controlling the spread of infectious diseases to ensuring maternal health and adequate facilities for children.”^[8] Similarly, with respect to education services, the same Committee underlines that “States should adopt a human rights approach to ensure that [education services are] of an adequate standard and do not exclude any child on the basis of race, religion, geographical location or any other defining characteristic.”^[9]
• Healthcare services: The questions on healthcare services focus on respondents’ experiences (or that of a child in their household who needed treatment and was accompanied by the respondent) with primary healthcare services (over the past 12 months) – that is, basic health care services provided by a government/public health clinic, or covered by a public health system. It can include health care services provided by private institutions, as long as such services are provided at reduced (or no) cost to beneficiaries, under a public health system. Respondents are specifically asked not to include in their answers any experience they might have had with hospital or specialist medical care services (for example, if they had a surgery), or with dental care and teeth exams (because in many countries, dental care is not covered by publicly funded healthcare systems). Attributes-based questions on healthcare services focus on 1) Accessibility (related to geographic proximity, delay in getting appointment, waiting time to see doctor on day of appointment); 2) Affordability; 3) Quality of facilities; 4) Equal treatment for everyone; and 5) Courtesy and treatment (attitude of healthcare staff).
• Education services: The questions on education services focuses on respondents’ experience with the public school system over the past 12 months, that is, if there are children in their household whose age falls within the age range spanning primary and secondary education in the country. Public schools are defined as “those for which no private tuition fees or major payments must be paid by the parent or guardian of the child who is attending the school; they are state-funded schools.” Respondents are asked to respond separately for primary and secondary schools if children in their household attend school at different levels. Attributes-based questions on education services focus on 1) Accessibility (with a focus on geographic proximity); 2) Affordability; 3) Quality of facilities; 4) Equal treatment for everyone; and 5) Effective delivery of service (Quality of teaching).
• Government services: The battery on government services focuses exclusively on two types of government services: 1) Services to obtain government-issued identification documents (such as national identity cards, passports, driver’s licenses and voter’s cards) and 2) services for the civil registration of life events such as births, marriages and deaths. This particular focus on these two types of services arises from the high frequency of use of these services. Attributes-based questions on government services focus on 1) Accessibility; 2) Affordability; 3) Equal treatment for everyone; 4) Effective delivery of service (delivery process is simple and easy to understand); and 5) Timeliness.

Selection of relevant disaggregation dimensions

• Relevant international legal frameworks: Indicator 16.6.2 aims to provide a better understanding of how access to services and the quality of services differ across localities and across various demographic groups. This aim is supported by international human rights law:
• Article 25 (c) of the International Covenant on Civil and Political Rights provides for the right to equal access to public service. In its report on the role of the public services as an essential component in the promotion and protection of human rights, the United Nations High Commissioner for Human Rights reminds that “States must bear in mind that there are demographic groups in every society that may be disadvantaged in their access to public services, namely women, children, migrants, persons with disabilities, indigenous persons and older persons. States need to ensure that the human rights of these groups are not undermined and that they receive adequate public services.”^[10] The High Commissioner also calls attention to the fact that “Poverty acts as a major barrier in relation to public services.”
• The obligations to ensure equality and non-discrimination are recognized in article 2 of the Universal Declaration of Human Rights and are encountered in many United Nations human rights instruments, such as the International Covenant on Civil and Political Rights (arts. 2 and 26), the International Covenant on Economic, Social and Cultural Rights (art. 2 (2)), the Convention on the Rights of the Child (art. 2), the International Convention on the Protection of the Rights of All Migrant Workers and Members of Their Families (art. 7) and the Convention on the Rights of Persons with Disabilities (art. 5). In terms of public services, this means that States have an immediate obligation to ensure that deliberate, targeted measures are put into place to secure substantive equality and that all individuals have an equal opportunity to enjoy their right to access public services.
• Empirical analysis: Statistical analysis of available datasets on citizen satisfaction with healthcare and education services^[11] shows that the demographic variables that are most strongly correlated with satisfaction with healthcare and education services are (1) income (by far the strongest determinant of satisfaction levels), (2) sex, and (3) place of residence (rural/urban). There is no statistically significant association between the age of respondents and satisfaction levels.

Percent (%)

This indicator needs to be measured on the basis of data collected by National Statistical Offices (NSOs) through official household surveys.

National Statistical Offices

Recommended set of complementary questions to address selection 16.6.2 bias towards ‘users’ of public services

• Since SDG 16.6.2 refers to people’s ‘last experience’ with public services, the indicator needs to focus on user experiences rather than on non-user perceptions. The experience of users is important, but it is equally important to understand the experiences and perceptions of those who turn elsewhere for services, or who do not access services altogether.
• For each service area, NSOs are therefore strongly encouraged to administer three complementary questions (see Methodology section) prior to the two ‘priority questions’ to be used for global 16.6.2 reporting. These additional questions will help capture the experience of both users and non-users of public services. They will help identify which population sub-groups who needed healthcare, education and government services did not access the services they needed, and what barriers prevented them from doing so. While the information generated by these additional questions is critical for policymakers to design service provision programmes that ‘leave no one behind’, it is left to the discretion of each country to integrate them or not, as some may already be collecting similar information through existing surveys.

Otherwise, the selection bias inherent in SDG 16.6.2, with its focus on users, can result in mismeasurement due to underlying inequalities in the propensity of various groups to interact with state institutions. In other words, a focus on ‘the last experience with public services’ implicitly means that this indicator includes only those respondents who were privileged enough to access public services in the past year. This means that those (such as ethnic minorities, migrants, the elderly, undocumented workers) who have not been able – or willing – to access the healthcare, education or government services they needed in the past 12 months, often as a consequence of multiple social and economic barriers arising from overlapping forms of marginalization will be undercounted by this indicator. There is a risk therefore that overall satisfaction levels reported on 16.6.2 will over-represent the experience of more privileged groups for whom access to public services is easier, because they have the financial, logistical and intellectual means to do so, and they trust that it is in their interest to do so.

Reporting on SDG 16.6.2 should be done separately for each of the three service areas. (NB: questions on education may refer to either primary or secondary education – and separate computation of results is recommended for the two levels, resulting in de facto four service areas). Computation involves the computation and reporting of the following three estimates, for each service area:

1. The share of respondents who responded positively (i.e. ‘strongly agree ‘ or ‘agree’) to each of the five attributes questions;
2. The simple average of positive responses for the five attribute questions combined; and
3. The share of respondents who say they are satisfied (i.e. those who responded ‘very satisfied’ or ‘satisfied’) in the overall satisfaction question.

For instance:

*Note: It is important for NSOs to clearly report, for each question, the number of respondents who selected “don’t know” (DK), “not applicable” (NA) or “refuse to answer” (RA), and to exclude such respondents from the calculation of shares of positive responses. For instance, if 65 respondents out of 1000 respondents responded DK, NA or RA on the first attribute-based question, the share of positive responses for this attribute will be calculated out of a total of 935 respondents, and the reporting sheet will indicate that for this particular question, 65 respondents responded DK/NA/RA.

While national-level reporting should cover all three estimates described above, global reporting on SDG indicator 16.6.2 will focus on the last two estimates (i.e. the average share of positive responses across the five attribute questions; and the share of respondents who say they are satisfied in the overall satisfaction question). Additionally, global reporting will also consider the share of positive responses of the five service attributes by the share of people who are satisfied for each of the four service areas (i.e.., primary and secondary education, healthcare, and government services).

Answer scales:

• To ensure the consistency of measurement in an international context, a standardised approach to response format is required. Available evidence from piloting and other NSO experiences suggests that a four-point Likert-scale with verbal scale anchors is preferable over the alternatives. A four-point scale offers the optimal range of response options for the concepts at hand, in terms of capturing as much meaningful variation between responses as there exists, while remaining understandable for respondents who are not very numerate or literate. Piloting experiences have revealed that offering too few response options (such as a ‘yes/no’ binary response format) would not reveal much variation and might even frustrate some respondents, who might feel their satisfaction level cannot be accurately expressed. Furthermore, the Guidelines on Measuring Subjective Well-Being (OECD, 2013) caution against using “agree/disagree, true/false, and yes/no response formats in the measurement of subjective well-being due to the heightened risk of acquiescence and socially desirable responding”. Meanwhile, piloting experiences have shown that respondents would be equally burdened by too many response categories (such a 7- or 10-point scale), especially if the categories are too close to distinguish between them cognitively.
• There are different schools of thought on whether an odd or even number of categories is best when using Likert scales. While taking away the middle category forces respondents to voice a positive or negative opinion, and some respondents might find this approach frustrating, several NSOs in developing country contexts favor a Likert scale without a neutral value (such as “neither satisfied nor dissatisfied”). Their preference is motivated by their long-standing survey experience which has shown that when a neutral value is provided, a large proportion (often a majority) of respondents will refrain from expressing their opinion ‘hiding’ behind this middle-point.
• The survey methodology for 16.6.2 therefore uses a 4-point bipolar Likert scale for all questions (for internal consistency), with the following scale labels: “strongly agree, agree, disagree, strongly disagree” for attributes-based questions, and “very satisfied, satisfied, dissatisfied, very dissatisfied” for overall satisfaction questions. “Don’t know” and “refuse to answer” options are also available, but should not be read out loud, so as to not provide an easy way for respondents to disengage from the subjects of the various questions. When respondents say they “don’t know”, enumerators should repeat the question and simply ask them to provide their best guess. The “don’t know” and “refuse to answer” options should be used only as a last resort.

2023-03-31

United Nations Development Programme (UNDP)

SDG indicator 16.6.2, measured from citizen surveys, is an important complement to other SDG indicators assessing various aspects of public service provision that draw from administrative sources, such as SDG 3.8.1 on coverage of essential health services^[1] and SDG 4.a.1 on school facilities^[2]. While these indicators focus on similar attributes as those measured by SDG 16.6.2, such as ‘accessibility’ and ‘quality of facilities’, they may not reflect people’s actual experience of education facilities or healthcare services due to the methodological challenges of collecting quality data from administrative sources.

Amongst SDG indicators assessing various aspects of public service provision, indicator 1.4.1, which measures the “proportion of population living in households with access to basic services” has particular relevance to indicator 16.6.2:

• Indicator 1.4.1 measures ‘Access to Basic Health Care Services’ by drawing on readily available data reported on SDG indicator 3.7.1 on access to reproductive health (Proportion of women of reproductive age (aged 15-49 years) who have their need for family planning satisfied with modern methods). Indicator 16.6.2 therefore provides important additional information by (1) broadening the scope of measurement from reproductive health to ‘basic healthcare services’ as internationally defined, and (2) by assessing five key attributes of healthcare service provision not assessed by 1.4.1, namely access, affordability, quality of facilities, equal treatment for everyone and doctor’s attitude, and (3) by using survey data to measure people’s satisfaction with healthcare services based on their last experience.

• Indicator 1.4.1 also measures ‘Access to Basic Education’ by drawing on readily available data reported on SDG indicator 4.1.1 on educational achievements (Percentage of children/young people: (a) in grades 2/3; (b) at the end of primary; and (c) at the end of lower secondary achieving at least a minimum proficiency level in (i) reading and (ii) mathematics). Indicator 16.6.2 therefore provides important additional information by (1) assessing four key attributes of education service provision not assessed by 1.4.1, namely access, affordability, quality of facilities and equal treatment for everyone, and (2) by using survey data (SDG 4.1.1 uses test scores) to measure people’s satisfaction with education services based on their first-hand experience with such services.

Indicator 16.6.2 can also be used to complement SDG target 10.2 on the promotion of the “social, economic and political inclusion of all, irrespective of age, sex, disability, race, ethnicity, origin, religion or economic or other status”, which only has one indicator measuring economic exclusion (SDG 10.2.1 – Proportion of people living below 50 per cent of median income, by age, sex and persons with disabilities). Indicator 16.6.2 therefore provides important additional information to measure progress against this target by providing data on social inclusion.

Similarly, 16.6.2 can also be used to complement SDG target 10.3 on “Ensuring equal opportunity and reduce inequalities of outcome, including by eliminating discriminatory laws, policies and practices and promoting appropriate legislation, policies and action in this regard”, which only has one indicator measuring felt discrimination on various grounds (SDG 10.3.1 Proportion of the population reporting having personally felt discriminated against or harassed within the previous 12 months on the basis of a ground of discrimination prohibited under international human rights law). Thus indicator 16.6.2 provides important additional information to measure progress against this target by helping to identify in which service area the incidence of discrimination is highest.

Finally, SDG 16.6.2, with its focus on ‘accessibility’, ‘equal treatment’ and other important attributes of public services, provides important complementary information to analyze results on SDG 16.5.1 on the ‘Proportion of persons who had at least one contact with a public official and who paid a bribe to a public official, or were asked for a bribe by those public officials, during the previous 12 months’. In other words, people may resort to bribery when the quality of public service provision is too poor, as revealed by SDG 16.6.2.

3