Appendix D: Notes on Data Completeness, Reliability and Methodology

Performance measurement is undertaken to assess and communicate progress toward organizational goals and is contingent on the availability of useful data that are representative of performance. Because data can be imperfect, there is a need to balance the desire for meaningful information against the cost of a "perfect" measure. In the use of its performance measures, the ministry strives to provide accurate and timely information, but at a reasonable cost. In an attempt to improve the consistency and quality of its performance reporting, the ministry has made efforts to follow the Performance Reporting Principles for the British Columbia Public Sector (co-published by the Office of the Auditor General of British Columbia and the Province of British Columbia in 2003).

The following provides a general overview of the performance data, reliability and methodology for the information in the annual report.

Number of days to process approval requests for contaminated sites

This measure refers only to approval requests regarding low- and medium-risk sites. For each approval request received by the ministry, the dates at which the submission reaches different stages in the approvals process are recorded by ministry staff. The ministry approval process involves verifying that the submission contains all required documentation, followed by stages of reviewing the documentation and requesting further information as necessary. The data used for this measure is the number of days between when a submission is received and when the requested approval is provided.

Percentage of monitored communities achieving the Canada-wide standard (CWS) quality objective of amount of fine particulates in the air under 2.5 microns per cubic metre (PM_2.5)

The ministry collects PM_2.5 data from approximately 40 air quality monitoring sites from over 25 communities across the province. The monitors are placed in communities that are densely populated or where air quality may be an issue. The monitors are audited twice a year by the ministry following U.S. Environmental Protection Agency protocols for auditing and record-keeping standards. Each community's data is analyzed using the Statistical Analysis System (SAS) program, and the end result for each community is checked against air quality results reported by other agencies (e.g., Environment Canada).

Total greenhouse gas (GHG) emissions in British Columbia

GHG emission data is collected by Environment Canada and per capita population statistics are derived from population statistics from Statistics Canada. The ministry relies on the quality of these data sources and the methodologies used by ministry technical experts and contractors for the analysis of the data.

The trends in environmentally significant variables, such as metals and nutrients, that indicate the health of water for 30 water bodies monitored under the federal/provincial agreement

The water bodies selected for monitoring are identified in a three-year business plan agreed to by the ministry and Environment Canada. They are selected based on the following criteria:

• whether provincial coverage is ensured;
• whether local activities may have a negative effect on the water quality of a water body;
• whether a water body is used and for what purposes (e.g., by residents for drinking, irrigation, livestock watering, or recreational purposes; and by wildlife and aquatic life); and
• whether water bodies are monitored by other entities.

Data are collected bi-weekly, or in some cases monthly, from each water body by trained samplers using established protocols. Once the samples have been collected, they are sent for analysis to laboratories that have been accredited by the Canadian Association of Environmental Laboratories (CAEL).

There are a wide variety of chemical, physical and biological indicators of water quality that are measured, including: major ions, dissolved solids, conductivity, trace elements, algae, zooplankton, nutrients, nitrate, pH levels, acidity, alkalinity, non-filterable residue and turbidity, colour, fecal coliforms, cyanide, adsorbable organic halides (AOX), temperature, total dissolved gases, dissolved oxygen, and flow. The trends in the above-monitored indicators are based on data collected regularly and consistently over periods of five to ten years or more. Data are reviewed annually for each of the 30 water bodies to assess whether there has been a significant variance from the trend. Each water body is subjected to statistical analysis approximately every five years (i.e., statistical analysis is conducted for five or six water bodies per year) as this is the timeframe in which it is expected that a change in the trend would become noticeable.

Number of aquifers identified and classified

Data is collected by professional hydrogeologists contracted to the ministry. Areas in which aquifers are to be examined are selected based on factors such as the high use of groundwater in an area, a possible groundwater issue, and availability of data and information. Each identified aquifer is classified using the BC Aquifer Classification System and then digitized to provide an inventory of the developed aquifers in the province. The identification and mapping of aquifers increases public knowledge and understanding of our groundwater resources in the province. For more information see wlapwww.gov.bc.ca/wat/aquifers/index.html.

Number of product categories with industry-led product stewardship

This measure identifies the number of product categories covered by the Post-Consumer Residual Stewardship Program Regulation under the Environmental Management Act. For more information please refer to wlapwww.gov.bc.ca/epd/epdpa/ips/index.html.

Percentage of high-risk environmental emergencies responded to effectively

The Regional Operations Branch of the Environmental Protection Division oversees environmental emergency management by the ministry. Environmental Emergency Response Officers use a Risk-Based Spill Assessment Model that was implemented in June 2003 to determine which spills are "high risk," and which the ministry is therefore responsible for attending. (The ministry responds to spills for which the overall risk rating is "high.") The model uses a formula to determine an overall risk rating, which is based on an assessment of the public threat, environmental sensitivity, incident status, response time by other parties, and response capability of other parties.

Measuring response "effectiveness" involves consideration of several issues, such as identifying the spills that require attendance by ministry staff, providing timely response, addressing the technical issues at the site, effecting proper response by parties responsible for the spill or by others when there is no responsible party, and ensuring appropriate clean-up. It can be challenging to measure effectiveness regarding these criteria individually or collectively. Currently, the measure involves an after-the-fact assessment of whether or not each spill was responded to effectively, and response effectiveness is reported based primarily on observation. The ministry is working on putting formal data collection methods in place.

Number of nationally listed species at risk being addressed by recovery teams

The ministry participates on the Committee on the Status of Endangered Wildlife in Canada (COSEWIC). "Under the Accord for the Protection of Species at Risk, federal, provincial and territorial governments have agreed to recognize COSEWIC as the source of independent advice on the status of species at risk nationally and to work together to protect these species."⁶ For this measure, the ministry is using COSEWIC data as the source for the number of nationally listed species at risk. Information about the number of those species that are being addressed by recovery teams is provided by the ministry.

Number of protected areas with approved management direction

Protected areas are protected by Order-in-Council or through an amendment to the Protected Areas of British Columbia Act. Data regarding the total number of protected areas is from a registry maintained by the ministry.

Approved management direction refers to a protected area having an approved management plan, master plan, management direction statement, interim management statement, or purpose statement and zoning plan. The plans address issues relating to management of habitats, ecological diversity, species, genetic diversity, and economic opportunities relating to tourism and recreation.

Data on the number of protected areas with approved management direction are readily available. Up-to-date information is maintained by the ministry on the status of the development and approval of management direction for all protected areas.

An appropriate measure relating to environmental standards for the Forest and Range Practices Act (FRPA)

The specific focus for this measure was identified in the 2004/05 – 2006/07 Service Plan — ministry progress in establishing designations and objectives for fisheries, wildlife and water quality using statutory authorities under the FRPA. Initially, the ministry will assess its progress by monitoring designations of wildlife habitat areas and ungulate winter ranges. As there is no direct relationship between the number of designations and the area covered by the designations, both measures are being reported as they are both important for demonstrating progress. Data are collected and maintained by program staff.

The percentage of known native species that are threatened or endangered in the province

The data and the criteria underlying general status assessments for this measure are presented in the federal government's report Wild Species 2000: The General Status of Species in Canada.⁷ This report was published for the first time using 2000 data. The next report is to be published in 2006 using 2005 data.

There are several data reliability issues related to this measure. To be accurate, the measure requires that we know of all existing native species in British Columbia and all species that are threatened or endangered. There are currently significant data gaps in both these areas, which is a challenge that is experienced in most jurisdictions that collect species information. Cooperation between both the provincial and federal governments should lead to progress in obtaining better data over time.

Number and value of partnerships

The measure developed for inclusion in the 2004/05 – 2006/07 Service Plan is the number of collaborative management plans with First Nations. These plans are identified and counted.

Visitor satisfaction with park and campground facilities and services

Approximately 4,000 campground park visitors are surveyed each year, usually between May 15 and just after the Labour Day long-weekend in the first week of September. The survey is sent to each park for distribution by park operators with provisions made for random sampling of subjects. A Visitor Satisfaction Index is derived from the survey responses. This is an overall rating, averaged for the province, that campers and park visitors provide for five categories: cleanliness of restrooms, cleanliness of grounds, condition of facilities, sense of security, and control of noise. Responses of "excellent" and "above average" are considered to indicate satisfaction.

The survey data are entered into the SurveyIt! program, which is used to perform all statistical calculations. (SurveyIt! is a software product designed to help manage complex surveys.) There may be subsequent follow-up sampling to confirm that the surveys were completed by legitimate park visitors.

Client satisfaction with fish and wildlife opportunities

Data for this measure are not yet available. The ministry intends to acquire the necessary data for this performance measure through a regular survey of hunters and anglers. The ministry's plan to implement a comprehensive electronic licensing and permitting application is currently on hold, thus preventing the electronic collection of survey data. At such time as electronic licensing is in place, a survey may be developed and administered to hunters and anglers who purchase licences over the internet. In the fourth quarter of 2003/04, the annual paper-based survey of hunters and anglers that is used to collect harvest data included a portion on client satisfaction in order to obtain baseline data for this measure for subsequent years. The 2003/04 preliminary results are not yet available.

Number of basic hunting and angling licences sold

The number of basic licences sold reflects the number of people participating in hunting and angling activities because a basic licence must be purchased before a person can hunt or fish or purchase a supplementary species licence. Data for this measure are obtained from the Government Agents Branch of the Ministry of Management Services and the ministry. The Government Agents Branch is responsible for the management of Angling and Hunting Licence Vendor Operations and the sale of all basic and supplementary licences throughout the province. The ministry also sells a small number of licences. Data are collected monthly and are reported by residency category and type of licence. Data are auditable as the number of licences sold must reconcile with revenues collected for sale of the licences. The data for 2003/04 are not yet available but may be included in the ministry's 2005/06 – 2007/08 Service Plan.

Number of recorded visits to B.C. parks

There are three components to visitation data: camper-nights, day-use visits (which account for 80-90%) and visits to marine parks. Traffic counters, infra-red counters and visual counts are used throughout the province to collect data regarding the number of parties that visit parks. The number of parties recorded for each visitation component is then multiplied by a factor that estimates the average number of people per party to give the total number of recorded visits. The factors are 3.2 for camping parties and boating parties and 3.5 for day-use parties.

This is a measure of total recorded visits, not total visits, as there are parks where visitation data are not collected. Nonetheless, as long as data are collected from the same areas each year, the trend in the data should provide a useful indication of changes in the overall level of park visits. Implementation of the recommendations of the Recreation Stewardship Panel has resulted in park visitor statistics no longer being collected in the past couple of years at a number of sites where they were collected previously. Efforts are now focused on collecting statistics in the busier parks and/or the core areas in parks, rather than in the smaller, less used parks. Therefore, current data are not comparable to data from before this change was implemented, and recorded visits would be expected to be somewhat lower than before the change. Comparing annual results for this measure will be more meaningful when trend data becomes available for the locations where data are now being collected.

Annual survey of ministry clients to determine if they understand what the ministry is trying to achieve and whether it is being done in the most efficient and effective way

The staff survey chosen for this measure was intended to help inform human resource strategies for the coming few years, and will be useful for making comparisons because it was also used widely in the public and private sectors. The survey was based on a 2001 National Work, Family and Lifestyle survey, conducted by Dr. Linda Duxbury of Carlton University and Dr. Chris Higgins of the University of Western Ontario, who had also surveyed 31,700 working Canadians at 100 major organizations, employing 500 or more people. The anonymous survey was open to staff from June 18 to August 18, 2003 and received a 70% response rate. The 35 questions and 160 items were responded to by 840 ministry employees. The survey is considered accurate within 1.5 percentage points, 19 out of 20 times.

Progress toward delivery of a three-year deregulation plan (measured as percentage reduction in ministry's regulatory count)

Regulations are counted according to the Regulatory Reform Policy of the Deregulation Office at the Ministry of Small Business and Economic Development.

Percentage of staff with an Employee Performance and Development Plan

This measure identifies the percentage of ministry employees with an Employee Performance and Development Plan (EPDP) by the target date of May 31, 2003. The information is reported by Strategic Human Resources Branch.

Measures of call-centre trends, including response time, geographic areas, and call types to enable proactive area-based planning/resource allocations and response improvements

Provincial Emergency Program staff enter each problem wildlife and enforcement report into an electronic database and create numbered Problem Wildlife Occurrence Reports and Enforcement Routing Sheets. The routing sheets are forwarded to Conservation Officer Service staff, and are also available through a web-based view application. Reports are sorted into various categories, including district, species and complaint type. Data are regularly audited and the written occurrence reports are regularly checked against the recorded telephone reports.

Other Information:

Annual Data

Where possible, the data in this document are reported on a fiscal year basis (April 1 – March 31). The EPDP date is May 31 each year to link closer with the start of the fiscal year work-plan process. In some cases, when data are collected and reported from external sources, calendar-year data are used, as indicated.

Financial Resources

For consistency purposes, the numbers (unaudited) reflected in the Resource Summary by Core Business Area are based on information included in the BC Public Accounts 2003 – 2004.

Expenses by Objective in the Report on Results

Presenting expenditures (unaudited) by objective is part of the ministry's ongoing commitment to improve the links between revenues, expenses and the Service Plan. Program management costs for each core business area are allocated based on total expenses for each objective. The Sustainable Environment Special Account allocation has been prorated across the Environmental Protection core business area objectives. The numbers (unaudited) reflected in the resource summary for each core business area are based on information included in the BC Public Accounts 2003 – 2004. As acquisition of financial information and reporting methods improve, amounts may be restated, thereby allowing more effective comparisons in the future.