Appendix D: Performance Measures Methodology

Goal 1

Objective 1: Rate of Customer Satisfaction (index)

Average customer satisfaction rating (index): This performance measure is based on an average of customer satisfaction ratings with Service BC — Government Agents and Service BC — Enquiry BC. Survey questions were based on the Common Measurement Tool¹ (CMT), a survey tool that has been adopted by a variety of jurisdictions to measure customer satisfaction with service delivery. The CMT uses a five-point scale, ranging from "very satisfied" to "very dissatisfied".

• In a survey administered by BC Stats, between January 20 and February 4, 2004, Government Agent customers were asked "How satisfied were you with the overall quality of service delivery at the Government Agent Office?". A total of 651 surveys were completed. The satisfaction rate was 96 per cent. The accuracy of the results has a margin of error of 1.9 per cent at the 95 per cent confidence level.
• In a survey administered by BC Stats between January 20 and February 2, 2004, Enquiry BC callers were asked "How would you rate your level of satisfaction or dissatisfaction with the overall services of the Enquiry BC Call Centre?". A total of 300 callers were surveyed. The satisfaction rate was 98 per cent. The accuracy of the results has a margin of error of 1.0 per cent at the 95 per cent confidence level.

The 2004/05 – 2006/07 Service Plan, included satisfaction measures for individual services. This measure now uses an index approach — a roll-up of average customer satisfaction ratings into a single measure. Due to the use of different survey methodologies (i.e. satisfaction with the BC OnLine Help Desk) and/or the inability to develop a methodology to gather information to report results, the services in the following table were not included in the index. In future, a consistent survey methodology will be applied to all Service BC services.

Percentage of customers satisfied or very satisfied with use of Community Access Terminals

Community Access Terminals are Internet terminals, located in all Government Agent Offices across BC, that provide citizens and businesses access to electronic government services and information. They provide a convenient bridge across the digital divide for citizens who may not have Internet access in their homes.

Two separate attempts were made to leverage other survey tools to also measure customer satisfaction with the Community Access Terminals during fiscal year 2004/05. However, both attempts yielded very small samples of customers who had used the Community Access Terminals, which in turn resulted in a very high margin of error. Rather than report a satisfaction level that is of questionable reliability, a targeted survey of Community Access Terminal users will be undertaken in fiscal year 2005/06.

Percentage of clients of the BC OnLine Help Desk service who are satisfied or very satisfied

Note that this is satisfaction with the Help Desk service associated with BC OnLine, and not satisfaction with the services offered through BC OnLine. BC OnLine provides customers the opportunity to provide feedback regarding the previous year's service performance and new services of interest via an electronic survey tool that customers could complete when they signed on to the BC OnLine service. This feedback tool was available over a two-week period, from June 9 to June 20, 2003. Over this period, 913 surveys were completed. The survey asked the customers to rate the service of the BC OnLine Help Desk using a three point scale: Very good, Satisfied, or Poor.

Percentage of British Columbians who are aware of government services and channels

Information needed to report on this measure was not available this year. A new approach to gathering this information will be developed in 2005/06.

Volume served: Service BC — Government Agents (customers); Service BC — Enquiry BC (customers); BC OnLine (transactions)

This is a new measure. The volumes for in-person and telephone are consistent with the previous year's performance. The consistency reflects the migration of customer contact to the online channels. The volume of transactions for BC OnLine has increased from 6.43 million in 2003/04 to 6.89 million in 2004/05. Service BC is developing strategies to move customers to more economical web-based services.

Source: Service BC.

Average Per Minute Cost For Service Delivery:

This measure is calculated by taking the average service transaction cost, divided by the average time per transaction. The resulting average cost per minute is then weighted to determine the average weighted per minute cost for service delivery. The combined average is calculated by adding together the average weighted per minute cost for service delivery for Service BC — Government Agents and Service BC — Enquiry BC.

	Average time per transaction	Average cost/minute	Average service transaction cost1	Percent of Service BC delivery time	Average weighted per minute cost for service delivery
Government Agents	6.58 minutes	$1.62	$10.65	93.5%	$1.51
Enquiry BC	0.93 minutes	$1.80	$1.68	6.5%	$0.12
Combined weighted average2					$1.63
Combined weighted average service transaction cost	6.21 minutes		$10.13

The measure incorporates the full range of services and information delivered through Service BC — Government Agent Offices and Service BC — Enquiry BC contact centres, and captures over the counter, telephone, and e-mail channels of service delivery. The measure also incorporates "fully loaded" service delivery costs, based on salaries and benefits, and overhead (building occupancy, travel, and network and telecommunication costs).

Research into inter-jurisdictional comparisons will be conducted to establish benchmarks with similar organizations.

Source: Service BC.

Goal 1

Objective 2: Percentage of communities with access to broadband facilities

For the purpose of this measure, the ministry has adopted the definition of "community" used by the Premier's Technology Council: any location in British Columbia with a place name and either a public school, a library, or health care facility. There are 366 communities across British Columbia that meet the definition.

Following the "digital divide gap analysis" in the Premier's Technology Council's Second Quarterly Report, a second gap analysis was published in their Fifth Report. These gap analyses list each community in the province, and the access to each type of broadband that is available. The performance measure shown earlier in this Annual Service Plan Report is drawn from those inventories.

Data sources:
2001/02: Premier's Technology Council, Second Quarterly Report, April 2, 2002; p.92.
2002/03: Premier's Technology Council, Fourth Report, March 27, 2003; p.9.
2004/05: Premier's Technology Council, Fifth Report, January 12, 2004; Appendix A, p.29.

In their sixth report released July 2004, the Premier's Technology Council recommended that government: keep up the momentum to extend broadband to the remaining communities as quickly as possible; and, work with communities to identify last-mile solutions.

(Premier's Technology Council, Sixth Report, June 25, 2004, p.8.)

The digital divide is closing faster than anticipated. "The Council originally anticipated that by the end of 2003, broadband service would be available to 80 per cent of the province's population, living in 150 of the 361¹ communities in BC. The remaining 20 per cent of the population scattered in over 200 communities represented the Digital Divide — those without broadband and therefore without access to the many basic programs and services it enables, such as e-learning, e-health and e-government. The Council is pleased to report that this initial estimate has been surpassed and 193 communities representing 89 per cent of the province's population now have broadband service." (Premier's Technology Council, Fifth Report, January 12, 2004, p. 1.)

As of March 31, 2005, 308 out of 366 communities have either direct access to broadband or an Internet point of presence from which the community could secure broadband access. This measure was derived in part by the provision of EB1 circuits in 2004/05, but has since been superseded by higher speed circuits called E10s as a result of the NetWork BC / TELUS agreements. Hence, the target for having all communities connected to a broadband access point is December 31, 2006.

Further information about the Premier's Technology Council, including the Council's reports, is available here*. For additional resources about the ministry's NetWork BC project, please visit http://www.network.gov.bc.ca/.

Source: NetWork BC.

Goal 2

Objective 1: Percentage increase in total Solutions BC revenue generated from the broader public sector

The measure relies on information published as part of the ministry's budget. Recoveries are tracked in three "STOBs" (Standard Objects of Expenditure — consistent accounting categories used across government).

Internal recoveries, from other ministries, are recorded in STOB 88 while external recoveries — those from the broader public sector — are entered in STOBs 89 and 90.

Within Solutions BC, there are four lines of business:

• Common Information Technology Services (CITS)
• Procurement and Supply Services/Strategic Acquisitions and Intellectual Property (PSS/SAIP)
• Corporate Accounting Services (CAS)
• Payroll Services

The entries for the lines of business within Solutions BC are totaled, and compared to either the total recoveries or to the value for the baseline year. For fiscal year 2004/05, CITS and PSS had substantial recoveries from the broader public sector while the other two lines of business recovered smaller amounts.

Reduction in Unit Costs (Workstations and POIM/Payroll)

The ministry has recently entered into long-term agreements with TELUS to deliver POIM/Payroll Services and with IBM Canada Ltd. to deliver Workplace Support Services. The baselines are established in the respective contracts.

Source: Solutions BC.

Central procurement operational costs as a percentage of value of total goods and services purchased

This measure evaluates the efficiency of the government's central procurement services. The cost required to operate the procurement services is contrasted with the value of goods and services managed. The industry standard, established by the National Institute of Governmental Purchasing, is that operational costs as a percentage of value of total goods and services purchased should be no more than one per cent. A lower percentage indicates greater efficiency and cost-effectiveness in the delivery of procurement services; however, too low a percentage may mean sub-optimal results.

This measure is calculated by comparing the total value of goods and services purchased through Purchasing Services compared to operating costs (salaries, benefits, travel, training, information technology, and contractor support costs) incurred by Purchasing Services to provide procurement services to government. The calculation for the current and two previous fiscal years is shown below. The 2004/05 performance result of 0.78 per cent is an average of the percentage value of the current and previous two fiscal years.

Procurement Operating Costs as Percentage of Value
(all figures $ millions)

	2002/03 Estimate (2002/03 Annual Service Plan Report)	2003/04	2004/05
Total value of goods and services purchased through Corporate Procurement Services	$398.60	$243.0	$519.4
Operating costs, Corporate Procurement Services	$2.9	$2.5	$3.0
Procurement Operating Costs as Percent of Value	0.73%	1.03%	0.58%

Source: Solutions BC.

Goal 2

Objective 2: Average customer satisfaction ratings (index)

Satisfaction ratings were established through surveys conducted in 2004/05 by BC Stats for CITS, POIM/Payroll, Corporate Accounting Service (CAS) and Procurement and Supply Services/Strategic Acquisitions and Intellectual Property (PSS/SAIP). The following chart provides detailed survey results, breakdowns of what is included in respective lines of business, and respective weightings.

	wt.	2004/05
Solutions BC		77%
CAS Overall, how satisfied or dissatisfied are you with the services provided by CAS?	20%	62%
iExpenses		63%
CAS Help Desk		81%
instructor-led classroom training		72%
web-based training modules		63%
application-specific CAS user manuals		62%
the Oracle and CODE user access set-up forms		67%
regular CAS communications		52%
CITS	30%	79%
Workstation		70%
Reliability		70%
Performance		69%
Lets me do my job		70%
Open/Save/Print		74%
Reliability		77%
Performance		69%
Availability		75%
Office Suite		80%
Functionality		80%
Email		88%
Reliability		92%
Performance		85%
Availability		87%
Voice		86%
Reliability		88%
Performance		86%
Lets me do my job		91%
Functionality		79%
Help desk		77%
POIM/Payroll	20%	78%
Help desk		71%
Pay Processing		84%
Leave Processing		78%
PSS/SAIP	30%	85%
IPP (consulting services only)		85%
Timeliness		96%
Service Quality		96%
Cost relative to value		63%
SAIT (consulting services)		68%
Timeliness		76%
Service Quality		86%
Cost relative to value		42%
PDC (everything, more or less)		90%
Timeliness		90%
Service Quality		93%
Cost relative to value		86%
QP (Print services not stationery)		95%
Timeliness		96%
Service Quality		96%
Cost relative to value		94%
BC Mail Plus (custom mailings, variable printing)		94%
Timeliness		97%
Service Quality		95%
Cost relative to value		89%
Purchasing Services (consulting services only)		74%
Timeliness		82%
Service Quality		82%
Cost relative to value		59%
Distribution Centre Victoria
Timeliness
Service Quality
Knowledgeable
Courtesy
Price

Source: BC Stats.

Average client satisfaction ratings (index)

This performance measure establishes the satisfaction of respondents to Solutions BC's client survey with how effectively services have been delivered. The performance result of 67 per cent is higher than the previous year's result of 52.5 per cent and exceeds the target of 57.5 per cent.

Source: BC Stats 2004 Solutions BC Client Survey February 2004.

Clients surveyed include: members of the Deputy Ministers' Committee on Shared Services, Assistant Deputy Ministers of Corporate Services, Directors of Information Systems, Senior Financial Officers, and representatives from central agencies. Clients represent ministries or organizations that purchase services from Solutions BC.

Following is the breakdown of the services rated by respondents:

Percentage of respondents who say services have been effectively delivered by Solutions BC

	% of respondents Satisfied/Very Satisfied	Weights
CAS	(reading from graph)
General Ledger/Budget and Chart of Accounts (BCOA)	76%
Accounts Payable and Receivable/Expenses	76%
Reporting (CODE data warehouse)	59%
Purchase Orders/Procurement	53%
Average in CAS areas	66%	20%
Payroll
Leave Transaction Data Entry	82%
POIM — PeopleSoft and other Corporate HR Applications	71%
Payroll Transaction Data Entry	71%
Average in Payroll areas	74.7%	20%
PSS
Procurement services/Corporate Procurement including BC Bid	94%
BC Mail Plus	88%
Queen's Printer printing services	82%
Distribution Centre Victoria (Office Products Centre)	71%
Warehousing services/disposal of surplus assets	65%
Average in PSS areas	80.0%	30%
CITS
Voice services (telephones)	94%
Email	88%
IT security	53%
* Hosting ministry business application (server)	29%
* IT help desk	35%
* Opening, saving, printing files (servers)	29%
* Workstations	24%
* Average in CITS areas (Transitioning Services only)	29.3%
Average in CITS areas (including Transitioning Services)	50.3%
Average in CITS areas (without Transitioning Services)	78.3%	30%
		Weighted %
Overall average including Transitioning Services	65.3%	67.2%
Overall average without Transitioning Services	74.9%	75.6%

Rate of success in meeting service standards

CITS: Workstation Support Services and Incident Resolution (Priorities 1–4). Meeting or exceeding service standards builds customer confidence. Gaps in performance offer opportunities to improve service.

	Target	Actual
Workstation Support Services July 2004 – September 2004	90%	85%
Incident Resolution (Priorities 1–4)	90%	79%
Average		82%

Service Commitments Met: CITS: Indexed Service Levels (further breakdown of previous chart).

Workstation Support Services (Interim Model — during transition to IBM).

Reporting Window: July 2004 thru Sept 2004

ID Management:

Transfer User to/from ministry
Password Reset
Modify ID
Disable ID
Create New User

Desktop Services:
Urban IMAC Service request

Source: Workstation Service Interim Delivery Model — Service Level Reports.

Incident Resolution (Priorities 1 thru 4)

Reporting Window: April 2004 thru Sept 2004

Service Components:

ASI-P
BCEID
Data Network
HOSTI
ITIMS
NT
OPENVMS
SECURITY
UNIX
VOICE

Source: ITIMS Reports.

Common Business Services: Description of index components for rate of success in meeting service standards

Category	Standard Measure	2003/04 Actual	2004/05 Target	Weight
Reliability and Availability
Payroll: Time On Line	System available during the scheduled hours of operation	n/a	98%	Methodology: The number of transactions in each application divided by the total transactions.
Payroll: PeopleSoft	System available during the scheduled hours of operation	99%	98%
Payroll: Human Resource applications	Systems available during the scheduled hours of operation	100%	93%
CAS: Financial Systems	Per cent production availability during scheduled hours of operation	99%	99%
CAS: Data Warehouse	Per cent production availability during scheduled hours of operation	95%	98%
PSS: Publications Index	System available 24/7	95%	97%
PSS: BC Bid	System available 24/7	99%	99%
PSS: Distribution Centres — web ordering	System available 24/7	99%	99%
Average Rate of Success		98.0%	97.6%
Timeliness and Delivery
Mail Preparation	Prepared and delivered in accordance with customer requirements	99%	99%
Printing (in-house)	Prepared and delivered in accordance with customer requirements	97%	97%
Distribution Centres	Orders turned around within 2 days	99%	99%
Asset removal	Within 15 business days of surplus declaration	95%	95%
Payroll processing	Changes to pay received by cut off processed in next pay period	n/a	95%
Average Rate of Success		97.5%	97%
Accuracy
Printing (in-house)	Error free print jobs	100%	100%
Mail Preparation	Error free mail preparation	99.5%	99.5%
Distribution Centres	Error free orders	98%	99%
Average in PSS areas		99.2%	99.5%
Reliability and availability overall rate of success		98.0%	98.04%
Timeliness and Delivery overall rate of success		97.5%
Accuracy overall rate of success		99.2%
Rate of success in meeting service standards (index)		98.2%

Goal 3

Objective 2: A rolling strategic plan that co-ordinates e-government decision-making and investment across government

Information pertaining to the roll-out of the strategy is provided by the Office of the Chief Information Officer on a regular basis.

Progress in implementing information management and technology infrastructure processes (COBIT-based scales)

This is a new measure for 2004/05. One method of assessing the effectiveness of processes is to apply them to a "maturity model" similar to the one below. The model describes a five-level evolutionary path of increasingly organized and systematically more mature processes. The framework underlying the maturity model used to assess processes supporting information and technology infrastructure for the CIO is COBIT (Control Objectives for Information and related Technology), Third Edition, issued by the IT Governance Institute. Level 3 represents an organization where processes are defined, Level 4 represents an organization where processes are managed and Level 5 where processes are optimized.

Goal 4

Objective 1: Percentage of staff with a current Employee Performance and Development Plan (EPDP)

The performance information on this measure was provided by the Corporate and Ministry Support Services Division.

Objective 2: Rating of key employee satisfaction categories (index)

The staff of the Ministry of Management Services were surveyed between Monday, May 17, 2004 and Friday, June 4, 2004. Of the 1,714 in-scope employees, 1,101 responses were obtained, resulting in a response rate of 64 per cent.

The objective of the survey was to obtain benchmark information about knowledge of the ministry's direction, attitudes and perceptions about working for this ministry, and view on communication needs. The responses were used to guide the development of the ministry's human resources plan, The People Strategy.

For the purpose of performance measures in the ministry's 2004/05 – 2006/07 Service Plan, the questions asked were "Overall, how satisfied are you with your job at the ministry " and "I understand how my work contributes to the ministry Service Plan". The results are tabulated below.

Annual Employee Survey — Index (Key Questions)