My notes, live, from Jack Holleran of Ernst & Young and Patricia Prince-Taggart of CA on ways to measure program effectiveness, with an analysis of qualitative and quantitative measures.
Jack put forth three primary objectives to compliance programs:
- To prevent non-compliance
- To detect non-compliance
- To enhance business processes and decision-making
He offered the following as the benefits of measurement:
- Enables you to “know where you are”
- Enables you to demonstrate effectiveness to Executive management
- Enables you to demonstrate effectiveness to Audit committee
- Enables you to demonstrate effectiveness to Regulators
- Enables you to identify and prioritize opportunities for improvement in ethics and compliance program (design and execution)
- Enables you to demonstrate business case, or value, that ethics and compliance program provides to the business
He offered this as his illegible diagram of a compliance program:
Qualitative measures
- Provide some indication of awareness of ethics and compliance program
- Tend to be subjective in nature
- Useful in identifying trends
Quantitative measures
- Provide objective insights into program effectiveness
- Tend to be hard data
- Useful for benchmarking your company to other organizations or within industry
Measuring effectiveness – the role of auditing and monitoring:
Evaluate each control for adequacy:
- As designed, will it prevent / detect? Alone, or with other controls?
- If design is adequate, test to verify control is operating as designed
Testing examples:
- Field work: policy application within business units
- Continuous testing: review of helpline calls, customer complaints
- Transaction reviews for red flags
- Risk-based reviews (e.g., FCPA, environmental)
- Surveys/focus groups to measure awareness, attitudes, knowledge
Establish procedures for conducting investigations:
- Confidentiality
- Case resolution procedures
- Post-resolution surveys of callers
- Checking for possible retaliation
You want to determine the “Effectiveness Gap”: the difference between the inherent risk and management’s effectiveness.
Ethics and compliance, like any business function, faces the internal challenge of demonstrating return on investment (ROI). Measuring effectiveness can enhance your ability to demonstrate ROI. Trending over time can produce insights.
Starting is the most important part of effectiveness. Doing nothing is not effective. You can’t be afraid to find out information.
Here are some other resources they recommended:
Metrics Qualification Tool
www.oceg.org/view/mqt
The Elephant in the Room: Program Evaluation & Performance Measurement
www.oceg.org/view/elephant
Measurement & Metrics Guide: Performance Measurement Approach and Metrics for a Compliance & Ethics Program www.oceg.org/view/MMG
Metrics Full Listing
www.oceg.org/view/mmglisting
Metrics & Measurement Guide Presentation: Beyond Effectiveness
www.oceg.org/view/MMGPreso
(These notes are taken live, so I apologize if I left out anything or misquoted someone. Please forgive any typos or grammatical errors.)