Data Linking for Program Monitoring, Evaluation and Reporting

Date modified: 2025-02-21

More information

Policy:

Terminology:

Hierarchy
View complete hierarchy
Print-friendly XML
Expand all Collapse all

A. Purpose of this document

The Treasury Board of Canada Secretariat (TBS) has developed this guidance to help program officials, policy analysts, evaluators and performance measurement practitioners improve measurement, evaluation, and reporting by strengthening results activities through data linking. It includes a step-by-step process for measuring impacts using this approach. The use of data linking can ensure access to high-quality data to measure impacts while protecting privacy.

In particular, this guidance:

  • supports program officials, appointed by deputy heads under the Treasury Board Policy on Results, in collecting and using valid and reliable performance data, as set out in section 4 of the Directive on Results
  • enables departments to support government-wide reporting of program impacts on gender and diversity as set out in the Canadian Gender Budgeting Act

When a program serves individuals or businesses, TBS recommends that the program’s enabling authorities be developed to allow for the collection of identifying data. These identifiers can help link program data to anonymized tax, benefits or other data managed by Statistics Canada. Departments are strongly encouraged to explore these avenues prior to seeking authorities for new or existing programs through budget proposals, memoranda to Cabinet or Treasury Board submissions.

It is worth noting that the authority for an institution to implement a program or activity also includes authority to carry out monitoring and evaluation in most cases. The authority to collect and use data in accordance with the Privacy Act must, however, be confirmed by the departmental legal services unit.

See examples of how data linking has made it possible to report on impacts in Appendix A.

B. What is data linking?

Data linking involves combining administrative program data with other sources of data to enable more detailed analyses than would be possible with just the original data set.Footnote 1

To enable effective program administration, departments collect program-related data from individuals or businesses. Statistics Canada also collects other information on these individuals and businesses, such as income or geographical data, often from other governmental institutions (for instance, from income tax returns, immigration records and employment insurance records). In fact, Statistics Canada may have a lot of information on individuals that, when anonymized to protect privacy, could allow disaggregation of socio-economic outcomes by gender and diversity factors. This is essential for effective results management and also supports reporting under the Canadian Gender Budgeting Act.

Pursuing data linking with Statistics Canada significantly expands the potential for program analysis of gender and diversity factors while minimizing the burden of new data collection. As well, through Statistics Canada, the linking happens in a protected environment (with appropriate anonymization, which reduces privacy risks). Statistics Canada has various linkage services, including:

However, a number of steps are required to enable this work. These steps are provided in Section D.

C. Considerations

While this guidance focuses on how to implement data linking, it also refers to what data linking can be used for. Indeed, to ensure compliance with section 4 of the Directive on Results, impact reporting plans need to be considered in the early stages of program and policy development. These plans, along with supporting data activities, are to be developed so that departments are clear on how they will assess success as set out in section 3 of the Treasury Board Policy on Results. To do so, program leads are encouraged to consult early in the process with departmental leads in data management, as well as with the department’s Head of Performance Measurement and Head of Evaluation. These latter officials will help define what data requirements are for the department and for specific programs. Program officials may also benefit from involving Statistics Canada at this early stage.

Reporting on gender and diversity impacts generally involves personal information. Within their authorities, as set out in section 4 of the Privacy Act, program officials must recognize their responsibility as stewards of Canadians’; personal information and be seen to be good stewards. Program officials must be clear on what information requirements they have and for what purpose they are collecting that information. As set out in section 4 of the Directive on Results, they must update the program’s Performance Information Profile accordingly.

Given the requirements of the Privacy Act, it is critically important that programs ensure that they have the authorities to collect, use and disclose this information. In absence of this authority, it is recommended that program officials consult with the departmental legal services unit to assess whether a legislative amendment is required for program impacts reporting. When personal data are collected solely for monitoring, evaluation and reporting, program officials are strongly encouraged to communicate this to the individuals from whom the information is being requested. In other words, in such a scenario, it should be made clear that the data will not be used for program administration in respect of any specific individual but rather for the measurement and evaluation of the whole program.

TBS has prepared a Privacy Implementation Notice that provides further guidance on how to manage personal information for program monitoring, evaluation and reporting purposes while protecting the personal information of Canadians. It includes guidance on the requirements related to:

  • privacy impact assessments
  • privacy protocols
  • personal information banks
  • privacy notice statements
  • information-sharing agreements
  • privacy protective techniques

Program officials should reach out to their Access to Information and Privacy Offices for more information on the above.

As specified in the Privacy Implementation Notice, program officials also need to ensure that they have the agreements necessary to pursue data linking arrangements with Statistics Canada. Statistics Canada has strict policies, directives and processes to ensure the effective management of microdata linkage activities while addressing and mitigating the inherent privacy risks that linkages can pose.

D. What are the suggested steps to integrate data linking into planning and reporting?

Phase 1: planning

1. Determine what questions need to be answered

Review the policy or program logic model or theory of change.

Work with performance measurement and evaluation officials to determine what specific questions need to be answered to demonstrate impacts on diverse group populations.

2. Determine the data and methodological requirements

Based on the above, in consultation with the departmental leads in data management, determine the data needed for the analysis:

  • What mix of quantitative and qualitative data is the best source of evidence?
  • What identity and/or socio-economic considerationsFootnote 2 are necessary for analytical purposes?
  • What information is needed to demonstrate outcomes?
  • What methods are to be applied to measure net impacts?
    • Depending on the methodologies applied, different variables may need to be collected.Footnote 3

3. Develop a monitoring and evaluation data plan

Before new direct data collection is considered, the program official should work with its performance measurement or evaluation officials, data office, privacy officials and Statistics Canada on options for data linking in its secure linking environments to leverage existing data.

  • This step may require a significant amount of time for discussion and approvals. Statistics Canada operates on a cost-recovery basis. As a result, programs will also need to budget for linkage activities. To get started, contact Statistics Canada early in the process (see Appendix B for contact information).

Where direct data collection is necessary, the program should develop a detailed plan to articulate what data needs to be collected, how it will be collected, when, and from where. Considerations should also be given to what data could be linked from existing sources, where disclosure is permitted, to eliminate duplication. Consult with legal services and privacy officials to determine which provisions of the Privacy Act, such as paragraph 8(2)(j) for research and statistical purposes, can authorize sharing without consent.

To be able to link data, programs need to collect individual and/or business identifier data.

  • Particular attention needs to be paid to how the data will be collected, who will collect it, and what authorities, agreements, and terms and conditions are required to enable this. Because of the other parties involved, this is an important consideration for grants and contributions programs as well as programs providing services to Canadians.

Focus would be on additional data where:

  • personal characteristics of clients or applicants are important for analytical purposes and where they are not available through Statistics Canada
  • details on outcomes that are not available through Statistics Canada (for example, details pre- and post-benefits)

Programs should first leverage program data already being collected to analyze program interventions and impacts based on client eligibility.

4. Identify challenges and refer the systemic barriers to TBS

Programs should take stock of obstacles that the program faces in the implementation of a linking-based data initiative, including partnership agreements or information technology limitations. Plans should be developed to address identified obstacles.

Any barriers that are systemic or beyond the organization’s capacity to address should be referred to TBS at cgba-lcbss@tbs-sct.gc.ca. These issues will be reviewed by an interdepartmental steering committee.

5. Finalize the plan and implement

Once the steps above have been followed and key needs, challenges, obstacles and barriers have been addressed (including mitigating any risks identified in the privacy impact assessment or privacy protocol or security assessments), the program or activity’s data collection plan details should be integrated into the program’s Performance Information Profile at the beginning of implementing the data plan.

Phase 2: collecting, analyzing and reporting

6. Collect and share the data with Statistics Canada

Put the data systems, infrastructure and governance requirements in place as identified in Phase 1 of the plan.

Collect the required data as set out in the plan.

Coordinate with Statistics Canada to deliver any data files. These files should contain only the relevant information required for the linkage.

  • As noted above, there is a data acquisition process at Statistics Canada that should be factored into timelines. The length of this process can depend on the number of files and level of data sensitivity.

Prepare or submit data to Statistics Canada for linking as appropriate.

7. Work with Statistics Canada to produce the analysis

Through Statistics Canada, analysis such as the following can be carried out:

  • basic participation statistics (for example, ratio measures and means)
  • simple comparative impacts
  • net impacts
  • cross program comparative impacts

Programs will need to have the capacity in place to review and understand the results provided.

Programs that wish to access the microdata to conduct the analysis can leverage various data access arrangements with Statistics Canada. These arrangements may already be in place with the sponsoring department. Members of the program team will need to be deemed as having Statistics Canada employee status in order to work directly with the data.

8. Reporting and supporting decision-making

Analysis carried out on linked data should be incorporated into program reporting and evaluation, and inform the future direction of the program, including to support new policy or program authorities and budget proposals.

Appendix A: Illustrations of Analysis Completed with Data Linking

This appendix includes specific illustrations of how disaggregated analyses can be used to:

  • demonstrate the differential impacts of policies and programs on diverse groups of Canadians
  • help identify where there are gaps in impacts that may need to be addressed

Illustration 1: Immigration, Refugees and Citizenship Canada’s Express Entry program

Question

What are the early economic outcomes of immigrants screened in using Express Entry?

Express Entry (EE) is Canada’s application management system for certain immigration categories. EE uses the Comprehensive Ranking System (CRS), an evidence-based points system, to identify candidates most likely to achieve high employment earnings and who are able to maximize their economic performance in the Canadian labour market.

Data linkage

The CRS program data were linked to the Longitudinal Immigration Database.

Results

The 2020 evaluation of EE assessed early economic outcomes of EE-screened applicants. The evaluation found that:

  • early economic results for EE principal applicants are positive: they are demonstrating high levels of labour market participation and solid results in terms of their employment income, as well as the type of occupation in which they are employed
  • EE principal applicants generally outperform their non-EE counterparts
  • EE female participants have higher levels of human capital than male EE participants
  • despite having higher human capital than males, early economic outcomes are less favourable for EE female participants; female EE participants had lower employment income ($44,600) compared to their male counterparts ($66,400)
  • the worse outcomes for female EE participants point to where further analysis could be done to inform future program design

Illustration 2: Employment and Social Development Canada’s Canada Emergency Response Benefit program

Question

Who benefited from the Canada Emergency Response Benefit (CERB)?

CERB provided financial support to employees and self-employed Canadians directly affected by the COVID-19 pandemic.

Understanding the gender and diversity composition of recipients was a key reporting requirement for the program.

Data linkage

CERB program, disability and labour market data were linked to tax, immigration and census data.

Results

A Statistics Canada study released in June 2021 (see Workers receiving payments from the CERB program in 2020) examined the characteristics of workers most and least likely to have received CERB payments.

The study notes that:

  • with respect to individual Gender-based Analysis Plus (GBA Plus) characteristics, younger workers, visible minorities, refugees and Indigenous workers were more likely to have received the CERB than workers without these characteristics
  • with respect to intersecting GBA Plus characteristics:
    • generally, women and youth in visible minority groups were more likely to have received the CERB
    • among Black workers, men were slightly more likely than women to have received the CERB
  • beyond the above, there may be further areas of analysis that can be explored

Illustration 3: the Regional Development Agencies’; Regional Economic Growth Through Innovation program

Question

Did the Regional Economic Growth Through Innovation (REGI) program lead the assisted businesses to grow?

REGI provides assistance through two program streams:

  1. the Business Scale-up and Productivity Stream (BSP), which provides support to businesses at various stages of development
  2. the Regional Innovation Ecosystems Stream (RIE), which aims to create, grow and nurture inclusive regional ecosystems that support business needs and foster an entrepreneurial environment conducive to innovation, growth and competitiveness

Did the businesses that received assistance through the BSP or RIE grow in terms of employment, revenue and labour productivity? Did they become exporters and/or did they invest in research and development?

Data linkage

REGI program data (list of businesses assisted by REGI) were linked to the Canadian Employer-Employee Dynamics Database.

Results

Two reports were provided to the Regional Development Agencies:

  1. Economic Assessment for the REGI Program (BSP Direct Stream)
  2. Economic Assessment for the REGI Program (All Streams excluding the BSP Direct Stream)

Key results include the following:

  • BSP clients have different characteristics than non-clients. In particular, compared to non-clients, they are larger, they tend to be more heavily concentrated in the goods sector, and a higher proportion of them are in Quebec
  • clients in general have higher revenue, employment, labour productivity, export, and research and development (R&D) expenditure growth rates, and are more likely to become exporters or R&D performers

Illustration 4: federal business innovation and growth support to official language minority-owned businesses, 2020

Question

How did federal business innovation and growth program streams support official language minority-owned businesses in Canada in 2020?

Official language minority (OLM)-owned businesses refer to businesses whose owner has English as their first official language spoken (FOLS) in Quebec and French as their FOLS in the rest of Canada.

Data linkage

The Business-Linkable File Environment and Business Innovation and Growth Support programs data were linked to the 2021 Census of Population and Longitudinal Immigration Database.

Results

In 2020, around 344,000 businesses were OLM-owned, representing 6.3% of all businesses in Canada. Of these businesses, close to 2,000 (0.5%) received $437.6 million (12.7%) of total business innovation and growth support funding.

Compared with unsupported OLM-owned businesses, supported OLM-owned businesses were:

  • 1.9 times less likely to be owned by women
  • 4.8 times more likely to report making at least $2 million in revenue
  • 10.5 times more concentrated in manufacturing

Illustration 5: Black-owned businesses in Canada, 2020

Question

What were the socio-economic characteristics of Black-owned businesses in Canada in 2020?

Data linkage

The Business-Linkable File Environment and Business Innovation and Growth Support (BIGS) programs data were linked to the 2021 Census of Population and Longitudinal Immigration Database.

Results

In 2020, there were an estimated 145,000 Black-owned businesses in Canada, representing 2.4% of the total number of businesses in the country.

Black-owned businesses are more likely owned by immigrants, younger people and the self-employed.

In 2020, close to 1% of federal BIGS went to Black-owned businesses. This represents approximately $42 million of the total $3.8 billion in federal funding through BIGS.

Illustration 6: a comparison of the performance of female-owned and male-owned small and medium-sized enterprises

Questions

Are female entrepreneurs in Canada at a disadvantage in terms of enterprise performance and access to financial resources?

Is enterprise performance sex-based in Canada?

Data linkage

The Business Linkage File Environment was linked to the Survey on Financing and Growth of Small and Medium Enterprises.

Results

Majority female-owned small and medium-sized enterprises (SMEs) sold 20.9% less in 2011 and 18.4% less in 2014 than majority male-owned SMEs.

Majority female-owned SMEs employed 25.1% fewer employees in 2011 and 19.5% fewer employees in 2014 than majority male-owned SMEs.

When controlling for key factors, the assessed risk of debt financing requests by majority female-owned SMEs was comparable to that of majority male-owned SMEs in 2014.

Majority female-owned SMEs had a higher probability of innovating in manufacturing, accommodation and food services, and retail trade in 2014.

Appendix B: Data Linkage Services and Resources at Statistics Canada

Statistics Canada has a range of data linkage services and platforms. The following provides a brief description and contact information to help departments get started.

Social Data Linkage Environment

The Social Data Linkage Environment (SDLE) at Statistics Canada promotes the innovative use of existing administrative and survey data to inform socio-economic policy through record linkage at the individual level. Through SDLE, program data at the individual level can be linked to a broad range of survey, administrative and census data files that span multiple domains, such as health, justice, education and income. Services and supports include:

  • assessing the feasibility of record linkage projects
  • offering advice on data sources
  • liaising with subject-matter experts
  • assisting with approval steps
  • conducting the record linkage
  • building custom linked analysis files according to client specifications
  • advising on analytical limitations and validation
  • providing training and outreach

For more information, email STATCAN.SDLE-ECDS.STATCAN@statcan.gc.ca.

Business: Linkable File Environment

The Business: Linkable File Environment (B-LFE) is an environment in which Statistics Canada’s business microdata are linked from different administrative and survey sources. Through the B-LFE, program data at the enterprise level can be linked to a broad range of survey and administrative data files, providing information on Canadian enterprises, including:

  • financial, financing, innovation, patent and technology characteristics
  • owner and employee tax, immigration and Census responses and imputations

The B-LFE team provides support to users and partners by working with them to identify variables based on their topic of interest. Once identified, the variables from the appropriate linkable files are extracted as data sets used in research, impact studies or customized tables.

For more information, email statcan.elfe-eefc.statcan@statcan.gc.ca.

Analytical Studies and Modelling Branch

The Analytical Studies and Modelling Branch (ASMB) is responsible for supporting a range of data linkage environments (that is, the Canadian Employer-Employee Dynamics Database) and the use of existing linked data files such as the Longitudinal Worker File, Intergenerational Income Database (IID) and IID-Census to support analytical activities for program development and evaluation. ASMB provides expert advice to policy departments on the development of data and analytical plans to support program development and evaluation as well as training in quantitative evaluation methods. The Branch is also responsible for providing access to microdata for those departments wishing to conduct analysis.

For more information, email analyticalstudies-etudesanalytiques@statcan.gc.ca.

Expand all Collapse all

"Page details"

Date modified: