Automated KYP monitoring has a weakness that's easy to overlook: it can only be as good as the data it runs on. A system that checks every security on the shelf every day, against a well-written set of material change definitions, will still produce the wrong answer if a security isn't mapped to its data, if a vendor field stopped updating months ago, or if a missing value is quietly treated as a pass. Worse, it will produce a clean-looking log that suggests everything was checked.
Regulators have moved firms toward systematic monitoring. The joint CSA/CIRO staff notice found annual monitoring alone generally insufficient and cited firms using automated systems to flag significant changes daily, weekly or monthly.[2] The same notice expects firms that rely on automated systems for KYP to describe those processes in detail, and firms that rely on algorithmic models to keep evidence of ongoing oversight that the model is functioning appropriately.[2] In other words, the data and the machinery behind monitoring are part of what has to be supervised.
This paper sets out how to audit the data behind a KYP monitoring system. It covers the regulatory basis in Canada and the US, an inventory of the data that monitoring depends on, six dimensions of data quality, the ways monitoring data typically fails, the responsibilities of the firm and the individual advisor, a set of audit tests, and an example written process.
It complements Material Change: When to Reopen a KYP Assessment, which covers what monitoring should look for, and The Continuous KYP Playbook: Who Does What, which covers the records monitoring should produce. This paper is about whether the inputs to both can be trusted.
No rule in either country is titled "monitoring data quality." The obligation comes from the duty to supervise, the duty to monitor products, and the expectation that firms can evidence what they did. Each one fails if the underlying data is wrong.
CIRO's Rule 3301 requires a dealer to monitor what it makes available for significant changes.[1] Section 11.1 of NI 31-103 and CIRO Rule 3904 require a system of controls and supervision sufficient to provide reasonable assurance that the firm complies with securities legislation, including its KYP requirements.[2] Joint CSA/CIRO Staff Notice 31-368 adds four expectations that apply directly to monitoring data.[2]
The notice's list of what KYP policies and procedures should cover includes the division of KYP work between the firm and its registered individuals, and says:
In its discussion of approvals, the notice addresses firms whose processes are driven by algorithmic models:
A monitoring system that evaluates every security against coded thresholds is that kind of process. Evidence that it is "functioning appropriately" starts with evidence that it is working on correct data.
Among its monitoring findings, the notice described firms that "maintained up-to-date information on securities (e.g., updated issuer financials) but had no evidence it was reviewed or considered as part of their monitoring process."[2] The reverse problem matters just as much: a firm that can show its system reviewed the data, but can't show the data was right.
On KYP assessments, the notice says: "While third-party reports can support KYP assessments, firms need to document their own analysis."[2] Most monitoring data comes from vendors. The firm can rely on it, but it remains responsible for knowing what it's relying on and whether it's fit for the purpose.
FINRA Rule 3110 requires a supervisory system reasonably designed to achieve compliance.[5] Where that system depends on automated monitoring, its design includes the data feeding it.
FINRA Regulatory Notice 21-29 reminds firms that outsourcing an activity or function to a third-party vendor does not relieve them of their ultimate responsibility for compliance with applicable securities laws and regulations, and describes practices for vendor due diligence and ongoing monitoring, including reviewing the accuracy of vendors' work product and overseeing system changes that affect critical functions.[6] Market data, fund data and monitoring platforms are all vendor services that firms commonly rely on for KYP.
The SEC's 2019 interpretation of the adviser standard of conduct requires a reasonable investigation into an investment "sufficient not to base its advice on materially inaccurate or incomplete information."[4] Reg BI's Care Obligation requires a broker-dealer to understand the risks, rewards and costs of what it recommends.[3] Both depend on the information about the product being correct and current.
| Expectation | Canada | United States |
|---|---|---|
| System of controls | NI 31-103 s.11.1 and CIRO Rule 3904 | FINRA Rule 3110 |
| Automated processes | Set out in detail in policies; responsible individuals identified | Part of a reasonably designed supervisory system |
| Algorithms and models | Document the model, its outputs and evidence of ongoing oversight | Covered by supervisory and vendor oversight expectations |
| Third-party data and vendors | Third-party material can support KYP; the firm documents its own analysis | Outsourcing does not relieve the firm of responsibility (Notice 21-29) |
| Information quality | Evidence that information was obtained and reviewed | Advice not based on materially inaccurate or incomplete information |
Before a firm can audit its monitoring data, it has to know what that data is. Most monitoring systems depend on more sources than the people who rely on their alerts realize.
| Data Set | What It Holds | Typical Source | What Goes Wrong |
|---|---|---|---|
| Monitored population | Every security on the shelf and every security held in client accounts | Product management records, custody and account systems | Transfers-in and delisted or closed securities left out |
| Security master | Identifiers, security type, classification, issuer and fund links | Internal reference data, vendor reference feeds | Identifier changes and reclassifications break the link to monitoring data |
| Market data | Prices, returns, volatility, spreads | Market data vendors | Stale prices, corporate-action errors, gaps |
| Fundamental and fund data | Financial ratios, ratings, MERs and fees, holdings, flows, manager names | Fund data vendors, filings, rating agencies | Fields that stop updating; methodology changes; lag |
| Events and documents | Fund amendments, issuer notices, rating actions, regulatory actions | Regulatory filing systems, issuer and manager notices, news | Events not captured, or captured late |
| Material change registry | The metrics, thresholds, frequencies and severities the system applies | Internal - owned by the product committee | Coded thresholds differ from the approved policy |
| Output records | Evaluations, alerts, reviews, decisions and notifications | The monitoring system and workflow tools | Records overwritten rather than versioned; missing no-breach evaluations |
For each data set, the inventory should record an owner, the source, how often it updates, and which material change definitions depend on it. That last link matters most: when a vendor field changes, the firm needs to know immediately which monitoring rules are affected.
| Dimension | The Question | Example Test |
|---|---|---|
| Completeness | Is every security in the monitored population covered, with every field each of its rules needs? | Reconcile the monitored list against shelf and holdings; count missing fields per rule |
| Accuracy | Do the values match the primary source? | Re-derive a sample of values from fund documents, filings or financial statements |
| Timeliness | Is each value as current as the rule's evaluation frequency assumes? | Compare each field's last-update date with its expected update cycle |
| Consistency | Do the same facts agree across sources and systems? | Compare vendor fees and ratings with the firm's own KYP records and client reporting |
| Lineage | Can each value be traced to its source and every transformation applied to it? | Trace a sample of alerts back to the raw source record |
| Point-in-time reproducibility | Can the firm show what the data said on a past date, not just what it says now? | Reproduce a past evaluation from stored data and confirm the same result |
The last dimension is the one most often missing, and the one an examiner is most likely to test. A monitoring record has little evidentiary value if the data behind it has since been overwritten and the firm can't show what the system actually saw when it made the call.
Monitoring data rarely fails loudly. The common failures produce no alert at all, which looks exactly like a security with nothing to report.
Auditing monitoring data is ongoing work, not an annual exercise. Some tests run automatically every day; others are periodic samples and reviews.
| Test | What It Proves | Suggested Frequency |
|---|---|---|
| Coverage reconciliation | Every security on the shelf and in client accounts is in the monitored population and mapped to its data | Daily, automated |
| Field staleness check | Each field has updated within its expected cycle | Daily, automated |
| Could-not-evaluate report | Every rule that failed to run is identified, not counted as a pass | Daily, automated |
| Source re-performance | A sample of values matches primary documents | Monthly sample |
| Registry reconciliation | Coded thresholds match approved definitions | On every registry change, and quarterly |
| Known-event back-test | The system caught real events it should have caught - for example, fee changes or manager departures found in filings | Quarterly |
| Point-in-time reproduction | A past evaluation can be rebuilt from stored data with the same result | Quarterly sample |
| Vendor change review | Vendor methodology and field changes were assessed for their effect on monitoring rules | On every vendor notice, and annually |
The known-event back-test is the most revealing. Take a set of significant changes the firm can find independently - a fee increase in a fund amendment, a manager change announced by the fund company, a rating downgrade - and check whether the monitoring system flagged each one, and when. Misses point directly to gaps in coverage, data or rules.
The example below shows what a written process might cover. It is illustrative; each firm's process should reflect its systems, vendors and business model.