Safety by Design: How Do We Measure What Changes?

When safety moves upstream, regulators still need a way to measure what changes downstream.

Safety by Design is moving into regulatory practice.

In Australia, eSafety has developed a detailed Safety by Design framework. In the UK, Ofcom is pressing platforms to put Safety by Design “front and centre of their operating models”.

The EU takes a similar direction without using the same label. Article 35 of the Digital Services Act requires VLOPs and VLOSEs to mitigate identified systemic risks. Those measures can include changes to the design, features or functioning of a service. Article 34 also requires them to consider how recommender systems and other algorithmic systems influence those risks.

Safety is therefore increasingly being addressed through the way a service works. That puts more weight on a question regulators are already trying to answer:

How do you measure whether a safety measure had the intended effect?

The measurement problem is already recognised

Measurement is part of eSafety’s Safety by Design framework. Its transparency and accountability principle calls for information on the efficacy of safety features. It also calls for analysis of abuse data, moderation effectiveness and enforcement metrics.

Ofcom has gone further into how effectiveness could be evaluated.

Its 2024 discussion paper groups example metrics into process, risk and harm metrics. It treats unintended consequences separately. The examples range from detection and removal to prevalence, exposure, user wellbeing and specific harms reported.

Ofcom also looks at how those metrics should be used over time. It says straightforward comparisons can help establish a baseline and assess whether outcomes are moving as expected. Historical data is specifically prioritised so regulators can examine trends before and after a safety measure is introduced.

Comparison between services creates another problem. Ofcom notes that metrics are not always compiled on a standardised basis. This can make cross-service evaluation difficult.

The Global Online Safety Regulators Network has identified the same need. Its members want common metrics for evaluation approaches and more comparable global data for trend analysis.

WeProtect reaches a similar conclusion in its 2025 Global Threat Assessment. It says effective implementation of Safety by Design should be guided by evidence. It also calls for industry transparency and independent accountability.

The challenge is therefore not a lack of interest in measurement. It is having evidence that can be followed over time and understood in context.

The DSA already gives us part of that evidence

For EU platforms, the DSA creates a substantial public record that can contribute to this type of before-and-after analysis.

• Risk assessments describe systemic risks and the measures used to mitigate them.
• Transparency reports provide aggregate information on moderation and enforcement.
• Audit reports provide the auditor’s assessment of compliance.
• Statements of Reasons add decision-level data.

Platforms submit these records to the DSA Transparency Database when they take certain content moderation decisions. The Commission says the database allows those decisions to be tracked in almost real time.

That continuous record becomes useful when a Safety by Design measure is expected to affect enforcement.

If a platform changes how it addresses a particular risk, the Statements of Reasons provide a public history from before and after that change. A movement in total enforcement can then be examined in much more detail. It may be concentrated in one type of harm, while the rest of the enforcement population remains relatively stable.

The level of insight depends on what the platform reports.

Some Statements of Reasons identify a specific harm or mechanism. Others only identify a broader domain. That difference determines how precisely the public change can be measured.

This is where Civility Bureau’s normalization becomes important.

Making before-and-after comparison possible

Comparing raw Statements of Reasons over time is not enough.

Platforms use different policy structures. They also disclose different levels of detail. Those structures can change within the same platform over time.

For Safety by Design measurement, this matters for a simple reason. The data before a change needs to remain comparable with the data after it.

Civility Bureau normalizes each Statement of Reasons at the level supported by the disclosure. A specific harm can remain specific where the evidence supports it. A broader safety domain remains broader where the platform does not disclose enough detail.

The same analytical structure is then applied over time.

This creates a more stable public history. A policy rename or a change in reporting structure does not automatically break the measurement.

It also creates a cross-platform view. The platform itself has deep insight into its own systems. We can show how comparable public signals are developing across the wider ecosystem.

That matters when a large shift appears. A movement that occurs on one service has a different context from a similar movement appearing across several services at the same time.

From change to validation

This is the role we see for Civility Bureau in Safety by Design measurement.

A Safety by Design change starts with an intended outcome. After the platform implements the change, we can track whether the relevant public enforcement pattern changes.

The platform has the internal information needed to explain what happened inside the service. Regulators or auditors can assess that explanation alongside the public evidence.

A risk assessment can describe the identified risk and the mitigation measure. Internal data can provide evidence on how the measure performed. The public enforcement record shows how enforcement changed.

Civility Bureau makes that public enforcement record measurable over time.

Measuring Safety by Design over time

Consider a platform that introduces a new design measure to address a specific child safety risk.

If the platform discloses that risk at a sufficiently detailed level, we can establish its public enforcement history before and after the change.

Six months later, the discussion does not have to start with a single aggregate number.

There is a historical public record. There is the platform’s explanation of what changed internally. There may also be evidence from the risk assessment, transparency reporting and audit process.

Civility Bureau gives regulators and auditors a consistent external reference across that period. It can also show whether the same public signal changed elsewhere in the ecosystem.

This is particularly relevant as Safety by Design becomes more established. The measures themselves will increasingly sit inside products, systems and algorithms. Their effects still need to be evaluated against evidence.

The DSA has already created an unusually large public record of enforcement activity.

We can help to turn that record into a measuring stick that can be followed before and after a Safety by Design change, and across platforms.

The platform can explain why something changed.
Civility Bureau can show what changed in the public evidence.