How do we get a better understanding of the scale of the work that Digital Champions are doing?

We recently wrote about how our approach to evaluating digital inclusion is not just about the numbers, with the help of England cricketing hero Ben Stokes. Ben was busy this week so we got someone else in to help with our next piece, which is — let’s be honest — all about the numbers. Do we contradict ourselves? Very well then.

Here’s Muhammad al-Khwarizmi of Baghdad (c. 780 – c. 850)

Muhammad ibn Musa al-Khwarizmi – father of algebra

This is a piece about our annual “activity snapshot” process, and how we use it to make estimates about how many people are being helped by Digital Champions each year. We have come up with a suitably sophisticated algorithm to help us derive accurate estimates of annual impact from the snapshot.

Our thanks to al-Khwarizmi, who first popularised algebra with his Compendious Book on Calculation by Completion and Balancing. The word algorithm comes from the Latinised version of his name. We don’t think we could do it without you!

The Citizens Online data team doing compendious snapshot estimates.

Taking a snapshot

Citizens Online’s approach to digital inclusion, Switch, is built around the provision of assistance with digital skills provided by a mixture of Professional, Volunteer and Embedded Digital Champions (DCs). DCs help individuals (End Learners, or ELs) understand the benefits of using the internet, and can show them how to do simple things online.

  • Professional DCs are dedicated outreach workers who are recruited by an organisation, partnership or by us at Citizens Online to work solely as a Digital Champion.
  • Volunteer DCs are recruited and trained by an organisation or partnership to support digital inclusion work, but are unpaid.
  • We emphasise the valuable role played by Embedded DCs, who work in a specific role (such as a Job Centre Plus, Citizens Advice branch, or an HR or training department) but who integrate Digital Champion work into this role.

Each year, in April, we ask all the DCs working with our projects to complete a simple form during one particular week, noting down the number of people they have helped and what kinds of activity or skill they helped with. This can be anything from logging on to the wifi at a venue, to setting up an email account, to creating a CV – among many other things.

This process captures extra detail that is not usually collected by the DCs.

We have now run three Activity Snapshot weeks:

  • 12-16th September 2016, completed by a total of 39 DCs
  • 16-22nd April 2018, completed by a total of 54 DCs
  • 15-21st April 2019, completed by 23 DCs

Rationale for the snapshot process

Citizens Online works with Digital Unite to support people to become Digital Champions through the Digital Champions Network (DCN), which provides learning, tools and a friendly community.

We encourage DCs to utilise the DCN’s session- and tally-based Activity Records, but we know from anecdotal reports that the DCN doesn’t capture the full range of DC activity.

Without further analysis, however, we have little idea what proportion of activity is captured by the DCN. Whether for reporting to our Key Performance Indicator targets with funders, or in order to evaluate and improve internally, we want to know how many people are benefiting from the support of Digital Champions we have employed, trained or recruited to the DCN.

What we found

Although there are some differences between the three snapshots – for example, the 2016 survey covered four projects, the 2019 survey only one – and the estimates were calculated in slightly different ways each time, we can draw some conclusions.

  • We believe the respondents helped at least 47,000 people over the three years.1[1]
  • Of the total 116 responses, each DC helped 10 people a week on average.
  • Embedded DCs helped on average three times as many people as Volunteer DCs.
  • DCs recorded as few as 0 to as many as 69 ELs helped in a week, including up to 24 in a single day.2[2]
  • We estimate that – including active DCs that did not fill in the snapshot – around 68,000 people were helped across the four projects.3[3]
  • The snapshots provide us with more information about the work DCs do. In 2019, for instance, “Foundation”4[4] tasks were 52% of all activities recorded.

The exercise confirms our view that only a proportion of activity is recorded, either by the snapshot or by the DCN:

  • The 2019 snapshot contained an example where one particular DC had recorded four activities on the snapshot recording form, but had only recorded one of those four on the DCN. This suggests that the snapshot process enables and encourages DCs to record their activity in a way that they do not generally do via the DCN.
  • We think the activity recorded in the snapshot is between 14-20% of the potential DC activity, if all trained DCs were as active as those participating in the snapshot.5[5]

End Learners helped per DC per week, all vs embedded vs volunteer:

Year ELs per DC (all) ELs per DC (embedded) ELs per DC (volunteer)

Calculations of annual numbers of End Learners receiving support from DCs:

Number of projects in snapshot421
DCs in snapshot395423
Total trained DCs in snapshot projects 266295235
Annualised estimate of ELs helped by snapshot respondents24,52018,0404,080
Annual ELs estimate based on trained DCs35,95725,3426,844

Our methodology and algorithm

The annual EL estimates presented here use a standardised methodology for each snapshot, based on several assumptions.[6] First, we make an annualised estimate of ELs helped by snapshot respondents, which simply multiplies the snapshot totals for a 40-week year (for the 2019 snapshot, adjust the estimates to take account of the relative quietness of the sample week). For trained DCs who did not take part in the snapshot the process is slightly more complex:

  • Trained DCs are a subset of the total DCs recruited in a project, those that have completed a training course on the DCN or equivalent face-to-face training.
  • Having subtracted the number of DCs who completed the snapshot, we combine the number of each type of trained DC by the average number ELs that type of DC helped in the snapshot (e.g. 61 non-snapshot VDCs in 2019 helping 1.9 ELs/week = 117 ELs).
  • For non-snapshot DCs we reduced the level of estimated activity to one-tenth the total of those who completed the snapshot, on the basis that snapshot respondents are the most active DCs (i.e. the above 117 ELs becomes 12).
  • Finally, we again apply calculations to make annual estimates (and in the above example, the number of ELs becomes 18 to adjust for the quietness of the 2019 week).

It’s potentially a little complex, but we stand by our calculations.

It’s not easy to build up a more general picture from small snapshot samples, but having carefully considered all the factors and the rich information we have received, we think it’s a sound piece of work. Full notes and explanations can be found in our Activity Snapshot Analysis document (pdf).

[1] This figure is the sum of annualised estimates associated with each of the snapshots, and does not account for people helped by DCs who continued to help people in years they did not complete snapshots. We also have no data from Plymouth and the Highlands for 2018 and 2019, nor data for Gwynedd for 2019.

[2] Recorded by one Embedded and one Volunteer DC in Highland in 2016, a Brighton and Hove Embedded DC in 2016, and a Brighton and Hove Embedded DC and Professional DC in Plymouth in 2016, respectively.

[3] As with the earlier estimate, this figure is based on the sum of the annual estimate for each of the three years, not the full programme duration and project extent.

[4] The Essential Digital Skills Framework details the Foundation skills necessary for people to use devices.

[5] We don’t expect all trained DCs to undertake activity at the same rate as the most active DCs, this estimate is a reflection of capacity: what might be possible were they to become so.

[6] Citizens Online has previously analysed the snapshots from 2016 and 2018 using slightly different methodology, and the numbers presented here supersede those earlier estimates.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.