POSTS

Accessibility Evaluation: Open University and RNIB

February 11, 2021

Summary

In February 2020 the UK's Open University (OU), working with the The Royal National Institute of Blind People (RNIB), produced a practical report on accessibility of mathematical content.

This post provides a high level precis of the report. Its focus is on helping you which part of the report, if any, are relevant to you. I hope it encourages you to read this well-written report on carefully considered research.

OU: Accessibility of alternative format for Math & Stats (PDF), (2020)

This is one of three posts made today, prior to a zoom TeX Hour tonight at 6:30 to 7:30pm UK time on accessible mathematics. The other posts are Accessibility Tools and Blind Math News. For more on the TeX Hour see contact.

[Disclaimer: I used to work at the OU as a LaTeX expert.]

Background

The Open University (OU) is the UK's largest provided of distance learning. Many disabled people prefer distance learning, as their home already has the adjustments required. The Royal National Institute of Blind People (RNIB) is the UK's leading and largest community of blind and partially sighted people.

In 2018 to 2019 the OU evaluated the accessibility of their learning materials in mathematics and statistics. At the heart of the evaluation was the RNIB providing first an expert assessment, and second user testing. The OU's report on this evaluation was published in February 2020, about a month prior to the first lockdown in the UK.

Executive summary

The report is a 29 page PDF document, produced using LaTeX. Its authors are Chris Hughes, Chetz Colwell, John Clarke, Kaye Williams, and Alison Bromley. The report is an OU document. It describes a funded project that lasted a year. The external cost was £11,000, all for expert consultancy provide by the RNIB.

This consultancy had two phases, expert assessment and user testing, with a development phase between the two phases. (This is recognized route for improving user experience.)

They found that 6 out of 8 users users would recommend studying using material in this format to other maths or statistics students wanting to study idependently at degree level.

Finally, based on the project the report recommends that the school of Mathematics and Statistics use HTML + MathJax as tested as alternative format across the curriculum in addition to its existing outputs. The report propose that the output be named Designed As A Reasonable Adjustment (DAARA).

The first 6 sections about about 2/3 of the report. They are

Introduction
Background
Aims and scope
Activities
Findings
Conclusions

Section 1: Introduction

This section describes the use of DAISY talking book at the OU, for Maths & Stats courses. It references the OU student charter and legal obligations on UK education providers. In particular websites and mobile apps must be made perceivable, operable, understandable and robust.

It defines these terms as

perceivable can the user clearly find and/or be told by their assistive technology that there is mathematical content available to them?
operable can they use their assistive technology to engage with the mathematical conent?
understandable can the mathematical content be presented meaningfully to the user so that it makes sense?
robust can all of the above be done reliably so that the user can be confident that mathematical content will always be presented to them in this way with minimal failures, changes, and inaccuracies?

Section 2: Background

This is a major section. First it gives a history, starting with Larry's speakeasy (1983) and finishing with the axessibility package of Armano et al. In between are the technologies MathML, MathJax and PDF/UA. Also mentioned are papers based on the experience of blind students by Karshmer and Bledsoe; Ferreira and Freitas; Cooper, Lowe, and Taylor; Cliffe; Godfrey and Loots.

It gives a short summary of approaches to making math content accessible. It also gives a longer summary of the experience of screen readers, NVDA and ePUB.

Section 3: Aims and scope

The report states:

The aim of this project was to evaluate, and develop, an alternative format that is specifically designed for screen reader users who wish to independently engage and interact with mathematical content.

It further states:

We decided to approach the RNIB, as their ‘tried and tested’ certification can be seen as something of a ‘kite mark’. The initial discussions with the RNIB made it clear that the ‘tried and tested’ certification would only be possible if the evaluation included testing of our VLE sites.

And so:

We were keen to keep the project focused on our alternative format, so abandoned the goal of achieving certification; we were, however, confident that the RNIB’s testing process would provide us with helpful information.

Proceeding from this they formulated requirements of the alternative format, and made an alternative format choice. They chose HTML, with mathematical content written in MathML, rendered using MathJax.

Section 4: Activities

This is a major section. As stated earlier, the RNIB provided first an expert assessment and then user testing (with a phase of development between).

This section contains considerable useful detail as to what was done, and by whom. I find it helpful to think of this section and the next (Findings) as describing (or providing) the evidence, that making the changes described would substantially improve the access blind and visually impaired students would have to the teaching materials.

Section 5: Findings

This is a major section, containing considerable detail. In particularly, it contains summaries of participants stated experience of reading and interacting with the elements on the page.

Noteworthy (and helpfully starred in the report) are

Technology problems between different sites somewhat derailed the first user test. It was solved by changing computers.
Participants liked that it was easy to interact with the maths, and then escape.
The different [tree] levels in the document participants found took a little getting used to.

In my view, some parts of usability are obvious, but for others the devil is in the detail. Or in other words, the details are important and so success requires more time and effort than expected. Reading this section will make this abstract statement considerably more concrete.

Again in my view, careful observation of user experience and listening to user comments is an important part of usability testing. It's also a major part of the scientific method when conduction experiments. Usability testing is a form of experiment.

Section 6: Conclusions

From the conclusions I highlight:

In hindsight, the first version that we submitted to the RNIB in December 2018 was not fit for purpose, and we learned a lot from resolving the 36 issues [the RNIB found].

We must expect there to be a learning curve for any users of such an output. … [W]e must ensure that training documentation is available for [our students] for reference at any point.

any potential users [of NVDA] would need a dedicated training session.

producing MathML from LATEX code is straight forward, however, producing MathML that works well and reliably in such a way that it can be engaged with using assistive technology requires care.

there are likely to be many technical challenges in implementing such an output, and it is not likely that we will be able to simply ‘flick a switch’ or ‘press a button’ and deploy the output.

there are likely to be other, as yet unknown, challenges to the implementation.

Finally, the report restates the recommendation, that the school of Mathematics and Statistics use HTML + MathJax as tested as alternative format across the curriculum in addition to its existing outputs.

The report proposes that the output be named Designed As A Reasonable Adjustment (DAARA).

The remaining sections

These are

7 Impact
8 Deliverables
9 Acknowledgements
10 Figures and Tables
11 References
12 University approval
13 Appendices