In this installment of Accessibility Conversations, Jeff Kline, a seasoned expert in the field of accessibility, sheds light on the crucial role of metrics in creating a culture of accessibility in their organizations.
Like any journey, accessibility initiatives must start with a clear understanding of where your organization stands and where you aim to reach. Whether you’re at the starting point of your accessibility journey or are already well underway, the key lies in evaluating your current status and crafting a clear roadmap to reach your accessibility goals. An essential component of this roadmap is the establishment of a shared set of actionable metrics. These metrics serve a dual purpose: they enable you to track and report your progress, ensuring that your investments in accessibility programs yield tangible results, both in terms of improved digital accessibility and a measurable return on the time and resources dedicated to these initiatives.
Building Your Accessibility Journey Roadmap
Sanjay Nasta: Today, we’re privileged to have Jeff Kline with us to discuss a topic of increasing importance: metrics related to digital accessibility. Jeff, in your opinion, why is it so crucial to consider these metrics when establishing an accessibility program for an organization?
Jeff Kline: Sanjay, you’ve touched upon an often-overlooked area. The core reason to consider metrics when designing an accessibility program is that metrics can help measure and drive progress. When you establish metrics and set goals around them, it propels an organization forward. It indicates the organization’s maturity and commitment to using and producing compliant products and services.
The only way to know if you’re making progress or stagnating in your accessibility program is to establish baseline metrics and then measure progress against them.
Sanjay Nasta: As Peter Drucker said, you can’t manage what you don’t measure.
In your experience working with different organizations, what metrics or indicators have you observed that these organizations commonly use to gauge their progress in digital accessibility?
Jeff Kline: Well, it genuinely does vary based on where the organization is in its accessibility journey. For those initiating an accessibility program, metrics might lean more on the qualitative side.
For example, a goal of your accessibility plan would likely be to train people across your organization in different aspects of accessibility. Within the first six months, you can set 50% as a metric of the entire population to be trained on the basics of accessibility and how they relate to their job. In a year we may want 100% of employees trained. The obvious metric to measure and track progress towards this goal of the percentage of employees trained.
As the organization matures in its accessibility initiatives, more quantitative metrics come into play, again driven by plan goals. You look at establishing baselines to where the starting points are, the create the goals and metrics for compliance across your organization.
For instance, the web group can set a monthly target of reducing accessibility errors by 10%. The procurement group can set a goal of including accessibility requirements in 50% of IT procurements within the first six months and then perhaps 100% by year-end.
As groups report metrics to management, it fosters accountability and encourages improvement. Once you have a meaningful metrics process established, it can help drive providing accessible solutions to your organization.
You can measure many things. As you advance, the focus may shift from training, establishing processes, and governance metrics towards ensuring detained WCAG compliance for example. It’s essential to refine those goals, aim for 100% compliance in all areas, and consistently maintain that standard without regression.
Sanjay Nasta: That’s an insightful point. So, would you say that the metrics an organization chooses reflect the phase or progression stage of its overall accessibility plan?
Jeff Kline: Absolutely. It’s a roadmap. Your metrics have to evolve with your journey.
Building Baselines for Measurement
Sanjay Nasta: The interesting thing was that you recommended more qualitative metrics upfront and then set quantitative metrics later as you get more experience.
Jeff Kline: At the start, it’s crucial to assemble the program and set qualitative goals and dates. The qualitative metrics drive the establishment of timelines and goals that will ultimately be used to drive actual IT compliance. So, initially, you’re not measuring the number of defects found versus defects fixed because the “infrastructure” of the other pieces is not yet in place to do that.
Sanjay Nasta: How do you set a baseline at the beginning of the project so you can measure progress against that baseline?
Jeff Kline: It depends on what you want to measure. For example, for education and training, you can do an initial inventory of the skills you have on board, look at what those skills are, and how many people have them, which becomes your baseline. From there, you want to do a gap analysis and build a training plan.
When you’re looking at website accessibility, the first time you do your scan (using a scan tool/service) or the first time you do your manual test, that will become your baseline, so you want to show progress from there.
Sanjay Nasta: It also gives you a nice easy way to report back to management the progress you’ve made.
Jeff Kline: Absolutely. All the measurements in the world aren’t any good unless you have somebody to who you send the reports, who sees them and asks the right questions to make sure that everything is on track and being driven properly.
Engagement and Sharing Succeses
Sanjay Nasta: What’s your experience on reporting metrics up to management and using that to validate the accessibility program to get the buy-in and budget that you need?
Jeff Kline: In some of the bigger organizations, metrics can set up friendly competition regarding compliance between areas of the organization. The group making the most progress will want to stay on top, and the group not doing so well doesn’t want to stay there.
Of course, metrics may also be used to drive the trajectory of investments and prioritization of resources for the initiative.
Sanjay Nasta: Do you have some examples of how you’ve used metrics in the past to improve the accessibility program that you were managing?
Jeff Kline: For example, in my work with the State of Texas, we conducted bi-yearly accessibility surveys across 140+ agencies. Our survey methodology was to maintain standard questions to track progress over time survey to survey for consistency, but then a subset was to deal with more dynamic aspects that may vary from survey to survey. Then, some of those may be viewed as something to track over a bigger time frame, so they get added to the consistent question set.
One of the most consistent results was that people couldn’t train their employees. They didn’t have the budget or skills, and they needed better training sources. Using that data, we justified and implemented a state-wide training program where any Texas state employee or employee of Texas state-funded higher ed(ucation) could access a huge training catalog that we had under contract with an accessibility service provider.
The statewide survey metrics drove instituting a training program.
The survey also included questions to understand compliance levels. Many agencies said, “We don’t have the money or the accessibility tools to measure compliance.”
So we used that input to justify and implement a statewide government scanning program with another accessibility supplier. Once we did that, we could see the improvements at agencies because they had the tools and skills to go in and remediate their websites and fix things.
Guidance for Managers New to Accessibility
Sanjay Nasta: One of the advantages you have is that you have a tremendous amount of experience and knowledge on the elements that you want to measure and track. I believe someone new to the accessibility role has to identify which behaviors are crucial for accessibility and then decide on the relevant metrics. For those just stepping into this role, it can be quite a learning curve.
Jeff Kline: It’s a challenge. When embarking on an accessibility initiative, having someone with a broad range of experience is invaluable. While expertise in accessibility is a plus, it’s not mandatory.
A seasoned project manager with a background in large-scale projects could ramp up on the accessibility world (non-technical aspects, which we are dealing with in this chat) and then fairly effectively set the groundwork. While the specifics of accessibility can be detailed, with input from technical experts, adept project managers should be capable. So it’s also essential that accessibility coordinators and leaders have a grasp of project management skills.
Qualitative and Quantitative – Striking a Balance
Sanjay Nasta: We talked earlier about qualitative metrics and quantitative metrics. How do you create a balance between the two? I’ve learned that if you don’t measure the right things, it can drive incorrect behavior. How do you balance between qualitative and quantitative?
Jeff Kline: Many of those qualitative items we discuss are in maturity models. For example, the new W3C maturity model has seven dimensions, each containing a series of proof points. The model looks at how far along an organization is implementing those proof points to reach the highest degree of maturity (optimize state). The model is based on four different levels of maturity, which can be seen as somewhat subjective levels depending on the proof point specifics. That would be a good example of qualitative as opposed to quantitative.
Sanjay Nasta: We talked a bit about how the accessibility metrics should be related to your goals and your accessibility plan for that year. Is there anything you want to expand on that?
Jeff Kline: The only thing I could say is some of those things may come with a price tag. That might need to be factored in as a dependency or assumption. For example, buying tools, training, learning management systems, or live instructor-based training costs money. It’s important to recognize this as part of your goal setting. You may say, “We want to have everybody trained, or all of our developers trained, in the next nine months on how to build accessible applications.” However, the tools or the training may not be available because it couldn’t be procured for seven months, putting the target date at risk.
Now you’ll miss your target because you only had two months to play catch up, or somebody says, “Sorry, we don’t have money until next year.” You have to temper those goals with the reality that there will be other factors/dependencies that enter into it. I can attest that in state government, procurement can be long and drawn out.
Quality Assurance, Resources and Final Thoughts
Sanjay Nasta: Are there any tools that are useful for setting up metrics for accessibility? Are there sources of standard metrics or survey tools you’ve used in the past?
Jeff Kline: One resource that I’m thinking of is, I mentioned, the new W3C maturity model. it’s in the second draft, and it is available online. You can look at the document as well as the spreadsheet tool that you can use to measure progress against all those dimensions and proof points. That should give you some ideas on the qualitative and quantitative metrics you need to measure. The software development lifecycle section can get into all the defect rates and remediation rates and things like that.
Sanjay Nasta: That’d be interesting because, in many ways, accessibility should just be part of the QA process. The technical part of accessibility is a QA function.
Jeff Kline: I would disagree with that and say if you put all your eggs in the QA basket, you’ll have a bunch of rotten eggs because accessibility needs to be included way up front in the requirements, design, implementation, and finally testing phase. Take it out of the main flow, and trying to “test in’ accessibility at the end will result in lots of disappointment….and high remediation costs.
Subscribe to Accessibility in the News
Stay informed! Get your weekly update on digital accessibility standards, private and public sector trends, litigation, events, and more.