Category Archives: Institutions

Let’s start measuring impact at art museums

In the last few months several major institutions have publicly scaled back on digital operations, often as part of larger cost-cutting plans. Others have reorganized, or lost experienced tech staff in response to diminishing executive expectations. The Metropolitan Museum of Art, for example, is scaling back on digital in order to cut costs. Staff of the vaunted Indianapolis Museum of Art Labs have moved on to private-sector work.

These moves disappoint me for two reasons. One, because digital can be a revenue center: in my time at the Met my team ran email marketing, and we grew that program from $1.1M in revenue to $3.6M over three years–judiciously adding about $70K/yr in staffing costs when we saw that it would substantially increase revenue. Two, because we have never really quantified tech’s contribution to the mission-related impact of our organizations, or even to many revenue-related goals. (Our email marketing was so successful partly because we had great materials to work with–but we don’t know how many people learned about art from it, or came to the museum because of it.)

Right after the news that the Met will cut jobs, they announced record visitation. That’s 700K additional visitors on top of record attendance in 2012, for additional revenue of roughly $9M (let’s say the average gate runs a little over half the suggested price, which is now $25), just from admissions. Since 2012, the Met has added over a million followers on social media, an increase of about 40%. Web visits in 2012 exceeded 60M annually. That means that daily, over 100 thousand people, maybe 200 thousand, interact with the Met online. Do we think that has no impact on bringing people in the door? Where they spend money? The answer is, the Met doesn’t know, and nobody knows, because we’ve never investigated the links.

When we at the Met in 2011 re-launched the Met’s website, one of our primary goals was to increase visitation and serve visits. It wasn’t just that we built an interactive map with gallery/exhibition/collections integration, or a mobile site that focused on services for people in or near the building; we tried to build a site with an architectural integration into the institution, and we succeeded to enough of a degree that even some art critics noticed. As Ed Rothstein from the NY Times said, “The images do not replace the objects; they are pointers to them. The site ultimately defers to the place by creating a close association with it.”

What we never built, though, was a system for measuring the impact of that work and attributing growth in visitation to any particular online activity. We discussed it; for example, I invited experts in impact measurement from the New York Hall of Science to introduce us to the topic, but interest was weak and I left shortly afterward, so I can’t speak to what followed. What I know is that no system like it was built, and that very few museums are considering systems like that today.

However, what’s lost is not the impact itself; online efforts are surely bringing more visitors in and educating them directly (and driving other goals like membership/donations). What’s lost is any ability to directly attribute visitation, or other mission-related impacts, to online efforts. This is a failure of museums and museum tech teams to quantify the value of what digital is providing for their museums.

(One question worth asking: how is that we’ve never been required to do this before? The answer is that most funders don’t ask for impact reporting. When I was at the Met, 2006-2012, I was surprised by the minimal data reporting requested by our big donors. I always wondered if they were afraid to ask for more meaningful data–if they thought we would struggle to provide more sophisticated numbers–or if they didn’t know what to ask for. Here’s an article from shortly after that time that discusses why three pretty sophisticated funders don’t really ask their grantees for impact reports. But! In some fields–like science centers–it’s common, and it’s growing in many fields, and in fact one reason I was inspired to write this post is that for one of my clients, I’m working on impact reporting now. A donor has required it, finally.)

So what do we have to do? Bring a growth-hacking approach to museum technology operations. This is the next big thing in public-facing Museum tech. The collections are online, the social media networks are in place; now let’s show that it’s all working. Museums are a little bit on their own when it comes to visitation hacking, since that’s not a goal in many other areas. On the other hand, there are models for how to measure mission impacts, like educating people on a topic, or changing minds and moving the needle on a social issue.

Here’s a simplified model of how your everyday advocacy online activity (website, social media program, whatever) growth-hacks attitude changes:

      1. Establish the attitudes of new users (with surveys for example).
      2. Survey exiting/return visitors to identify attitude changes.
      3. Correlate attitude changes with behavioral patterns on the site/online tool.
      4. Optimize tools to increase the proportion of visitors who exhibit behavioral patterns correlated with positive attitude changes.
      5. Iterate, to make sure you aren’t suppressing repeat visitation with an overly mercantile approach.

(Of course, you could also be working on the incoming traffic end, with Facebook ad campaigns for example. But ultimately that’s more or less linear with expenditures, provided you’re A/B testing messages, etc. So it’s not really a performance measure.)

This would work perfectly for online sales, donations, tour registration–anything that takes place fully online. And we can do the same thing for visitation, or mission impacts like education/appreciation of art history–it’s just harder, and less accurate. The problem is that we have to correlate online behaviors with physical visits that happened at some other time. Mostly. But, we can:

  1. Note when people have our websites and webapps up on their phones as they near, or enter the building, and what they were doing online just beforehand (this one might not be possible with social media).
  2. Ask people entering the building, or inside if they came following a website visit or social media interaction, and ask them what they did online. Then optimize for & promote those behaviors.
  3. Continually revisit those measurements as we optimize for and promote behaviors associated with visitation, and show that the proportion of visits driven by online efforts is improving, and the overall number is increasing (and that the cost of increasing the number is smaller than the revenue each new visit represents).

What are the impediments?

First, a bias in museum-tech circles against the need to demonstrate success in numbers so mundane as revenues and visitation. Aren’t there times you thought you shouldn’t have to do this? Isn’t 60M annual web visits a good thing all on its own?

Second, in an atmosphere of cutbacks, how do we convince our institutions to spend money on this?

Third, we don’t have a strong history, in museums, of understanding online data and statistics, and almost no knowledge of impact measurement. We know some numbers are going up or down, but probably not why the really important numbers are going up or down.

To the first issue: shake it off. Care about visitation; or focus on mission-related goals like introducing new audiences to art, and measure your impact there. Figure out if your Facebook feed or your website is actually educating people, and optimize your efforts to maximize that.

To the second: don’t spend money on it. Build the data-tracking into your app development, prioritize it among your team’s daily duties, and do the surveying like you do your guerrilla user-testing–for free. Talk to people waiting in line; see if you can get special treatment of some kind for people who give their time to your two questions.

To the third: this is where leadership comes in–commit yourself to understanding how it’s done and how to derive meaning from the numbers.

Go for it!

Thanks to Rich Cherry and Douglas Hegley for conversations that dramatically improved this post.

 

 

Search-dominance and NYPL’s home page redesign

An earlier version of this post appears on the New York Public Library’s blog. I’m highlighting it here as a success in applying generalized web UX research to the specific needs and content of an institution.

Among my first tasks upon taking leadership of the Website Department at New York Public Library in late 2012 was to update its old and ineffective home page. Driving the strategy were a few important points that distinguished NYPL and its web site from many others:

  • a huge proportion of visitors used the home page simply as a pathway to catalog searches, leading to book-checkout transactions.
  • With 88 branch (neighborhood) libraries, and four large research libraries, NYPL was a HUGE institution, with a TON of great events and classes going on all the time, that hardly anybody knew about.

I wanted to support that aggregate search-dominant visitor behavior, but also to make people aware of all the other amazing things the library did (and still does). We needed a great, beautiful home page that didn’t make search any harder, but put those events right in your face.

Continue reading

What to include in a web design RFP, RFQ, RFI

A lot of the work that I do on the side is helping nonprofits investigate and choose vendor partners for major tech upgrades. Website rebuilds, Art Management (Collections) systems, fundraising systems, CRMs, that kind of thing. I frequently get asked what to include in an RFP (or sometimes get asked to write them).

Here is a list of typical content across many of the requests for proposals/requests for qualifications/requests for information that I’ve been part of sending out.

Continue reading

What zoos’ websites could be doing for them

Museums, especially art museums, have made tons of progress in figuring out what it means to be a museum in the internet-connected world. All significant art museums now have most or all of their collections online, engaging social media strategies, and effective email marketing programs. And all these efforts are all fundamentally content-driven (i.e., they use the art as the draw), serving the mission at the same time they drive visitorship.

I would draw a contrast here between museums, on the one hand, and zoos and aquariums on the other. Zoo and aquarium sites are heavy on marketing–though without always being savvy about fundraising–and usually provide pretty good visitor information. But they do almost nothing to serve their missions online.

Continue reading