Measuring Impact

The Pitfalls of A/B Testing and Benchmarking

May 7, 2012 by brightplus3 Leave a Comment

14615953_a61bad3c43_z — Improvement begins with measurement, but the ruler can also limit your audacity to try wildly new approaches (photo by Flicker user Thomas Favre-Bulle).

Google is famous for, among other things, crafting a deep, rich culture of A/B testing, the process of comparing the performance of two versions of a web site (or some other output) that differ in a single respect.

The benefit: changes to a web site or some other user interface are governed by real-world user behavior. If you can determine that your email newsletter signup button performs better with the label “Don’t Miss Out” instead of “Subscribe,” well, that’s an easy design change to make.

The practice of benchmarking – using industry standards or averages as a point of comparison for your own performance – has some strong similarities to A/B testing. It’s an analytic tool that helps frame and drive performance-based testing and iteration. The comparison of your organization’s performance to industry benchmarks (e.g., email open rates, average donation value on a fundraising drive) provides the basis for a feedback loop.

The two practices – A/B testing and benchmarking – share a hazard, however. Because a culture of A/B testing is driven by real-time empirical results, and because it generally depends on comparisons between two options that are identical in every respect but one (the discrete element that you are testing), it privileges modest, incremental changes at the expense of audacious leaps.

To use a now-classic business comparison: while Google lives and breathes A/B testing, and constantly refines its way to small performance improvements, the Steve Jobs-era Apple eschewed consumer testing, assuming (with considerable success) that the consumer doesn’t know what it wants and actually requires an audacious company like Apple to redefine product categories altogether.

Similarly, if your point of reference is a collection of industry standards, you are more likely to aim for and be satisfied with performance that meets those standards. The industry benchmarks, like the incremental change model that undergirds A/B testing, may actually constrain your creativity and ambitiousness, impeding your ability to think audaciously about accomplishing something fundamentally different than the other players in your ecosystem, or accomplishing your goals in a profoundly different way.

The implication isn’t that you should steer clear of A/B testing or benchmarking. Both are powerful tools that can help nonprofits focus, refine, and learn more quickly. But you should be aware of the hazards, and make sure even as you improve your iterative cycles you are also protecting your ability to think big and think different about the work your organization does.

And if you want to dive in, there are a ton of great resources on the web, including a series of posts on A/B testing by the 37Signals guys (Part 1, Part 2, and Part 3), the “Ultimate Guide to A/B Testing” on SmashingMagazine, an A/B testing primer on A List Apart, Beth Kanter’s explanation of benchmarking, and the 2012 Nonprofit Social Network Report.

Our First Book Launch: The Nimble Nonprofit Hits the Streets (and Barnes & Noble)

April 26, 2012 by brightplus3 Leave a Comment

Screen shot 2012-04-26 at 11.14.27 AM — The Nimble Nonprofit is now available at Barnes & Noble ($4.99)!

Yesterday Trey and I launched our first book, The Nimble Nonprofit: An Unconventional Guide to Sustaining and Growing Your Nonprofit, with a ton of help from our Bright+3 colleague Ted Fickes.

Weâ€™re only a day into it, but itâ€™s been great fun so far: a ton of awesome reviews on Amazon, a bunch of great Twitter traffic, and even an unsolicited and really favorable full-on book review (thanks Bonnie Cranmer!).

In addition, I now have a “Jacob Smith” author page on Amazon. I wasnâ€™t expecting much when I logged in to set it up, but I must not have paid author pages much attention previously because it turns out theyâ€™re actually set up pretty well. In addition to what youâ€™d expect (profile, photo, etc.), they also allow you to bring in a Twitter feed and an RSS feed, which is a nice touch.

And great news if you are a Nook fan: The Nimble Nonprofit is now available at Barnes & Noble!

The book is in review at Apple, and as soon as it launches there we’ll announce it.

Weâ€™re thrilled to sent our little book out into the world, and we welcome your comments, critiques, and thoughts â€¦ send them our way:

email: [email protected]
Twitter: #nimblenpo
web: http://brightplus3.com/

The First Bright+3 Book Launch: The Nimble Nonprofit

April 24, 2012 by brightplus3 1 Comment

I am thrilled to announce the launch of The Nimble Nonprofit: An Unconventional Guide to Sustaining and Growing Your Nonprofit.

The nonprofit world truly is in a state of flux. Much of what used to work doesn’t anymore. The need to invest in growing ass-kicking staff and to develop sustained organizational capacity has never been greater, yet the difficulties of doing so are growing as quickly as the need. In The Nimble Nonprofit we cover a wide range of what we believe are critical challenges facing the nonprofit sector:

cultivating a high-impact innovative organizational culture;
building and sustaining a great team;
staying focused and productive;
optimizing your board of directors;
creating lasting relationships with foundations, donors, and members;
remaining agile and open; and
growing and sustaining a nimble, impactful organization.

We mean for The Nimble Nonprofit to be a guide – an unconventional irreverent, and pragmatic guide – to succeeding in a nonprofit leadership role, and to tackling this incredibly challenging nonprofit environment. We aimed for a conversational, practical, candid, and quick read instead of a deep dive. If you want to immerse yourself in building a great membership program, or recruiting board members, or writing by-laws, there are plenty of books that cover the terrain (and some of them are quite good).

But if you want the no-nonsense, convention-challenging, clutter-cutting guide to the info you really, really need to know about sustaining and growing a nonprofit, well, we hope you’ll check out The Nimble Nonprofit.

This is our first book, and the publishing industry is a state of disarray, so – following the spirit in which we wrote the book – we are taking an unconventional path. We decided to publish strictly as an e-book, and we decided to self-published (with a bunch of help from Ted here at Bright+3). We are offering the book through the big three e-bookstores (Amazon, Apple, and Barnes & Noble, and we might add a few more to the mix), and we’ve priced the book at $4.99, which is much less expensive than the vast array of other nonprofit books.

As of right now, the book is available on Amazon (and it’ll hit the other two stores shortly). If you’d like to score a copy of The Nimble Nonprofit and enjoy reading it on your Kindle, iPad, or another tablet, jump on Amazon and grab it (did I mention it’s only $4.99?).

And, because our main goal is contributing to the conversations around these critical questions, we are also making a .pdf version of the book available for free.

We suspect that most readers will agree with some of what we argue and disagree with other parts, and because we challenge much of the conventional wisdom about building strong nonprofits, we’re pretty sure that some folks will disagree with a lot of what we write. And we look forward to the conversations. Please send us your thoughts, critiques, comments, and ideas

email: [email protected]
Twitter: #nimblenpo
web: http://brightplus3.com/

Tell us where you think we’re wrong and where we’ve hit the nail on the head, and please share with us other examples of nonprofits doing a great job of tackling these challenges and where they are just getting it wrong.

Happy reading –

Jacob

(P.S. The Nimble Nonprofit is available right now on Amazon.)

Building email lists one opt-in address at a time

December 6, 2011 by Ted Fickes Leave a Comment

How organizations build email lists is no small issue. Many groups are investing significant resources in staff, consulting, advertising, events and vendor contracts (in particular partnering with Care2, Change.org and similar communities) to increase their list size.

An issue that often comes up as these programs take off is do we use “opt-in” or “opt-out?” What does this mean? Opt-in means that a new subscriber must make a proactive decision to join an email list by clicking a checkbox, filling out a form or, in the case of double opt-in by replying to a confirmation email and essentially telling you twice that they want to be on your email list. Opt-out happens when someone is added to an email list without clear prior acknowledgment and must actively opt-out if they want off the list.

We were intrigued the other day to come across a post on ClickZ looking at the results of over 300 million emails sent to sets of subscribers added to lists via opt-in and opt-out methods. Email professionals will generally discourage organizations from using opt-out methods (though, as discussed below) typical subscription practices aren’t far from opt-out and many organizations use opt-out frequently through email appends. And opt-out is pretty much assumed in political campaign marketing where lists are bought, sold, traded and given away all the time.

Opt-out is cheaper than opt-in. You may put money up-front to rent/buy an email list or run an email append against your mailing list or other house file. But dollar for dollar it costs less than opt-in. Generally, the more subscribers need to do to indicate their interest in subscribing the less likely they are to subscribe.
[Read more…] about Building email lists one opt-in address at a time

Google Analytics Tips and Resources

August 16, 2011 by Ted Fickes Leave a Comment

Google Analytics is free but getting actionable data from it takes a bit of planning and time. We wanted to highlight for you here some great resources and tips for using Google Analytics in your organization or business.

The sky is the limit (of course) as you can spend almost endless amounts of time sifting through data and creating Google Analytics reports. At the end of the day though you need your website help you achieve some pretty discrete tasks. These could be things like:

give your readers more of what they’re looking for;
make information easier to find;
get more visitors to subscribe to your email list;
get more visitors to buy your products or make donations;
see more visitors sharing links to your blog posts, videos and social media networks.

New to Google Analytics or just want to get some insight personalized to your organization? Check out our free Analytics Assessment.

Google Analytics can help you see if you’re achieving these tasks and meeting the goals you set for your online program. But you won’t find this data by just looking at just pageviews and visitor numbers.

Here are a few tips and resources to help you measure what matters and make more data-driven decisions using Google Analytics.

[Read more…] about Google Analytics Tips and Resources

Testing, testing, testing…is this message on?

May 30, 2011 by Ted Fickes 1 Comment

So you’re sitting there lamenting the somewhat lethargic results of recent email campaigns and wondering if a little tweak to your email or landing page would improve results. Maybe change up the subject line – add or remove the organization. Maybe add a photo or two to the message. Maybe change the placement of a link or form or call to action on a landing page. Would that get more conversions you wonder? There has to be an easy way to bump this up, you think to yourself.

So you post an email to a handy helpful email list largely made up of folks doing similar work asking if a subject line change would help. The feedback is extensive but largely anecdotal. Hardly anyone offers up actual data and most of the stories are second or third hand… “well, a group I used to work with put the name of the organization in the subject line and it helped a little, I think.”

And you think, “well, that good but it’s not exactly the same situation I’m dealing with here. It’s a good story but doesn’t exactly apply to my list.”

My god, man… then why not test it on your list!

The thing is, testing on one’s own list and pages is pretty darn easy (though we can make it quite complicated and involved, sometimes for good reason and other times not) but rarely done.

Okay, so the ease of testing depends on the tools at hand. If your email system/online CRM is pretty unwieldy or you just don’t know how to use it then little tweaks here and there can be massive potholes in the road, not small bumps. If you don’t know how to move things around on your site – or have the staff to do so – then little changes can be tough.

Yet what I’ve found is more often a lack of interest or curiosity about testing. More often, folks are resigned to the results they get or, if not sure, just don’t know how to proceed. What to test? How to set it up? Is it worth the time?

Valid questions all. But I think the lack of a learning culture is more the culprit. More on that in a future post but first… what to test. [Read more…] about Testing, testing, testing…is this message on?

Be More Like a Business!

April 12, 2011 by brightplus3 2 Comments

chart_1 — Nonprofit or for-profit, this organization's revenue trend has a problem.

Nonprofits hear the “be more like a business” refrain a lot, despite plenty of evidence that being like a business isn’t all it’s cracked up to be.

I’m persuaded by Jim Collins’ analysis, which more or less argues that within the business community you’ll find the entire spectrum, from extremely effective organizations to highly dysfunctional ones, and the issue is ferreting out the practices that make the good ones good. He extends this to the social sector, and essentially makes the same argument: the challenge isn’t for social sector organizations to be like businesses but to adopt the practices of the really well run businesses, as well as the really well organizations from whatever sector. I don’t think he’s ever evaluated the public sector, but I suspect his argument would be the same . . . learn what we can from the practices of the organizations that really kick ass (and I’ll bet there’s more of this than you might think, but that’s a post for another day).

In other words, conversations that focus on how the nonprofit sector can be more like the business sector are asking the wrong question. A better question: how can nonprofits learn from the best practices of highly impactful organizations within their own sector and across the business and public sectors?

Our frequent insistence on asking the wrong question leads to two kinds of problems. On the other hand, the nonprofit sector is often so allergic to the very idea of the business sector that it runs away from it, and eschews anything that smells like it might have come from the business world (the “we don’t do things that way” reaction). The problem is that there are a bunch of practices you find among the best-run private sector organizations that nonprofits would benefit from using. It’s not “be like a business,” but “adopt the applicable practices of the best businesses.” The most successful of the private sector organizations are often strong at things nonprofits are often weak at: capacity investments, staff cultivation, and staff management among them.

And then there are nonprofits that run in the other direction, thinking that salvation lies in being like a business, and they adopt private sector practices in seemingly wholesale and indiscriminate fashion, which often produces some of the weirdest and worst dysfunctions, since it involves the hybridization of worst practices from both the nonprofit and private sectors.

Nonprofits really are, at root, businesses. They are different in some important ways from traditional profit-oriented businesses, but they are similar in many more ways. Nonprofits live and die on cash flow, for example, and they have to offer a sufficient and appropriate value proposition to their supporters. But the implication isn’t “do whatever private sector businesses do,” since that would be dumb, but rather that the laws of business physics really do apply even if you are serving a social good of some kind. The sense of nonprofit exceptionalism that pervades much of the nonprofit world, the notion that we are special and different because we are a nonprofit, is harmfully misguided. If you apply a discriminating eye toward the private sector, however strong your allergy may be, you’ll find a ton of practices and accumulated wisdom that will help your nonprofit do good, better.

The Nonprofit Dashboard Roadmap

April 3, 2011 by brightplus3 7 Comments

167428102_79e33842cd — Photo by Flickr user istargazer.

Dashboards have been used heavily in parts of the private sector for a long time, but they seem to be making inroads in the nonprofit and government worlds as well. The premise: just as with a car or airplane dashboard, a nonprofit staffer can quickly glance at the gauges and clearly understand the real-time status of key systems and indicators across the organization. The understanding won’t be deep – they may need to probe more thoroughly to fully understand anything they are seeing on the dashboard – but they can very quickly get a sense of how well the organization is functioning and if any problems are emerging.

In our view, dashboards have the following characteristics:

They are dynamic. They display information that is changing on a regular basis.

They rely heavily on gauges or other data visualization displays to convey information in readily understandable formats.

They allow for quick status assessments, although they may enable deeper inquiry

Even with these criteria, however, the dashboard concept is used to describe a diverse range of displays, each distinct in function and design from the next. As the dashboard concept takes great hold among nonprofits we are encountering some confusion among our clients about which is which and what people mean when they use the term.

We took at stab at delineating five distinct types of dashboards:

1) Business Intelligence Dashboards
These display detailed information about a particular area of an organization’s operations. Many customer relationship management systems, for example, like Salesforce and Raiser’s Edge, include dashboards to make it easier for the development staff to track fundraising activities, donations, and other performance measures. Fundraising and advocacy management tools like Convio use dashboards to display campaign status. Google Analytics, with its robust dashboard system displaying key web site metrics, is another example. Technical dashboards help specialized staff keep a close watch on what’s happening within their organizational purview. Technical dashboards are typically inward facing, so that only staff and perhaps board members can view them, but they can be outward facing as well. The Indianapolis Museum of Art’s dashboard is an oft-cited example.

2) Status Dashboards
Organizational status dashboards, like the one the software company Panic described on their blog, are another variant. In contrast to technical dashboards, which tend to focus on a single functional area within an organization, status dashboards display less information from a wider range of functional areas across an organization. A status dashboard is the answer to the question: what is the critical information everyone in the organization should be able to view all the time? Rather than probing deeply into any one area of an organization’s operations, they offer a broader overview of value to everyone.

3) Accountability Dashboards
Increasingly, we are seeing dashboards used in external accountability contexts: a nonprofit or local government that wants to share its real-time performance data with its donors and its community. The Town of Oro Valley in Arizona maintains a financial dashboard displaying the town’s financial performance compared to past trends. It’s not a great example in that it is only updated monthly, and not in real-time, but it’s at least in the ballpark. Over at PlaceMatters (where I spend part of my week), we’ve been doing a lot of work on sustainability dashboards, web-based tools that openly share a community’s performance against its sustainability goals. Incidentally, we described the Indianapolis Museum of Art dashboard as a Technical Dashboard because of its depth, but it really serves as an Accountability Dashboard as well.

4) Tracking Dashboards
These can be inward or outward facing, and typically show visualizations of unfolding data streams in real-time. These aren’t organizational in nature but, rather, are tracking events that are taking place outside the organization. The data stream may have implications for an organization, but it isn’t specific to that organization. Al Jazeera’s “Region in Turmoil” dashboard shows the volume of Twitter traffic by country in the Middle East as a proxy for the level of political activity.

5) Scenario Comparison Dashboards
These are typically designed to compare likely outcomes of a range of future scenarios across a range of key metrics. For instance, MetroQuest uses dashboards to compare multiple regional development scenarios across factors. CommunityViz, a GIS-based data visualization and decision support tool, allows user to analyze the environmental and other community outcomes from a range of land use scenarios, and it uses dashboards to display those outcomes across a range of factors.

We shouldn’t entirely neglect the category of “Displays That Are Called Dashboards But Aren’t.” It probably isn’t useful to use ‘dashboard’ to refer to web pages full of relatively static narrative information, for example. One example is the recycling portion of the Emory University Sustainability Dashboard.

We would welcome your thoughts. Does this seem like the right breakdown? Are we missing anything? What are the terrific examples of each category?

« Previous Page