Bright+3

Bright Ideas Change the World

Bright Ideas

The Pitfalls of A/B Testing and Benchmarking

by

Improvement begins with measurement, but the ruler can also limit your audacity to try wildly new approaches (photo by Flicker user Thomas Favre-Bulle).
Google is famous for, among other things, crafting a deep, rich culture of A/B testing, the process of comparing the performance of two versions of a web site (or some other output) that differ in a single respect.

The benefit: changes to a web site or some other user interface are governed by real-world user behavior. If you can determine that your email newsletter signup button performs better with the label “Don’t Miss Out” instead of “Subscribe,” well, that’s an easy design change to make.

The practice of benchmarking – using industry standards or averages as a point of comparison for your own performance – has some strong similarities to A/B testing. It’s an analytic tool that helps frame and drive performance-based testing and iteration. The comparison of your organization’s performance to industry benchmarks (e.g., email open rates, average donation value on a fundraising drive) provides the basis for a feedback loop.

The two practices – A/B testing and benchmarking – share a hazard, however. Because a culture of A/B testing is driven by real-time empirical results, and because it generally depends on comparisons between two options that are identical in every respect but one (the discrete element that you are testing), it privileges modest, incremental changes at the expense of audacious leaps.

To use a now-classic business comparison: while Google lives and breathes A/B testing, and constantly refines its way to small performance improvements, the Steve Jobs-era Apple eschewed consumer testing, assuming (with considerable success) that the consumer doesn’t know what it wants and actually requires an audacious company like Apple to redefine product categories altogether.

Similarly, if your point of reference is a collection of industry standards, you are more likely to aim for and be satisfied with performance that meets those standards. The industry benchmarks, like the incremental change model that undergirds A/B testing, may actually constrain your creativity and ambitiousness, impeding your ability to think audaciously about accomplishing something fundamentally different than the other players in your ecosystem, or accomplishing your goals in a profoundly different way.

The implication isn’t that you should steer clear of A/B testing or benchmarking. Both are powerful tools that can help nonprofits focus, refine, and learn more quickly. But you should be aware of the hazards, and make sure even as you improve your iterative cycles you are also protecting your ability to think big and think different about the work your organization does.

And if you want to dive in, there are a ton of great resources on the web, including a series of posts on A/B testing by the 37Signals guys (Part 1, Part 2, and Part 3), the “Ultimate Guide to A/B Testing” on SmashingMagazine, an A/B testing primer on A List Apart, Beth Kanter’s explanation of benchmarking, and the 2012 Nonprofit Social Network Report.

The Digital Nonprofit Toolkit

by

For most nonprofits, a suite of digital tools can be a critical asset for enabling your team to do amazing work (not to mention simply enabling yourself to kick ass). As consultants to nonprofits that have hands in some start-ups, we don’t have the same needs and use cases as many nonprofits. But we’ve spent years working in nonprofits and are collaborating with many on a daily basis.

Here’s what we are using these days:

File sharing: We use Dropbox a lot, although Google Docs can be a good alternative if you’ve got a shared account (and we suspect that the new Google Drive is going to give Dropbox a serious run).

Email: I mostly use MailChimp…it’s inexpensive, reasonably easy to use and integrates with a wide array of third-party apps. Creating and changing templates can be annoying at times but, hey, nobody said HTML email was as easy as making toast. Documentation is solid, the Chimp folks blog frequently and seem genuinely interested in nonprofit implementations.

I am increasingly a fan of SendGrid, as well. They’ve got awesome customer service, it’s much easier for a non-techie like me to design and modify the email templates, and they are slowly rolling out free and low-volume pricing options. If you do high volume email, especially if you want to build your own internal UI, then SendGrid seems like the obvious choice.

We realize that most mid-size to ginormous nonprofits (and many small groups) are going to be using tools from Blackbaud, Convio, Salsa or other companies that mush together email, advocacy and fundraising. If you’ve got the budget and you’d prefer the combined multi-function product instead of stand-alone elements, they can make a lot of sense.

Emailing large files: YouSendIt and Dropsend are both solid. They can be pretty useful when the other party isn’t using DropBox.

Document collaboration: For all its quirks, we haven’t found anything that beats Google Docs (which is now being folded into Google Drive).

Bookmarking, Notetaking, Writing: Ted has become a consistent and almost fanatical user of Evernote (and if Instagram can be worth a billion dollars then why not Evernote?). It covers the bases, it’s easy to use, and it’s very accessible on multiple devices. Our main complaint is that its formatted content doesn’t paste well into other tools.

Cloud storage: For simple storage, I’m a fan of Rackspace, and I especially love their “fanatical” customer support. We know others who like Amazon Cloud, as well.

Backups: I use a pair of external hard drives and Time Machine, but I might end up exploring something cloud-based (maybe as a supplement to my hard drive-swapping approach). Anyone out there really in love with a particular cloud backup solution?

Social media dashboards: Hootsuite has been the hands-down winner for me, although it’s really just a Twitter dashboard. I like the UI, it’s easy to use, and it does what I want. You can include other accounts, like Facebook, but it doesn’t work as well for those (which I think is generally true of dashboards like this). Ted is happy with Hootsuite as well but is less enthusiastic about it … he finds the interface to be clunky, and there are some annoying issues when trying to manage/admin client Twitter accounts (e.g., if the client is already using Hootsuite free version to manage their Twitter account you can’t get access to it via your own Hootsuite, which is just silly).

Time tracking/Invoicing: When you’re an independent consultant or small shop tracking your time is both a pain in the arse and one of the most critical parts of surviving. We’ve been very pleased with Harvest. Heck, most nonprofit enterprise time tracking systems could learn a lot from Harvest and similar systems.

Project management: Basecamp and Wrike are my two favorite project management tools. They take different approaches, the former built more on a “Getting Things Done” type of structure while Wrike is a little more traditional, but they both have good UIs and solid features. I’m just starting a project using Smartsheet (because it integrates with a very cool public input tool we are using called Crowdbrite), which I’ve not used before … I can report on it in a few months.

Blogging/web platform: WordPress, especially when it’s used with the Genesis framework. It has its quirks, and it can be tough for non-techies to build out a site with any real customization (although I can recommend great web designers if you need anyone), but every blog I use now is built on WordPress. It’s robust, the UI is solid once the site is built, and it looks really good. We’ve seen (and built) some great sites that are much more than blogs using WordPress. It can be done. But think hard about Drupal, especially if you’re building a broad content and/or community-rich site.

Music: Because who doesn’t need music sometimes while they work … I’m a fan of Spotify and Pandora, although our local classical station (KVOD) and the terrific Santa Monica indy station KCRW get a lot of my streaming as well. Ted is a big Rdio fan, which I haven’t tried yet (but should because it’s far better than Spotify, Ted claims). He also threw in a vote for KCRW and for Denver’s KUVO. We are both fans of Seattle’s KEXP for edgy alt-rock.

Link shortening: I mostly use Bitly. It’s simple, free, and has decent analytics. Hootsuite integrates ow.ly, though, which also works fine.

Password management: It took me a while to warm up to it, and because I have multiple Dropbox accounts the Dropbox-based syncing didn’t work (they have other sync options), but I’m now a solid fan of 1Password. I keep track of one master password and it keeps track of everything else. Very handy once you get over the hump of actually using it.

Online stores and e-commerce: The combo of Shopify and Stripe seems to work well for managing online stores and the related e-commerce transactions.

What did we miss? Other great options we should cover?

Photo courtesy actna.net (which has a pretty good article on digital tools that should be in a journalist’s toolbox).

Giving Feedback: A Starting Point

by

This was originally published as a guest post on Frogloop on April 30, 2012.

Some approaches to giving feedback are more useful than others. Singapore 2010 Youth Olympic Games photo via Inc. Magazine (Creative Commons).
Over the course of more than 100 interviews with nonprofit staffers while writing The Nimble Nonprofit, “managing people” turned up on the list of “skills that my executive director sucks at” more than any other. And giving feedback is one of the specific skills that nonprofit managers seem especially weak at.

It’s not just nonprofit executive directors that struggle with this challenge. Anyone at a nonprofit who manages people – a senior staff attorney managing a team of six lawyers, a development director who supervises a grants manager and a membership manager, a communications coordinator directing the work of a media consultant, a program associate managing an intern, a site manager supervising a group of volunteers – giving feedback is an important skill for all of these folks.

And giving feedback, especially negative feedback, is hard. It’s awkward to tell someone you aren’t satisfied with her performance, or to explain to her what she’s doing wrong, especially when you know she puts in long hours, she works hard, you don’t pay her as much as she deserves for the work, and there’s a good chance she’s your friend in addition to being your direct report. It’s not surprising, then, that so many managers give vague, soft-pedaled feedback that obscures the criticisms.

Swinging too far in the other direction is a problem, as well, where the feedback comes off as a brusque attack on the recipient’s character, integrity, and basic self-worth. You might think you’re doing him the favor of candor and clarity, but if it sounds like you are questioning his basic value to the organization your feedback won’t probably have the desired effect, either.

Some advice:

* Offer both positive and constructive feedback every time.
* Be direct, clear, and pleasant (which is very different from being nice, as Mike Monteiro points out).
* Provide concrete guidance on how to improve or remedy the problem.
* Give feedback frequently.
* Explicitly tie your feedback to your previously established and explicit expectations about their job performance.

The goal in giving feedback is – or at least should be – really, really simple: enable your staff to kick ass. Your feedback needs to validate and support. It needs to communicate in concrete terms what they’re doing well and what they need to improve. And it needs to be clear (so they understand it) and actionable (so they can do something with it).

A lot of smart folks have written a lot of smart things about giving feedback, and it might be worth spending some time skimming through some of the literature for ideas. The Harvard Business Review blog, Dummies.com, and the Manager Tools podcast are just a few good examples.

But these five principles, mixed in with a healthy dose of your own intuition and thoughtfulness, and topped with a commitment to try, learn, and improve, are a good place to start.

A Challenge to Authors and Publishers: Support a Terrific Nonprofit for Free

by

Last week, Seth Godin on The Domino Project blog issued a challenge to authors and especially publishers: donate ebooks to the nonprofit Worldreader, which loads them onto Kindles that are then distributed to kids in sub-Saharan Africa.

The cost to the author and the publisher for each ebook donated to Worldreader? Zero. No one loses out on sales (the people who end up with these Kindles weren’t going to buy those books anyway) and the marginal cost of distributing an ebook is zero.

Most of the authors we know have written books for more specialized audiences, which, like our book The Nimble Nonprofit, don’t really have audience among kids in Ghana (as Worldreader pointed out when they very graciously declined our offer to donate our book), but for those of you that have written books which might be appropriate, please consider contacting Worldreader and offering to donate your book to their efforts, and please consider as well calling your publisher and encouraging them to do the same.

By the way, while the cost of donating your ebook is zero, the benefits are pretty amazing. As Gigaom reported last week, the results of the Worldreader program have been pretty impressive.

The scarcest resource at nonprofits is bandwidth. Period.

by

Ever been in or around a nonprofit and heard something like:

Think of what we could do if we raised another $500 (or $50,000 or $5,000,000)!

What would you do?

In our experience, the answer is often something like “we would do this new campaign or program by hiring another person or part-time researcher or a consultant.” Or the need for new money/people is driven by the desire to take on a new project.

Let’s face it, ambition and heart are huge in most organizations. Nonprofits and the people in them want to do good. One more campaign. One more program. We can do it.

Stop. Organizations (even, ahem, cushy ones) are stretched thin. Even if you’re managing time well and not burning out chances are good that you’re tackling too much and maybe turning out some mediocre results (which, by the way, would probably be improved by the extra funding that would pay for more people – right?).

You never ever have enough bandwidth

This piece is inspired by Mark Suster’s recent piece on the scarcity of management bandwidth at startups. Mark points out that as a VC he meets with leaders of new companies all the time and his most common reaction to hearing them describe what they’re doing is basically, “whoa, that’s way too much.”

At a tech startup this is seen most often in new features. Every new feature, project, or marketing idea adds complexity and, most likely, is something that will need to be supported forever regardless of whether or not it works.

Complexity adds to staff time which is a burden on management. A new idea or project (no matter how brilliant) is a shift of focus. Even if you cut previous programs you’re redirecting staff and management time to something new and adding to lead time.

The time and mental energy it takes to get a project rolling is rarely accounted for in nonprofits (or startups, apparently). Shifting gears (or adding gears, as the case may be) means more planning, more meetings, more reporting. Less doing.

Mark Suster focuses on scarce management bandwidth at startups. And for good reason. I think the problem at nonprofits is bandwidth in general. This can inevitably become a debate over scarce resources (if only we had more money, better computers, better facilities, more volunteers) and for some direct service agencies its true that more resources often means more service provided and more people helped.

Yet in most cases this is an issue of not focusing, unclear goals, weak management, being pushed by leaders (inside CEOs and outside funders, for example) to do more. Resulting work can often be of the mile wide and inch deep variety – broad but shallow.

Many startups will flounder and go out of business at this point as revenue/capital/enthusiasm dry up. Nonprofits, however, can plod along with diminished resources. This may create a culture of diminished expectations, where doing okay is worth a pat on the back, a raise or, heck, a step up into directing programs.

Bandwidth is precious. You get used to stretching it, doing too much, accomplishing less than you should.

Protecting and creating bandwidth

What’s to be done? Here are a few ideas.

  • Say no. That is hardly as easy as it sounds. We know that all too well.  But pushing back on ideas (however great) and requests is the best way to stay focused and not get spread thin. This means saying no or some version of it to staff, board, donors, community leaders and more. If you have clear, program-driven explanations people will understand and probably even appreciate your focus and honesty.
  • Realize that “nonprofit management” is an oxymoron. Nobody gets into nonprofit work because they love management. Few people with nonprofit management responsibilities receive training or support. Most just end up responsible for a team and would rather be doing direct work themselves. This means that as a leader/manager you need to be willing to seek out and accept help in becoming a better manager. And trust your team to do more of the direct work.
  • Make timesheets matter. This isn’t the same as “make sure everyone fills out a timesheet.” No no no. If staff can’t stand timesheets its probably because they don’t seem to matter. It is an HR or grant reporting thing. As a manager, sit down with staff and figure out how to connect time to goals and outcomes. What is the work that actually goes into getting a job done? Be creative with tracking time and continually reflect back with the team on how time connects to outcomes.
  • Kill regularly scheduled meetings and conference calls. Or at least shed some of them. Ever notice that a new project often means a new weekly conference call (and dozens of emails ahead of time to figure out when to have that call/meeting)? Is work getting done in those regular meetings? Or are they “check-ins?” Think carefully about who is there, why, and if you need to meet at all.
  • Reflect. Reassess. Repeat. Get into the habit of checking on progress towards goals, what’s getting your team there and what’s getting in the way. Do more of the former that’s getting you there and less of the latter. Genius. Not really. But we’re surprised by how often it’s simply assumed that work and being busy equals progress. Further, try not to set up weekly check-in meetings to assess progress. See above. Have more conversations. Ask pointed questions. Listen more and listen well.
Good luck and back to work.

Photo: Drowning under a mountain of paper by net_efekt, Flickr.

A Nonprofit Turnaround Story: The Santa Cruz Museum of Art & History

by

I’m an energetic fan of Nina Simon’s commitment to designing deeply participatory engagement, her willingness to experiment and take risks, and her transparency and candor in writing about her experiences.

This morning, on her Museum 2.0 blog, she reflects on her one-year anniversary as the director of the Santa Cruz Museum of Art & History. It’s a great post, in part because of her candor about things that worked and things that didn’t. We would all benefit if more nonprofit executive directors wrote a post like this once a year about lessons learned.

I also love the post because of her characterization of their strategy for recovering a near-fatal financial situation:

“Just doing it. We didn’t go through an extensive planning process followed by deliberative, careful steps forward. We had a vision, a short list of goals for the first year, and an energetic (if underfunded) attack. Over the past year, we’ve developed several planning methodologies and approaches to our work—such as our exhibition philosophy and community program development process—and we did it iteratively through a series of experiments. We tried and tested and played and worked our way forward, and we’re still doing it. It is, as Kathleen McLean puts it, “museum as prototype,” and it is exhilarating, thoughtful work for all of us.”

The approach aligns well with an argument Trey and I make in The Nimble Nonprofit: strategic plans can be really valuable, but do them quickly, write them in pencil, and aggressively experiment, learn, and adjust along the way. It’s a powerful model that enables groups to sidestep the typical – and painfully laborious and lengthy – strategic planning process while still taking the time to set a clear, thoughtful course. Importantly, it fundamentally integrates a learning cycle into the execution. The prototype model is powerful so long as it’s explicit and so long as there’s a real commitment to frequent assessments and adjustments. The assessments themselves shouldn’t resemble a traditional strategic planning process, either … they have to be smart, targeted, and fast.

Huge congrats to Nina for the post, and even huger congrats to the entire Santa Cruz Museum team for pulling off a remarkable turnaround.

Why Twitter is Bigger Than You Think

by

Photo by Flickr user stevegarfield.
Edison Research posted last week about their new research on how Americans use social media. One interesting finding: a huge percentage (89%) of Americans twelve and older are familiar with Twitter while a much, much smaller percentage (10%) actually use it.

This has been a pretty consistent finding over the past several years, so while it’s interesting, an even more interesting finding is the explanation for that gap. Edison concluded that “44% of ALL 12+ Americans report seeing tweets in other media (radio, TV, newspaper or other websites) ‘Almost Every Day,’ and 80% of Americans overall claim to have ever seen tweets in other media.” This is big deal, because it means that Twitter’s impact is outsized relative to its market penetration and because it means that its impact is actually quite complicated, filtering through other media in addition to whatever direct impact it has.

Edison points to three important implications:

  • Regardless of how you use Twitter, most Americans (as in an actual majority of Americans) view Twitter as a purely broadcast network.
  • As such, Broadcasting is far from dead, and social isn’t killing it. Social is changing it, but in terms of how most Americans consume tweets, Twitter is just another cable network.
  • If you are measuring anything based upon unstructured data mined from Twitter (particularly influence), you are missing nearly 80% of the potential impact of Twitter by not taking the cross-media and offline impact of Tweets into account.

They won’t be releasing the actual study until Blogworld NY in early June, so we may have to wait until then to really dive into their findings, but it may have substantial implications for what an effective nonprofit Twitter strategy might look like.

Some of the most important nonprofit uses of Twitter are relatively confined, involving conversations amongst engaged nonprofit folks. Twitter is where a lot of those conversations happen, and a nonprofit Twitter strategy might fruitfully focus on engagement in those conversations. It’s not clear that the Edison study impacts that type of strategy very much.

But for nonprofits that want to use Twitter to engage with people outside of those conversational circles, paying attention to how tweets and Twitter memes escape the Twittersphere and penetrate other channels and other conversations is really important.

The Board Of Directors: Guest Post From Matt Blumberg

by

Photo by Flickr user EMSL.

This is a cross-post of a guest post originally written by Matt Blumberg for Brad Feld’s blog but re-posted on Fred Wilson’s terrific A VC blog (with thanks to Fred for permission to run it here on the brightplus3 blog).

Did you follow that? The blog post (despite its complicated journey) is a great read. It’s focused on what makes for a terrific board of directors member, and while it’s written in the context of a private company, most of what Matt writes here is very applicable to nonprofit boards.

**************************

In last week’s guest post Scott Kurnit advised entrepreneurs to put a friend on the Board and keep co-founders off. This week we’ll continue the theme of “who should be on your Board?” with a re-run of a post that Matt Blumberg wrote for Brad Feld earlier this year. The topic is “what makes an awesome Board Member.” I am the person who made the point about firing executives. Brad Feld is the person who downed two shakes in one meeting.

– – – – –

I’ve written a bunch of posts over the years about how I manage my Board at Return Path.  And I think part of having awesome Board members is managing them well – giving transparent information, well organized, with enough lead time before a meeting; running great and engaging meetings; mixing social time with business time; and being a Board member yourself at some other organization so you see the other side of the equation.  All those topics are covered in more detail in the following posts:  Why I Love My Board, Part II, The Good, The Board, and The Ugly, and Powerpointless.

But by far the best way to make sure you have an awesome board is to start by having awesome Board members.  I’ve had about 15 Board members over the years, some far better than others.  Here are my top 5 things that make an awesome Board member, and my interview/vetting process for Board members.

Top 5 things that make an awesome Board member:

  • They are prepared and keep commitments: They show up to all meetings.  They show up on time and don’t leave early.  They do their homework.  The are fully present and don’t do email during meetings.
  • They speak their minds: They have no fear of bringing up an uncomfortable topic during a meeting, even if it impacts someone in the room.  They do not come up to you after a meeting and tell you what they really think.  I had a Board member once tell my entire management team that he thought I needed to be better at firing executives more quickly!
  • They build independent relationships: They get to know each other and see each other outside of your meetings.  They get to know individuals on your management team and talk to them on occasion as well.  None of this communication goes through you.
  • They are resource rich: I’ve had some directors who are one-trick or two-trick ponies with their advice.  After their third or fourth meeting, they have nothing new to add.  Board members should be able to pull from years of experience and adapt that experience to your situations on a flexible and dynamic basis.
  • They are strategically engaged but operationally distant: This may vary by stage of company and the needs of your own team, but I find that even Board members who are talented operators have a hard time parachuting into any given situation and being super useful.  Getting their operational help requires a lot of regular engagement on a specific issue or area.  But they must be strategically engaged and understand the fundamental dynamics and drivers of your business – economics, competition, ecosystem, and the like.

My interview/vetting process for Board members:

  • Take the process as seriously as you take building your executive team – both in terms of your time and in terms of how you think about the overall composition of the Board, not just a given Board member.
  • Source broadly, get a lot of referrals from disparate sources, reach high.
  • Interview many people, always face to face and usually multiple times for finalists.  Also for finalists, have a few other Board members conduct interviews as well.
  • Check references thoroughly and across a few different vectors.
  • Have a finalist or two attend a Board meeting so you and they can examine the fit firsthand.  Give the prospective Board member extra time to read materials and offer your time to answer questions before the meeting.  You’ll get a good first-hand sense of a lot of the above Top 5 items this way.
  • Have no fear of rejecting them.  Even if you like them.  Even if they are a stretch and someone you consider to be a business hero or mentor.  Even after you’ve already put them on the Board (and yes, even if they’re a VC).  This is your inner circle, and getting this group right is one of the most important things you can do for your company.

I asked my exec team for their own take on what makes an awesome Board member.  Here are some quick snippets from them where they didn’t overlap with mine:

  • Ethical and high integrity in their own jobs and lives
  • Comes with an opinion
  • Thinking about what will happen next in the business and getting management to think ahead
  • Call out your blind spots
  • Remembering to thank you and calling out what’s right
  • Role modeling for your expectations of your own management team
  • Do your prep, show up, be fully engaged, be brilliant/transparent/critical/constructive and creative.  Then get out of our way
  • Offer tough love…Unfettered, constructive guidance – not just what we want to hear
  • Pattern matching: they have an ability to map a situation we have to a problem/solution at other companies that they’ve been involved in – we learn from their experience…but ability and willingness to do more than just pattern matching. To really get into the essence of the issues and help give strategic guidance and suggestions
  • Ability to down 2 Shake Shack milkshakes in one sitting
  • Colorful and unique metaphors

Disclaimer – I run a private company.  While I’m sure a lot of these things are true for other types of organizations (public companies, non-profits, associations, etc.), the answers may vary.  And even within the realm of private companies, you need to have a Board that fits your style as a CEO and your company’s culture.  That said, the formula above has worked well for me, and if nothing else, is somewhat time tested at this point!