Podcamp Session on Social Media Metrics: Thank You Jeremiah

Flickr Photo by whatchamakallit

UPDATE:  The presentation slides and wikispace is here.

I'm doing a session at Boston podcamp called the Social Media Metrics/ROI Game.   I'm slightly out of my comfort zone here because I've only been following this topic closely for a short time, although I have co-authored a guide for nonprofits on technology evaluation and ROI.  I also don't want to get too deep into geeky measurement crap.  And, above all, the session needs to be a conversation about Social Media, not a lecture.

So when I heard that one of the leading subject matter experts on social media metrics - Jeremiah Owyang - was the guest of honor at a local social media networking event, I decided to go.  And, the networking was good, but the best part was the opportunity to ask Jeremiah a few clarifying questions to his face!  Thank you!

There is a lot of discussion on the topic on web strategy blogs and from social media strategy gurus like Owyang.  There's also a lot of talk in the nonprofit blog space prompted by a recent Netsquared ThinkTank and some great thinking on the topic, particularly by Wendy Harmon of the Red Cross, mostly focused on ROI.   I'm looking to blend different perspectives and have a workshop session that can speak to nonprofits and grassroots media makers.  Also, most of the audiences I'll be talking to are not web analytics geeks either, so will need a basic and condensed overview too.

  So, here's the session description:

Social Media Metrics/ROI Game

Standardized metrics for measuring the ROI of social media strategies are a moving target.  Many are discovering that traditional web analytics and metrics (like page view) are not appropriate.  The current conversation is about the need for new metrics and methods for objective measurement.   What metric(s) you select depends on your strategy map, tool selection, and context.   Different metrics matter for different social media makers for different reasons. This session will use a training game as a conversation starter for participants to begin to identify the most important metric mapped with strategy. The session will provide a brief summary of the recent thinking on this topic by thought leaders in the field, but mostly provide an opportunity for small group discussion and sharing about current practices on this important topic.

Session Learning Goals

• To understand why social media metrics are different from traditional web metrics
• A snapshot of current and evolving thinking about social media metrics on the social web
• To provide a simple framework for thinking about how to apply social media metrics to improve your marketing strategy and begin to measure ROI
• A conversation starter about social media metrics with your fellow social media maker peers here in the room and (hopefully) to be continued on the social web.

When standard metrics are discussed, people easily fall into the "mine is bigger or better than yours"  comparisons or "numbers data out of context thinking."   Our brains naturally migrate to scoreboards, report cards, and stock market returns   For nonprofits, it isn't about making money.  The focus is on:  improving the effectiveness of the marketing strategy or content,  not wasting limited resources, and having more people to take action (push for cause, donate money, or whatever). 

I'm going to give a very brief overview (15-20 minutes at most) that will serve as a conversation starter for small group work.    Here's a brain dump of some key points.  I was very grateful for Jeremiah's assistance in thinking about the social media metrics.

• Many people are coming up with their take on how to provide useful metrics and measurements on the effects of social media for a nonprofit organization or individual social media maker as well as the conversation in the corporate sector. But define some terms first:   
  

• Measurement is the process of determining the result of a strategy (source: J. Owyang)
• Metrics are the attributes or factors that are important to understand (source: J.Owyang)
• Standardized Metrics are those attributes that an industry might use to compare different organizations, media outlets, etc. (Think Nielsen Ratings).
  
• Mapping is the process of developing a map of all things you are doing to generate conversation, sharing, and collaboration as a social media maker. (source: Chris Brogan)
• ROI (Return on Investment) determines whether the cost of your marketing investment paid off.  Should we be spending our time or money on this social media strategy?  Traditionally, this looks at dollars out and dollars in. It may also look at outcomes or reaching goals.
  

• Three assumptions:

• What is more important how.  We should pay attention to the social media standardized metrics discussion and translate it into useful ways to apply it to nonprofits and grassroots media - but not get bogged down in geeky measurement.

• It's Hard To Put A Dollar Amount On Priceless. (Need to track down the cost of the most expensive Van Gogh)  It can be difficult to put a dollar value to intangibles in social media.  Think about that Master Card commercial as illustrated here.

• Data is not like Martha Stewart - it is far from perfect.   Don't get distracted by the data you may or may not have, but look at more than one source of data (both quantitative and qualitative), look at trends and always link information to decision-making. (We're not going to debate:
  Google Reader Stats Are Bullshit  - but if you want some tips on how to do that, see Avinash Kaushik's Convert Data Skeptics)
  

• A Two-Minute History of Web Metrics:   Tim Berners-Lee, a long time ago, invented the world wide web.  Life was good.  You typed in a URL and a file with text and links (and later pictures and video) would be delivered.  Then some smart human created server logs and it was like  Prometheus bringing fire to marketers.  The log files captured lots of data that gave us a lot "measurable" information about what people were doing on web sites.    As web sites got bigger, their log files got bigger and even smarter humans created analytics software that could generate cool reports based on standard metrics like page views, unique visitors, referring sites, pages visited, and time on site,  etc  (Source: Avinash Kaushik's "Web Analytics An Hour A Day")

• We have a history and culture and expectation for objective web site measurement.   Let's imagine walking into and out of  a supermarket. If you didn't purchase anything, no one would know you're there.  If you purchased something, say laundry detergents, then they would know a bit more about you.  Visiting a web site is radically different if you look at from view of Web data collection.  The web site logs know what aisles you walked down, in what order, how long you stayed in each aisle, what detergents you took off the shelf and put back on, how many times your visited before, how you were referred to the store in the first, and so forth. (Source: Avinash Kaushik's "Web Analytics An Hour A Day")

• Let's define some of these "traditional" and long standing metrics:

Basics: Visits, Visitors, Page Views, Visitor Info
Engagement: Time on Site, Bounce Rate
Traffic:  Referrers, Entry/Exit Pages, Google Search Terms
Content:  Click analysis, goal conversions, funnel analysis

• So, why can't we simply use existing metrics and analytics software and call it day?

• The Web is changing!  The emphasis is on collaboration, sharing, and conversation.  (If there are podcasters in the room, you know that already!)  Social media has rocked the world of web analytics and metrics!

• In December, 2006, Steve Rubel predicted that the Page View is Dead!

"The page view has served us well. It has established a universal way to measure web sites. However, the metric is about to become a moot point. The page view does not offer a suitable way to measure the next generation of web sites. These sites will be built with Ajax, Flash and other interactive technologies that allow the user to conduct affairs all within a single web page - like Gmail or the Google Reader. This eliminates the need to click from one page to another. The widgetization of the web will only accelerate this."

• Question:  Why does a Star Fish Have Five Rays? (Answer: I don't know).  So, what the heck does a Star Fish have to do with Social Media Metrics?  (Answer: I still don't know, but I saw it on Scoble's white board)  He describes five social media metrics and suggested they be combined in some way for a standardized measure. 
  

• I found the categories useful to thinking about it, but most valuable I asked Jeremiah some specific questions about how to put these  metrics in a more holistic context -- outcomes, measures, strategy.  The categories were audience, engagement, loyalty,  influence, and action. (What's interesting is the acronym aelia stands for Jerusalem) 

Audience:  Who is coming? Demographics  How many people visit the site? (Raw numbers aren't as important as trends)  Unique Visitors

Engagement= Interaction + Attention

How many people comment on posts? What is your post to comment ratio?  What do they click on? How long do they stay? 

I used some metrics and measurement techniques suggested by Avinash Kaushik to look at the commenting.   I also think quality of the conversation or shape of it is important.  Are you really getting off your soap box?

Loyalty:  How many subscribers does your podcast or blog have? What is the trend?  Going up and to the right? 

Dave Winer says why rss subscriber numbers alone don't matter and Kaushik has some advice on measuring this 

Influence/Authority:  Scoble defines this as % of posts that show up on Techmeme, Digg, my Link Blog, Slashdot, StumbleUpon, etc.  Avinash suggested Technorati.  This is hard to pin down.  Does the number of embeds fit in here? 

Intent/Action:  What is the most important goal or outcome of your organization, work, or business?  What is your conversion rate for getting me to sign up, volunteer, donate, purchase an item, etc leading towards that goal?  Owyang says that qualitative information, anecdotes are very important. 

• This naturally leads us to Chris Brogan's thoughts on the ROI and Measuring Social Media
  

There is a symbiotic relationship between strategy and metrics.  Chris talks about the importance of mapping your social media strategy and identifying a key metric or two for each strategy that tells you whether you've been successful.   He gives a great list of what goes into the map

• List of the most likely places a human will encounter the media I produce.
• Methods for listening to conversations off-blog and outside my media.
• Touch Points along the value chain and how my media reaches each one.
• Path back to a central data capture for reporting and strategy monitoring.
• Pinpoints to corrective measures taken from initial strategy path to current efforts.

• How do we put this all together?  Let's look at three different ways to think about your the analytics data:

Therapy:  You look at your site meter or views in real time.  Yes, people are listening.  They like me, they hate me - ah darn, just wasted three hours ...

Detective Work:  What's going on here? Why are the comments higher on one blog post or podcast and lower or another?   What can we adjust?  This can be helpful for continuous improvement, but it is time consuming and may not be effective.

Answers Business Questions Related To Goals:  (examples)

• Here's a simple framework for thinking through these.  A logic model. (I need some concrete examples - help!)

• Goal:  What is your most important organizational, business, program, or project goal? 
• Outcomes: What changes? What happens?  What would success look like to you?
• Metrics:  What are the attributes or evidence you will measure?
• Measurement:  How will you measure?  What data?  What source?
• Map:  What social media strategies will you use to reach your goals? A whole other workshop

At this point will want to do a small group where one person is the client and the others will ask these questions, focusing on the first three. Each group will have a cheat sheet with the questions one side and the metrics/definitions on the other.

Here's what I need:

Anyone willing to go through the framework above and share an example from real life or make one up so it isn't all concept? 
What isn't clear or could be simplified?
Am I missing anything?

Recent Posts from Different Perspectives on the Topic:  Need To Explore

ROI of Blogging/Social Media

Charlene Li, ROI of Blogging, January, 2007
Jeremiah Owyang, ROI of Blogging, Intangible
Jeremiah Owyang, From the Drawing Room to the Board Room

Avinash Kaushik, Tips for Measuring the Success of Your Blog
Jeremiah Owyang  Social Media Measurement category
Jeremiah Owyang Factiva Social Media Roundtable Answers What We Should Measure
Stow Boyd, Conversation Index

KM/Informal Learning/Business Perspective

Jay Cross, ROI of Informal Learning
Dennis Howlett, ROI is so Business 1.0 Not
Altul Rai, Measurement of Business Processes
Elsua, Making the Business Case for ROI
Stephen Downes, Can't Count Friends or Count on Them
Greg Verdino, Key Takeaway from Forrester

Nonprofit Examples/Thinking

Wendy Harmon on ROI (Diagram)
Joitske H., ROI for Development Orgs
Froggy Loop Social media ROI Calculator

Metrics

Comparison of Old/New Web Metrics
Facebook Metrics - ROI of Friendship - Dave McClure's ARRGH Model

Relationship Measurement was defined by Jim and Laurie Grunig and Linda Hon in this paper..
Forrester has a new "engagement index" as does Eric Peterson

Comments

[Jeremiah Owyang]

Beth, thank you for spending time with me, you're an excellent student ready to absorb information and apply, wow fantastic post.

My new employer has also written extensively on the subject, and I look forward to learning about their methodologies, and perhaps even improving mine. I'll share more details with you as I get past the three week mark!

[KD Paine]

I think you're missing a couple of things, the biggest one is a measure of your relationships. See http://www.instituteforpr.org/research_single/guidelines_measuring_relationships/
Once you've started the conversation, it's important to measure how well the other guys think its going.
Also, since transparency is key to trust, you need to have an objective measure of your transparency.

[Beth Kanter]

@JO - thanks and will look forward to the example.

KD - Do you know how one might measure transparency?

[Michael Bailey]

There are several layers of transparency, and not all people are equally comfortable at each level.

One level is when a company acts like they are being open and honest with you, but in reality they are hoping that you are just a lemming and will follow along without asking any questions.

Another opposite level is when things are being told exactly how they are - of course, this level of honesty often meets with people thinking that there "must" be something between the lines.

And several grey-layers in between.

[Jay Cross]

Beth, this is a gigantic topic, for corporates as well as NGOs. I won't let that get in the way of a few high-order observations.

First of all, metrics are simply measurements that matter. Because they matter differently to different people, metrics are inevitably relative. (The administration has different concerns from the funders, neither of which may be the way the customers see it.)

Overall metrics won't be for social relationships or training or blogs: you can't parse reality into those categories. They each influence one another. You're looking for something simpler: what we put in and what we get in return.

ROI and its bean-counting brethren are industrial-age vestiges of a time when, for accounting purposes, people were worthless. (A skilled person was carried on the books at the same value as a new-hire: nothing.)

To rise above the trees, the rule of the forest is cost:benefit. Again, this is not monetary, as least not much. Cost includes time that could be spent on other things. Benefits include customer satisfaction, increased capacity to serve, or more dedication to the mission.

Naysayers will argue that you can't manage what you can't measure. That is absurd. The higher the level of management, the more uncertainties. Decisions you can back up with numbers are no-brainers. The tough part is judgment calls. Very few decisions or evaluations are made on the basis of numbers alone.

So how can one tell if something is worth repeating? Or putting more energy into? Ask the stakeholders. Anyone who understands statistics can specify how large a sample you need to take the temperature of an entire group. A few in-depth interviews will probably be more revealing than thousands of fill-in-the-blog questionnaires.

I've been looking at this issue for a dozen years, and the one certainty I have found is that's there's no silver bullet for this.

[Russel Montgomery]

Hi Beth

These are my comments.

This is not my area of expertise. I have never warmed to the subject of measurement. However I can make this observation about your structure.

For me the key are your learning goals for the session and you have stated those reasonably clearly in the centre of your post. Basically you want to fire up a conversation among your participants, getting them to compare two broad forms of measurement wrt their useful in measuring ROI in this context.

That's all good. Keep that central in your mind and forget the detail. Let them fill out the detail.

I don't know your target group and I don't know how hard you will have to work to establish and maintain the conversation. Some folk love converstaion. Some folk just want to be fed. You will have assess this before hand.

As for all the content that you have here. It demonstrates that you have done your homework... but don't flood them with this. If they know they will be bored. if they don't it will go over their heads. I would make it available but not "present" it.

Rather... think about your focus questions. What do you really want them to talk about? What questions will guide the conversation?

On the day, if the conversation goes off on some tangent, if it is fruitful and they are engaged, then let it go there. if you have your focus questions in mind you can always bring it back later if that is appropriate.

And most of all... relax and go with it.

Now all this is based on the assumption that you truly want a conversation.

If, however, you want to present information, then use the 20 slides, each for 20 seconds rule. It's called Pecha-Kucha. Here are a couple of references.
http://davidwarlick.com/2cents/2007/08/25/why-havent-we-heard-of-this/
http://youtube.com/results?search_query=pecha+kucha&search=Search

When presenting... keep it short, keep it sharp, keep it sweet.... and then let them do with it what they will.

I hope this helps

Russel Montgomery

[Beth Kanter]

@Jay: I think what I'm after is how can you use the logic model formula (goal, outcomes, metrics, measurement, map) to make good judgment calls or improving what you're doing?

@Russel: I appreciate your instructional advice. I'm going to make a little sticky note with the learning goals on it - to keep myself on track while facilating.

[denith]

I thought this social media session is the same as what we did in Cambodia.

But going underwater is great, I love views over there. What were you holding in the pix?

[Beth Kanter]

Denith,

This is session is different. That's not me, it is a photo from flickr. I'm not sure how this might translate to Cambodia. Does the word "metrics" even exist in Khmer?

[Holly]

Beth -

As usual, you're the brightest bu;b in the box? How do you get SO SMART about everything you take on?

I know that one way you get so smart is by leveraging your social networks! But are you measuring what you're getting out of them? :)

The problem I have with measuring new media is that metrics are designed to measure in one direction - from point A to point B. They aren't great at measuring in multiple directions at once, and that's the key benefit of social media - it takes people and messages in several directions at once. So I think that you can pick INDICATORS of ROI in social media, but it's nearly impossible to use metrics to determine real ROI from social media.

This of course, assumes that you can actually measure a "real" ROI anyway. Which sounds like a whole other blog post.

[robin]

Beth,

This is a great, very thoughtful analysis of the problems with measuring social media. The biggest problem I see agencies struggle with is trying to put this in a standard bucket of ROI, and the only model that exists is for web analytics. Its not the same though, for the reasons that Holly, the last commenter mentioned -- social media goes in many directions and you can't predict what people are going to say, where and what they will post, etc.

It's good to get baseline measurement of activity, sentiment and themes that are hot in UGM BEFORE you start on a campaign, and to measure that against the competition. This at least gives you a baseline to see what has changed afterwards (are the themes shifting related to what you are doing? is sentiment more positive? Has conversation increased? is it hitting the key influencers?)

[Hans]

Thanks Beth - I'm turning some folks I'm working with onto this post. Hope it can help us think about the outcomes from a wiki we're working on.

Cheers.