SXSW 2011: Metrics-Driven Design

Joshua Porter (@bokardo)

Doug Bowman

Doug Bowman is the lead designer at Twitter. A few years ago Doug wrote a blog post about leaving Google, where he was hired three-years prior. He was really the first real designer hired at Google. There are very few hires like that who get so much press and people talking about it. Doug’s post was a post that he was leaving Google, and usually when you write these sorts of posts they follow the same type of formula: this is my last day, I’m sorry about leaving, and my co-workers are great. However, Doug’s post was nothing like that at all: “Unfortunately for me, there was one small problem I didn’t see back then.” And that’s the problem we’ll talk about in this session: 41 shades between each blue (the best they found was HEX #2200C1) to see which one performed better, and an argument over the width of a border (1, 2, 3px). Bing was/is using a different HEX color, which their UX Manager estimated being a loss of $80 million dollars. The data testing culture becomes a crutch for every decision, paralyzing the company and preventing it from making any daring design decision.

Spectrum of Design

Intuition-Driven (Doug)

  • Instinctive
  • Subjective
  • Daring
  • Follow other people’s practices
  • Trust your gut

Data-Driven (Google)

  • Everything is tested by small numbers, and small variables
  • This process is very slow
  • You rely on data for decision making
  • You don’t trust your gut

Imagine that Your Design is a Mountain

Your existing design is on the side of a little mountain. With an engineer’s approach you can only get to the top of that little mountain. An engineer quickly gets to a diminishing return as they can only go so high with the current design approach. It’s really dissapointing  because you’ll eventually hit a ceiling where you cannot go any higher, known as a “Loci Maxima” in calculus.

Our goal is to be at the top of the largest mountain where our goal is the best (or even just a better) design.

  • Optimization asks: What works best in the current model?
  • Design innovation asks: What is the best possible model?

What are Metrics?

Metrics are simple numbers that measure the effectiveness of your business.

  1. Metrics reduce (but don’t take away all) arguments based on opinion.
  2. Metrics give you answers about what really works. They can also lead you down a rabbit hole, but if you do testing and you have valid data they can give you answers about what really works.
  3. Metrics show you where you’re strong as a designer. They also show you where you’re weak as one.
  4. Metrics allow you to test anything you want. Metrics actually empower you to try anything, where as before you’d have to sell someone on a crazy idea.
  5. Clients love metrics.

Principle: Your metrics will be as unique as your business.

Vanity Metrics are thing such as old-school graphical hit-counters, which you shouldn’t rely on.

The Usage Lifecycle

  • Interested
  • Trial/beta User
  • Customer
  • Passionate Customer

Acquisition Metrics

  • How much did it cost to gain your customer? (Cost per Acquisition, CPA)
  • If your CPA is higher than their life-time earnings, then it’s not worth it.
  • Comparative Metrics: knowing where users came from, and their cost and outcome based on those.
  • The best acquisition outcome is still from Email Lists.

Engagement Metrics

  • Hits
  • Page views
  • Visits
  • Unique Visitors
  • Returning Visitors
  • Registered Users
  • Customers
  • Frequency
  • Time on Site
  • Daily Active Users

Cohort Analysis: Engagement over time. For instance, the number of customers remaining after every month from sign-up. This is very valuable.

Satisfaction Metrics

Net Promoter Score: How likely is it that you would recommend our company to a friend or colleague? (ligart 1-10. 1-6 is “Detractors”, 7-8 is Passives, and 9-10 is Promotes). “Maybe we didn’t have a high viral coefficient score, but we had a great satisfaction metrics.”

Emergent Metrics

Having friends inspires continuous use.

“Find the people you know” is a good example of this.

SXSW 2011: Seth Priebatsch, Keynote: The Game Layer on Top of the World

Seth Priebatsch
Chief Ninja

The Last Decade: Social Layer -> Connections

  • Last decade
  • All about connections
  • Facebook’s Open Graph
  • Construction is over

The Next Decade: Game Layer -> Influence

  • Next decade
  • All about influence
  • No set foundations
  • Construction has just begun
  • Seeks to act on individual motivation: why, how, when we do things
  • Has the opportunity to be 10x larger than the Social Layer

The Game Layer: real behavior in the real world, emulated in the digital world.

What can the game layer do for me?


School is a game, it’s just a poorly designed one. With a school you have: motivated players, challenges, rewards, rules, allies, enimies, levels, appointment dynamics, countdowns, initiatives, penalties, etc. The problems with schools is they’re not Engaged since grading is broken, and there’s cheating. Engagement is an incredibly critical concept which all game developers are aware of. Because the grading system in schools is broken, it has created the “Moral Hazard of Game Play.” It’s replaced the real reward for a letter, and these become chores.

Grades are failing as rewards. They are simple game levels Valedictorians, percentiles, and honor student status. The problem is these titles are boring. As well, these are game mechanics where people can loose, and this is not what we want in education. So why not create a grading dynamic where you start at 0 points, and then you gain points to focus on the positive. This way you cannot fail, or go negative point. It’s all about focusing on the end result.

So what about cheating? Princeton removed professors from the class room when tests were being taken and just required everyone to write down the code of ethics, and the realization

Customer Acquisition


  • Free Lunch: show the customer that although they’re skeptical, it’s still ok.
  • Communal Gameplay: the community needs to hit 50 people before “the deal is on.”
  • Countdown: Time left to buy. Everyone understands it, but it creates this huge exponential spike of activity right before it’s up.
  • Email List


Status: the idea of being a regular at a place. American Express uses status very well because they have levels, and they make you feel better based on the color of your card.

The “Level Up”: Jumping from one level to another.

Inclusive Ownership: everyone owns it, and so everyone benefits from it.

LBS -> Mainstream

How do you move location based services from something that isn’t mainstream, to something that is?

Big Partners + Big Money = Big Results. The problem is just a wee-little bit of the world uses these location based services. So how do we get everyone in on it?

Quantitative Easing: what the federal government does to flood the economy with new dollar bills, without actually having the value. They’re doing this because the “game” is too hard, and sometimes it makes sense to look at the rules and refactor them.

Tightly vs. Loosely Location-Based: basically, to play tightly based location you MUST be at a location. You cannot just be somewhere else. Because of this, it limits the number of people who can be engaging at that place at any one time. If you loosen this rule, then the number of people increases substantially.

Reward Schedules: everyone in the space naturally gets the question, “What do I get in exchange for doing this?” Everyone who’s doing LBS has some form of reward, badge, specials, etc. Rewards work really well. The problem is as once you’ve checked in and gotten the reward, you wont get it next time. As well, this reward will not be offered everywhere, just at certain places.

Global Warming

Not any one thing can solve these problems. While one thing cannot solve these massive problems, it can still give us the tools to move from something impossible, to something that’s just really hard to solve (but still possible!).

SXSW 2011: Diversity in the Digital Age

Anne Hunter (@followanne)
VP Advertising Effectiveness

How does the Internet change how we define communities and ourselves? What does diversity mean in a world of technological grazing? What does diversity matter at all?

This topic is looking at Diversity in what we classicly define as diversity, and what Diversity is on the internet. A digital medium, like the internet, can have the same kind of effect on communities that a television can have in local environments as it causes them to change.

Definition of Common Diversity:

  • The condition of having or being composed of different elements; variety, especially.

Definition of Digital Age Diversity:

  • The concept of diversity encompasses acceptance and respect.
  • It means understanding that each individual is unique and recognixing our individual differences
  • These can be along the dimensions of race, ethnicity, gender, sexual orientation, social-eco status, physical abilities, religious beliefs, political beliefs, or other ideologies.

Statistics on how people use the internet:

  • Genders: 54% of internet users worldwide are male. Mobile phones play a large part in this. Males, especially in the United States, are more likely to have smart phones. As prices drop, women being to buy these phones and actually consume more content than males through these devices.
  • Ages: The younger the generation, the more internet use they do. As we get older our generation will use the internet more than currently elder generations.
  • Locations: The Asian Pacific is the largest area for internet usage, and it’s not the USA. They are 38.9% of the worldwide usage.
  • How Often: Typically, every person in the world access the internet 18 days a month.
  • How Long: Statistically each person spends 26 minutes per use of the internet.
  • How Quickly: They are on broadband, they have a cell phone, they send text messages, but other countries do not have smart phones yet. This is, however, rapidly changing. Typically new technology is first obtained my “wealthy” males, and then later passed to the rest of the world and genders.

What websites: Globally 73.9% visit Google, 66.7% visit Microsoft Sites, 50.6% visit Facebook, 48.0% visit Yahoo, and 31% visit Wikimedia.

But Internet use is not typical! In January 2011 there were 416 thousand people in southwest Brazil were reading pet content online. 381 thousand people pensioners (older individuals) in the UK went online looking for love on personals sites. 1.3 million men between the ages of 25-44 in Vietnam were looking at cosmetic websites.

Does the ability to connect with other like-minded people mean we’re more diverse? Well, the internet has really made human “grazers.” We’re like Human Cows. The power of $0.99: On average 18 songs are purchased on iTunes per person. However, less than 2 songs per album. There’s not a lot of commitment for any one certain artist. People are looking at 9.8 artists per month, per person. Are there areas in music where people are more committed, buying specific genres, etc? Well, New Age is the most common, then Classical, Metal, Jazz, Inspirational, Rock, Christian, Hip hop, …, and Latino. Commitment to a particular artist varies worldwide. Brazil and Sweden are the most committed, and the USA is the least.

Even specialized content consumption changes online. Essence print magazine reaches 40% of all African American women every month in print. And almost all of these women who receive this are African American. However, online only 1/3rd of people who access their website are African American. This is because online you can link to an article and sample pieces of the magazine without having to commit to the entire thing which is targeting a demographic. This is a change we have not seen before, because access to content has always been very specific.

Video content consumption changes online. 180 million USA video viwers. 85% of internet users viewed at least one video. 13 hours of viewing time per view. Unlike sitting down and watching a sitcom, the average video length we watch is 3.9 minutes per video. This is because we’re “grazing.”

If we are grazing more, does that mean we’re committed to things less? Typically core audiences are still more engaged online. African Americans spend 62% more time per person on than non-African Americans. African Americans consume 2x as many pages on than non-African Americans.

The appearance of Diversity, the reinforcement of community. Sites link to other sites that are relevant to the topic, or ideas. Recommendation engines for example does this. If we start interacting with a certain group, they’ll start providing us with other knowledge and links for related materials. This happens with the advertising we see so that we are more interested in this.

How the internet introduces me. The first thing Facebook says about us is our job. Then it lists college and major, residence, relationship status, and then hometown. In a verbal conversation typically these things do not come up. However, with the internet we use this to introduce one another. Who you are is defined by different definition of your community. This online almost makes us less diverse, but gives us the abilities to connect.

Diversity prevents Vulnerability. The more diverse our MHC genes are the stronger our immune system. Inbreeding can make populations vulnerable to diseases because you do not have the genetic immune system built up.

Diversity breeds innovation. Societies with diverse natural resources prosper. Environmental barriers prevent the spread of ideas and societies innovated from new ideas.

The internet is a tool for benevolent and nefarious excuses. To harness the power of the internet, we must actively steward all forms of diversity. We need to go with full heart and say to ourselves, “Am I learning and reinforcing my ideas to help myself, or to limit my diversity to see the other side of the story?” As we go forth, we need to make sure we are active stewards of this.

In the Fortune 500 companies, the most gender diverse boards outperformed by 66% with women on their boards. In 2011 the World Economic Forum found equality drove prosperity as countries with almost parity between men and women to succeed the most.

SXSW 2011: Why Everything is Amazing But Nobody is Happy

Daniel Hope

Zachary Burt
Mr. Awesome
Awesomeness Reminders LLC

Daniel Hope has an interest in the effect of Social Media on relationships. Zachary Burt works at Awesomeness Reminders which calls people every day to tell them how awesome they are.

Entitlement is having something awesome, but hating it when it doesn’t work as expected. It’s in our nature to complain, which is fine. We can move forward into embracing our need to complain. The seven deadly sins are violated in almost every social media product.

Zachary created a website where people could go and vent to someone they don’t know about how bad their life is. This is win-win because the person venting is revealed, and the listener gets a fun story of how bad someone’s life is. Just because something is evil or bad (the seven deadly sins), doesn’t mean it needs to be negative.

Daniel recently was hit as a pedestrian when crossing the street by a vehicle. The driver was using their blue-tooth head set, and couldn’t drive and talk on the phone enough to hit them. This is an “evolution fail.”

Turning off your cell phone before you engage with someone is a good way to say, “Hey, I’m here now. I’m here for you.” It’s a great way to communicate and interact with someone, instead of spending the conversation not listening and just reading your texts or email.

Multitasking is also really difficult. With so much stuff going on, we cannot focus on one thing. So what are some ways we can start our day at being more intentional (asked to the audience)?

  • Don’t use Twitter before noon.
  • Someone in the audience has a program that disables their internet for a couple hours while they write, and then turns off his phone. He tells his mom if someone dies then they’ll still be dead once he turns his device back on.
  • A good technique is to work for 20 minutes, and then take a 5 minute break. It’s almost a form of multitasking, but being intentional with the way you’re doing it.
  • Someone makes a list and puts in the fun things first, and then the bad things. They’ll do the bad things first knowing that once they’re done then they’ll have the fun things to do.

One of the underlining things is to simply measure things to see your output. Because what can be measured, can be managed.

How can we add a dimension into our conversation to identify emotion into our verbal communication online? It’s kind of like an emoticon, but something more in depth. Because with an emoticon it gives you a concept in the mood of the message, but you almost need to know the person personally for it to make sense. Sarcasm, for example, is very hard to demonstrate through a text-message without actually knowing the person. However, if they’re your friend it’s not hard to convey that at all.

If you focus on just one thing, at the end of the day you’ll be more happy than if you multitask multiple at the end of the day.

We are getting better at emotionally connecting with people and things we cannot see with our own eyes. Good Morning America said that if you were wearing a $2000 suit you’d jump into a lake to save someone drowning even if it would ruin the suit without even thinking about it. However if someone asked you for $2000 to dig a well in a foreign country to save a village of people from dehydration you probably wouldn’t consider it. It’s all relative to what’s in front of you in the here and now. You almost have a sense of responsibility for those things in front of you. In an email, however, you may just not consider responding right away.

The Happiness Journal is a concept where you write down things that you’re thankful for. And then when you’re having a bad day you can look through this Journal to feel better about your life. Also just getting feedback about projects helps. For instance, doing an Open Source project lets you see that others want to help you, be grateful for your product, and care about your abilities.

SXSW 2011: How Not to Design Like a Developer

Chrissie Brodigan (@tenaciouscb)
Engagement Lead

She found it difficult to contribute design into an open source project. Most free software projects fail, usually around 90-95% of these. It’s not a lot different than what happens in our day-to-day job. For a project to be successful it needs to attract users and developers. Intricate systems usually are the number one cause for failed systems.

Open source software is free, flexible, customizable, full of great features, and they’re able to make you feel “warm fuzzies.” Open source gets really awesome once you get in there and become involved.

Double D: Developer/designer conundrum. Those awkward conversations between developers and designers. Designers and Developers are willful, independent, idealistic, prideful people… We love our work! Sure, we can argue that Developers and Designers are cosmetically different in style, dress, attitude, etc., however in the end of the day we all love Pandas (yeah, the cute little animal). Hehe.

Documentation matters rather it be developer, user, or design documentation. They’re all different types of documentation, as they’re not the same thing.

Confidence matters since your project’s presentation builds confidence.

The most important thing here is “dissolving demotivation.” One of the big things is how do we get Designers to contribute to the Open Source community? The reality is you cannot. You cannot ever give great designers what they want. The best thing if you do find a great designer is to hide them from everyone, and shelter them. Designers take a lot of pride and credit in what they do, but being told not to use your name is scary. Developers see logical problems, but they use those problems as excuses to bypass design. The excuses are usually pretty compelling and convincing. When you have developers who are putting themselves out there, admitting they’re chaotic, no one will want to work with them.

Bad Habits

  1. Workarounds: developers think and engineer in workarounds for speed. Designers design to avoid workarounds.
  2. Going Rogue: developers misinterpret design logic and make a decision work the way they think it should/might work. Designers don’t provide design documentation to any a developer’s “why” and instead makes it work the way they think it should work.
  3. Being Trendy: Designers design for trends vs. designing for maintenance and iteration. Developers write code for the future. Designers will never say, “man, I’d love to see where this design is going to go in the future in the next version.” This is bad.
  4. Source Code: developers practice version control and code review. Designers don’t treat their output assets as source code or practice team-friendly version control. As a designer if you make a pseudo-version control and keep versions of your design as you create it, then it’s easy to roll back, see what’s already been done, or learn what hasn’t worked before. This is what developers do to produce really good code, and this should be in design too. Work with your designers to teach them how they can treat their design source code in version-control.

Solutions for Designers

  • Open Source design can work, and work very well! The best place to start is to open an IRC channel for your project. use #project_namedesign.
  • Design graphic lite: don’t rely on graphics. Because of HTML and new browser web-fonts we have the ability to rely more on that, and less on the graphics.
  • Design practice design-specific version control. There are very few software solutions for this at the moment. Promote design-specific bug tracking.
  • Make micro opportunities by starting small, don’t use your own name, and you’ll still have small success somewhere.
  • Design and documentation for localization. Your project can be designed locally, but still be shared globally.
  • Refactor together: apply user leanings and amend hastily written code.
  • Include “forkability” as a part of the project’s design ethos. Designers think that what they make wont be used by others, and when they are they consider that intellectual theft.

There are a few projects that are helping Designers play nice in the Open Source World

A good way to get people involved and excited is to design design contests. Mozilla’s Firefox logo actually came from a community-drive design challenge.

Free software projects include software innovation for designers. For designers lettering.js does down to the letter control of your web fonts. Another good place is

Good design is a powerful (non-markety way to spread the word about your project without feeling like you’re selling your soul.

So, where are the Open Source designers? This is definitely a problem, because they’re not at free software-centric conferences and never on the IRC. They are definitely here at SXSW.

Thank you: @estellwyle, @lxt, @mozcreative, @paulirish, @nateabele of lithium, @scottmac of facebook (PHP developer).

“Go hacktivate designers!”