Desperately Seeking Validation meetings this month

As discussed earlier, we are having one short and one long meeting this month. Here are all scheduled upcoming events.

The short meeting will be at the Panera east of 86th and Michigan at 6:30 PM on Monday, January 9th, 2012.

The large meeting will be at a place TBD on January 21st from around 11 until ?.

Subscribe now for location updates and future event notices.

Posted in Announcements | Tagged | Leave a comment

The important differences between Lean Startup and bootstrapping

At the Indianapolis Lean Startup Circle November 2011 meetup, James Paden gave a great talk about the differences between Lean Startups and Bootstrapping. He highlighted the major difference being that bootstrapped startups are unlikely to try to spend money most of the time, while Lean Startups do so strategically to get validated learning quickly and once they have validated learning (product/market fit.) James cited Steve Blank’s post titled Lean Startups Aren’t Cheap Startups and Ben Horiwitz’s The Case for the Fat Startup.

Check out the slides here:

(Having trouble seeing the slides? Click here)

In an effort to get practice with the fine line between doing things the “Lean Startup way” and the “bootstrapping way”, two of the co-organizers of the Indianapolis Lean Startup Circle came up with some scenarios to work through. They stress the complicated nature of having few resources and needing to get validated learning. The people who attended the meetup seemed to really enjoy the exercise. Check out the exercise here!

Posted in Event Summaries | Tagged , , , , , , | Leave a comment

Indy Lean Startup Circle – September 2011 Presentation

Anthony Panozzo presented on at the Indianapolis Lean Startup Circle’s September 2011 meetup. The slides cover an overview of the book Four Steps to the Epiphany by Steve Blank as well as an in-depth look at the first step. Enjoy!

(Having trouble seeing this presentation? Try here.)

Posted in Presentations | Tagged , , | Leave a comment

Desperately Seeking Validation Meetup Saturday

Join us to practice Lean Startup techniques in Indianapolis! Email Anthony for more details or to get on the list for next time. Also, check out our upcoming events.

Posted in Announcements | Leave a comment

Indy Lean Startup Circle – August 2011 Presentation

At the inaugural Indianapolis Lean Startup Circle event, Frank Dale presented on an overview of what the Lean Startup movement is all about (and what it isn’t about.) Afterwards, the meetup split into two groups discussing business problems that the members had. Specifically, they tried to focus on how to approach problems by using Lean Startup principles.

Here are Frank’s presentation slides:

(Having trouble seeing the slides? Try here.)

Posted in Presentations | Tagged , , | Leave a comment

Indy Lean Startup Circle – October 2011 Presentations

First, Wes Winham presented on the build-measure-learn loop, with examples from Eric Ries‘s book The Lean Startup.

Wes’s great slides can be seen here.

Next, Anthony Panozzo presented on at the Indianapolis Lean Startup Circle’s October 2011 meetup.

Posted in Presentations | Tagged , , | Leave a comment

Desperately Seeking Validation Update

Desperately Seeking Validation is a group in Indianapolis that works on practicing Lean Startup techinques. After working together for several months, the members have decided on the following format for future events:

Generally we are trying to do one small and one big event per month.

The purpose of the small meeting is to talk about ideas, shoot the breeze, figure out what people are excited about working on for next time, do some retrospecting / blogging, and generally prepare for the big meeting. The small meeting can happen at a coffee shop or wherever. It could be a weekday night or sometime on the weekend.

The bigger meeting also happens once a month and falls on a Saturday so we can go from 10 or 11 until ? with some R&R afterwards. The day is devoted to taking ideas or projects and furthering them along the validation side. Super-intense development is probably not the focus here, but if you need to quickly build something for an MVP, this might be the right time. The general plan is to have some food available so we can last a few hours until lunch, and then hit the ground running again after lunch. We can debrief with some games or your vice of choice. The nice thing about a longer meeting is that people can come by for just a portion of it if they are can.

If you are interested in hearing more about events, please subscribe to the blog and contact Anthony to get on the mailing list. Also, check out our upcoming events.

Posted in Announcements | Tagged , , , | 1 Comment

Lean Startup Presentation

Here is a presentation that Anthony gave to about a year ago about Lean Startups. I think it’s still relevant today, although my understanding has subsequently improved.

Here’s a breakdown of the slides, with a short description of them.


Why Do I Care?

Why is this interesting to me?
Interested in creating great products for people to use and not wasting my time
Have been following this community for awhile, and thought I would share what I have learned and see if others are interested in this topic

What Is A Startup?

Widely accepted definition, and the one I’ll use is:
An organization delivering a new product/service under extreme uncertainty.

Key Takeaways

Context is king
Cannot understand where a practice is useful and apply it correctly unless you know the right contexts to use it in
Most of the case studies that I talk about apply to small product companies, but the implications are much bigger
Would like to plant some seeds for future applications and conversations

Types of risks organizations typically face

Technical risk

Phase III trial -> 4.5 years and $50 million
Something that is not currently technically feasible
If you develop and distribute a cure for cancer, the only question is how many can you make? :)
Industries / startups affected: biotechnology, health care, energy

Market risk

Customer acceptance
I don’t have the problem you are solving
I don’t need your solution badly enough to buy it
Market adoption
Do people want what you are building?


How have things in the valley traditionally worked?

Case Study: Webvan

There were many large failures (WebVan story)

Customer Development

Enter Steve Blank and the Customer Development model
Blank had several companies that succeeded and failed, and tried to come up with a good model for why the companies had the outcomes that they did

Redefined startup as: an organization formed to search for a repeatable and scalable business model.

Not necessarily just for products aimed at end consumers, as Steve Blank’s original study focused on enterprise customers like banks
– not repeatable? you’re consulting / contracting
– not scalable? you can’t scale returns

Use Webvan as example of problems

Customer Development Model

The part on the top of the CD Model slide image is what companies typically do
What they should be doing instead is learning and iterating

Form hypotheses and test them. Two are critical: problem and product concept
Another one that is important is pricing
Blank’s model states that there are various things to do and validate at each step of the process

Customer discovery – achieving problem/solution fit (does the customer think this is a problem?)
Customer validation – achieving product/market fit (a repeatable and scalable business model)
Customer creation – driving demand (crossing the chasm)
Company building – scaling the company

At each step, consider making an exit

Get out of the building
A startup is not a smaller version of a big company
Theory of market types (reminds me of Blue Ocean Strategy)
Creating a new market
Long timeline, be prepared to have slow burn
Do not understand customers well
Bringing a new product to an existing market
Resegmented market (niche, different cost / performance points)
Lots of competition, might fail quickly
Major point is that if you use the incorrect market type model to make decisions, you will not do well
Find a market for the product as specified, not the other way around
Learning and iterating versus linear execution

Product Adoption Lifecycle

Most companies used to focus on “crossing the chasm” (Geoffrey Moore)
However, most companies aren’t learning enough to even get to the chasm


Visionary customers can “fill in the gaps” on missing features if the product solves a real problem


One of the more potentially ambiguous concepts in LS theory
MVP is minimum set of features needed to learn from earlyvangelists – visionary early adopters
MVP could be something as simple as: [and give examples of these]
– a survey targeted to potential customers
– landing page that describes a proposed solution
– a mock up of a solution (paper prototype, HTML prototype)
– an extremely buggy, barely working product that contains only the most important features and doesn’t scale to more than ten users

Technique of giving a link that takes someone to a page where they ask them about the feature
Think of how this could change the way we build applications
Instead of eliciting feature requests and such, have links to things that users might want and when they try to explore there

Lean Startup

The latter is exactly what Eric Ries created at IMVU
He took a class by Steve Blank when Customer Development was still just theory and applied it to his startup

Leverage a commodity technology stack – what would have cost big bucks can be had for cheap now

Ries contends that:
Traditional waterfall development takes a known problem and generates a predictable or planned solution
Agile development takes a known problem and generates an unpredictable solution based on feedback

Lean startup tries to find an unknown solution to an unknown problem

Going Faster

OODA loop applied to startups
We are trying to increase throughput not of features, but of validated learning


When you learn, take what you learned and change your product accordingly
Consider these three pivot types:
Pivot on customer segment
Perhaps your consumer product is gaining traction with enterprise customers
Perhaps your product aimed at teens is actually popular among moms over 45
Pivot on customer problem
Solve a different problem for the same customer segment
Pivot on a specific feature
PayPal found users loved the email payments of its service, so switched to do this

Isn’t all of this wasteful?
Incorporate thread of MTDF and waste being critical to learning


Continuous Deployment

Beyond continuous integration

What it looks like to ship one piece of code to production:
Run tests locally (unit, Selenium)
Everyone has a complete sandbox
Continuous Integration Server
All tests must pass
Incremental deploy
Monitor cluster and business metrics in real-time
Reject changes that move metrics out of bounds
Alerting and predictive monitoring (Nagios)
Monitor all metrics that stakeholders care about
If any metric goes out of bounds, wake someone up
Use historical trends to predict acceptable bounds
Key is that this infrastructure gets built up over time as your needs and understandings change (through the 5 whys below)

When customers see a failure

Fix the problem for customers
Use 5 Whys to improve defenses at each level

“Great test coverage is not enough. Continuous Deployment requires much more than that. Continuous Deployment means running all your tests, all the time. That means tests must be reliable. We’ve made a science out of debugging and fixing intermittently failing tests. When I say reliable, I don’t mean “they can fail once in a thousand test runs.” I mean “they must not fail more often than once in a million test runs.” We have around 15k test cases, and they’re run around 70 times a day. That’s a million test cases a day. Even with a literally one in a million chance of an intermittent failure per test case we would still expect to see an intermittent test failure every day. It may be hard to imagine writing rock solid one-in-a-million-or-better tests that drive Internet Explorer to click ajax frontend buttons executing backend apache, php, memcache, mysql, java and solr. I am writing this blog post to tell you that not only is it possible, it’s just one part of my day job.”

Important to note that this can be done with desktop / mobile products as well by having them fetch updates regularly, etc.

Triggers in code to ensure that functionality that is under development is not run by end users

What are the advantages?
What are the pitfalls?

Other Techniques

Key component: Analytics
Back in the day, it was tough to figure out how you got leads and what the effectiveness of marketing was
Newspaper or magazine ads might have had a special code or number, but overall, not good at figuring out ROI
Lemonade Stand advertising example – It was pretty easy to figure out the algorithm for how much marketing was good
Overview of Google Analytics

Three keys of metrics
Avoid vanity metrics like page hits
Instead, focus on things that will provide results you can act on or answer hypotheses you have
You and everyone else understands what the metric measures
Are the metrics correct or verifiable?

Search Engine Marketing to test viability and naming of products

4HWW example – selling shirts with and without market testing
4HWW example – how Ferriss figured out the name for his book

Statistical hypothesis testing

A/B and multivariate split testing
Huffington Post example
Pros: can tweak small changes and optimize designs
Can increase variables, might be harder to interpret or get sample sizes you desire

Should be transparent
Can’t really use for huge changes, as users will notice
Hard to figure out which larger solution better

Triggers in code

Common Misconceptions

Lean does not necessarily mean cheap
Lean does not mean have few members
Lean startups do not necessarily follow tradition Lean principles

Implications For You

Useful for people’s side projects because they won’t become discouraged
Don’t just put things out there – find a way to learn from those that are using your product first, the earlyvangelists
A/B testing to get the most bang for your effort
Consider writing a book as a process that needs information to succeed
Almost everything you do, from open source to writing blogs is a product
Use these techniques to learn about your “customers”

Implications For Contracting Companies

A client with significant market risk in their application might benefit from these techniques
Almost no one is doing these sorts of things well, so if we got someone who wanted it and had someone who could do it, we could really help them out
If we are trying to put out a product, we might consider some of the things I talked about
Kanban is not truly delivering continuous development to the customer like it does for other organizations
If you release every two weeks, it is zero to two weeks before your client will see a correction
Changes that cause problems in production are not seen until long after they have been implemented
Talked to CMS about this, and the main reason that we do Kanban is for internal quality, reducing WIP, etc.
Can stop building things that are never/rarely used, understand what is the most used
Do you really need this sub sub sub page in the application? Would it be better to fix a bug on the third-most used page on the site?
Better alerts for when things go awry in the system
If someone has set up an alert system for when business metrics go out of normal bounds, shouldn’t we have better alert systems?
Why should the customer have to email you when something goes wrong? Shouldn’t you already know?
Do you know when people can’t sign up for a service by linking it to business metrics?
Allows us to hedge against over-delivering because the customer sees things in production as soon as they are fixed
Recruiting or advertising could use multivariate testing techniques to refine our message

Appendix / out-takes

Required reading on Continuous Deployment:
Overview of process


From Dave McClure
Essentially a user acquisition and retention understanding mechanism, useful for good baseline metrics for improving your flow
A: Acquisition – where / what channels do users come from?
A: Activation – what % have a “happy” initial experience?
R: Retention – do they come back & re-visit over time?
R: Referral – do they like it enough to tell their friends?
R: Revenue – can you monetize any of this behavior?

Posted in Presentations | Tagged , , | Leave a comment

6/22 Mini-meetup Notes

Here are the notes we captured from this event.

Here’s who showed up:
Anthony Panozzo (blog) (twitter)
Alex Toumey (twitter)
Kyle Shipley (twitter)
Wes Winham (twitter)

Things we did:
Create shared ideas list
Motion to create videos/descriptions of the different pitches
Listen to Wes’s ideas
Create a mini-checklist for ideas or validation
Explored the sports stats trackingn idea.

Non-professional sports stats tracking (Wes)
“People want a way to engage with people they know who are in sports.”
“It’s easy to represent a game as statistics, and so you can engage people in that way.”

Rough cut potential hypotheses:
People care about tracking statistics
People care about tracking statistics about games that they are looking at
It’s important how long the kids are playing the sport (year-wise)
People care about improving over time
People can enter in the information quickly enough to make the data useful to them (or to us)
Someone at every game is tracking stats in the stands
People are willing to record stats for the whole game
There is a large enough market to justify having this app
Assumption that there are enough people to make a viable business
I think we have enough evidence to say this is not the highest risk hypothesis
Yearly packages make more sense than monthly due to having seasons?
So general pricing questions
People are interested in stats that are better than their local newspaper
Picking a sport is the right way to go
Thinking the competitive sports niche is the right approach
Maybe acquisition or adoption would be the highest risk
Called four friends who played AAU basketball and two people who played soccer
Talked to some Ultimate people who were ambivalent
Pretty easy to do research / talk to people who might be in the market

Questions to ask people who you talk to about this
Do you do any paper stats tracking?
How do you do it?
How often do you do it?
What do you do with the stats when you are done?
Do you share them?

General consensus to talk to people we know in the competitive youth sports space

Posted in Event Summaries | Leave a comment

May 2011 Customer Discovery Day


Matt Barloh (twitter)
Anthony Panozzo (blog) (twitter)
Alex Toumey (twitter)
Kyle Shipley (twitter)


We got started at 11 am on May 22, 2011. We ate some kolaches, and then went through a quick retrospective. Next, we remembered the ideas from last time and brainstormed some new ideas. We chose an idea to run with for the day, made some paper prototypes, and got out of the building to collect some much-needed feedback. Finally we wrapped up with dinner and followed up with some loose ends in the next week.

Retrospective and brainstorming

The first thing we did was discuss the previous month’s event. We agreed on some general things to improve on and what went well. Potential improvements included getting outside of our first order social network for customer discovery and general validation. Also, having product ideas that did not rely on having a two-sided market would generally be easier to test, or at least test more quickly.

We also considered having more frequent but smaller events because it is hard for everyone to commit. We’ll try for every Wednesday that is not a local Meetup. We can work on tools to make future learning faster, or work through the process on some new ideas. We’ll try our first smaller event on June 22 (contact us for more details.)

Anyway, we threw around some ideas, and then decided to pursue an idea that we discussed about last time. Basically it’s a way for people to trade things they don’t care about for things that they do care about, locally. The idea’s genesis was “ChatRoulette for stuff”, so the internal nickname was StuffRoulette.


Paper prototyping

Once everyone was on the same page, we asked ourselves: how can we get the most validated learning today? While it might have been useful to come up with a business model canvas like we did in April, the real risk seemed to be whether people would actually use a product like this one if it existed. I think we were in a building kind of mood as well, so we decided to see if we could make a single landing page that people actually understood. Rather than put something up on Twitter when we were finished, we decided to instead get feedback by going to a local coffee shop and asking people if they understood what was going on.

The first image was our initial shot at getting a landing page that was understandable. To us, it was pretty clear what it meant. We left a “window” in the piece of paper that we could slide new objects through, and would do this while showing the prototype off. Matt and Alex were tasked with grabbing people before they left the coffee shop, and started each conversation with the following general statement:

{attention grab} We’re working on a new website, and we haven’t written any real code for it yet. We wanted to see if you could take a look at this paper mockup that we have and understand what the site is about. Go ahead and take a look at this paper. What do you think this site does or might do?

At first we got feedback like: “I think I answer questions for guitars” or “I don’t know, you tell me.” (from an older gentleman :).) So talking to real live people indicated that it was a bit unclear. We then got that the question mark portion was a bit confusing and replaced it with “your item here” instead. We emphasized the trade portion and cleaned up the buttons a bit. These are changes that in hindsight are obvious but would have taken a long time to figure out if we didn’t get the feedback that we did.

After changing to the second layout, we got more rapid recognition of what the site was supposed to do. Some people seemed pretty excited about the concept. It was easy to quickly get feedback, and people were receptive to hearing someone ask their opinion on the matter.

Moreover, while we were showing people the prototype and gradually changing what was unclear about it, Kyle was working on a new Rails app that just focused on this page. When we made a change, he would work on implementing it until the next person came around to give us their opinion. In this manner, we were able to parallelize development and learning what actually made sense to people. I suppose that this wasn’t a pure customer development/discovery event, and more like a rapid prototyping exercise. Nonetheless, it was useful and everyone seemed energized.

Finishing Up

We only went a few hours, and then broke up after dinner. There was some followup work that was done the next Wednesday on the web end to make something that could actually be published, and it can be seen here.

Our next meeting will be a shorter one, on June 22. Contact one of the members listed at the top for more details.

Posted in Event Summaries | Tagged , , , , | 1 Comment