Augmented reality and the ultimate user manual

Most user manuals are worthless. They're chock full of poorly written text and confusing diagrams. Worse still, the gap between problem and solution is vast because we're forced to apply a linear format (a guide) to a specific question. Where's a search box when you need it?

But here's an idea: What if instead of leafing through pages or scrolling through an online manual, you could simply see your way through a task? Just slide on a headset and work your way through a bit of customized, augmented-reality education.

That's what Columbia University computer science professor Steve Feiner and Ph.D. candidate Steve Henderson are trying to do with their Augmented Reality for Maintenance and Repair (ARMAR) project. They're combining sensors, head-worn displays, and instruction to address the military's maintenance needs. Take a look at this project video and you'll quickly see how the same application could extend to all sorts of use cases:

In the following Q&A, Feiner and Henderson discuss the genesis of ARMAR and its practical applications. They also offer a few tips for anyone who wants to develop their own AR-based instructional project.

Mac Slocum: What inspired ARMAR?

Steve Feiner: ARMAR was inspired in part by earlier research projects that we have done in Columbia's Computer Graphics and User Interfaces Lab, investigating how augmented reality could be used for maintenance and assembly tasks.

This work dates back to 1991, when we began work on KARMA (Knowledge-Based Augmented Reality for Maintenance Assistance). The earliest work on ARMAR itself began in 2006, with initial funding from the U.S. Air Force Research Lab, when Steve Henderson began his Ph.D. studies at Columbia.

Our application domain of the LAV-25 light armored vehicle turret was the result of funding from the U.S. Marine Corps Logistics Base, beginning in 2007, to investigate how AR might be applied to future field maintenance of military vehicles.

MS: Is ARMAR in active use?

Steve Feiner: ARMAR is a research project and has not been deployed.

MS: Can you walk me through the ARMAR user experience?

Steve Henderson: The user can see five kinds of augmented content presented on the see-through head-worn display:

1. Attention-directing information in the form of 3D and 2D arrows, explaining the location of the next task to perform.

MS: What tools and technologies does it employ?

Steve Henderson: The initial implementation of ARMAR was built as a game engine mod using the Valve Source Software Development Kit. Over the past semester, ARMAR has been reimplemented using Goblin XNA, our lab's open-source platform for developing augmented reality applications.

Steve Feiner: We also take advantage of a wide range of head-worn displays and tracking systems available in Columbia's Computer Graphics and User Interfaces Lab. These include a custom video see-through head-worn display that Steve Henderson built specifically for use in the project (using a Headplay display and two Point Grey Firefly MV cameras), a Vuzix iWear VR920 with CamAR video see-through head-worn display, and an NVIS nVisor ST 60 optical see-through head-worn display. The tracking technologies that we use include InterSense IS900 and IS1200 hybrid trackers, NaturalPoint OptiTrack IR optical tracking, and the VTT ALVAR optical marker tracking package.

We typically run the application and head-worn display on a desktop PC with an NVIDIA Quadro FX 4500 graphics card. When applicable, we run the NaturalPoint OptiTrack on a separate laptop. But, there's no reason why the application itself couldn't run on a high-end laptop.

In addition, there are now wireless HDMI solutions that could be used to effectively cut the cable from the computer to the head-worn display, eliminating the physical connection to the computers.

ARMAR is a research testbed, and not a ready-to-deploy production system. Therefore, we are free to explore different combinations of technologies, without having to commit to them as part of a turnkey solution.

MS: The video shows what appears to be the G1 mobile phone. Is that an input device?

Steve Henderson: The Android G1 phone is used as a wrist-worn controller that displays a simple set of 2D controls and detects user gestures made on the touch screen. Gestures are streamed to the computer running ARMAR through Wi-Fi. The G1 allows the user to move between maintenance steps, and control the explanatory animations that the system can present -- starting and stopping them, and changing the speed at which they play.

MS: How small can you make ARMAR?

Steve Feiner: Our emphasis has been on developing a research testbed in which we can design and formally evaluate the effectiveness of new ways to assist mechanics in learning and performing maintenance tasks. Therefore, we haven't had to worry about choosing specific hardware on which a production-quality implementation could be fielded right now, let alone making it really small.

That said, Moore's Law, in concert with competitive hardware development and strong consumer demand for ever smaller and more powerful devices that can support 3D games, is driving down the size and cost of the mobile devices on which ARMAR and its descendants will be able to run. And, the capability for transmitting wireless high-resolution video could also help eliminate the need for cables to/from the head-worn display, eventually allowing the system to use eyewear that looks much like current glasses. These could be connected wirelessly to a small smartphone-sized waist-worn computer, or even to a nearby stationary computer whose size then becomes much less important.

MS: Could something like ARMAR be ported to mobile phones? Could it exist as an app?

Steve Henderson: Yes. But, note that an app that used a current mobile phone's built-in camera and display, held in the user's hand, won't accommodate many tasks in which the maintainer needs to devote both hands to the task itself. As mobile phones mature, however, we believe they will soon be designed to interface with -- or even be built into -- tracked eyewear, making them an ideal platform for ARMAR.

MS: What's been the most challenging aspect of development?

Steve Henderson: It's been challenging to track the user's head within the cramped confines of the turret. We do not have a full replica of the turret in our lab, and were not able to permanently install any tracking infrastructure in the actual turrets where we did our studies.

Using stereo video see-through head-worn displays under Direct3D has also been challenging. There are no explicit provisions for stereo in Direct3D and the formal support for stereo displays provided by graphics card vendors does not address merging rendered graphics with separate left-eye and right-eye video. We were lucky to have NVIDIA provide us with an unsupported software development kit for handling this on their graphics cards.

MS: Has anything gone smoother than you anticipated?

Steve Henderson: Our recent reimplementation of ARMAR using the GoblinXNA framework has gone very smoothly. Our initial prototype design, which leveraged the Valve Source software development kit, required custom implementations of several core functions required for augmented reality applications (e.g., tracking and camera control). GoblinXNA provides these functions implicitly, which has allowed us to spend more time on the design of the actual augmented reality interface. Additionally, implementation of the wrist-worn controller was very straight forward using the Android Software Development Kit and Eclipse Integrated Development Environment.

MS: Do you see applications in other industries?

Steve Feiner: There are many potential applications of AR to explaining industrial tasks, in both training and production. Essentially, it could be used in any domain in which personnel use conventional documentation, ranging from paper manuals to computer-based electronic manuals.

MS: How about consumer use?

Steve Henderson: There are many day-to-day tasks in which consumers currently need to consult written or computer-based instructions. Think of assembling a bicycle or a piece of furniture, making a complex recipe, wiring a home entertainment center, or fixing a balky lawnmower. These are just some examples of tasks in which systems like ARMAR could make the task easier and faster to perform, and make it more likely that it's performed correctly.

MS: If someone wants to pursue a similar project, what guidance would you give them? What should they watch out for? Where should they start?

Steve Feiner: It's important to be aware of, learn from, and build on relevant ongoing and past work. Researchers have been exploring AR and publishing their work for over 40 years, beginning with Ivan Sutherland's research on head-tracked see-through head-worn displays.

The leading conference in this field -- the IEEE International Symposium on Mixed and Augmented Reality, and its direct predecessors -- dates back to 1998. So, we would strongly recommend that someone who wanted to develop a similar project (or, for that matter, any AR project) become familiar with what others have done before, to find out what worked and what didn't.

It's also important to have a close working relationship with subject-matter experts in the field in which the application will be developed and to be able to run user tests with the members of the population for whom the system is being designed.

MS: What's the next step in making this technology more widely available?

Steve Feiner: In the work we reported on at IEEE ISMAR 2009, we showed how AR made it possible to locate maintenance tasks to perform more quickly than state-of-the-art electronic documentation. And, we're now concentrating on improving users' speed and accuracy in performing tasks that involve orienting and positioning parts during assembly and disassembly. Making the technologies on which we're working available to others will involve additional funding to address other domains and to make robust production implementations of the software.

73.6% of all Statistics are Made Up

How to Interpret Analyst Reports

The headlines in the media are filled with that latest stats.  Stats sell.  The stats are often quoted from the latest reports.  People then parrot them around like they’re fact when most of them are complete bullsh*t.  People throw them around at cocktail parties.  Often when they do I throw out my favorite statistic:  73.6% of  all statistics are made up.  I say it deadpanned.  Often I’ll get some people look at me like, “really?”  ”It’s true. Nielsen just released the number last month.”

No.  It’s irony.

Or as Mark Twain popularized the quote most attributed to the Prime Minister of Great Britain, Benjamin Disraeli, “there are three kinds of lies: lies, damn lies and statistics.”  The quote is meant to highlight the deceiving but persuasive power of numbers.

So, where is this all coming from, Mark?  What are you on about?  Anyone with a great deal of experience in dealing with numbers knows to be careful about the seduction of them.  I’m writing this post to make sure you’re all on that same playing field.

Here’s how I learned my lesson:

I started my life as a consultant.  Fortunately I was mostly a technology consultant, which meant that I coded computers, designed databases and planned system integration projects.  OK, yes.  It was originally COBOL and DB2 – so what? But for my sins I got an MBA and did “strategy” consulting.  One of our core tasks was “market analysis,” which consistent of: market sizing, market forecasts, competitive analysis and then instructing customers on which direction to take.

It’s strange to me to think that customers with years of experience would ever listen to twenty-something smarties from great MBA’s who have never worked in your industry before – but that’s a different story.  Numbers are important.  I’d rather make decisions with uncertain numbers than no numbers.  But you have to understand how to interpret your numbers.

In 1999 I was in Japan doing a strategy project for the board of directors of Sony.  We were looking at all sorts of strategic decisions that Sony was considering, which required analysis and data on broadband networks, Internet portas and mobile handsets/networks.  I was leading the analysis with a team of 14 people: 12 Japanese, 1 German and 1 Turk.  I was the only one whose Japanese was limited to just a sushi menu.

I was in the midst of sizing the mobile handset markets in 3 regions: US, Europe and Asia.  I had reports from Gartner Group, Yankee Group, IDC, Goldman Sachs, Morgan Stanley and a couple of others.  I had to read each report, synthesis it and then come up with our best estimate of the markets going forward.  In data analysis you want to look for “primary” research, which means the person who initially gathered the data.

But all of the data projections were so different so I decided to call some of the research companies and ask how they derived their data.  I got the analyst who wrote one of the reports on the phone and asked how he got his projections.  He must have been about 24.  He said, literally, I sh*t you not, “well, my report was due and I didn’t have much time.  My boss told me to look at the growth rate average over the past 3 years an increase it by 2% because mobile penetration is increasing.”  There you go.  As scientific as that.

I called another agency.  They were more scientific.  They had interviewed telecom operators, handset manufacturers and corporate buyers.  They had come up with a CAGR (compounded annual growth rate) that was 3% higher that the other report, which in a few years makes a huge difference.  I grilled the analyst a bit. I said, “So you interviewed the people to get a plausible story line and then just did a simple estimation of the numbers going forward?”

“Yes. Pretty much”

Me, sarcastically, “And you had to show higher growth because nobody buys reports that just show that next year the same thing is going to happen that happened last year?”  Her, “um, basically.”

“For real?” “Well, yeah, we know it’s going to grow faster but nobody can be sure by how much.”  Me, “And I suppose you don’t have a degree in econometrics or statistics?”  Her, “No.”

I know it sounds like I’m making this sh*t up but I’m not.  I told this story to every consultant I knew at the time.  Nobody was surprised.  I wish it ended there.

The problem of amplification:

The problem got worse as the data flowed out to the “bulge bracket” investment banks.  They, too, were staffed with super smart twenty somethings.  But these people went to slightly better schools (Harvard, Stanford, Wharton, University of Chicago) and got slightly better grades.  They took the data from the analysts.  So did the super bright consultants at McKinsey, Bain and BCG.  We all took that data as the basis for our reports.

Then the data got amplified.  The bankers and consultants weren’t paid to do too much primary research.  So they took 3 reports, read them, put them into their own spreadsheet, made fancier graphs, had professional PowerPoint departments make killer pages and then at the bottom of the graph they typed, “Research Company Data and Consulting Company Analysis” (fill in brand names) or some derivative.  But you couldn’t just publish exactly what Gartner Group had said so these reports ended up slightly amplified in message.

Even more so with journalists.  I’m not picking on them.  They were as hoodwinked as everybody was.  They got the data feed either from the research company or from the investment bank.  And if anybody can’t publish something saying “just in, next year looks like a repeat of last year” it’s a newspaper.  So you end up with superlative amplification.  ”Mobile penetration set to double next year reaching all time highs,” “venture capital market set to implode next year – more than 70% of firms may disappear” or “drug use in California growing at an alarming rate.”  We buy headlines.  Unless it’s a major publication there’s no time to fact check data in a report.  And even then …

The problem of skewing results:

Amplification is one thing.  It’s taking flawed data and making it more extreme.  But what worries me much more is skewed data.  It is very common for firms (from small ones to prestigious ones) to take data and use it conveniently to make the point that that want to make.  I have seen this so many times I consider it routine, which is why I question ALL data that I read.

How is it skewed?  There are so many ways to present data to tell the story you want that I can’t even list every way data is skewed.  Here are some examples:

- You ask a small sample set so that data isn’t statistically significant.  This is often naivete rather than malicious
- You ask a group that is not unbiased.  For example, you ask a group of prisoners what they think of the penal system, you ask college students what they think about the drinking age or you ask a group of your existing customers what they think about your product rather than people who cancelled their subscription.  This type of statistical error is known as “selective bias.”
- Also common, you look at a large data set of questions asked about consumer preferences.  You pick out the answers that support your findings and leave out the ones that don’t support it from your report.  This is an “error of omission.”
- You change the specific words asked in the survey such that you subtly change the meaning for the person reading your conclusions.  But subtle changes in words can totally change the way that the reader interprets the results.
- Also common is that the survey itself asks questions in a way that leads the responder to a specific answer.
- There are malicious data such as on Yelp where you might have a competitor that types in bad results on your survey to bring you down or maliciously positive like on the Salesforce.com AppExchange where you get your friends to rate your app 5 out of 5 so you can drive your score up.

That doesn’t happen? “I’m shocked, shocked to find that gambling is going on here.”  We all know it happens.  As my MBA statistics professor used to say, “seek disconfirming evidence.”  That always stuck with me.

Believing your own hype:

And this data subtly sinks into the psyche of your company.  It becomes folklore.  13% of GDP is construction – the largest industry.  40% of costs are labor, 40% are materials and 20% are overheads.  23% of all costs are inefficient.  18% of all errors come from people using the wrong documents. 0.8 hours are spent every day by workers searching for documents.

It’s important to quantify the value of your product or service.  I encourage it.

You’ll do your best to market the benefits ethically while still emphasizing your strong points.  Every investment banker I know is “number 1″ in something.  They just define their category tightly enough that they win it.  And then they market the F out of that result.  That’s OK.  With no numbers as proof points few people will buy your products.

Obviously try to derive data that is as accurate as possible.  And be careful that you don’t spin the numbers for so long and so hard that you can’t separate out marketing estimates from reality.  Continually seek the truth in the form of better customer surveys, more insightful market analyses and more accurate ROI calculations.  And be careful not to believe your own hype.  It can happen.  Being the number one investment bank in a greatly reduced data set shouldn’t stop you from wanting to broaden the definition of “number 1″ next year.

Here’s how to interpret data:

In the end make sure you’re suspicious of all data.  Ask yourself the obvious questions:

- who did the primary research on this analysis?

- who paid them? Nobody does this stuff free.  You’re either paid up front “sponsored research” or you’re paid on the back-end in terms of clients buying research reports.

- what motives might these people have had?

- who was in the sample set? how big was it? was it inclusive enough?

- and the important thing about data for me … I ingest it religiously.  I use it as one source of figuring out my version of the truth.  And then I triangulate.  I look for more sources if I want a truer picture.  I always try to think to myself, “what would the opposing side of this data analysis use to argue its weaknesses?”

Statistics aren’t evil.  They’re just a bit like the weather – hard to really predict.

And as they say about economists and weathermen – they’re the only two jobs you can keep while being wrong nearly 100% of the time

We typically define innovation as an idea that is brought into valuable action.  We use the more inclusive "valuable action" because schools can innovate by teaching more kids or using different teaching techniques.  Governments can innovate by delivering better services.  It doesn't have to be a product.  If we take the definition at face value, then creating a somewhat new product based on reduced functionality that makes the solution more affordable and is just as acceptable could be innovation.  But what's more important is that we occasionally forget one critical factor in innovation:  it's not about what we, the developers and product managers want.  It's about what the customers and users want and need, and are willing to pay for.

If I create an excellent product, but it is too difficult to use or too expensive to acquire, then even though it may be very innovative, it may not be successful.  The ultimate goal of innovation is to create the right solution, for the right customers just as they become aware of their needs, and are able to acquire the solution.  If my timing is too early or too late, sure, it's innovation, but I don't capitalize on the market opportunity.  Timing is critical - too early and I educate the consumers and the second and third entrants win the market share.  Too late and I've missed the chance to gain the lion's share of the market.  Aligning to the consumer base is important as well.  If I create very interesting products and services that don't meet a need in the consumer base, I may be innovative but won't be around for very long. 

Ultimately, then, innovation is about recognizing needs or opportunities before others do, and validating that those needs are important and relevant to the targeted consumers.  If making a product that is over-engineered or missed the market window more relevant and appealing to customers by removing unnecessary features, then I'm willing to call that innovation.  The folks who brought us Blue Ocean Strategy used the Strategy Canvas to look at competitive features in most industries, and argued that we could radically increase or decrease offerings in most of those features.  For example, seating choices in airlines.  The major airlines suggested that being able to choose your seat at purchase is very important.  Southwest suggested that it's not important at all, and for some consumers they are right.  In fact Southwest "defeatured" a lot of the factors around airline travel - no food, no seating choices and at one time no frequent flier miles.  Yet by simplifying they opened the door to a different class of traveler, and by scaling they now compete with the majors.

Too often we are so interested in creating the "best" product or the newest product or service that we fail to realize that many innovation opportunities are available for us if we'll only adjust our thinking.  Far too many products and services only partially meet the needs and expectations of customers.  That means there are opportunities to make products or services that are much better, and much worse, than exist today.  As innovators we are expansive and interested in the really new, when sometimes radical adjustments of the existing are what is called for.

This blog is part of our Inspired Ethonomics series.

From lip balm to dish detergent, it's hard to find a product these days where at least one brand doesn't have an eco-label. But what do these labels really mean? Do consumers really change their buying habits because a cute bunny or a healthy-looking tree is pasted on the side of a bottle of shampoo or a roll of paper towels?

The short answer is yes. Research from the U.S. Environmental Protection Agency (EPA) and some NGOs seems to suggest that consumers who see labels like the dolphin-friendly image on Starkist tuna or the "Totally Chlorine Free" stamp on paper products, tend to prefer those products over others. Of course, price points and other variables come into play, but in general, a green label encourages people to shift their purchases toward environmentally-friendly products.

In recent years, these labels have proliferated. And they've started to compete for increasingly eco-minded consumers' attention. For example, both the Sustainable Forestry Initiative (SFI), an industry group, and the Forest Stewardship Council (FSC), formed by NGOs, have labels for "green" paper, pulp, and wood products. But environmentalists complain that SFI standards aren't high enough and even undermine the value of the FSC label.

Scholars at the Erb Institute and Resources for the Future found in markets where multiple eco-labels compete there was a wide range of labels of varying stringency. That way, every firm could get some type of label, even if it represented the weakest of environmental efforts. NGOs, on the other hand, preferred fewer labels that actually represented tough "stretch goals" for companies. The problem, then, when NGO and industry labels competed, the industry labels representing weaker standards tended to steal a lot of market share from the tougher NGO labels. Thus, there was a risk that more labels equaled worse environmental performance by companies.

The result is that consumers are getting confused. They don't always know which labels support what standards, so multiple labels on a product might seem like a good thing to shoppers who don't have perfect information about the pictures on the side of the box or the end of the canister. Does the average buyer even know that SFI and FSC are competing groups, or that one represents looser environmental standards? Worse, will the proliferation of labels cause consumers to throw up their hands in frustration and ignore them altogether?

Ultimately, what is at stake is the impact of consumer behavior on the environment--as the number of eco-labels increases, it's harder and harder to tell which ones matter, environmentally speaking. The labels become shorthand for going green, but they might not actually lead to greener purchasing decisions because the whole labeling system is so complex and opaque.

So how can we make eco-labels more meaningful? One option is for government to step in and create a common standard, as was done with the U.S. Organic Foods Production Act of 1990. Of course, government intervention means tax dollars, and it means some industry groups will balk at the oversight. (Some big retailers, however, welcome government leadership in this area.) Another option is for a respected third party (like Consumer Reports or Underwriters Laboratory) to step in and rate the labels themselves. Either way, clear standards should be agreed between NGO and industry players, and those standards should be represented by a few labels whose meaning is made clear through education programs initiated by NGOs and the EPA or other public agencies.

A survey by Accenture reveals interesting findings about the Indian youth (as well as millennials* worldwide)

• Youths in India and the Netherlands constitute the most active downloaders of open source technologies, at 61 percent and 77 percent, respectively.
• Social profiling is most common in China and India, where more than three in four Millennials use social networks more than half of the time when trying to learn more about peers or superiors.
• As far as posting information online (i.e. blogs/tweets) is concerned, Indian kids take a back seat.
• One in four Millennials globally – and four in five in China and India – use social networking to investigate employers, superiors, clients, and service providers.
• Only 6 percent of mid-Millennials expect to use only corporate applications at work
• Self-display is rampant on Webpowered social networks in some countries, as is the use of such networks to research employers, prospective clients, and colleagues.
  The most open and casual behavior when writing about themselves and friends online is found in China, Japan, and Brazil. By contrast, Millennials in India, Canada, and
  France are far less open online.
• 72% of Indian youth said that the state-of-art technology is an important consideration in selecting an employer.

Youth and Technology

Is Email Dying?

While older Millennials (ages 23-27) globally still spend an average of 6.8 hours a week writing or receiving work-related emails, mid-Millennials already in the workforce spend just 4.2 hours a week on email.

Among that group, real-time alternatives are gaining ground, such as text messaging via mobile phone (3 hours) or instant messaging (3.2 hours). The trend is even more pronounced among high school and young college students.

The research was conducted across 5,595 employees and students, between 14-27 age group, in 13 countries; and the essence of the research is that IT employers have to reinvent their policies, and ways to reach out to the new generation (and back home, there are reports that suggests Social Networking Kills Productivity for Indian Corporate Employees by 12.5% )

What’s your opinion?

Download the report.