In his book, The New Know, Thornton May makes a case for data analysis becoming the next frontier of corporate evolution. Having spent the past 15 years getting their transaction systems in place, businesses will now turn their attention to making sense of the massive amounts of data they are collecting. The result will be a transformation of corporate productivity fueled by deep insights into customer needs. Business analysts will become the new rock stars of the organization.

This is one in a series of posts that explore people and technologies that are enabling small companies to innovate. The series is underwritten by IBM, but the content is entirely my own.

May is a futurist who is executive director and Dean of the IT Leadership Academy. A popular speaker, he is known for combining deep insights with a stand-up comic’s delivery and a gift for storytelling. His writing has appeared in the Harvard Business Review; Financial Times; The Wall Street Journal and many other academic and business journals.

I caught up with him to talk about The New Know. 

I was struck by how well researched your book was, with lots of real-life examples.

Not only am I talking about The New Know, the whole thing was a new know for me. It was ethnographic study of what people are doing in this space. Analytics is the business reengineering of the mid-90s. All of the major service firms are moving into this space in a big way, but not many people have a good definition of it.

I created a new acronym: FODDRS. That forecasting, operations research, data mining, data integration, reporting and statistics. Those have traditionally been six disparate disciplines, but you need the totality of them today.

Why did you set out to celebrate analysts?

We’ve come through the Bataan Death March, which is enterprise software. Everyone in the global 2000 has burned in through a painful process their enterprise transaction systems. So they have their systems of records in place and now and now people asking what do we do with all this data? Let’s analyze it. In 2010, the analyst is to the value creation dance what the CIO was in the 80s and 90s: unloved, misunderstood but poised to play a major role in the future.

Will new tools make analytics accessible to the common business person in the same way that spreadsheets made forecasting, accessible?

We are living in a much less tolerant society. You don’t need analytics if you’re always guessing right, but the minute you guess wrong, you’re going to be held accountable for how you made that decision and what you knew when you made that decision. We’ve created a whole class of people who are willing to forensically spank you if you’ve made decisions wrong.

Is this a generational issue?

I think the next generation of leaders is going to be not only aware of analytics but also masters of it. The next real social issue we deal with as a nation is executive compensation, and the only way you can justify the pay levels of these guys is to become more analytical. The people who simply guess are being found out.

Can you think of any recent examples where an executive has been pilloried for making bad examples based upon not analyzing data?

[Former BP CEO Tony Hayward] is a classic point. There is a presumption on the part of regulators and customers that senior executives understand what the key processes of the organization are and what their operational health is. At BP, that was not the case. We’re living in a complex society and the ticking bomb is not understanding how operations work.

But operations and organizations are getting more complex. Can one person know it all?

You can’t know it all but you can have processes for knowing. It’s OODA loops: observe, orient, decide and act. It’s not artificial intelligence but augmented intelligence. We’ll be delivering to the human actor the appropriate dashboard or operational reality at the right time.

We’re going to have to rethink and reinvest in our entire technology infrastructure. Our infrastructure was built to do transactions, not to manage information.

What will the new infrastructure look like?

For example, Partners Healthcare is looking at what outcomes they want to have happen at that point of care, then they design systems to deliver information to the front lines right when it’s needed.

We haven’t had to do this before. We used to be in the Soviet system where people were grateful for what we put on the shelf. It used to be product-centric vs. customer-centric. I will create this product and you will buy it.” Now the customer is saying “Here’s what I want and you build it for me.”

There are 360 million Americans today. Speaking analytically, that is a trivial database. You can actually know every one of those individuals.

At what point does this become creepy? Google CEO Eric Schmidt has said that Google has a lot more data than it ever uses because it would be creepy if they did use it.

Information management becomes a species survival trait. Where it doesn’t become creepy is where people extract utility and value from their information management ecosystem. Kaiser Permanente has shown that you will live longer if you have mastery of your health care data. It’s not creepy that way.

Historically, we’ve been living in California where nobody knows how to park a car, but they know how to do valet. We don’t know how to park our data and how to take care of it. In K-12 we’re going to be teaching kids how to manage their information. I see kids graduating from high school seven years from now getting the equivalent of the top information security credential of today.

Do you think hoarding information rather than sharing it is basic human trait?

I think that’s an acquired trait during the information age. Today, the more information you share, the more powerful you are. This is why Apple had trouble early on. They are a closed system and this is why RIM and Android are going to be able to catch up catch up. An open system will always outperform a closed system.

Another thing is that people now know who is hoarding information. It’s like being the Amish on steroids; If you’re not behaving appropriately in the group, you are shunned. And that is having some social consequences like teen suicides, because there aren’t a lot of places to hide. We are at the big bang of social, economic, political and evolutionary change.

The technology we now have in the portfolio today is so amazingly powerful and it works. People with technology imagination, who can dream of creating value with technology now have the tools to do that.

The old think was that information overload is a problem. We’ve got to change our thinking. Having all this information available to us is not a bug; it’s a feature. Speaking realistically, there is nothing we cannot know. So an organization can now say, “What do I want to know about my customer?” We can create that.

The other day [Intel CEO] Paul Otellini was being interviewed by Charlie Rose and was asked what is going to be obsolete next and he said “ignorance.” We have migrated to a point in our society where it is no longer socially acceptable to be ignorant.

At companies that do this well, how is the organization different?

One thing is they celebrate curiosity, they ask “What if?” and they tend to have more porous boundaries. And they realize there are different kinds of smart in the enterprise. They conduct experiments. We’re migrating into this scientific age of business where you can launch a product prototype and let people react to it and fix it almost instantaneously. Michael Schrage says we should all become the chief experiments officer.

But most companies don’t operate that way. The culture punishes failure.

We’re migrating away from that. Every company is trying to realize that it’s not really a failure; it’s learning. Even organizations that are traditionally viewed as hidebound, like the U.S. military, treat every mission As an analytical exercise. They analyze what they learned and what they have to change. And not only does a particular unit learn, but they all learn from that. Those learnings from the field work their ways into the curriculums at the service academies within two to three weeks. That’s unheard of in academia.

Doesn’t this challenge our hierarchies?

You’re coming closer to getting organizations and humans within them to almost be on the same metabolic rate. The industrial discipline of coming to work at a certain time is an unnatural act. We’re migrating to a world in which we can now learn together. The organizations that will succeed are those that have an unambiguous reason for being. Kaiser Permanente’s is to improve health, the U.S. military’s is to keep people alive. People who exist just to sell more in this quarter will be lost. People have to have a North Star to come back to. The criteria for experiments has to be to advance the mission.

You quote Clayton Christiansen at some length. He has demonstrated that companies that listen too closely to their customers and don’t realize that the business environment has changed can kill themselves. Is there a risk of listening too closely to your customers?

Tom Davenport has pointed out that airplanes today have amazing avionics, but the pilots will tell you it’s still a good idea to look out the window now and then.

The organizational question is: Can you create an organization that people are excited about engaging with? GM is a very switched-on organization now. They are very excited about analytics. If you look at the traditionally underperforming members of the New York Stock Exchange who are now bringing in the analytical firepower of an Accenture or a SAS, you should buy their stock because you’re going to see a material improvement in performance within six to nine months.

There is so much waste in the enterprise today.  People don’t know what’s going on in their enterprise. We reengineered in 1995 and then stopped. The whole term “process” has disappeared from the vocabulary.  You’re going to see process mining, where people are mining their processes and optimizing them. And you’re going to make some serious money.

Look at what Gary Loveman did at Harrah’s, what the Boston Red Sox did: data tied to real mission delivers value. This may be what really excites children to get sharper in math.

Will this bring us out of this economic malaise?

Oh, yes. We still have 90% employment. The economy is unbalanced, but there are parts of the economy that are red-hot. We have to redo the US infrastructure, migrate to clean fuels, improve the quality of healthcare; there will be enough work to go around, but people will have to be smart to do it. If I was a young individual coming out of school, I would be studying analytics right now. The Master of Statistical Analysis will be the new MBA. It will be the ticket to success over the next 20 years.

We’ve also got to change our approach to education. A lot of kids went to school to show how smart they were, so they stayed away from things that were hard. We may go back to the Greek concept of people becoming better and more well-rounded citizens.

People say we’re no longer manufacturing in America. We’re actually manufacturing more than you can believe, but we’re so good at it that there aren’t as many jobs anymore. Where we dropped the ball was not reinvesting in their human beings. Corporations used to have training and development programs and sadly, those disappeared. There’s a whole generation in corporate America who have never been to a corporate training program. Training not disappear in the U.S. military, which is why ex-military people make such great employees. People are now doing it themselves with online education courses. I believe that retraining will be a major part of our economy.