Cubegeek

Deep Variances in Essbase

(from the archives - Fall 1996)

In my travels, I have just come upon a neat trick. I've never seen it posted anywhere and it solves a problem I remember being unable to solve a few years ago.

Consider a simple model with a geographic dimension 5 levels deep, a measures dimension of sales, and a scenario dimension with actual, plan and variance. How can I get the model to indicate 'deep variances' without having to drill down to them if my high level variances are within spec?

Create 'red flag' and '%red flags' in the scenario dimension.

Let's say that I have a 10% absolute value variance tolerance set.

The member formula for redflag would be

if (@abs(variance%>=10)
    redflag = 1;
else
    redflag = 0;
endif

Create a 'neutral flag' member in the variance dimension and hardcode the member formula to be 1;

redflag% = redflag % neutralflag; (two pass)

geo	actual	plan	variance	variance%	redflag	redflag%
NY	100	115	(15)	(15%)	1	100%
NJ	100	97	3	3%	0	0%
CT	100	85	15	15%	1	100%
East	300	297	3	1%	2	67%

Notice that even though your high level variance is only 1%, 67% of your lower level forecasts are out of bounds with two red flags.

simple, neat.

What Should IBM Do With Red Hat?

Q: How does the Red Hat acquisition improve the IBM hybrid cloud offering?

A: I don’t think it particularly does. Not by itself. It’s very late for IBM to be trying something like this with one big fat caveat and that is that IBM strive to dominate the mid-market. That is to say that everything IBM has ever done with AS/400 has to die, right now, at IBM’s command. They need to stop supporting that operating system and make RHEL run on all IBM iron.

Then they need to reorient itself in the market and stop trying to make money on high end hardware until they can price it like clouds. In other words, they need to be able to sell two big IBM mainframes that run in a skanky environment and say that it can run 1000 containers simultaneously. That way you don’t have to go to the cloud until you outgrow that - and that path would be seamless. IBM mainframes therefore have to be drop dead rugged and shipped overnight in a container to any location with enough electricity. That’s attractive. That’s something nobody else will do. And that’s something that can be facilitated by a company the size of IBM. Why? Because IBM will finance the whole thing.

What IBM must do, in the end, is beat Salesforce.com. If it cannot beat Salesforce, then it has no business in the cloud. Because it cannot and will not beat AWS, GCP or Azure. IBM needs to master the downmarket as I said before[1].

Everything IBM must do with RedHat is make RHEL administrators and software devs feel like they have an inside track to a comprehensive set of cloud management APIs that give them a reason to stick with IBM for 5 years. That means whatever the IBM cloud looks like today, it had better port over to RHEL in short order and the RedHat people need to be really excited about that.

I don’t think IBM management is so courageous, and I do believe that the RedHat people are just going to be very happy with their stock premiums. I’d bet the best of them are ready to ride off into the sunset and that this purchase is their last hurrah. The alternative is that IBM provide an exciting vision that encompasses cloud, mid-market, devOps and big rugged IBM hardware.

IBM vision? Let’s wait and see, but if I see another animation of an MRI brain scan coming from IBM marketing, I’ll know they’ve run out of ideas.

 

 

The Carl's Jr Story

I actually have a story here. Let’s see if I wrote it down. Nope. OK So I’m long past the restrictions of the NDA. Here goes.

Some long time ago, we put up an analytic database feed against the POS transactions of several stores of Carl’s Jr. It was kind of surprising that nobody had ever done it before. But they were basically wanting to know who bought what food and when. When they finally did analysis on it, it not only told them what was their most popular items for sale but it gave them a very good idea about their daily traffic.

Prior to this, they only had daily summary data. We broke it down to 15 minute intervals during a day. Since the average store was open 16 hours a day, that obviously that produced about 64 times as much data as they ever analyzed before.

Now here’s where the magic happened. They used to hire people to work 8 hour shifts if they were full time and 4 hour shifts if they were part time. So you’d have a manager come in at 5 to open and work until 1 or 2. Then there would be a part-timer working from 5 to 9, then another 9 to 1, etc. The night manager would close the store at 10 and work till 11. Hard work. Until we did inter-day analysis on the bigger data set, we had no objective way to see what the managers and part-timers could kind of see, which was:

1. People wanted breakfast food past 10AM
2. People wanted lunch food before 10AM
3. People started eating lunch around 11, not just after noon.
4. Sometimes people were changing shifts during periods of high activity.

The last point was huge. Think about it. You’re standing in line to order and suddenly when there were two people at registers, one of them closes and it’s like sorry, time for me to go home. They fixed that right away.

Basically Carl’s ended up not only changing what food was served, but also hiring people for shorter periods. We proved that a store could be more efficient and profitable by having three managers in shifts that overlapped when foot traffic and drive-thru traffic peaked, which was not around the traditional times expected.

Remember also that entry level people working the morning shift were not necessarily trained to cook lunch or dinner food. Big difference. So essentially we changed the definition of ‘breakfast’ and found a much more efficient way to schedule work, that resulted in part-time staff getting cross-trained for multiple shifts which gave them more flexibility.

Carl’s went on to do other IT stuff too. They were a lot earlier than McDonald’s in allowing customers to swipe their own debit cards at the register. And if you know Carl’s, their menu includes all kinds of crazy burgers and Mexican food too. So training their cook staff to be flexible paid off. Their menu is a lot more flexible than others.

Human Memory is not Computer Storage

Steven Pinker is my go to authority in this. He explains something I now understand about how memory is ‘retrieved’. Part of intelligence is the way you organize information in your mind. The better you organize it, the easier it is for you to remember.

In fact, ‘memory’ is not memory, it is the active re-creation of ideas and events based on the way they were organized in the first place.

This has a great advantage so that you can abstract ideas and events and then deal with them as somewhat different versions of a basic concept. So if you become, say, a baseball play by play announcer, you don’t have to remember the path of every ball from the mound to the plate (which might take a ‘gigabyte’ of visual storage), you just say the pitch was a ball or a strike. I can say ‘baseball game’ and immediately (if you understand baseball) you have the context for a class of events. When you’re watching baseball and you don’t understand the game, you have to store a lot more information. When you understand baseball, it’s easy.

Another example is you are talking to a stranger vs you are talking to a friend. With a friend you have context, therefore less storage required. This is why two people who were at the exact same party will remember the same conversation differently. They hear the same words and sentences, but they make different assumptions about what the person speaking was trying to say. So you could have a third person describe motivations and then suddenly have an aha moment, “Oh that’s why they said that, I thought they meant..” People remember differently because interpret and organize context differently, and thus they re-create events differently.

The more associations you give a particular event or idea, the more ways it is accessed in your mind. If you associate songs, or smells or how your body was feeling, you remember better. If you have an experience with a framework of theories, you remember better. If somebody keeps telling you the same story and you can hear their voice telling you, you remember better.This is (simplistically) the way that the mind works with the brain being the apparatus. If an old person has a healthy brain, then nothing changes. But if they have a disorganized way of creating memories, it won’t make sense coming out.

So it’s really not about how much storage, but how efficiently ideas and events can be re-created based on their intelligent way of contextualizing such ideas and events. This is why intelligent people seem to have better memories or *more* memory even though their brains are the same size. Their context for performance of certain things is better, they can get ‘in the zone’ quicker. And scatterbrained people remember crazy stuff all the time. Same ‘memory’ different organization, different re-creation.

 

The New Driver's Ed

In the news is a story of a driverless fatality in Tempe, Arizona. In the dark of night, a woman walking a bicycle across two lanes of traffic was struck and killed by one of Uber's new autonomous vehicles. Behind the wheel was a driver who was momentarily distracted, probably by a tablet in his hands, while the autopilot of the car cruised along at just under 40 miles per hour and ran the poor woman down. The article I read about the story quoted at least two experts who said that the lidar equipped machine should have easily seen the pedestrian and avoided the accident. I was most interested in the final paragraphs. 

Raj Rajkumar, a Carnegie Mellon University professor and founder of an autonomous-vehicle software company that he later sold, suggested the laser sensor may have had a blind spot around the vehicle because of its position mounted on the roof. Still, he said, he would have expected the software to have reacted.

“Legally speaking, the pedestrian may be at fault,” Mr. Rajkumar said. “But mature, reliable self-driving vehicle technology would have done better by slowing down or changing lanes, and this major incident would have been prevented.”

I have been in the IT industry for decades and one of the most fabulously successful products ever was the iPhone. I've been using mine for 7 years. My mother has been using hers for 2 months and she gets lost all the time. She is computer literate, but a lot less so than I. The distance between what I can digest as a professional at this stage in my career and what ordinary folks can understand is the gap into which a great deal of VC money is thrown in expectation of aggregating many multiples of that in consumer spending. Yes, I have a very new, very advanced, very capable iPhone device in my pocket and I use it all of the time. It works exceedingly well for me because I am highly discriminating in my consumption of applications, and I know how application designers think when they are building for the iPhone. In that way I am immune to bad apps, and I don't get lost. Half of all wisdom is knowing what to throw away. Simple works best.

So while journalists will have ready access to founders and professors who will state what might be professionally obvious, pedestrians and the rest of the world are puzzled as to exactly what went wrong. I suspect as the story of autonomous vehicles grows, we will all find out sooner or later, most likely later. It has always been my opinion that as cars learn to pace themselves in traffic that we humans are going to have the hardest time understanding their protocols. Indeed the future of driver education will not focus so much on skills of the drivers as to understanding how cars have been programmed to think about themselves and about traffic. That's a long, dark and narrow road for all of us to travel. 

What's legal today ought to be perfectly obvious. We have had crosswalks for generations. We have had crosswalks in law for generations. These things are perfectly obvious because we've had a very long time to learn the legal and physical truth about crosswalks. Lidar? Not so much. There will not be human and animal analogs for the way lidar perceives or the way cars will communicate with each other. We will have to learn new concepts and adapt. Or we will simply have to drive. 

What is it like to drive in traffic with cars that don't have human drivers? We are going to have to change our perception. We know 'idiot drivers' and are quick to identify their idiosyncrasies on the road, because we understand human inattention. He's weaving like a drunk. She's obviously on her cellphone. He's obliviously blasting beats. She's a student driver.  He's an old man. But Tempe offers a perfect example of a driver unable to recognize his own vehicle's inattention. 

 I've been a gearhead all my life and am probably unusually attracted to and informed about cars given my inexperience in actually repairing them. So I am fairly geeky into racing and the skills associated with demanding driving. I will trust autonomous driving machines when they start winning rally races and drifting competitions. Analogously, when computers were only reliable in playing an excellent game of tic-tac-toe, the mastery of chess was the benchmark for intelligence. I think that's were autonomous cars are now, four squares of tic tac toe rather than just three, with three being mere cruise control and lane alerts. It ain't chess, and it's far from go. 

Here's the thing though. When machine learning figures driving out, it's still going to think like a machine and not like a human. That means there will be hacks and vulnerabilities machines will have navigating 3d spaces that we humans will have to study very diligently to understand. I expect a lot of that diligent conceptualization to happen at Carnegie Mellon, but not in DMV parking lots. We can only hope that traffic law changes very slowly. 

Dehumanizing AI On My Mind

Last week I was in Bogota again. It was less interesting than IAH. In Houston's new terminal I was accosted by the 'Stacked' business model. Stacked is a restaurant chain that got started here in Southern California several years ago. Their idea changed the way people were served. A hostess would direct you to a table at which a custom iPad had the menu. You ordered custom burgers, shakes and fries. You had no dedicated server, but a swarm of people would bring food and take empty plates away according to the instructions you pushed out of your tablet. You could order any time and add to your bill which you also paid for from the iPad.  At Houston, there were literally hundreds of custom iPads at bar seats and tables inside and outside of themed restaurants. The swarm of servers went 50 yards in all directions.

It was disturbing to me that this form of automation was replacing one of the most intimate human jobs there are, then again, airport food. But what was really annoying was the fact that the idle screen on each iPad had dozens of casino-style video games with their own currency you got from ordering food or could pay for with frequent flyer miles. Clever, I know, but sucking all of the society out of a transportation hub felt quite crass and dehumanizing.

I've recently written about where data mining fits in my scope of business decision making and it's very interesting. I like AI and machine learning, but I haven't gotten to the point at which I'm comfortable saying 'ML'. Most of my experience with AI comes from old theory (backwards chaining and forwards chaining), my understanding of Markov and years of experience in video game playing. These days I'm completing another romp through the giant world of Skyrim as a maxed out conjurer. I spawn AI companions in all of my battles, especially dramora lords. But let me recap where in the world of business computing AI / machine learning lives from my perspective.

So first let me recap that my interest is in augmenting the decision making capabilities of individuals and organizations. I think that in a large set of domains, there are data available that can be captured and processed into knowledge and actionable intelligence.  In the context of business, these are data-centric applications and in the context of business process management and transformation my aim is to enable 'data driven business'. In other words, after Peter Drucker, I am catalyzing the professionalization of business management by taking the seat of the pants guesswork out of much of it. Nevertheless, I recognize and respect how business leaders generate insights and visions of their business operations. So sometimes a good decision support system merely disciplines the accurate communication of performance indicators already known to the principals.  In the context of AI and machine learning, we can generalize that there are two kinds of data driven businesses.

A. Data driven businesses whose leadership knows what works and seeks to discipline processes.

B. Data driven businesses whose leadership is unclear on what works and needs to build or re-engineer processes.

Somewhere in the middle there is generally a margin for improvement. But those operating in relative data darkness closer to B, are more vulnerable to outsized claims about the possibilities provided by AI.

So without getting into a mountain of details, the most important thing that determines the quality of AI is the amount of data it has access to, and the size of the domain it is expected to perform in. What's changing, therefore about the nature of what AIs do is the scale of data that cloud native architectures are able to deliver. For example, speech recognition at Google is superior to that at Apple, especially when it comes to the context of asking for navigation directions. Apple famously kicked Google Maps out of its IOS offerings several years ago. Consequently Siri has some shortcomings when it comes to recognizing spoken street and city names as compared to Google's voice recognition.

When customers and prospects ask about AI in the cloud, I tend to push back. The first thing I want to know about their possible application is how well their data is managed in the first place. From my perspective, if you don't have a serious structured data lake up and running, you are probably not going to get a lot of value training your AIs. We are at the very beginning of AI, and those things that we have AIs for, we tend not to think of them as frightfully scary AIs. For example, probably the most mature AI in existence today is speech recognition. Think of all of the millions of people who have iPhones who have been speaking to Siri for years. And yet every Siri user finds its comprehension inferior to that of a 12 year old. That's billions and billions of transactions and Siri still can't recognize the name of the avenue just down the block. 'Del Amo' simply doesn't work in Siri, which thinks I'm saying 'The lommel'. What the heck is a lommel?

AIs can best be thought of as business models, and their capitalization is the data behind it. If you're going to have a sophisticated AI model, you need to throw a lot of data at it over time and you're going to have to know how to validate what it does.

There is an ugly undercurrent I perceive in the idea of replacing low wage workers with AIs. And I think it is instructive to think of 12 year old children in this regard as well. For there are dimensions of satisfaction and delight that people can take in the performance of young people that would be completely absent from a machine doing the same thing. So I guess I'll conclude with the following story.

In 2000, I went to the Sydney Olympic Games and sat in the at my apartment watching the long jump competition. As the competition began, we were treated to a short documentary about the introduction of an automatic pit machine that sat on rails at the far end of the sand pit. It's job was to level the sand after each jump. This had traditionally been done by volunteers in every Olympics past, but we were regaled by the story of the dogged determination of the engineer who poured his life savings into the project. After years of testing, he finally got his design right. Then came the problem negotiating with the Olympic officials to have it trialed. Finally he got the approval, here it was, ladies and gentlemen. In the afternoon of the first day of competition it rained. The 500 pound machine broke and required four people to detach it from the rails and retire it causing a 45 minute delay in the competition. Now the officials had to find somebody with rakes and sand, something a 12 year old boy would be honored for life to do.

People ought to be wary of the machinations of clever engineers in their ability to take honor out of work and give tasks to machines, especially when it is something that uses poor substitutes for human intelligence. When we take human beings out of what could be honorable work we depopulate social spaces and make them sterile and mechanistic. There are appropriate places for AI, but the best use of AI is to augment human capability not to replace it. This is an old lesson, but one that bears repeating. If you don't know what you're doing, don't put an AI in to figure it out for you. If you do know what you're doing let the AIs be additional eyes and ears. Let them amplify your own senses, not replace them.  Either way, you're going to have to get the data in first.

Where Are My Drugs Now?

I have had the unique opportunity to be involved in shepherding members of my family through a maze of healthcare delivery systems. I can't tell you how much I'm looking forward to Amazon to gain a presence in this industry.

The system that clearly doesn't exist and should is a secured link between medical prescribers and pharmacy inventories. If I were Jeff Bezos I would get busy readying my M&A people to purchase Caremark. CVS is, to my knowledge, the most thorough and extensive business that handles the delivery of pharmaceuticals. But they also deliver other services, like roadside assistance. Strategically, Amazon has the most to gain by extending their ability to coordinate and deliver more services. So sentimentality aside, Walgreens name recognition might work better for the short term, and lord knows they need a makeover, but Caremark is a better target. As for Right-Aid, there's a bit too much overlap with Whole Foods. So Caremark.

I learned a long time ago that delivery and inventory controls were a very significant part of the value of Philip Morris profitability. In exchange for signage in convenience stores, they offered free point of sale systems. Those systems in a coordinated fashion allowed the tobacco giant to figure out how much of their product was leaving retail. Since they knew how much they were producing on the manufacturing end, they could estimate how much was in transit and inventory  in the part of the supply chain they did not control, logistics and transportation. The competitor there was McLane Logistics. PM was able to figure out how much inventory to sell the McLane and squeeze inefficiencies out of their delivery system.

This is something that is clearly not done in the pharmaceutical business, and whomever gets there first will have a huge leg up. There are few things more nerve wracking for patients than to have a prescription on hand and not be able to fill it in a timely manner. That doctors and pharmacies depend on office hours and phone calls is absolutely ridiculous.

Amazon as Conglomerate

Here's the highlight reel. When Peter Drucker invented the MBA part of the idea of professional business management was that ordinary individuals could be trained to run a business. Not a generalized business, but a specialized business in some category. (NAICS codes). The way Harvard has been teaching case studies was that within the scope of business and product analysis such a cadre would be measured and their public company made an indicator by P&L and balance sheet metrics to the public shareholders. IE, IBM competes in its NAICS sector, becomes a market leader taking product share away from competitors, then you know to buy the IBM ticker. All the analysis is around product competition and market share and the effectiveness of management has everything to do with its ability to execute in that narrow area.

 

This doesn't work for general contractors. This doesn't work for hospitals. This doesn't work for software. This doesn't work for conglomerates. Shareholder maximization only works for certain kinds of single focus businesses. So long as the balance sheet works, in this narrow way of looking at business performance, you can justify all kinds of trickery which is not good management or good governance.

 

So there are only a few management teams, relatively speaking, who know how to make a company profitable on narrow profit margins. A conglomerate can have a balanced portfolio of businesses that won't suffer the business cycle. That's how Berkshire Hathaway makes money. By running multiple businesses in different NAICS and not making their whole company dependent on a killer app. Each of these businesses, after a time, contribute to a large pool of funds that allows the directors to make experimental buys in new areas, instead of mergers in the same area. Bezos has said specifically that Amazon can afford to make billion dollar mistakes, because the underlying businesses are already fully capitalized and don't need to keep growing or be leveraged before they are mature.

 

People have wanted AWS to be spun off from Amazon for years, and Wall Street has been hedging on the stock price because before it became a market leader, he never broke out it's P&L and balance sheet from the whole of the company. Wall Street for the most part doesn't know how to do company management analysis either. That's why index funds are winning. Anyway, conglomerates are safer from hostile takeovers because the executives actually understand how to run their businesses and there is no incentive to sell out the business. That's why a class of Silicon Valley VCs and entrepreneurs loathe the idea of working for Amazon. There's no exit strategy. You don't exit conglomerates. They scale horizontally.

 

It's interesting how well Bezos has run it as a conglomerate, given the bad news recently about GE. I don't think, outside of P&G, that most Americans understand conglomerates. So I'm hoping that Amazon does a stock split and gets listed on the Dow.

Where Have All the Xeroids Gone?

(from the archives 2005)

I talked to my boy Marty this long weekend. He runs part of the facilities department over at Xerox El Segundo. His was a tale of woe. I owe a lot of my professional development to the management style of Xerox - continuous quality improvement and all. So it was hard for me to hear what he had to say.

I started my first internship for the Big X back in 1983, my first summer in college. Since I had some real-world work experience, I was fortunate enough to land a spot with the MBA candidates and worked in the Centralized Printing Systems Division several levels beneath the legendary Hoshi Printer. Yes, Printer was his last name. Last time I checked, he was making moola at Cars.com. So I had seven years of Xerox on my resume by 1990 when I went full-time into the software business.

About that time, Xerox had proven two things. Number one, it was master of the copier and printer business. Number two, it didn't know squat about the computer, software and networks business. Xerox stock limped along in the low 20s for my entire career there and its employee count was about 100,000. Here in El Segundo, where we used to actually sweat about being a target of Russian nukes, there was a vigorous set of employees of Xerox. The legendary A&E building housed first rate engineers and brown shuttle vans took folks around the campus of about 18 buildings. I myself had worked in CP-8, CP-10, XC-1 and XC-2. I sometimes had lunch in one of the 'M' buildings, the big M1 cafeteria. As an intern, I toured through M2 where some of America's first 'surface mount' technology was being deployed for Xerox printer circuit boards.

Marty says that there are about 60k people who work for Xerox now. A&E is down to about 600 employees and only two of the eight M buildings still even belong to Xerox. Over in Xerox Centre, office space had been empty so long that Xerox was renting to all comers for as little as 1.50 per square foot. The boom days of El Segundo are long gone.

Today, Lexmark printers are no joke. HP dominates the 'decentralized' print world. Warnock made Adobe and all the rest is history. But it remains stunning to me that these very buildings, the whole of the Xerox campus has been so drastically reduced. Where have they gone? How long are those who are left going to stay?

It's all design work, so I've been told. It's what we do best in the US, come up with new ideas and figure out ways to sell them. But I miss the days of end to end, where you could be on campus and know you were 10,000 stong. Where the spirit of a company pulling together, from design to test to manufacturing to sales and marketing distribution and support were all in walking distance. The virtual corporation is real. We in IT and our crafty MBAs made it possible. Now El Segundo is less than it used to be.

I wonder what's going on at Northrop.

 

via cobb.typepad.com

Xerox is in the news as FujiXerox takes over.

(from the archives 2005)

I talked to my boy Marty this long weekend. He runs part of the facilities department over at Xerox El Segundo. His was a tale of woe. I owe a lot of my professional development to the management style of Xerox - continuous quality improvement and all. So it was hard for me to hear what he had to say.

I started my first internship for the Big X back in 1983, my first summer in college. Since I had some real-world work experience, I was fortunate enough to land a spot with the MBA candidates and worked in the Centralized Printing Systems Division several levels beneath the legendary Hoshi Printer. Yes, Printer was his last name. Last time I checked, he was making moola at Cars.com. So I had seven years of Xerox on my resume by 1990 when I went full-time into the software business.

About that time, Xerox had proven two things. Number one, it was master of the copier and printer business. Number two, it didn't know squat about the computer, software and networks business. Xerox stock limped along in the low 20s for my entire career there and its employee count was about 100,000. Here in El Segundo, where we used to actually sweat about being a target of Russian nukes, there was a vigorous set of employees of Xerox. The legendary A&E building housed first rate engineers and brown shuttle vans took folks around the campus of about 18 buildings. I myself had worked in CP-8, CP-10, XC-1 and XC-2. I sometimes had lunch in one of the 'M' buildings, the big M1 cafeteria. As an intern, I toured through M2 where some of America's first 'surface mount' technology was being deployed for Xerox printer circuit boards.

Marty says that there are about 60k people who work for Xerox now. A&E is down to about 600 employees and only two of the eight M buildings still even belong to Xerox. Over in Xerox Centre, office space had been empty so long that Xerox was renting to all comers for as little as 1.50 per square foot. The boom days of El Segundo are long gone.

Today, Lexmark printers are no joke. HP dominates the 'decentralized' print world. Warnock made Adobe and all the rest is history. But it remains stunning to me that these very buildings, the whole of the Xerox campus has been so drastically reduced. Where have they gone? How long are those who are left going to stay?

It's all design work, so I've been told. It's what we do best in the US, come up with new ideas and figure out ways to sell them. But I miss the days of end to end, where you could be on campus and know you were 10,000 stong. Where the spirit of a company pulling together, from design to test to manufacturing to sales and marketing distribution and support were all in walking distance. The virtual corporation is real. We in IT and our crafty MBAs made it possible. Now El Segundo is less than it used to be.

I wonder what's going on at Northrop.

Data Mining & Easter Eggs

Q: Why do they call it data mining? Isn't it knowledge mining?

A: No, it's data mining. Knowledge is two steps up the added value chain. Once something becomes knowledge, you don't really need data mining. So let's first talk about the added value chain.

Data is less valuable than information. You can have bad data. It’s still data. You cleanse the data and organize it then you have information. Encrypted data is still data. It’s not information until you decrypt it. That’s the difference between data and information.

Good, clean, organized data = information. It’s not knowledge until it tells a story. That means a human can understand what it is. Knowledge is information in context. If I say “35 Units are in Department 7” and “27 Units are in Department 8” I have information. But it’s not knowledge until I have complete information. I don’t know the date. I don’t know how many departments there are.

Intelligence is actionable knowledge with appropriate qualifications. We could get into what make makes something actionable. But that has been done here[1].

All of these elevations in value come from human organizing data into information into knowledge into intelligence. But humans have a particular way of organizing and thinking about where to find intelligence. Think of how children search on an easter egg hunt. They don’t brute force search every square inch for things that look like eggs, they try to emulate the thinking of ‘Easter Bunnies’, the parents who hide them. Machines, on the other, search more exhaustively and find patterns that humans don’t consider relevant. A machine might notice for example that light colored eggs were hidden an average of 2 inches from tall vertical surfaces. Children would not notice, they just count the eggs. But machine can surface patterns in the DATA that create a different kind of information, knowledge and intelligence. For example, data mining an easter egg hunt might show based on proximity of egg colors and the color of occluding objects, that some of the people who hid the eggs might be colorblind. Their inability to distinguish certain colors made the hunt easier for non-colorblind children to find the eggs, and therefore the hunt rewarded faster running children without sophisticated hunting tactics. That is surfacing information in the data that humans would never have the inclination or patience to discover, which is the point of having machines do the mining rather than humans.

As soon as you have a machine-sensed pattern in the data, it changes how you organize the data into information. That is what adds value and it changes the way you think about analyzing easter egg hunts in the future.

Footnotes

[1] Barriers to Added Value in Data Analysis