Cobb

Jeremy Winters at ChefConf 2012

The Logarithmic Shadow

If you've listened to any gangsta rapper for long enough, you know that he will eventually tell you that he could have you killed. Generally, the person who would do the killing would be one of the rapper's former associates, and the price of the deed will be relatively small.

Knowing that there are people who would never kill and that there are others who would kill for cheap, or even free, poses an interesting problem. It is the problem of absolute vs relative value. This is a problem that I have found particularly troublesome in my rise through American society. It cuts several ways.

I characterize this problem as 'the logarithmic shadow', for my lack of the generally accepted term. I observe that people abstract in such a way that small differences are seen as large and large differences are seen as small. The truth of the differences are obscured in shadow, as exemplified by logarithmic scales.

Hacker's 20%
Andrew Hacker, the famous social scientist, took a survey in his book 'Money'. He found that when he asked people of various income levels what their ideal raise would be - a raise they would say could take care of all of their financial needs, the answer was more or less universal. It was 20%. People's expectations are relatively finite.  For the man with $10 shoes, those costing 2 dollars more would be all he desired, but for the man with $200 shoes, he required 40 dollars. 

Look at it the other way. For the man driving a Ford, he would need to get a BMW in order to be complimented. But for the man already driving a BMW, he'd probably need to buy an Aston-Martin to get a similar compliment from his peers. 

The Cocktail Party
A welder is invited to a cocktail party celebrating the opening of the new building he helped construct. He happens to overhear a conversation between an architect and an attorney, both who make a great deal more money. The architect makes an aesthetic statement about the beauty of a particular archway in the lobby of the building. The welder knows that there were many difficult welds used in the steel of that arch. In fact, one of his subordinates injured himself. The attorney takes the architect's statement and elaborates on the beauty. The welder pipes up, smiling with pride, yeah it cost Joe his finger. The conversation halts, abruptly with embarrassment. The subject is changed.

The Cookie vs The Credit Card
A new technology called e-Commerce becomes available so that ordinary people can shop online. In order to accomplish this, a technology called a 'cookie' is stored on the buyer's computer.  It just so happens that a very sophisticated thief can get personal information from you by hacking your computer and accessing this cookie. People are horrified by the possibility. Yet every day these same people speak to travel agents to book their business travel. These travel agents know their favorite airlines, hotels, car rental, and meal preferences. Whenever people go to restaurants, they willingly hand their credit cards over to waitstaff who make much less money.

Flies on the Belly
Marta is a starving sick illiterate child in a poverty stricken country with a population of 2 million.  A healthy altruist comes on television to tell us that for a mere $20 a month, Marta can get food, clothing, shelter and education. Tens of thousands of people respond netting $20 million in contributions. Two years later the same commercial comes on television. Nobody wants to contribute any longer.

--

These are all paradoxes of perception. I could probably structure them more properly were I to spend more time, but they all illustrate to me the difference between effort expended and results achieved, between things of equal import that can generate asymmetric reactions. I am coming to understand very specific kinds of flaws in human perception owing to the fact that I have had a career in translating such perceptions into computer aided systems of representation. A precise translation still often yields misinterpretations. People cheat in over- or underemphasis in order to communication something of value.

What is particularly fascinating in all of this to me are the border conditions of perceptual change and the manner in which shifts are communicated. 

Facebook vs Arpanet
The first time I sent an email, I was still in college. It was through a system called DECnet. I was part of a small project, assisted by a guy named Richard Z. White who was working at Raytheon. We established what was then called NSBENET. It worked for a short period of time, then went unused. Even among college engineering students, it was considered geeky and complicated.  A few years later, in 1985 or 86, I began sending email messages in the current way we do, client server store and forward GUI based email. In fact, I wrote back in 1986 how cool it would be if we could have our computers store and forward digitized music. At the time, CDs were first being sold, and I can remember a time when it was almost impossible to get reggae music on Compact Disk. The social context of spending many hours in a week in front of a computer screen was decidedly declasse. Today, a generation of parents who have probably experienced email for the first time in the late 90s are fretful of how many hours their children spend on Facebook.

--

This is the cross-fertilized field of cognition, economics and philosophy. It is fascinating to me. One last example.

The Ignored Prospectus 
Ever since 2007 behavioral economics has become a dominant theme in discussions of what's going wrong in the market. People who have relied on the 'efficient market hypothesis' have now been faced with evidence that the invisible hand is a lot more clumsy and slow than they ever thought. That's because people, whose behavior ultimately determines the price of things in the market, often misperceive risk. These aggregate misperceptions establish themselves as market prices which are not efficient and they generate bubbles which certain conditions can conspire to sustain. There is probably no better example than Greenspan's indictment of 'irrational exuberance'. If there is, it's probably 'too big to fail'. Both are examples of unsustainable prices for which 'the efficient' market did not correct precisely because the aggregate perception expressed a desire that was contrary to rationality. In other words, if enough people suspend disbelief, you can sustain castles made of sand longer than rationality dictates and 'efficient' markets are not exempt.

So one of the many studies coming out to explain where we went wrong had very definite conclusions about how people perceive risk. You could draw very specific warnings out in the language of any security for sale, and people will generally ignore those expert warnings in favor of their feelings on the matter. It's rather like a reverse transferrance. No matter how big the warning label, once people have decided to buy, they go with their gut.

--

I expect upcoming books by Taleb and Ariely to aid my consideration at length of how people's cognitive errors manifest themselves with regard to how they take up information. I'm particularly interested in this from the perspective of abstracts of big data. There are many phenomena that we have heretofore been physically unable to probe, owing to the massive amounts of data. As our technical ability increases, and more computer mediated interpretations become the norm, what kinds of cognitive errors will we make?

When The Techs Take Over

I have heard it said that there are now as many semiconductor chips on the planet as grains of rice. It is an arresting thought. And as I take my technical interests towards open source software, cloud and devops, I am breathing a whole new gulp of oxygen in what goes on in the state of the art. Take these guys for example. 

There are several ideas behind the speculation I'm about to lay down. The first is based on this reality. The computer sector is outperforming the rest of the economy by leaps and bounds. It's not only doing well, it's doing very well. Somewhere over at TableauPublic, which I can't find and has been making me leave this draft in Park for too long, is a set of charts that show which companies have cash and which have debt. When you look at companies like General Electric it's downright scary. When you look at companies like Apple, they seem to do no wrong. Times change but what if the double dip comes and times get desperate?

Over the long term, there doesn't seem to be any end to the number of excellent and productive ideas that have always been basic to IT. And it gets easier to see them become reality. Back just 10 years ago, I couldn't even imagine today's future and I'm in the industry. Everyone was saying 'the last mile' could not be overcome - that is fiber to the home. Everyone was saying that video on demand was impossible. Now Netflix is bigger than Blockbuster. People were trying and failing. After the dot com bubble, nobody had any idea what to do. The idea the Amazon would do what it has done with EC2 was inconceivable by most of us. When Microsoft was forced to separate IE from Windows and allow other browsers in, they speculated that a company like Netscape would take market share by doing things just through the browser, but nobody dreamed that Google would become what it is. In the year 2000 only the geekiest of us had MP3s and the consensus was that nobody could break the hammerlock of DRM. Now Apple runs a DRM free universe of music on demand. Moore's law has ten more years, at least. That means that the iPhone 7 might have a terabyte on it. 

All of those innovations are extra cool, but the fundamental economics of IT can still be expressed in very simple ways. The key term is 'disintermediation'. What will computer and communications tech allow people to do for themselves that they used to need third parties for? To jump the gun a bit, I will suggest banking, and when that happens ohh doctor!

First off, the fundamental thing that IT gives is the ability to overcome time and distance. It enables human intercommunications on levels never achieved in the history of mankind. In and of itself, this is an economy pulled out of a hat. Without disintermediating planes, trains and automobiles, there are new ways that people interact that make IT a non-zero sum game. Look at a movie from the 80s and find all of the plot holes and crazy situations that could have been obviated by today's cell phone networks and GPS. 

Whenever I hear it said that reading and writing online saves a tree, I also know that it saves oil and electricity. What people tend to forget is that computing owes its very existence to the ability to engineer devices that sense very minute changes in electrical state. IT is energy efficient by definition.

What's happening to crowds these days is something very unusual. People have their own networks, and communicating with them and their paths makes an extraordinary difference in attention and focus. Many people may not quite know what to think about virtual friends, but the opportunity to have multiple online networks is a new social skill. Your telepresence will be a bigger status symbol than your car.  The hardware and networks into your home will be more important than your lawn. For those of us in the business, that future is now. 

Let me ask you. Do you remember the classified ads? 

Newspapers and broadcast TV are being redefined. You might laugh about Hulu today, but people laughed at Tivo five years ago. If you think Apple TV is important now, wait until film is completely passe. What's happening on the edges of digital networks is critical. Devices like the Red camera system and the iPhone are changing everything. Given what I know about the infrastructure that is coming, YouTube and Facebook and eBay are just the beginning. 

I think that in the not too distant future, much of banking is going turned on its head. If you think about banking as a collection of discrete trusted financial transactions, there are many classes of these things that can be disintermediated from the banks of today. It's already happening at Square. And look at what people are doing for themselves at Prosper. 

Back in 1996, I was thinking about the cloud before it had such a name. In my conception, I couldn't imagine that people would trust ISPs to retain their digital assets. So I imagined that only banks would have the financial ability and trust. Now Google does it for free, and Backblaze does it for $50 a year. When you accumulate about 10TB of digital assets, you're going to have to trust people too, and they will be a different set of names. Let us not forget that when I started out in this business, corporate managers would not *think* of using computer systems to transact trusted financial matters. The idea of a manager approving an expense reimbursement without physically touching a receipt was unthinkable in 1990. Now think about Western Union, and money orders. That's a shrinking business. 

In short, there are more and more ways the American economy is going digital, and there are fewer and fewer barriers. Young entrepreneurs are building software systems that disintermediate back office protocols. Every workflow in every business is being optimized by IT - it may be some time before you don't have to go to the DMV to renew your license, but I bet you a team of 10 people could build the system in six months. 

White collar and intellective work is changing. Permanently. That is going to have a tremendous positive effect on the energy efficiency of the American economy. In education, in the professions, in business, finance, banking, insurance, medicine, law, publishing, music, film and all the visual arts.

FUMH

Enderle explains some of the reasons that Mark Hurd is considered a pariah in the industry. Now fled to the barbarian king Ellison, Sam Palmisano of IBM is starting to pitch crap at a weakend HP and Hurd in particular. 

It rankled CEOs from Palmisano to Michael Dell to be compared to a guy who by most measures was slowly killing his company to maximize quarterly returns.  A kind of vampire, Hurd represented the inordinate power financial analysts had and how they were forcing decisions that were crippling long term. However any company that didn’t listen to analysts was pounded in the market with an expensive impact on executive bonuses and longevity.  It was understandably upsetting for Palmisano and others to see Hurd receive massive rewards for sucking the life blood out of HP while their better parenting went largely unnoticed and often unrewarded.

Aha.

And so now some of the reasons why this website exists are becoming clearer. The news there is that Hurd is getting booted off the board of News Corp.

I can tell you that it is music to my ears to hear that this particular kind of exec is getting flamed. HP is one of those great loved companies to engineers, and the very idea of having a finance goober run the joint into the ground just by tweaking numbers to please Wall Street analysts is horrid. So now that this one has outed himself as an asshat. To earn the appellation 'The Turd' takes some doing, even among vulgar grouches. But now the saga has taken root in my attention sphere.

I've always admired HP. They're having it rough up there.

Rights and Wrongs in IT and Software

Ceolas writes about The Industry:

Performance based? Nope, it hasn't been performance based for almost ten years. It's seen a loss of innovation and invention. Microsoft's former VP elaborated the problems within the company in an op-ed in the NY Times that have killed it's future, what he didn't mention, was the fact that Bill Gates has always only been a one trick pony who got lucky, and isn't interested in invention and innovation so much as denying others with the ability, the opportunity to succeed where he has failed. He also didn't mention the fact that the preceding decade has been one where Gates displaced US citizen workers with cheap Indian and other visa workers. Un/under educated, and in no way skilled or talented. Anyone who works in the industry knows that while they might like to act like sharks, they're incompetent and capable of doing little more t han steal the efforts of others. Everyone knows the real story behind the failure of the Indian tablet pc. It was a stolen prototype, one that hadn't been fully developed. India's tech house of cards will fall, because it's based on a lot of hot air and theft. Everything they touch, they foul.

 

A lot of this is right, and I would like to discuss it.

First off, Gates and Microsoft are not the spark of the industry, they are a marketing powerhouse and a very hermetic universe. They are almost incapable of innovation because everything is done the Microsoft way. My experience is that they are simply too thick to be agile and that they spend all of their time selling to mass markets. At Microsoft scale, they are a one size fits all company, and even when they break things up, like the different versions of Windows 7, it's still just too huge. There are very few people who think the way I do who expect much from Microsoft in terms of cutting edge. There are many boring things that Microsoft does well, but few truly exciting things that Microsoft does that win over hearts and minds of techies like me. They have just been too obstructionist too many times and so it is no surprise at all that a company like Google could just come through and steal every bit of glory Microsoft ever had. 

There was a sense two years ago, when I spent a lot of time in Redmond, that MS spends a great deal more time being a defensive company and that their captive audiences really make MS believe that they change the world. Microsoft is starting to show its age. They should voluntarily break themselves up. And they should make a tiny, new OS.

As for the Indian quotient. I have to say that there are just too many flavors to make broad statements, but I know where the dissatisfaction is coming from. I don't blame the Indians so much as I blame corporate IT for outsourcing their own brains. And corporate IT has just discounted itself below the threshold of where people with software skills are willing to work. And this is what leads to the incredibly sclerotic world of enterprise software, where good ideas take forever to materialize and Oracle, SAP, IBM, and a handful of pretty unimaginative business models are entrenched. 

Metathought on Google Wave

I saw something last night from Google that made perfect sense to me. It was that the same two brothers who were the brains behind Google Maps are the brains behind the next generation of email. It's called Google Wave and it is a brilliant blending of instant messaging, email, intelligent conversation threading and blog content management. It might have been more but I fell asleep watching the 80 minute video.

It solves a great number of problems that we emailers have been working our way around for 20 years and promises to put us on the Axiom for 700 years.

I have been considering the nature of a society, or a moment in time and a crevice of society, that celebrates and rewards toolmakers. And I think I have seen the end result. It oddly comes from another show that I watched last night (aside from Wall-E) which was on the Wealth Channel. It turns out that one of the guys that gets on the list for the most expensive cars in the world was showing off his Ferarri 430 Scuidera, his bright yellow Rolls and his Bugatti Veyron. All were customized to excruciating perfection, exactly the way he wanted it done. He kept referring to them as works of art, which they certainly were. This made them one of a kind and useful to nobody else in the world except as the ultimate status symbols, which was why they were on the Wealth Channel in stunning HD. The toolmakers in this case were celebrated by signing their names on the carbon fiber under the hoods next to the prominent steel embossed serial number plates. (His Bugatti was the fifth one imported into the USA). So, essentially and especially given the HD documentary, he has converted these automobiles into instant collectors items. The toolmakers don't have their tool's utility as its own reward, but the customization of it to please a completely other sensibility.

This sensibility has long been in evidence in the computing industry, which has long overproduced any dozen redundant technologies in 8 flavors. It is why the Linux world exists in such variety even though all we ever needed was Red Hat, a viable alternative to BSD, MacOS, Solaris, HPUX, AIX, System V and Windows. Instead there are some 3 or 4 dozen operating systems including the embedded world which is a complete mystery to me, each with dozens of programming languages. So it comes as no surprise that only the juggernaut of Google can do something utilitarian like perfect email once and for all. Further it comes as no surprise that it only takes two or three really smart people to do it.

So what is Web 2.0 and all that social software and clouds and grids all about?

That's a big question for another day, but the short answer is the computer equivalent of suburban sprawl. Nobody, save perhaps the NSA, has sat down and built an actual computer and software system from the ground up for 40 years. There is no planned community. There are only applications and marginal evolutions of applications for an interoperable world that has 60 ways to do the same thing which frustrate us all into looking towards Google to finesse them all back together in unifying ways. Surprise.

But Google is not going to do that, which is why I mentioned the NSA. You see outside of getting applause, stock options and face time at conferences, the rock stars of software development have no transcendent purpose. Only those who have had such purposes have really mattered, and I think we are in short supply these days. So it makes me wonder exactly how Google defines the 'evil' they will not do. It's certainly inclusive of the 'evil' the NSA must do.

From a technological perspective, Google is the embodiment of infinite scalability. Its compute engines are the Saturn Five of the computing world made all the more impressive by their tethering of millions of actual horses to make the horsepower rather than giant, fallible boosters. But I am confounded by the idea that the Rasmussen brothers, the brains behind Google Wave, would be so satisfied with the Google compute environment for what it is that their minds conceive a compute beast with such power could do. It's the old trade off of what kind of millionaire you want to be - the rap star wants $17 worth of respect from 5 million fans. The investment advisor wants $5 million of respect from 17 clients. They have obviously decided to catch the attention of the twittering masses.

So what is the transcendent purpose of making email, instant messaging and blogging incrementally simpler for 50 million Google users? Surely that will not enable one more significant writer on the planet.

This is what I notice every time a new Web 2.0 3.x or whatever program comes out of the software programming world from American rock star programmers, and was at the very beginning of this demo. It's all about 'Hey let's find a new restaurant' or 'Check out my vacation pictures'.

I think I've made my point.

So how about Google Medical Records, bitches?

On Complexity & Emergent Behavior

Riffing off Tim Burke:

There is a fundamental problem with building efficiencies into systems. There are essentially, with regard to white collar productivity, two schools of thought, Deming and Hammer. Deming for constant quality improvement and Hammer for business process re-engineering.

The Deming approach requires a certain understanding of the process in which all workers are engaged. From the standpoint of a software designer, it can all be conceptualized in terms of a workflow. Person A has tasks 1,2,3 Person B with function X has tasks 4,5. Task 4 requires 2, etc on down the line. The willingness of software engineers to absorb the details of workflows in business is highly limited - and the ability of managers of such processes to communicate their complexity and possible efficiencies and chokepoints, etc. is also highly limited. Furthermore the willingness for software engineers and business managers to deal with each other on a long-term basis is slight. Both see their time more profitably spent elsewhere. This is especially costly to the Deming scenario, and so what generally happens is a Hammer implementation.

This does damage to the process by obviating some complexity. So you actually have more complex and new systems shoved into a process which is actually dumbed down because of the communications limitations of managers and engineers. Then what ends up happening is that those employees in the process chain make up the difference by partially working with, and partially working around the new system in ways they have no incentive to explain, and often in ways their managers are unaware of. The outputs of these systems as well as their inputs may be manipulated because those processes to which information technology is applied are rarely end to end systems. So there is a complex adaptive dynamism at work and in the end describing these systems are like describing the flow of molecules of water down a stream. They are chaotic, somewhat random, extraordinarily complex, yet at a macro level comprehensible and predictable.

Once a system becomes predictable at a macro level it gets frozen into place as the practice of the business with all of the inefficiencies and mysteries locked in.

And then the boss changes and all of the internal stresses on the system lose particular incentives.

The Chalkboard

My brief comment in response to the thread.

What strikes me odd about this is that there are thousands of (apparently) companies that keep pointing to youth and (maybe) understanding the technology that connects people together, but they seem to have no respect (or clue) to what people do after they are young. I don’t need another friend and friending as a verb is ridiculous to people who are married with children.

What would be transformational is if someone took another look at the most important things people do in life and built tools around the individual, especially the information intensive individual.

I work in the business intelligence business, and started out with at Xerox long ago under the influence of people like JSB. What I have come to learn is that people interacting with computing systems do indeed come up with verbs, and the most intense interactive experiences end up having their own verbs. Podcasting is an example, as is ‘friending’, ’surfing’. And yet it has only been recently that we have been ‘buying’ or ‘banking’ and obviously nobody has been ‘voting’.

None of these things work as transactions as well as they do as interactions. It’s not just a simple set of verbs but a set of interactions around important functions - this is the approach that works best, when a person’s activities become describable as a workflow. The problem is that we’ve got a bunch of discrete actions in most of our social computing - rather like old video games before the invention of the sandbox, or old top-down programming before simulation loops - that don’t give us the sensation of swimming or anything continuous, like life itself.

I would argue a la Chesterton that we not tear down walls before we understand why they were erected. I’ve enjoyed auditing video lectures of physics ‘at’ Berkeley as much as any student in that classroom. But I doubt many students would be comfortable with my voyeurism - there is a temporal wall in the non-live rebroadcast that gives some propriety, and we all need a certain amount of that.

I would also be wary of giving over to a Friereian view of ‘equality’ in the exchange of information from teacher to student and vice versa. I say so as an old man who is not entirely convinced that all questioning of experience is useful, moral or productive. Pay very close attention to what is given and what is taken, because whatever arrangements we conclude are superior must inevitably be sustained in order for them to be anything more than academic. Keep that in mind especially in view of the factoid about how much laptops cost vis a vis wages in the Third World.

One of the things that was fresh in my mind as I blasted through that comment was this set of improvisations and meditations on the improvement of the Bloomberg Terminal. As you all know I'm very much into Bloomberg these days - information monks from way back, they are, Yoda like in their inscrutable wonkiness. But also I'm thinking of sandbox decision making and how the shape of an environment is determined by the rules many of which are invisible.

We don't often have sandbox-like decision making abilities in our compute environments. They are very discrete. We simply don't have the economics to achieve that yet. Not until the price of sandbox environment development comes way down are we going to be able to break down walls of education and have real simulations of important interactions with realworld implications. I can visualize it all myself - I've been working in the environment forever - but it is only beginning to become economical for SMB much less the general public.

It Seemed Like a Good Idea At the Time

There's something about soccer that best captures the spirit of HD television. But I have to tell you that a lot of sports is looking really good this season. I've been watching and recording the HD broadcasts on Universal HD and they are spectacular. Until I speed them up, and then I realize I don't know what I'm watching.

For a moment this morning I had an insight about machine intelligence from the Culture. I was fast forwarding through a bunch of commercials and then through some of the soccer game, and I realized that I was catching a bit more information on HD than I used to on the standard definition Tivo I used to own. But I was surely at the limits of human comprehensibility - there was no way possible for me to make sense of the soccer game itself. I could only recognize that it was a soccer game and nothing more, even though it was not literally a blur - several images were clear as stills - I couldn't process it with my organic brain as fast as the DVR could play it.

Suddenly I remembered a conversation between ships from 'The Excession' in which they had decided to take some action after a discussion that went for several pages in the book. But one of the ships suggested that before they do, that they ask their passengers what they thought, after all things had changed. One ship objected saying that it had polled the passengers 2 minutes ago and they had assented to go. The other said, well you know how long it takes them to decide things. You have to give them time to think.

A comic routine I heard on XM 150 the other day made a running joke about the common excuse 'It seemed like a good idea at the time'. It is the ultimate excuse. There are a lot of reasons that our human psychology makes the decisions it does, we are subject to certain inevitable biases, not the least of which is that we often make patterns that don't exist. We make errors in cognition due to a lack of time or focus or based on irrelevant data that somehow seems relevant at the time. For example, this very blog piece started off as something about the Olympics, but it is actually more interesting (now) as a thought piece on human vs machine cognition.

Being in the BI community, it is my job to see how machines can help humans make better decisions. The better I understand the business, the better I can tailor the system I'm building to assist the a limited scope of decisions to be made by the audience. When I described my gig to Unk last week, I got down to a specific job that I'm doing for a specific customer. It turned out that he had a similar issue during his days shuffling budgets as an administrator - and so he actually repeated that case back to me flawlessly.

There are lots of reasons we decide poorly, but when we are under time pressure, I think that is the reason we make the most mistakes. I'd make a final observation which is that some folks decide forcefully but then are amended quickly. That tends to be my decision making style - to set up a framework in which a good number of choices can be taken (in a fast OODA loop context) and harness as many observers in the process as possible. The quick decision coupled with authoritarian style can be very effective within organizations, but also dead wrong. In a proper framework, quickly wrong, might not be so bad.

Speed is key.

Korean Beef

According to a quick glance, South Korea does about 78 billion dollars of business with the US annually. However they stopped the importing of American beef five years ago out of fears of mad cow disease. That strikes me as strange. We've never had a problem. Some may say that mad cow is inevitable in all beef supplies, but that doesn't seem a cogent argument against ours. It's about 20 billion worth of business that we've been doing without and in April the Bush Administration signed a deal to restart the flow. That's a lot of beef.

Well it turns out that South Koreans don't trust American beef, no matter what kind of deals our governments have put together. And so thousands filled the streets recently to throw down against riot cops in the streets of Seoul. The Democrats are opposed to the trade agreement between Korea's Lee Myung-bak and GWBush, but it's unclear why. Lee has something like a 25% approval rating in his home country, and there is probably no easier way to excuse any behavior in the US than saying it's the opposite of what GWBush would do. So the South Korean government is caving in by playing on what appears to be a popular idea that cows younger than 30 months old are less susceptible to BSE. Don't ask me about how prion-based infections work, half the websites I looked for on BSE are down (except for PETA's of course).

Bottom line politically is that it comes down to a matter of trust. The number of people who are going to be able to make any actual determination on the quality of American beef is not going to change significantly. The Koreans in the street are going to take it on faith, just as they are taking it on faith that American beef is unsafe. There's an interesting unravelling going on here.

My interest in this is based on my own kind of information theory with regard to the ability of people to establish and maintain policy that is both transparent and well-abstracted. I will follow up in that regard again in speaking about how politics makes people emotionally deranged and affects their willingness to distinguish Natural Truth from Political Truth.

You see there's a bioengineered solution to the problem just at hand. But what Venter understands is that engineered genomics is predictable whereas random DNA fiddling is not. That is precisely the difference between bioengineering and agriculture.

Bloomberg & Government Transparency

NYC Mayor Michael Bloomberg is a pioneer who is at the very beginning of a revolution started simultaneously with Walter Wriston. The second fruits of his pioneering spirit are beginning to bloom. The first fruits were, of course, his fortune in the securities information industry created by the machine and company that bear his name. The second fruits can be seen here in Citywide Performance Management.

Microsoft + Yahoo = Monster Zero

What I learned about Microsoft is that it already has three and a half heads. It's now poised to add a fourth. Since I've been purging myself of MS over the past several weeks I don't have any idea where I put my notes that told me exactly what the acronyms are for the three of Microsoft's 10 billion dollar businesses. But the biggest point, which led to my leaving the Kiretsu was the realization that Microsoft is a technology galaxy of loosely federated product groups. This has specific advantages when it comes to providing a platform, but specific disadvantages when it comes to building product stacks, and product stacks are what sell in the enterprise software space. For my tastes, and for the tastes of every SAP, Oracle and IBM customer under the globe, Microsoft's product strategy doesn't have the requisite focus. I won't belabor the point.

What's shocking about this particular idea is that it is a startling admission that MSN is a failure. I can't imagine anything more or less than a huge, huge culture shock.

What's obvious is that this is an attempt to kill Google, or replicate parts of its business model. But Google's business model is actually besides the point. Advertising revenues through online are really means to a Google end which is to build a single computer for the world and world scalable applications for that computer. Microsoft is just adding functionality to a massive installed base that has never and will never talk to itself the way Google customers talk to themselves. Which is to say Microsoft enables software gadgetry with no overarching design. Adding Yahoo to the gumbo.. well it only makes discrete sense not holistic sense.

So Microsoft is going to grow another head. I think it makes that which is ugly even uglier, and that which is dangerous more dangerous still. It will show Microsoft's ability to fight in yet another domain and it will put another lovely bushel of mulch into the compost, but don't expect all the fermentation to yield anything smooth for another four years.

Cognos To Disappear Inside IBM

I have three gut reactions to the news that Cognos is going to be acquired by IBM for 5 Billion.

1. OK it's official and now there is 'nobody' left standing. 'Nobody' meaning Information Builders, Microstrategy, SAS and the open sourcers. SAS is not going anywhere and was never the same kind of player as everybody else. Microstrategy is probably not an attractive target, nor is IBI. They will not be assimilated and will basically die off, like everybody else.

2. Cognos is going to disappear inside of IBM. It will be absorbed into the Blob of Websphere and will be IBMized into marketing oblivion. Very little of Cognos branding will survive once the product integration is complete. It will find itself barely recognizable amongst other brilliant IBM technologies like Cube Views etc. The good news is that if the DB2 engineers get their hands on it, Report Net will ultimately and finally work as it had been advertised.

3. Microsoft stands alone, again. We (now that I've sold out to the dark side) have a lot to prove in the enterprise space, but what we won't have is a huge product integration story to have to sell. That is actually a big deal in the enterprise space and the FUD works in Microsoft's favor in the short term. Whether or not MS reps can and will capitalize on this moment is anybody's guess. Even though I'm on the MS extended squad, I'm still from Missouri.

Outside of that, I will speculate that this finally represents the top of the bell curve in an industry that has taken too damned long to mature. I walk around with my jaw dropped when I think about what a focused industry might have done with BI had any of these monster companies done this acquisition work in 1999 when they should have. Then again, maybe you're nothing until you're a 5 Billion dollar company as far as the IBMs of the world are concerned.

Which is why everybody ought to be scared as hell of Google. One of these days, the 20%ers are going to recognize what they can do with Gapminder + BigTable. The lightbulb is going to go off in their heads and they're going to see how a bunch of their technologies can be put together into a zero footprint xPM stack. If they do it within 5 years, then they will make all of the big BI acquireres look dumb. What Google doesn't realize is that their ability to structure data from multiple sources is a huge strength considering that they've built an empire with data people have volunteered to them.

What is BI, in the end? It's gobbling up and spitting out very small chunks of data each cell of which has an extraordinary amount of metadata associated with it. Nobody has done that in parallel scale. The entire paradigm is ripe for re-invention. And this time, I hope they put some nicer graphics into it.

More Solutions 2007

Several  impressions and reflections on the last Hyperion conference ever.

MOLAP Wars
The good news is that some of the best Hyperion folks on the planet are still with the company. That is to say some stellar Essbase product managers and developers are on the team and not going away. That includes guys like Marciante, Nader, Hite and Tai as well as very capable new folks I've never seen before. The even better news is that it's Essbase again. Not Hyperion Analysis Services Enterprise 9X or some huge mouthful. So when it gets right down to it, the Essbase we know and love is there in all its beauty.

The bad news is that I didn't have time to sit down with the Essbase guys and talk about futures. It was too crowded. But I did see some slides that said 'Trickle-Feed' and some that said 100s of Gigabytes OK. Since I have heard last year that the aim was for ASO to handle a full set of BSO functionality, eventually, I will have faith. So there is no doubt that Essbase has a future.

In fact, I think the case will be made abundantly clear going forward that the Essbase-centric paradigm of OLAP development will continue into the future. It is the platform for the future of BI. The Oracle-Hyperion people know exactly where the strengths and weaknesses of OLAP are, but Essbase still  has got the advantage. That advantage depends on the combined company's ability to reposition the meaning of BI.

Yeah I said it was murky, but that's because the presence or absence of Essbase is not necessary to make sense of the application stack. It's not until you get into the details of how capable these applications are going to be at the individual level that you begin to recognize how key Essbase is to them. That's why Oracle is challenged to present BI in a whole new context.

The New BI Paradigm
People have been saying 'drill down' and 'slice and dice' for two decades now. And you would think that they would have gotten tired of that. Oracle is going to show them how retarded that is with their new technologies going forward. The reason is that now that System 9 is the big dog on the block, the APIs to Essbase are going to support a class of workflows that have never been so integrated before.

What the Hyperion engineers know, better than perhaps anyone on the planet, is how analytic workflows run. They have been doing complex OLAP for many years. They understand the cyclical nature of OLAP queries and iterative analysis, planning, goal-setting and review. Now this is what they are calling the BI paradigm, and what is new and exciting is that they are going to be exposing more of the data mining features of Essbase (that have been around for years) into the BI paradigm.

Obviously Essbase has always been able to handle multiple scenarios of data. There isn't one real application that we deliver that doesn't have at least three. What's new is that they are going to bring forecasting into the foreground with tools that will allow end users to drill down into a set of products (for example) and mine the past data, as well as the past generated data and automate the process of selecting heuristics, employ them and expose those heuristics for adjustment.

This means in Hyperion Planning, it will automatically fit curves at every level and generate a base forecast for you. It will crank those through and allow you to select for factors that effect the outcome, and recycle. This is way more intense than "take last year's sales number and add 10% and spread it". This is more like figure out which products have the most affinity with this new product we have and predict its sales, calculate the profit and forecast the profitability. Now take into consideration my promotions budget, and reforecast sales volume. Now what's the profitability?  Now lets cut costs here and reallocate to tweak the price against this demand curve. What's profitability now?

That's the new paradigm of BI. It's not just reporting, it's smart extrapolation of financial and operational history and application of data mining for the individual user forecasting. It is a revolution just waiting to happen.
 

Merger News
One of the comments that struck me in elaborating details of the merger was that Oracle was impressed with the amount of work that Hyperion produced per capita. As a relatively small company, the extent to which Hyperion absolutely dominated BPM was stunning to the Oracle folks. In addition, I've heard tell that they are staffing up their pre-sales and post-sales consulting groups. So all that points to more reasons to believe that as a wholly owned subsidiary, Hyperion will not lose, but possibly gain headcount within Oracle.

Crystal Ball
I've been in this business for a long time and even I had a hard time following the presentation by the guy from Decisioneering. What I can tell you is that a whole new era of risk management is coming to be integrated into the planning cycle. That is because unlike the products I've seen before with SAS and SPSS, this stuff is totally integrated with Excel. Excel makes the world go around, and Essbase connects that world with the universe of business data. But I just learned how to do a 10,000 case Monte Carlo Simulation and a 10 factor sensitivity analysis in under an hour. I saw curve fitting and confidence intervals put in place and I saw a 4 year forecast built in front of my eyes that was simply brilliant. And guess what? They did it at Ludicrous Speed.   

MapReduce
By far the coolest thing that I learned at the conference, aside from the fact that I've still got my presentation chops, is that the guys at Tellme are using MapReduce to get massive data into Essbase. What does that mean? It means that I now have a framework for integrating VLDB data into an analytic platform.

Tellme basically generates 100GB of data daily. Huh? What? That's what I said. 100GB daily. it comes from processing 5 million telephone calls which generate on the order of 150 million detail records, PER DAY! They've got something like 30 MySQL master / slave arrangements in a two-tier MapReduce network. They issue a query through this very cheap, open source network of PCs in about 4 minutes. That's right, in 4 minutes they have processed all of the aggregate data they need into an ASO cube. That, my friends is scalability.

SmartSpace
The second coolest thing at this conference was SmartSpace. Basically, this is the future look and feel of BI. All I can say about it is this: Forget portals. Forget Web 2.0. It's all about the widgets. SmartSpace is, make no mistake about it, a killer app for Windows Vista. I could not take my eyes off of the screen. It was absolutely gorgeous. It's a single signon, intergrated IM, multi-canvas workspace for System 9 that deploys sharable widgets for every kind of display and control available to the Hyperion stack. It turns a PC for BI what Photoshop does for a Mac. There are auto-hiding toolbars, floating controls like a calc script initialization widget, for example. I didn't think anything would make me take a second look at all this again, but this is the front-end of the future. Oh by the way, it also works like a thumbdrive. You 'mount' a SmartSpace and double click and you get the hierarchy of System 9 Workspace right in the Windows explorer. I'm in the Beta, buddy.

The Birth of EPMS

The general session of the Hyperion conference here in Orlando is over and ... I like it. This one was light on hype, and pretty much got straight down to business as it was dominated by the presentation of Thomas Kurian who will inherit and run the Hyperion development organization.

First thing first. I've been pretty positive that the merger was a good one, and Hyperion president Godfrey Sullivan communicated that the vision of the two companies was very much aligned. He said that the entire thing went very smoothly and you can be fairly sure that he was straight about it. The whole thing went down in four months and now Hyperion is officially a wholly owned subsidiary of Oracle.

What is very clear here is that Kurian is all business and has a rather deep understanding of the products and their capabilities. But there is a very subtle genius in this acquisition which is going to make a fairly impressive change in the marketplace. But let me give you one piece of background that I discovered yesterday. The Hyperion salesforce is safe. This is a big deal. So it appears to me that Oracle is big enough and healthy enough not to go headlong into 'synergies' and cost cutting and that they probably won't until they have significantly changed the market. Even so there's a whole lot of product to sell, which is the genius. Kurian's product strategy is clear. He's going to a modified Six Sigma business model that boils down to the phrase "Insight into Action". There are going to be vertical suites for all of the major industries, and all of the major products are going to be represented. The secret sauce? Contextual navigation.

You may have heard that before. The whole idea of the Hyperion Portal which died somewhere back in 2000-2002 was all about that. The problem was that even though it was a great idea, nobody with Oracle Financials was going to manage their business from the context of new BI technology. Kurian's emphasis on drilling down to application-level detail and changing assumptions in the ERP and CRM systems shows that this is how customers prefer to manage their operational business. It always has been a difficulty for Hyperion to sell end-to-end, not because of a technical limitation but to a lack of an operational integration story. Those days are over.

It's All System Nine, Baby
Oracle buys System 9 completely. System 9 will be the framework for all of the BPM going forward. It is now recast as Enterprise Performance Management Systems. Before long, everybody will be calling this the Management System. It integrates financial reporting, operational BI and CRM+ERP. The vision is bold, it's sensible, it's workable and it looks damned near inevitable. It's also going to cost a lot of money because all of these systems are going to have a lot of baggage. However, there's a lot of time to work all that out.

But here's my quick sketch on it in terms of emphasis.

Hyperion Winners:

• System 9: New Industry Standard
• Planning: Huge win
• Crystal Ball: Big Win
• Hyperion Scorecard: Win
• Hyperion Strategic Planning: Win
• Hyperion FM: Big Fat Win
• Hyperion MDM: Big Fat Win

Oracle Winners:

• Oracle BI EE: Big Win
• Oracle Alerts: Big Win
• Oracle BI Packaged Apps: New Industry Standard

Muddy:

• Hyperion EIS
• Hyperion Essbase

In the new era that's coming, all of the analytics from every cross-platform thing is going to trickle up through System 9. All that's left is to figure out a new Semantic Layer and Oracle BI Foundation. This is where it gets muddy. Obviously Essbase is the keystone of all of the Hyperion Applications today. But at the enterprise level, you don't care about any of that in terms of presenting the solution to customers. Depending upon how smart Oracle gets in that Semantic Layer (dimensions, hierarchies, metrics, calculations.. etc) pipes through to the BI Foundation can go any which way. In this regard, sure we'll be serving up multidimensional models for analysis and reporting, but they won't necessarily be optimized for Essbase. But that will all be internal to the new development group. I have a lot of confidence that Essbase won't be going away as a product, but in the context of EPMS, it's just a small fraction.

Basically the number of integrated applications available has just skyrocketed. Making all of it work together is going to be a lot quicker and smoother than most people imagine and making sense of the products is going to be less complicated than I thought. The big vision works, the integration is on a fast track, this train is moving.

Why Essbase Looks Muddy
The architectural look of this whole new stack boils down to something fairly straightforward. All of the major Hyperion apps work through a middle tier which is a combination of a relational repository, a Java server and one or more Essbase Cubes. But once the semantic layer is muscled out, you may be able to swap Essbase out for Oracle. That's going to take years, and lots of extentions to OWB, but systems like Star Analytics' SIS could facilitate all that. Now that Oracle owns Essbase, we'll find out price/performance wise if that's a good direction to head, but I'm convinced that the two companies are going with the market leading technologies as a guide. I don't know if Express is behind Oracle BI EE, but Kurian was talking about Essbase as a source on the same level as Hyperion IR and Oracle BI EE. Meaning that pure Essbase apps may be just considered point BI solutions which don't play a major role in the EPMS world going forward.

So I conclude that Oracle is putting behemoth resources behind this. Their combined R&D with Hyperion's, which was no joke, is going to put huge pressures on everybody else in the space. The story already works and already looks architecturally sound. With the combined customer base, the possibilities are very broad and deep.

Fire or Poison?

Deca-BDE is a controversial chemical. According to the SeattlePI:

Stung by the Legislature's groundbreaking ban of a controversial fire retardant, the chemical industry isn't giving up.

Even with Gov. Chris Gregoire's approval all but assured -- she's scheduled to sign the legislation today -- it still didn't extinguish the chemical manufacturers' fight. They've run full-page ads in the state's largest newspapers. They wrote a letter to Gregoire urging a veto. Since 2005, they've spent more than $220,000 here lobbying against a ban.

Why the big spending? Because there is potentially more at stake than prohibitions on flame retardants called PBDEs in Washington furniture, televisions and computers.

A ban here could be the beginning of the end for PBDEs, or polybrominated diphenyl ethers. It could mark a move toward more aggressive protections by local governments -- at a time when the federal government is largely ineffectual in its regulation of long-used but potentially dangerous industrial chemicals.

"The larger story is that there is a highly organized and very well-funded effort to change the way in which chemicals are regulated around the world," said John Kyte, spokesman for the Bromine Science and Environmental Forum, a trade group representing PBDE manufacturers.

One of these days, if a guy like me gets rich in the business, I would have a business model that would put together publicly accessible knowledge bases that would allow environmentalists, industry trade groups, academics, legislators and the general public to look at the same data at the same time and make more scientific conclusions than they are doing now.

My jaundiced eye says this is basically going to come down to some study saying that the increase in the number of cancers in some lab mice goes up x% when they stuff Deca down their throats, and on the other hand we're going to see a repeat of the empty crib commercial because little Johnny's pajamas didn't have fire-retardant. Then somebody else is going to charge that Johnny can't read because the flame retardant in his pajamas is giving him brain damage.

In my business, I make systems that would allow the manufacturer of pajamas (or furniture, or whatever) identify the commodity in his supply chain which is controversial and allow him to swap that out and run a simulation of what a new and improved 'organic' product would cost. I'm talking hours here. With blog-speed, this could become its own marketing campaign. The trick is to make the market that fast.

Even though I'm caustic about environmentalist thinking (mostly because it is ignorant of manufacturing processes and puts an arbitrary infinite cost on those things it wishes to avoid), I have no problem with their end goals. But it should fall to the consumer to have and know the alternatives. I want to be able to go to Dollar Tree and buy pajamas for a buck and know that they are not fire-safe, or that they are fire-safe with DecaBDE for 2 bucks or that they are firesafe with the new post-Washington State controvery Sierra Club approved fire retardant for 3 bucks.

Open the process and break the interest group oligopoly on product R&D. Or sleep naked and buy a fire extinguisher.

Boot Time

I haven't bothered with Windows Vista, and with any luck I won't have to for several years. XP Pro is as good as it gets under Microsoft, and I'm not taking the home network any further along the Microsoft upgrade path. We'll go to Mac first. I refuse DRM, and XP does everything I want.

Over at ZD Adrian Kingsley-Hughes asks why long boot times matter. I essentially get his point, but what he overlooks are the number of software installs, not to mention seemingly random software patches from MSFT that require soft boots. Also, whether we want to admit it or not, there are many companies who require that people turn off their PCs in order to save energy. I know that's dumb and such businesses are on the verge of collapse, but...  But the biggest reason long boot times are annoying is because they don't have to be.

Leaving aside the brilliance of Knoppix and other Linux on a CD rootkits that can embed very cool Windows utilities, we know that there are a whole lot of extra things that go on during the boot cycle depending upon what software you have installed on your machine. Why not just push those back to the post boot phase? When I'm logging in I know that a whole lot of profile driven stuff is loaded onto the machine. How many times have I sat waiting to get a wireless connection in an airport hoping the last few minutes of battery life don't slip away before the hourglass over my toolbar disappears? What's going on here? Something. There could be a class of applications that I want to be able to use right away just after boot, and just like power consumption priorities, I ought to be able to determine readiness priorities.

The Google No Brainer

Arizona State goes from spending 500k/year to zero. Google's threat to Microsoft Exchange is so potent, that it's not even funny. Check out this story:

When Adrian Sannier became the university technology officer at Arizona State University in the fall of 2005, he was handed a simple yet monumental brief by ASU president Michael Crow: make IT the primary driver in Crow's ambitious "New American University" project. The goal is to raise ASU's academic standing while increasing the number of resident students from around 65,000 to 90,000 by the end of this decade.

The first step in the school's new IT strategy was an alliance with Google's Enterprise Solutions division. Last October, ASU became the first large institution to deploy Google Apps, a comprehensive suite of productivity applications that includes e-mail, search, calendars, instant messaging, and even word processing and spreadsheets.

Initially offering new e-mail accounts based on Google's Gmail service (but retaining the "asu.edu" domain) on an opt-in basis, Sannier and his team found that students were making the switch at the rate of around 300 per hour. Today, more than 40,000 ASU students and faculty have made the switch, and he expects to shut down the University's in-house mail servers near the end of this term.

Since the e-mail switch-over, Sannier has been rolling out additional applications including calendar (which users can now share online, a capability the old university system didn't have), IM, and search. Within the next two months he expects to offer personalized home pages as well as online word-processing documents and spreadsheets based on Google Apps.

The cost to ASU: zero.

Ray Ozzie better get busy integrating his Groove stuff into the next gen Microsoft collab-platform. Oh wait, it's done. Who knew? Who'll pay?

 

AS/400 OMG!!

I can remember System 32, OK? Then came System 36 and I think there was a System 38 in there too. But I can really remember when IBM announced the AS/400 as the godbox of all end-user computing. Can you believe that I got certified in LU 6.2?

Because I'm so long in the system tooth. I've had the pleasure of getting data from an AS/400 source and putting it into a client/server interface. You realize of course that people had to be convinced. There are fewer things more aggravating that having to put up with the nonsense of AS/400 gurus as they begin fulminating about the inferiority of every other API and what a waste of time it is to replicate data from the 400 onto another platform. I patiently explain to them that their bosses bosses boss wants historical information charted in color and they give me one of these grimaces usually found on the face of Michael Moore when the security guards escort him from the premises. Yeah but I'm right, they smirk to themselves.

It is at times like these when I think of my girlfriend's sister's boyfriend who married her and moved up to Stockton, California. He was the god or RPG and drove a Mitsubishi Starion. Whoa. OK I exaggerate, it was a Sapporo. But I still couldn't bring myself, ghetto child that I was, to give up life in Los Angeles, even though he had a house and made 36 Thousand Dollars. That was 1980, and I had to hold on to my dreams. I knew RPG, in fact I knew RPG II. It was torture, I couldn't submit to its petty demands, and I wasn't ready to get married.

If you happen to find yourself staring into the wall eyes of an AS/400 guru, do as I do. Count backwards from 10 slowly and tell them how much respect you have for the innovative microcode of the AS/400 instruction set. Then turn around and run like the wind. If you actually have a mandate to work with this dweeb then keep a stiff upper lip and do what you need to do to get your data. Prostrate yourself.

Seriously, you should do whatever possible to get all necessary AS/400 data into SQL Server. You should get a nightly job to push transactions from the 400 onto a separate NT server and then use the ODS on NT.

Generally a customer will have green screens running against the 400. I know there's something of a conceit that the 400s are big hardware, but there really are no 400 databases out there that are too large for MSSQL. The 400 is really all about capturing transactions and they can't afford to run ad-hoc queries.

The best way to manage it is to add a native query job on the 400 side and dump that into a flat file on the 400 in a new outbucket directory. You can then FTP or map to the outbucket and initiate a transfer from the NT machine.

This has three primary advantages.

1. The 400 administrator will have control of the query and when it runs and where the output goes. He can optimize all that on his side. Nothing leaves his control.

2. The only thing that is initiated from the outside is a file transfer. That shouldn't interrupt anything on the 400 side.

3. Once you have the data on NT you can Perl it, DTS it into MSSQL and manipulate it any way you please. You get out of the 400's hair and you're essentially free.

The important thing is to make sure you get everything you need in that one big fat nightly query. Chances are you won't be able to get master data but ask anyway. Your better chance is to run master data from the data itself.

Now go have a chai spice. You win.

More on the Merger

I've had a chance to ponder as well as listen to other folks talk. Here are some of the best ideas of mine and theirs.

1. In the short term, nothing will happen.
We always forget this, and then reality sinks in. Nothing significant is going to happen in the next 4 to 6 months in the product lines. People will be getting over the shock in the marketplace and in the organizations. Associated companies will be getting their bearings and the very few people who knew about this blockbuster of a deal will surprise us again. From what I hear, this secret was kept very well. Although it has been rumored for years (and I wish I could find some of my old comments and speculations on those rumors) it blew everybody away.

2. Current Hyperion customers might upgrade, just to be on the safe side.
If you're sitting on an older version of Hyperion software, chances are you'll upgrade just to buy yourself time. This means busy work for the consulting sector. It's easy if you're already in System 9 of Hyperion, and easy if you're pure Essbase. But if you have a mix of Hyperion products moving all of them up will take a little doing. Still this is the only downside hedging I would expect current Hyperion customers to do.

3. Brio guys aren't breathing hard.
Somebody needs to really sit me down and explain what it is that Business Objects has that Brio doesn't. Whatever BO might have, or Oracle Discoverer might have the Brio folks aren't sweating it. The Brio customers were loyal in their transition to Hyperion and they weren't disappointed. Oracle rebranding doesn't seem to concern them much at all. That's interesting.

4. Product Integration takes time.
Peoplesoft people are still calling Peoplesoft Peoplesoft. So for the time being, it's entirely reasonable to expect that Hyperion will be a 'wholly owned subsidiary' as far as customers are concerned. It's going to take a while for Oracle to learn what Hyperion customers like about Hyperion and what they don't like. So in order to do product integration smartly, they'll most likely run out the current planned release schedule on the Hyperion suite and then start looking to merge. Again, I'm thinking Data Integration Manager (Informatica OEM) and Analytic Integration Server will be the first likely casualties to all of Oracle's data pump tech. Then again, current customers must be served. Perhaps by Solutions 2007 in Orlando in a couple months, we'll hear a few more specifics.

5. IT objections mitigated.
Already, dithering customers have committed to Hyperion based on the fact of the merger. Oracle shops are now open to Hyperion BI. This is bad news for Business Objects seeds in particular. This is one issue that has been the bane of my existence, being one of those rare individuals who thinks both relationally and multidimensionally I have suffered endless tirades about what MOLAP can't do, by people who have no hands-on. When I explain to people that Essbase is a modular database technology that has Hybrid, MOLAP and Aggregate Storage, that it is partitionable both logically and physically, they still find it hard to believe because nobody thinks of Hyperion as a database company. Now instead of showing them case studies with 500 concurrent databases or 50,000 concurrent queries I can just say two words to cease all hostilities. "It's Oracle".

People forget that Hyperion Essbase was Teradata's choice for marting in their Active Data Warehouse strategy. So I will reiterate my trash talk against MSTR. You guys are in for a surprise. The database guy in me is just giggling silly, and I haven't been so pleased since IBM OEM'd Essbase for DB2OLAP.

6. Verticals
Back when I was talking with people who oughta know, the idea for the killer Essbase apps was a certification program. That is to say Essbase had (and still has) the capability of being a brandable kind of analytic engine, as in Analytics powered by Essbase. I was personally involved with the deal that almost got Essbase embedded into Siebel, and I've since heard where the strengths and weaknesses of the chosen technology are. In any case, Hyperion has dreamed of moving towards verticals and now with Oracle's coverage of vertical markets, there are opportunities galore for Essbase to be exactly that. Of course it will require some thoughtful people to make it happen, but the Oracle introduction can be there. The technology is ready. Today.

Oracle + Hyperion = Gorilla

Oracle is buying Hyperion. SAP is buying Pilot. Cognos and Business Objects stand alone.

This is huge. I'm still overwhelmed by the implications.

I don't doubt anything this article says:

Hyperion and its popular Essbase product, has "almost no overlap" with existing Oracle products, Phillips said. But, there is overlap between the PeopleSoft's performance management products Oracle acquired in 2003 and Hyperion's products. The PeopleSoft products will become part of a new enterprise performance management division, Phillips said. Oracle has had a "small" performance management product that "wasn't that pervasive in the market," it considers Hyperion a complementary, best-of-breed vendor, he said.

Instead, Oracle has focused on building out the underlying infrastructure required for BI, he said, and the Hyperion acquisition will give it critical analytical applications to layer on top of its data management and BI technologies. Specifically, the acquisition gives Oracle an enterprise planning system, a financial consolidation product, a powerful OLAP engine and a "dedicated" field sales organization. It also believes the move will help it in its battle's with rival SAP. Many SAP customers use Hyperion, Phillips said, and Oracle is achieving "critical mass" within SAP accounts.

"Now Oracle's Hyperion software will be the lens through which SAP's most important customers view and analyze their underlying SAP ERP data," he said in an earlier statement.

From my perspective as a consultant it's all good. I'm confident that Hyperion's BPM apps are going to go through pretty much as is. It puts the Informatica OEM in a wobbly position, and it will probably delay integration of the more recent product acquisitions like Crystal Ball and Upstream. The integration of Hyperion MDM works technically but it will have to compete with Oracle's Customer Data Hub. I don't know how successful that product is. MDM hasn't sold well on the Hyperion side although everybody who gets it loves it.

There's basically no question that this is excellent news for us core Essbase folks and for those applications based on Essbase. The war between Essbase and Oracle's Express is ages old, and while performance wise, old hackers like myself and those on the Express side could get into lengthy debates, there was no question the Essbase was the more interoperable product. Now that it is plug-compatible with the new Yukon DTS, Oracle will have another way to kick MSFT around.

I say this is really bad news for Microstrategy. When the Oracle technical folks recognize what they'll be able to do with Oracle + Essbase, Microstrategy's whole high end BI story will crumble. In many ways this is a technical match made in heaven.

What's iffy is of course matters of marketing and product synergy. That is to say if I were a product marketing manager at Hyperion, I would get that resume out. Sullivan, in his letter to customers this morning declared in about as obvious as is possible to say that Hyperion sales reps have some security. So that whole organizational end of Hyperion is in jeopardy.

There have been a lot of us who have been worried about Hyperion development, ie without Gersten where would the engineering organization be. I think that Oracle would make a huge mistake to do anything but hang onto a lot of that engineering staff and keep them happy. Obviously System 9 integration will become a priority as product feature planning levels off. The good news is that there are not many major flaws in System 9.

Hyperion's tech support has taken a dive in recent years. The Oracle Developer Network will be a huge boost for that end of the business. If I were smart, I'd probably spin off a training business and start raking in dough for anybody out there who wants to become certified.

There's a lot more to think about, but on the whole there's basically one thing to know. There is now a new gorilla in the BI and BPM space.

The Clocksite

Here's an idea and a term to coin. Clocksite.

A clocksite is a website put together to establish or debunk any economic or social theory that makes predictions about some important occurrence or keeps track of some indicator of public interest.

The term originates from the now familiar practice of billboard advertisements which are calibrated at a certain rate to estimate some important metric of public interest, like the number of smokers who die from lung cancer, the estimated number of dead in war, the number of dollars in the federal deficit or most famously the 'number of minutes to nuclear holocaust'.

A clocksite however, would employ multiple metrics and varying types of analysis and would allow for a more sophisticated and nuanced analysis of the issue at hand.  It might be open for collaboration or some officially sponsored by a partisan group.

As more open source tools become available to present analytic data on websites, we should expect more clocksites to follow the open publishing models that are now being experimented with by the folks at Swivel and ManyEyes. I have been interested in this matter for many years as I have been involved with building such sites with proprietary tools for business.

OODA Diagram

Saas vs Appliances

Don Linstedt has been watching something that I think I should be watching, and that is the emergence of appliances into the enterprise applications space.

Every so often a future suggestion that I've discussed (made by more than just me) actually happens. Now I'm not the kind of guy to normally say "I told you so." However on this occasion, I feel it's important to announce that the market is changing, dramatically - and that software vendors NEED TO TAKE NOTE!!! EAI is now available in plug-and-play appliance format. In this entry we'll discuss what this means, and how it will affect ETL/ELT, EII, and EIEIO (old Mac Donald had an appliance... E-I-E-I-O).

Cast Iron Systems announced in 2006 their EAI appliance. They've made some incredible enhancements, and were recently reviewed again: http://www.infoworld.com/article/07/01/26/05TCcastiron_1.html?APPLICATION+DEVELOPMENT

While the review was neither favorable nor unfavorable, it highlights an important point I've been making about appliances: they will become a combination of hardware and software vendors, specifically ETL/ELT and hardware, EII and hardware, EAI and hardware, and eventually BI and hardware will hook up. Why not? Hardware increases sales, increases price points - at the same time, it can increase ease of use.

He's right. And moreover if these appliances work as advertised, then they will be the counterbalancing force to the ever increasing complexity of tiered services. That means IT can take control where it was losing ground to complexity and simplify at the same time.

My experience is telling me that IT is having increasingly difficult times managing all of the tiers required for the web-centric enterprise. When a repository is not on the same subnet as an application server which is located somewhere else than authentication, stuff gets crazy and nobody wants to debug that. We get a cascade of bouncing servers because the people who installed whatever ear file aren't around any longer or nobody quite knows why some rows are locked in a database whose name doesn't really explain which application is using it. Let's not even get into the prodigious number of bot users which administer tasks for each and every one of the applications dancing in unison. It has been a nightmare, and because of that nightmare I had predicted that more and more companies will find SaaS an increasingly irresistable option.

BUT..

This appliance technology promises plug and play enterprise applications and that's even more irresistable, especially as vendors collaborate. I'm excited.

MDM + CDI Convergence

I haven't been writing so much about technology and my business. That's about to change, because there are actually some very interesting things that are beginning to happen that I may be taking part in. So let's talk shall we?

The first bit of news is that Hyperion has invested into Purisma. Hyperion is the mothership of BI and BPM and it's where I've cast my lot for over a decade starting with Arbor Software in the fall of 1996. I am, by the way, one of those rare creatures I call a 'double DBA'. I understand both relational and multidimensional database structures. My guess is that I could extend that a bit both ways if I were so inclined. Like most old heads, I have some experience in the hierarchical database technologies of yore. And like most new jacks, I have some experience in object models and object databases. Unlike most people I have a whole hodload of experience cobbling enterprise systems together in the real world under the creakiest and most robust IT architectures. I can tell you stories about hybridized VM dispatchers to 64bit parellel databases that would blow your mind and stories about Excel clouds on OS/2 & token-ring that would have you reaching for the Pepto Bismol. Suffice it to say that I've earned my bones as a data architect; it's a hell of a job and I love it. 

Now here's the deal with Purisma. It's got Bob Hagenau. Bob was the guy behind Acta, and Acta was the coolest ETL on the planet a few years back. It got absorbed by Business Objects because Hyperion was slow to the draw and shy of acquisitions. If it's doing nice things today, I have no idea. But it was the key to getting data out of the 'data jail' of ERP systems, especially the high security prison of SAP. As such they were Sapphire certified and even understood Z tables and weirdness in the SAP data model that most ABAP coders mumbled about and SAP tried to hush away. So I have some confidence that Hagenau is not a time-waster and that when it comes to complex data, he'll make some of the jagged edges smooth.

Purisma's business is simple. Master customer data. Hyperion's business is simple, monetize and benchmarkify everything for management. Those are the simple explanations behind the two of the three three letter acronyms that are most significant in this piece. CDI is Customer Data Integration. That's Purisma. BPM is Business Performance Management, that's Hyperion. The killer app between the two will be customer profitability across all channels.

Now there have always been customer profitability models and they're not too hard to build. What's hard is that what your department calls a customer and what marketing calls a customer and what finance calls a customer are three different things. And all of you have incompatible systems that don't let your company decided once and for all what a customer is. In comes the third three letter acronym, MDM. MDM is Master Data Management.

Hyperion purchased a leading MDM vendor a couple years ago. They have a hot product which is now called, drum roll please, Hyperion MDM. What it does is centralize name calling in your organization. Huh? What? Name calling? OK think of the product you are familiar with in your company? What is it called? OK now what's the serial number? What's the product code? What do the engineers who built it call it? Who in marketing changed the name? When? How did they spread the word? What do people in accounting call it? What's the code for it in the accounting system? What about the inventory system? What about when you OEM it to another company? Complex? Well it's your product, you ought to know all of these things. You don't, because you haven't managed your master data. You don't have a chief name caller in charge who has octopus arms that coordinate the names of everything in every system. MDM gives you that ability.

In this regard, CDI is a very particular MDM application. It allows you to understand all the names of all your customers no matter which channel you touch them through. Something in the back of your head says, didn't Siebel promise this years ago with their CRM systems? Sure they did, that was marketing, this is technology. My nickel says Purisma does the job way better than Siebel does, because Siebel is not selling customer data management, they're selling CRM. If you get CRM but not a single view of the customer from Siebel, you don't send it back. If you don't get a single view of the customer from a CDI vendor, you send it back. Different incentives, different results.

So Hyperion, who gets the BPM theory and builds some pretty damned good products, is being smart in seeing the future of CDI and investing in Purisma. But are they hedging against their own MDM product? No. MDM is expensive and takes some doing to implement, it's not easy to sell either. But so long as Hyperion can cover the base of CDI and MDM through whatever agreement they're making with Hagenau and company, they are not going to miss the opportunity to position their company as knowledgeable about the space, and so their money is where their mouth is.

All that remains is for field commanders like myself to staff up some troops and actually battle test these technology weapons against the enemy of corporate information entropy. I'll let you know.

What Happens In Google Stays In Google

Here's what I think about the Google purchase of YouTube. I think that as soon as Google hosts all the YouTube content, there will be no way of knowing the monetization potential of any piece of contested material without Google's participation. In otherwords, everybody who guesses how many hits, and therefore how valuable some piece of content is in terms of eyeballs will have to ask Google and Google would have to be compelled to divulge that information. Then what happens? Google can make a quick calculation of what it's worth, in terms of ad revenue, to deal or no deal. If it's no deal, Google cuts the piece of content loose and its value drops to zero.

The trick is having the entire infrastructure of the Googleplex which would host the valuable content. That makes them the equivalent of the broadcast TV networks, and if entertainment lawyers are dancing naked because they think that YouTube is a litigation free-for all just waiting to happen, they will quickly find out how difficult it is to sue the network.

All of that seems to be a big fuss over content that's primarily only valuable because Google broadcasts it, some very small fraction of which has any commercial value in any other form. Only Google has the free billions of equity required to do any of this so, it's essentially a win situation for them. It seems to me that this is something only Google could do because they're the ones investing in the Googleplex.

Three Hours Flat

One of the reasons I'm resurrecting my blog is because of continuity. I owe continuity to my virtual partners, even if they're tired of hearing me repeat myself. A boring familiar voice is better than silence, and as I've mentioned, I'm occasionally worth reading.

What makes me waste a little time on this blustery Wednesday is the fact that I have learned, within three hours of it becoming 'public', of Google Real Estate. I don't know how many degrees of separation I am from the leaking source - it doesn't really matter; I just find that to be a fairly amazingly short period of time, and I feel fortunate to be in the loop of know.

I've been thinking about explaining the basics of BI to noobs as well as senior managers whose companies have yet to exploit the economies. In answering an imaginary question of where the industry will be in 10 years (actually the iQuestion was 'Where do you see yourself in 10 years?'), I said that there will be four players. They will be Microsoft, Oracle, SAP and ... Google. I'm hoping to see myself with Google, the reason isn't obvious but I have mentioned it before:

Google does not charge the user. All of Google's software and hardware is at Google, behind locked doors, just like the IBM data center. To Google, you are a user, not an owner. You couldn't own the Google software because you would clog up Google's precious computers with your idiot code which couldn't possibly be as efficient and secure as Google's own code. They are masters of the algorithms and queue management. Google releases no product that doesn't scale to infinity. Google never gets a virus. Google is a free IBM datacenter. Google is the opposite of IBM despite all these similarities because its business model does not involve charging the user. Microsoft and IBM are the same because they charge the user. Microsoft makes you configure your machine and calls it freedom.

Furthermore, Google deals in human scale media. Google does not deliver compacted abstracted codes. Google delivers whole books, whole libraries of books. Google delivers satellite images of the whole planet. You are incapable of asking too much of Google, and their orientation is to deliver that all to you for free. Plus they take away all of the headaches that Gates delivered, the world of limitations of PCs. Clunky, insecure software written by anybody.

Google is one of the only computing products that hasn't bloated. Even Linux is running into controversy over kernel bloat. Pigs aren't flying, they're recompiling RPMs on their anti-Microsoft boxes. Same paradigm, same headaches. Free software approaches perfection, but there is no guarantee. Google guarantees, and that is why Google is rich and there are only two or three Linuxes left worth mentioning, all equally user poor.

But Google could charge the corporate user because Google has got what every enterprise desires: fundamentally unlimited scalability. They have the opportunity to license the way they do business and thereby revolutionize the IT business. Google is the ultimate ASP - they always have been, and they've been providing incredibly fast information for years now. Google can simply wait for every Tom, Dick and Ajax to write their little Web 2.0 app, and then determine whether or not it scales into a nice business model. They plug it into the Googleplex and voila.

It is not a stretch for me to see them doing this with 2 million spreadsheets, which is very likely what your corporation's finances are really running on despite MSFT's, Oracle's and SAP's protestations to the contrary.

So here's the question I posed to my BI noob interviewer. 'It takes about 15 minutes for you to find out whenever a car bomb goes off in Baghdad. How long does it take you to learn that your business has lost a customer?'. The answer to this question is generally sub-Google speed, which ultimately is unacceptable. The gap between real-time analytics is not so much technology as it is IT methodologies and captialization. This is something most companies would rather have, but not pay for up front. Some enterprising practitioners (like me) and some Google people could meet one day and come up with an ASP business model for enterprise BI. If and when that happens, MSFT, Oracle and SAP will find that they are a lot slower to the middle-tier webware market than they thought.

I am consistently surprised that people remain skeptical of fat clients. We are seeing 3GHz PCs with 100GB of disk and still people go Web. The hows and whys of that we can discuss later, but it is clearly a fait accompli. Whatever happened to port 80? It's now a thousand different ones for everyone's 'administrative client'. Client server has been swallowed up by very specific clients, ie browsers running interpreters. Whoda thunk? That being the case, applications development companies are essentially web development companies.

I have a hard time believing that people who built WebMethods or Vignette (nothing personal) are going to be developing the applications of the future. The narrow scopes of what IT organizations and (outsourced) consultants build these days unfortunately don't stand a chance against the big huge apps Google rolls out.

One final analogy. Once upon a time, Bloomberg relied on hardware. He had a near monopoly when it came to (bond trading I think). The smarts of that moved to PCs and the second revolution was exemplified in the day-trading bubble. (By the way, once upon a time the same was true of Metaphor DIS, specialty hardware to fat-client software to running on browsers in the BI space). Clearly the next evolution in tiered computing will be making browsers smart enough to do all fat clients do, cleaner. Now it's all about the scalability of the back-end. I have a hard time believing that the same decision makers who are gung-ho about punting to procurement departments when it comes to <strike>outsourcing</strike> staffing IT are going to think twice when offered the opportunity for 2 second response time without capital spending.

Mining the Mind of Steve Krause

Back in the days when I was complaining that 30 year olds were running corporations, the gang of us at Hyperion eCRM had to deal with the equation hype + money - common sense = success. At least that's the way it was in the days of irrational exuberance. As one who always likes to review history in the light of renewed appreciation or scorn, I find it fascinating to find out whatever happened to.. along the lines of my career path and pointed pontifications. It's for that reason that I started my little jag on Xerox History. In the meantime, as the F500 slouches towards real security, BPM and data mining, I have fun digging up data on my own industry.

Steve Krause is my latest find. He puts up a nice practical post on Last.fm and a competitor that I never heard of or paid attention to. It fits rather congruently with my 'Do As I Say' theory. Note that the ultimate judge of the appropriateness is aesthetic consistancy, an entirely human creation.

It also turns out that Krause was a competitor at Personify. It's amazing that they were able to burn through half a billion dollars. If those happy days ever come back, never give a sucker an even break.

Dynamic vs Transactional Engineers

Every once in a while I kick something over the Drezner's site. Especially when he writes on outsourcing which is one of my pet peeves. As a highly technical architecture level consultant that does hands-on work, I really don't blog about it as much as I probably ought to. I've thought that I could run multiple blogs and keep all those worlds from intermixing, but I think I may begin to blur that firewall too. So on the matter of the quality of engineering and technical talent from the emerging second world, I had this to say:

I think that this distinction between dynamic and transactional engineers is very useful and accurately describes what I see in the software industry as well.

Even when chinese and indian programmers are on staff in American companies, there are notable differences. You never quite know what you're getting until you sit folks in a room and start talking about the systems to be built.

Enterprise systems that are to make a difference in the productivity of the target customers are notoriously difficult to assemble, even when using simpler standard technologies that procurement departments are demanding be offshored. That is why companies like Accenture continue to make money in the American and international markets. The skills of management consultants that can do a tight handover to technologists are in high demand, but even so these are applications that tend not to be robust. Anything that takes more than six months to build will suffer from changes in the business environment, turnover in personnel and integration with other systems that themselves are being changed.

Again, the allure of cheap labor in this area is that SQL is SQL. Not necessarily so. It's actually getting more complicated to do these applications properly, primarily because of an attitude the 'best practices' can be built into every application. This means that a lot of abstraction of problems is done, and a number of experts who don't do hands on work are employed. All this is done at the expense of homegrown (meaning inside the client company) experience which is the great hidden expense of outsourced systems.

It comes down, in my view, to a decades old clash between management philosophies. Deming vs Hammer. The Deming method says to evolve the way people are working with technology and business processes. That's highly integrative and evolutionary. It means you have to do a lot of listening and translating. The Hammer schools says, throw out the old and make everyone start from scratch. That's re-engineering. Companies that get re-engineered outsource and offshore better than companies that evolve. Companies that evolve are more productive because the culture of the company teaches everyone what the focus of the business is. They can be more nimble, but there's a steep learning curve.

I'm from the Deming School (an old Xeroid from the McKinsey makeover under David Kearns) and have been in the enterprise sofware business for two decades. The Deming way is harder, and it's often against the best interests of management and technology consulting companies, not to mention software vendors, to evolve a good company to great, but the real loss is that so many American corporations are literally outsourcing their own quality improvement. They don't want to grow their own MBAs, they want somebody else's. This dependency is what both depletes American talent, and keeps consultants like me in new BMWs.

Meanwhile over here at Nissan, things are beginning to get very interesting as deadlines start to loom. While Toyota is poised this year to make 100k more cars than General Motors, I'm finding that the systems that are being built in the wake of Sarb-Ox may have a long-term payoff for American businesses. Because over here in this Japanese company, typical of a large number of companies I've seen, the spaghetti and spontanaity of financial planning and accounting boggles the professional mind. In my close circle, we have a term called 'The Official Ass'. That's where a lot of numbers are pulled from. The number of companies that do real demand planning are few and far between. Demand planning is very difficult to do with most kinds of accounting systems that companies have, and so companies pull numbers out of a collective hole in the ground. That is particularly typical of non-financial companies - that is companies whose business is not primarily the management of money. When your financial staff is considered overhead... well.

It's one thing to understand how to build this database and that database. Sure that stuff can be outsourced. Should it be? It really depends on whether you're building system to be building systems or building them to improve the way a company is run. The latter can only be done by dynamic engineers. I like to think of myself as one of those types, but its not very often that we are called to do all that.

Dynamic vs Transactional Engineers

Every once in a while I kick something over the Drezner's site. Especially when he writes on outsourcing which is one of my pet peeves. As a highly technical architecture level consultant that does hands-on work, I really don't blog about it as much as I probably ought to. I've thought that I could run multiple blogs and keep all those worlds from intermixing, but I think I may begin to blur that firewall too. So on the matter of the quality of engineering and technical talent from the emerging second world, I had this to say:

I think that this distinction between dynamic and transactional engineers is very useful and accurately describes what I see in the software industry as well.

Even when chinese and indian programmers are on staff in American companies, there are notable differences. You never quite know what you're getting until you sit folks in a room and start talking about the systems to be built.

Enterprise systems that are to make a difference in the productivity of the target customers are notoriously difficult to assemble, even when using simpler standard technologies that procurement departments are demanding be offshored. That is why companies like Accenture continue to make money in the American and international markets. The skills of management consultants that can do a tight handover to technologists are in high demand, but even so these are applications that tend not to be robust. Anything that takes more than six months to build will suffer from changes in the business environment, turnover in personnel and integration with other systems that themselves are being changed.

Again, the allure of cheap labor in this area is that SQL is SQL. Not necessarily so. It's actually getting more complicated to do these applications properly, primarily because of an attitude the 'best practices' can be built into every application. This means that a lot of abstraction of problems is done, and a number of experts who don't do hands on work are employed. All this is done at the expense of homegrown (meaning inside the client company) experience which is the great hidden expense of outsourced systems.

It comes down, in my view, to a decades old clash between management philosophies. Deming vs Hammer. The Deming method says to evolve the way people are working with technology and business processes. That's highly integrative and evolutionary. It means you have to do a lot of listening and translating. The Hammer schools says, throw out the old and make everyone start from scratch. That's re-engineering. Companies that get re-engineered outsource and offshore better than companies that evolve. Companies that evolve are more productive because the culture of the company teaches everyone what the focus of the business is. They can be more nimble, but there's a steep learning curve.

I'm from the Deming School (an old Xeroid from the McKinsey makeover under David Kearns) and have been in the enterprise sofware business for two decades. The Deming way is harder, and it's often against the best interests of management and technology consulting companies, not to mention software vendors, to evolve a good company to great, but the real loss is that so many American corporations are literally outsourcing their own quality improvement. They don't want to grow their own MBAs, they want somebody else's. This dependency is what both depletes American talent, and keeps consultants like me in new BMWs.

Meanwhile over here at Nissan, things are beginning to get very interesting as deadlines start to loom. While Toyota is poised this year to make 100k more cars than General Motors, I'm finding that the systems that are being built in the wake of Sarb-Ox may have a long-term payoff for American businesses. Because over here in this Japanese company, typical of a large number of companies I've seen, the spaghetti and spontanaity of financial planning and accounting boggles the professional mind. In my close circle, we have a term called 'The Official Ass'. That's where a lot of numbers are pulled from. The number of companies that do real demand planning are few and far between. Demand planning is very difficult to do with most kinds of accounting systems that companies have, and so companies pull numbers out of a collective hole in the ground. That is particularly typical of non-financial companies - that is companies whose business is not primarily the management of money. When your financial staff is considered overhead... well.

It's one thing to understand how to build this database and that database. Sure that stuff can be outsourced. Should it be? It really depends on whether you're building system to be building systems or building them to improve the way a company is run. The latter can only be done by dynamic engineers. I like to think of myself as one of those types, but its not very often that we are called to do all that.

The Grinchbag Theory

(soon to be translated into Suessese)
The story of the grinchbag is simple. It is an IT parable based the story of the Grinch Who Stole Christmas.

Whoville is where all the users live, and throughout the year they gradually pile up their little Who requirements and expect at the end of the year that the IT Santa Claus will fulfill all their desires.

Now the Grinch may be anybody in this tale, but the point of the grinchbag theory is for whatever reasons, all of the goodies and wishes are accumulated into one giant bag and taken away from Whoville. And whenever you put everything into one grinchbag, it's awfully hard to move all at once.

So if you're holding the grinchbag at the top of the hill and you change your mind for any reason, you're going to have to grow your heart three times and attain superhuman strength to save Christmas. The fact of the matter is that you won't. You can only hope that the Whos will sing Dahoo Dory without their goodies, because the only thing superhuman strength does is save you from being crushed by the grinchbag.

The moral of the story is to spread goodies throughout the year and not try to gather up everything in one grinchbag. The Whos are going to sing with or without you. You really aren't in charge of their spirits, which you can't possibly understand because you don't live in Whoville.

The Grinchbag Theory

(soon to be translated into Suessese)
The story of the grinchbag is simple. It is an IT parable based the story of the Grinch Who Stole Christmas.

Whoville is where all the users live, and throughout the year they gradually pile up their little Who requirements and expect at the end of the year that the IT Santa Claus will fulfill all their desires.

Now the Grinch may be anybody in this tale, but the point of the grinchbag theory is for whatever reasons, all of the goodies and wishes are accumulated into one giant bag and taken away from Whoville. And whenever you put everything into one grinchbag, it's awfully hard to move all at once.

So if you're holding the grinchbag at the top of the hill and you change your mind for any reason, you're going to have to grow your heart three times and attain superhuman strength to save Christmas. The fact of the matter is that you won't. You can only hope that the Whos will sing Dahoo Dory without their goodies, because the only thing superhuman strength does is save you from being crushed by the grinchbag.

The moral of the story is to spread goodies throughout the year and not try to gather up everything in one grinchbag. The Whos are going to sing with or without you. You really aren't in charge of their spirits, which you can't possibly understand because you don't live in Whoville.

OLAP Blog Integration

I'm also about to crank up the volume on getting some BI & OLAP bloggers coordinated. I think it's sad to see a bunch of stragglers out there with no trackbacks or comments. I'm going to play cat herder and see what I can come up with.

Part of the problem is that there isn't a good back and forth with industry experts. Rumor has it, however, that John Kopke is a blast in front of customers. I know he understands the technology and I used to work with him at Pilot Software back in the days. And yet there are a lot of people trembling with fear at guys like Nigel Pendse. Now there's a showdown I'd like the blogosphere to witness.

I've been neglecting the dialog because I've been downmarket for a while and basically nobody's talking because smaller companies don't have time to listen. But now that I've come back to the state of the art and I see what Hyperion has been doing in terms of rolling out new products and upping the ante and expanding the scope of BI'able applications, there's plenty to say. The question is, who's saying it? I certainly would have been, but that's not what Cubegeek turned out to be. Let's give it a shot in this brave new world.

UPDATE
Introducing The Cubegeek Blog

OLAP Blog Integration

I'm also about to crank up the volume on getting some BI & OLAP bloggers coordinated. I think it's sad to see a bunch of stragglers out there with no trackbacks or comments. I'm going to play cat herder and see what I can come up with.

Part of the problem is that there isn't a good back and forth with industry experts. Rumor has it, however, that John Kopke is a blast in front of customers. I know he understands the technology and I used to work with him at Pilot Software back in the days. And yet there are a lot of people trembling with fear at guys like Nigel Pendse. Now there's a showdown I'd like the blogosphere to witness.

I've been neglecting the dialog because I've been downmarket for a while and basically nobody's talking because smaller companies don't have time to listen. But now that I've come back to the state of the art and I see what Hyperion has been doing in terms of rolling out new products and upping the ante and expanding the scope of BI'able applications, there's plenty to say. The question is, who's saying it? I certainly would have been, but that's not what Cubegeek turned out to be. Let's give it a shot in this brave new world.

UPDATE
Introducing The Cubegeek Blog

Master Data Management

One of the most exciting pieces of software to come down the pike in many years is one picked up in a recent acquisition by Hyperion Solutions. It's one of the reasons I have to be fairly jazzed about the kinds of systems I'll be able to build in the coming months. Formerly called Razza, it's Master Data Manager.

If you had asked me a month ago what was the best way to make money in the Enterprise Computing business, I would have told you Master Data Management. I wouldn't have used that precise term, I would have probably said something like this:

One of the biggest problems for me, in building systems with the tools I have is always the political problem of getting all the people talking the same language. A significant reason why DW initiatives fail is because the metadata is all over the place and everybody spends too much time chasing the data down rather than analyzing it. All I need are my tools (speaking of Essbase outlines) and then I get functional people and technical people speaking the same language, because everybody can see how the numbers and entities roll up. The reason Informatica is making all kinds of money in this space is because they promise to solve this problem.

Well here's what IDC says.

Master data management is a challenging, long-standing problem. But recent attention to business performance management and compliance represent a new opportunity to deal with the issue in a way that can improve both information accuracy and organizational agility.

With Hyperion's MDM, I believe the problem has been solved. As soon as I get a copy I'll get deep into the details, but basically this is a collaborative tool that will allow enterprises to manage all of their dimensions, whether they change slowly or quickly, back through history.

Imagining the worst spaghetti possible, a partial migration between ERP systems without the benefit of ETL, a MDM Server would get everyone on the same page. How many times have I had people squawk about the complexity of Peoplesoft Trees and complain that their reporting systems use one drill down and their interal reporting systems use another and that the Business Objects Universe was painstakingly coded with another? And how many times have I had to be the one to reverse engineer all that rot and put into my systems? Too many to count.

I'm going to have a field day with this tool. Believe that.

Master Data Management

One of the most exciting pieces of software to come down the pike in many years is one picked up in a recent acquisition by Hyperion Solutions. It's one of the reasons I have to be fairly jazzed about the kinds of systems I'll be able to build in the coming months. Formerly called Razza, it's Master Data Manager.

If you had asked me a month ago what was the best way to make money in the Enterprise Computing business, I would have told you Master Data Management. I wouldn't have used that precise term, I would have probably said something like this:

One of the biggest problems for me, in building systems with the tools I have is always the political problem of getting all the people talking the same language. A significant reason why DW initiatives fail is because the metadata is all over the place and everybody spends too much time chasing the data down rather than analyzing it. All I need are my tools (speaking of Essbase outlines) and then I get functional people and technical people speaking the same language, because everybody can see how the numbers and entities roll up. The reason Informatica is making all kinds of money in this space is because they promise to solve this problem.

Well here's what IDC says.

Master data management is a challenging, long-standing problem. But recent attention to business performance management and compliance represent a new opportunity to deal with the issue in a way that can improve both information accuracy and organizational agility.

With Hyperion's MDM, I believe the problem has been solved. As soon as I get a copy I'll get deep into the details, but basically this is a collaborative tool that will allow enterprises to manage all of their dimensions, whether they change slowly or quickly, back through history.

Imagining the worst spaghetti possible, a partial migration between ERP systems without the benefit of ETL, a MDM Server would get everyone on the same page. How many times have I had people squawk about the complexity of Peoplesoft Trees and complain that their reporting systems use one drill down and their interal reporting systems use another and that the Business Objects Universe was painstakingly coded with another? And how many times have I had to be the one to reverse engineer all that rot and put into my systems? Too many to count.

I'm going to have a field day with this tool. Believe that.

Tableau Software: Visualization Comes to BI

As part and parcel of my boredom with politics and lack of inspiring things like juicy battles overseas with which to occupy my mind, I will indulge a bit of the geekside. Something tells me I ought to do so elsewhere, in fact I probably will although I really am divided about all that with regard to implementation etc. I like to keep my politics and my profession in separate parts of the plate.

Anyway the reason for the excitement at the current moment is my discovery of Tableau Software. This is the sweetest front-end I've seen since I first laid eyes on Wired for OLAP. It's a pure visualization tool that has about as much wow factor as anything in BI. It could singlehandedly destroy the concept of prefab dashboards, however chances are that's not going to happen for some time.

What's best about it is that it is pretty simple to use, give dramatic presentations which are very rich and informative and it uses a the single best realtime fat client on the planet as its backbone, Microsoft Excel. Tableau hasn't wasted a bunch of time and energy on the n-tier fantasy, instead they have made use of both corporate inertia and Moore's Law. Guess what ladies and gentlemen, PCs can handle big fat clients, and this one delivers.

What I'm doing these days is ramping up my own data warehouse on the home network, which has about 9 boxes right now in various states of function. I've managed to keep various scraps of master data from a zillion clients over the years and I have a fake data generator that I built from scratch. So there's about to be a world of research I'm going to be able to accomplish here at Lab 107, (a wholly owned subsidiary of Metro Decisions).

Hell, I might even snarf some interesting data from blogs and other public data sources just to show off. Stay tuned, the geekfest is about to begin.

Tableau Software: Visualization Comes to BI

As part and parcel of my boredom with politics and lack of inspiring things like juicy battles overseas with which to occupy my mind, I will indulge a bit of the geekside. Something tells me I ought to do so elsewhere, in fact I probably will although I really am divided about all that with regard to implementation etc. I like to keep my politics and my profession in separate parts of the plate.

Anyway the reason for the excitement at the current moment is my discovery of Tableau Software. This is the sweetest front-end I've seen since I first laid eyes on Wired for OLAP. It's a pure visualization tool that has about as much wow factor as anything in BI. It could singlehandedly destroy the concept of prefab dashboards, however chances are that's not going to happen for some time.

What's best about it is that it is pretty simple to use, give dramatic presentations which are very rich and informative and it uses a the single best realtime fat client on the planet as its backbone, Microsoft Excel. Tableau hasn't wasted a bunch of time and energy on the n-tier fantasy, instead they have made use of both corporate inertia and Moore's Law. Guess what ladies and gentlemen, PCs can handle big fat clients, and this one delivers.

What I'm doing these days is ramping up my own data warehouse on the home network, which has about 9 boxes right now in various states of function. I've managed to keep various scraps of master data from a zillion clients over the years and I have a fake data generator that I built from scratch. So there's about to be a world of research I'm going to be able to accomplish here at Lab 107, (a wholly owned subsidiary of Metro Decisions).

Hell, I might even snarf some interesting data from blogs and other public data sources just to show off. Stay tuned, the geekfest is about to begin.

Pirate Code

I have heard some way out schemes in my life, but this one takes the cake. A pirate ship off the coast of Los Angeles with H1B failures coding enterprise systems? Yeah right.

The demand for highly qualified programming staff is high, but not that high. What people forget is that the trends toward the demand for highly competent software staffs is growing, and the demand for highly engineered software is diminishing. That is to say that Open Source will grow, and it doesn't matter where that comes from, but that implementing software *on site* is the toughest and most demanding job in the industry. It is the equivalent of changing a tire on a moving car. The moving car is the business of the enterprise, and they're not going to paddle off to some boat in order to get their specs.

I'm betting that the market is going to get tougher, not easier, and that the necessity of having personal contact is going to be greater, not less. For all the marvelous things we do with software, our interactions are only going to get more complex. It is with software as it is with law - even though it belongs to everyone and is sortof open source, when you need yours, it's all about intimate contact with the squad who is going to take you through it.

For software engineering, I can see that such flighty ideas might have weight. The question is, what kind of character wants to work on a rusted out ship? I think the best programmers are going to want to drive nice cars and get dates...

Pirate Code

I have heard some way out schemes in my life, but this one takes the cake. A pirate ship off the coast of Los Angeles with H1B failures coding enterprise systems? Yeah right.

The demand for highly qualified programming staff is high, but not that high. What people forget is that the trends toward the demand for highly competent software staffs is growing, and the demand for highly engineered software is diminishing. That is to say that Open Source will grow, and it doesn't matter where that comes from, but that implementing software *on site* is the toughest and most demanding job in the industry. It is the equivalent of changing a tire on a moving car. The moving car is the business of the enterprise, and they're not going to paddle off to some boat in order to get their specs.

I'm betting that the market is going to get tougher, not easier, and that the necessity of having personal contact is going to be greater, not less. For all the marvelous things we do with software, our interactions are only going to get more complex. It is with software as it is with law - even though it belongs to everyone and is sortof open source, when you need yours, it's all about intimate contact with the squad who is going to take you through it.

For software engineering, I can see that such flighty ideas might have weight. The question is, what kind of character wants to work on a rusted out ship? I think the best programmers are going to want to drive nice cars and get dates...

More Work Talk

After, or during a long day slaving over hot algorithms and queries, the last thing I want to blog about is work. In fact, I've probably talked more about crap I have absolutely no clue about than things that I'm probably world class in, which is OLAP and data warehousing.

These days I'm getting a little fed up with having this blog be of no use in spreading the word and working with work. Since my mind is made up about this upcoming election and I really don't think that people who are undecided should even bother, I'm not going to take up much more space on that tip until after the election. Plus P6 and I get to kiss and make up. So starting today I'm going to start blogging about OLAP and DW, which may be a very uncompelling subject, but...

But I have work for people who are interested in this very cool field and you all need to be aware that I have a little bit of moving and shaking ability. My little business, Metro Decisions is ready willing and able to start responding to the interest in the field, and quite frankly I am really tired of telling the recruiters and agents that call me every week: "Sorry, I'm booked and I don't know anybody who is available to work now."

If you're interested. Start talking. Right here. Right now. There's lucrative work sitting undone and several industries which are untouched by some truly cool technology that I'm trying to bring. My other site, CubeGeek.com can use a little assistance so I'm goosing it here. You can see all of the jobs referrals that I get. I will continue to post them there and perhaps refer to them here. This will make Cobb a bit more boring, but it can lead to economic happiness.

By the way, I'm also getting into more traditional web development stuff as well, so if you're ready to talk that, let's do it.

The Hatred Continues

Those who hate Microsoft use Linux.
Those who love UNIX use BSD.
-- Anonymous

I can tell that the headaches are about to start. I can feel a new brace of wind from the ever squabbling factions in the mind-numbingly tedious OS Wars. In today's installment, the open sourcers are crowing that some new winky blinky has been dropped from a future release of Longhorn, an MS OS version that's somewhere on the horizon to be released in the next year or two. Not only that, there's scoffing about XP's new Service Pack and bragging about Apples new iMac.

As the bored chicken said, cockadoodle-whatever. Is there anything really new in computing?

IBM Taking BI Seriously?

Within the past week IBM has acquired Alphablox and added a new Microstrategy bundle to their offerings. Is this another attempt to forestall a Yukon invasion, or are they just filling up company checkboxes?

My guy on the inside is as confused as anyone but his angle is that it is just another way for them to fill out there Websphere story. Alphablox is very labor intensive and if you know it, it's a good way to bill consulting dollars, if not build a very efficient system. I don't think the guys at Cognos or Business Objects are shaking in their boots. However, for those big huge 'we don't really know what we're doing but we sure do have a methodology" type projects, there will be a lot of Rational guys filling out forms. That could be a good thing, I suppose.

OWL & Metadata Management

There's an interesting nexus between some ideas Cameron and I dreamed up when thinking about XRepublic and this new notion of OWL and worm's eye metadata.

The point is that to harness the power of distributed thinking in assigning value to metadata on a subject matter, some categorization is necessary. You want to avoid re-inventing the wheel, but you don't want valuable one-offs to go unnoticed.

In the context of a bottoms-up deliberative space whose purpose is the generation of consensus, I came up with the idea of a Taxonomy Hike. If you care about your idea, then you can walk the distance it takes to attach it to an indexing scheme that people will look for. For example, if you want to weigh in on transracial adoptions, you might want to hook that up under the Parenting category. Cameron suggests Hipboning that subject in a dynamic way such that you can gain valuable insights by taking tangents that don't work in a hierarchichal taxonomy.

The following is from the XRepublic talk about the Wonk Path.

Taxonomy Hike
At some point, the Wonk needs to do some research. The research that is done online in the context of discussions and artifacts of the XRepublic necessitates a Taxonomy Hike. The Hike requires the Wonk to show a demonstrated effort to understand the context of similar discussions. Since there will be a Master Taxonomy which incorporates all of the discussions, the system should deliver the most 'considered' objects relating to the subject. This assures the Citizens that wonks have done their homework. After a wonk has taken the hike, she may be presented as a wonk and begin crafting a resolution, litmus test or other partisan artifacts.

Hipbone Room
The Wonk at any time in the process should be able to free-associate in the research space. In the Hipbone Room, some gaming is done to connect different concepts. An excellent introduction to Hipbone Games is given by Charles Cameron. As Charles Cameron has showed me via Hipbone, one of the most attractive things about dialog is that sometimes you happen upon a connection, a vibe as it were, which is completely unintentional but gives a great deal of insight to another class of problem or issue. XRepublic is more of a funnel for weighty arguments as determined by peers and is such directed towards a specific purpose. But it is also the aha of a tangential discovery that can lead to greater insights.

How indeed does some matter of Niger relate to a resolution of war? Nobody, but nobody was discussing that in March of this year when it came to the top reason pro or con. Nevertheless, something tangential in a different house, perhaps talking about Northwest Africa might be hipboned to the conversation somehow. The best we could come up with was the idea of a 'hipbone room' in which people free associate and build weblike structures in which seemingly disparate ideas could be linked to each other for another layer of contemplation.

In the context of BI and decision support, the need is the same. If you notice an interesting pattern of consumption by eyeballing your POS in a retail business - say you find a lot of men buying Diapers and Beer, you need the ability to flag that notion. There should be some way to hike the taxonomies of the global data warehouse from the data supply side which has a quick worm's eye facility onsite. Doing this before the data gets assimilated into the top-down paradigm of the DW might even obviate the need for data mining. At the very least it embeds human clues into the system.

I'm rather unclear about exactly what OWL is, but this page suggests it's a wider purpose facilitator for creating ontologies. I would expect that ontologies are broader than taxonomies, but make more sense from the worm's eye view. After some time, people might agree that a Diaper/Beer ontology needs recognition and acceptance into the formal Taxonomy.

How's that for a groupthink killer?

Worm's Eye Distributed Client Side Aggregations

I just had a flash of inspiration. One of the problems with the cube paradigm is as Neil Raden says here in his Netezza white paper. It doesn't take advantage of cheap hardware scalability and it abstracts the most educated guesses of strike teams that don't always strike true.

What if we reversed the paradigm of query aggregation with the understanding that end-users have significant hardware resources at their disposal? The idea here is that an analyst will poke at peek at detail and then come to some preliminary conclusions given their worms eye view of the business. Therefore you give them very specific access to real-time data in their area of expertise and then allow them to speculate about the effect on the business. In other words, rather than using standard top-down metadata to pre-build aggregations for marting, allow ad-hoc aggregations using bottom-up metadata decided on by some collaboration of worms-eye views of the business.

Arboretum

'Arboretum' is the code name for a project that I've been needing to do forever. This is a programmer's tool for understanding and maintaining hierarchies of all sorts, but in particular those associated with enterprise software.

Every time I go to a customer I have a huge headache in getting their charts of accounts straight. It's such a simple concept that you would think that somebody would have made this kind of tool before. But no.

So I'm going to specify this and sell it into my client base. The trick is that it will support plugins that talk to all of the major packages. If I can do this, I'm going to make a fortune. All I need is time and a programmer.

Watch And Learn

I am coming around to the realization that my field of software is an Oldsmobile. Part of this realization has to do with the grid computing infrastructure I mentioned last month. Part of it has to do with the insanity of VB.NET which I am trying to digest. (more on that later). A good fraction of it relies on my absolute ignorance and fascination with the tools, techniques and outputs of the CG trade (Do check out Rockfish). But a lot of it has to do with watching my own kids and monitoring their expectations.

As Stowe Boyd points out, kids don't care about snail mail. And one of these days, the tools of this trade are going to be a lot more sophisticated.

Yesterday, as I returned Splinter Cell Pandora Tomorrow & Red Dead Revolver to the local Blockbuster, I tried to conceptualize a new set of visualization tools for Enterprise computing. When it comes to graphical interfaces, we are so very retarded in terms of the way we conceptualize the functioning of our businesses. Even the simplest videogame is worlds ahead of what most businesses use. The number of execs I have talked to who would pay an arm and a leg for stoplight charts and dashboard dials are legion. But no gamer over the age of 7 would settle for the lack of realtime, or subtlety in corporate accounting systems. As I staff up at Metro Decisions, I'm only going to take gamers. I decided that a month ago.

A pal in NZ is starting up a company which appears to have broken through the OLAP barrier. It was bound to happen. Now that reality is at our doorstep, the next phase really has to kick in. And we will not sell it to the current crop of businessmen, but to the next generation which is happening now.

I have watched the industry for a long time, and they squander resources. They make excuses the same way they make work. Young minds and egos have no time for such ossification.

I argued with F9 yesterday. She says as a general principle the movie is better than the book. The subject was L'Engel's 'A Wrinkle in Time' and I wouldn't concede the point. But perhaps I should. What's in books that can't be digitized? It's always that the book covers things the movie does not. But we do have ways to visually and auditorially master the third person omnicient, and the craft will advance from here. What's expensive about shooting film in the current format can be overcome by videogame narrative. Kids do spend 100 hours in single games. I know.

There's a great potential awaiting the right team of programmers. You may not have heard it here first, but this blog is to remind me of certain things as well.

Deep Variances

Deep Variances
M. Bowen
August 2001
---

Abstract
Deep Variances are high level indicators of problems at a deeper level of hierarchy. It can reveal fudge factors and other low lying problems which might not otherwise be noticed at a high level. The effect is that it quantifies problems in a manner suggestive of the *breadth* of a particular problem in contrast to the *severity* of that same problem. Severity errors propagate normally up a dimensional hierarchy when they are sufficient, but a breadth indicator such a Deep Variance can give an early warning even when certain high level balances seen ok.

Example
For example, imagine a high level of variance volatility at a low level of a dimensional hierarchy. There are high negative and high positive variances. These 'balance out' in aggregation. So an individual considering overall progress might not think to look into the detail.

On Data Mining

Over at Crooked Timber, social scientists and economists take potshots at data mining. Over here in the Business Intelligence field, there's a lot more potential for it.

As an added value part of a product suite that I am developing, I will be offering some data mining. Coming from the realm of enterprise computing, Data Mining has a mixed reputation, but that stems primarily from the difficulty it is to implement and maintain as a system with regard to the technical expertise required to properly interpret data and that required to weed through BS products.

Of the many DW projects I've been on, there have only been a very small few who expressed interest in data mining and only one or two who have invested. Part of the reason is because unlike social scientists, those folks that I deal with have more real information than they need to make intelligent decisions, furthermore their business models are not immediately amenable to new discovery.

In the first case, as is well known in retail, when you have millions of transactions at the checkout counter at your disposal, you already have more than enough information to handle most inventory and product profitability problems. Since the retailer's problem is pricing according to supply and demand, mining things like purchase affinity is only icing on the cake. For the most part, all the merchandise in the store is going to remain in the same aisles and there are only endcaps to change. So figuring out the 28 products to feature there (especially considering the market research already done by suppliers of endcap goods like potato chips and soda) doesn't require extraordinary precison.

Despite the apocryphal tales of diapers and beer sold together by dads making a run, there aren't a great number of data mining success stories in retail. It doesn't make that much of a difference to the bottom line. Speaking of hearsay, a certain large retailer has confided that they have many many terabytes of data and it's difficult enough for them just to store it much less make mining passes over it for interpretation.

In the second case, there is always a proverbial prophet crying in the wilderness about some problem with a company's business model. The example I love to give had to do with what actually happened in one of the biggest and best systems I put together back in the early 90s. At Philip Morris USA, the proud owner of the world's second most powerful brand behind CocaCola, there was some slight fear about the market share dominance of Marlboro. As I designed and built their tracking system which gave monthly market share numbers (when that was the most frequent numbers were published) with the aid of an economist and *the* statistics text, we programmed some modified confidence intervals. These told us that it was reasonable to assume that the newly arrived bargain brands would actually eat into Marlboro's lunch.

At the time, such a thing as discounting Marlboro was practically unthinkable. PM had declared as much publicly and it was well known that if PM ever reduced the price of its premium cigarettes, it would spell the beginning of the end for the entire industry's legendary profitability. Considering all that, it really didn't matter what our fancy computer projections said the impact of 'Basic' and other generic tobacco products.

Some time later, however (we like to think based upon the information we were able to show) PM actually did discount Marlboro. The stock dropped several percentage points and the industry swooned. Then people got over themselves and adjusted to the new normality. Nevertheless, these changes took place in spite of the psychographic data and the company's sense of the that data which said brand loyalty would survive price competition.

I primarily think of data mining in the context of multidimensional analysis. 'Bucket Shaping' is how I will use it in my next application. Predicting which factors people use as customers is a dicey business, and it's reasonable to pay a marginal amount to gain a marginal edge. Honing that edge and finding the real cost benefit is no simple matter and certainly not used to the same ends of independently verifyable theoretical ends as with social scientists, but marketing is non-trivial work, and marketing managers do buy it.

MS Reporting Services - First Look

They didn't make it pretty. They made it work. From what I could get from the demo this moring of Microsoft Reporting Services, Microsoft has a very big hit on its hands. Sell your shares of Crystal right now.

There are a lot of interesting things to know about MSRS. The most important thing to know is that they did a very good job of putting some hot functionality into it so that it's not a joke product. Coming out of the gate, MSRS is capable of handling a good 75% of enterprise reporting requirements. They didn't show much on the security part of this so I might hedge on that a bit, other than that it appears to be a very competent product.

Licensing is 'free'. Basically it rides on top of your SQL Server license, and I'm not sure how they enforce that in code, but there is basically no barriers to entry with regard to getting started. This means a lot of companies are going to start hedging their bets on purchasing products from the competition until their SQL jockeys get their hands on it. And guess what, SQL jockeys are just the intended audience.

Although this has a translation doohickey from Access Reports, there is gobs of lovely SQL behind this piece of work. That means SQL jockeys and hacks will intuitively understand it and crank out many many briefing books and satisfy a lot of needs in short order. That language called SQR just became a useless skill.

It's difficult for me to understand the MS upgrade path with regard to their licensing, so I'm not in a position to determine if it has a reasonable chance of upstaging current implementations of BO, Cognos, Crystal, and Brio. After all, most enterprise reporting projects don't originate from SQL Server. So somebody's going to have to pay to transfer Oracle seat money to grow up the piddly SQL Servers all over the place MS is betting will come into use. But since the learning curve seems deceptively shallow for MSRS, a lot of apps developers may very well jump ship.

The wizards look fairly nice. After all, page layout is generally a no-brainer it's made just for wizards. There are very handy table, list and text objects to drag and drop around the design tool which allows you to preview your reports in realtime. Data is provided through XML / SOAP from .NET so as a data source you navigate to an http url on a local or remote server. It paints everything rather quickly and it isn't very difficult to see how you flip back and forth between the design screen and a QBE thingy very much like the one in Access to select your data. So I gather that this will allow a reasonable individual to develop reports rather quickly and efficiently. (You can use stored procedures too).

The Reporting Stack has 4 services that sit on top of SQL Server only Repository. Rendering, Security, Data Processing, & Delivery. By sitting on top of SQL Server, it's going to have a very strong management layer built in. From what I can tell, you can manage Dev, QA and Production servers from one console and do migrations back and forth automatically.

More Later...