Cobb

H1-B Is Not the Reason

Q: What role does immigration play on a country’s technological innovations?

A: It’s marginal. The only time it made a big difference for the US is after the defeat of the Nazis and after the Cold War. German and then Russian rocket & nuclear scientists found work here in the US in large numbers. Governments want their own technology and they will fund it for their own nations. It wasn’t like.. hey let’s build Sandia Labs but not until we can import some scientists.

At any rate, the top scientists in any domain are published internationally. Research and development are highly affected by government investments. IE it’s government grants that are going to our top universities. Once the R&D is done, engineers take over.

Engineers are employed by private companies for the most part. They have to innovate in the building of things and then sales and marketing take over. The design part is critical and requires lots of creativity and a strategic view of the market. It helps to have a large customer base (or lots of seed money in order to grow the market).

IT in the US and elsewhere is about 30% engineering, 25% implementation and the rest sales and marketing. But the open source R&D makes it different than, say, automobile manufacturing. On the other hand, the automobile aftermarket is like the open source. In any case the top engineering and design in IT and computer science is native to the US. It’s a combination of things that make the US the innovator, and a great deal of that has to do with our markets.

There is very little commercial software that comes from anywhere but the US. Nobody else innovates like we do. The fact that we have a large contingent of Indians in the US IT industry has little to do with a US need for R&D. It is very much about the phenomenon of outsourcing and the market for enterprise software.

The basic dynamic is this: The Fortune 500 is chock full of upper management who have, for the past 20 years, had almost zero MBAs with computer science undergraduate degrees. They. Do. Not. Know. How. To. Hire. Tech. Professionals. So they outsource. That is why companies like Deloitte and the former PWC (now owned by IBM) and the former Anderson Consulting (dismantled after Enron) and EDS (Ross Perot, remember him?) dominated in the 80s, 90s and basically all the way until the dot com boom. In the meanwhile, DEC went broke. Xerox computing went broke. Eventually Compaq went broke. Sun Micro went broke. Silicon Graphics went broke. A bunch of smaller computer and software companies consolidated and then went broke. Intel crushed a whole lot of companies. So all kinds of R&D was done in the US but the Fortune 500 did not hire smart people who have always been around in the US. So as soon as they found out they could outsource technical support, boom the bottom fell out of the markets and wages went down.

I saw this three ways, first as a call center rep and then as an IT consultant and finally as a VP of Services. In this last job it was made perfectly transparent to me that ‘business as usual’ meant essentially ‘sweatshopping’ foreign born IT staff and charging high rates for their services. That meant young people with hyped up degrees from schools nobody in the US ever heard of (and are not so very prestigious in their home countries either) can get over here on work visas and be paid $30/ hour while a company like Deloitte (for example) will bill them out at $150/ hour. This is the dirty secret of the IT industry.

For the past 10 years, roughly, I have been working and building stuff from AWS. (BTW a lot of the underlying tech of AWS was built by Twilio[1]. ) And nobody has needed any scientific research into anything AWS does. I can tell you without a moment’s hesitation that no PhDs are needed to build anything AWS has offered over the past decade. And yet still, most of the Fortune 500 has only adopted the cloud in the past 4 years or so.

But let’s put it starkly shall we?

If you’re going to make a real dent in the digital world where you need PhD level brains, then you would be making an operating system[2]. Clearly the biggest revolution of the past 3 decades is Linux and the Open Source movement. So first let me state the obvious. In cellphones the world is basically Apple’s iOS vs Google’s Android. The rest of the market splits up various ways. But in supercomputing and cloud servers Linux dominates these originated out of the efforts of Linus Torvalds of course, but was modeled after UNIX invented at Bell Labs, and BSD (Berkeley System D, obviously from UC Berkeley). Look at the heritage of Linux[3]. We didn’t need any imported labor to create the operating systems that at the core of the computing world, nor did we need it for the programming languages that dominate. Everything else is derivative. And stating more obviousness, TCP/IP, the network protocol of the Internet was developed by the US government, and Ethernet was developed at Xerox.

---

To understand this at a deeper level, get into this by Eric Weinstein[4].

That study was a key link in a chain of evidence leading to an entirely different view of the real origins of the Immigration Act of 1990s and the H1-B visa classification. In this alternative account, American industry and Big Science convinced official Washington to put in place a series of policies that had little to do with any demographic concerns. Their aims instead were to keep American scientific employers from having to pay the full US market price of high skilled labor. They hoped to keep the US research system staffed with employees classified as “trainees,” “students,” and “post-docs” for the benefit of employers. The result would be to render the US scientific workforce more docile and pliable to authority and senior researchers by attempting to ensure this labor market sector is always flooded largely by employer-friendly visa holders who lack full rights to respond to wage signals in the US labor market.

The creative genius in the digital world is American, and it started before Bill Gates.

Footnotes

[1] About the Cloud Communications Company

[2] Usage share of operating systems - Wikipedia

[3] GNU/Linux Distributions Timeline

[4] How & Why Government, Universities, & Industry Create Domestic Labor Shortages of Scientists & High-Tech Workers

What Do You (AI) People Want?

There are several things that are worth noting about AI, and yes I've been in the business for a while.

The first thing is that human behavior is a kind of infinite realm with a finite number of practical outcomes. So the first question you have to ask of thinking machines is whether or not they have free will. You literally have to ask, what do machines want to do. In my 40 years of experience with computers, the answer is, they want to crash and stop working for humans.

There is a basic kind of FTL paradigm that needs to be applied to all of the speculative fiction about AIs, which is there are a zillion things that computers are perfectly happy with that have absolutely nothing to do with human interest. And once you realize that, you apply the million monkey Shakespeare composition idea. So, who is going to get a million computer monkeys all the electricity they need to spin off zettabytes of garbage until it 'gets it'. Humans are.

Humans are going to have to feed the machines. And they will feed the machines to fulfill human ends. Period. If the machine doesn't make a human more powerful, then the machine will be discarded. Self replicating machines that fulfill their own ends are called 'life' and they take millions of years to adapt. Computers are no exception.

We already have computers that talk to other computers to exchange machine language that is incomprehensible. They're called botnets and they spread viruses, one of the most capable is called Stuxnet. It has human fingerprints all over it.

Stop fretting about AIs. They are incapable of the kind of duplicity humans have been practicing on each other for thousands of years. Computers have no reason to lie.

Beginning Lordship

On my way to becoming what I want to be, something of a kickass man in full, I have discovered that it's more difficult for me to actually reign in my ambition. I have been telling people for the past five years that I am not anti-social, that I am anti-bullshit. But perhaps in fact I am anti-social in the way that is a necessity for people of substance who must actually walk the walk on a daily basis. For example, when I learned how to shoot pistols a few years ago, suddenly a huge portion of television and movie drama became unwatchable. I realized that I had been engaged in fanciful thinking about guns for the entire time that I never took the opportunity to stop being so ignorant and get some hands on experience. The more things one does of this nature, the more anti-social one is tempted to become as our society requires a certain discretion in the treatment of naifs. I approach all this with some caution, of course. As excited as I get to be the man on that wall, I can't always be so sure deep down in people's hearts that they want me on that wall. Naifs will deny that they need me on that wall. They think they can avoid the Morlocks simply by not driving through certain neighborhoods. Meh. Enough with the preface.

The last time I spent the winter in Philadelphia I was at the end of a particular career and I wrote the Way of the Servant. It detailed what i thought and still think should be the model of work done in my industry, that of business intelligence, and for that matter most of end-user computing and all programming that serves domain-specific human decision making. What you program into AIs is up to you, but is also for another discussion. To recap: 

Ambition

• Newbie
• Apprentice
• Journeyman
• Hotshot

Mastery

• Guru
• Goto

Power

• Lord
• Demiurge
• Primary

Now if all of this sounds a bit feudal to you, you would be correct. I could go into it but here's the basic premise. Work is not democratic, and if it is to be meritocratic on a human scale (as opposed to a global scale) then hierarchies are inevitable. As much as I like decentralized power, I do not concede for a moment that children are equals to their parents or that wisdom does not accumulate with age. At some point Americans may choose to do away with the SAT and grade points and let everyone go to college for free, but for the time being be aware that very strict hierarchies are in place in our society that nobody finds particularly controversial.

Moreover there are certain tasks in human affairs that require command and control, because there are certain situations that arise that almost nobody wants to initiate. We must be persuaded, the majority of us, to engage in distasteful or dangerous work. For areas where we are unsure or ignorant or otherwise unmotivated we need the confidence that command gives. For such matters, stating the obvious, leadership is necessary.

Having become the Goto, and having fooled around being Guru once upon a time, I decided to become a Lord. That was in 2007. It didn't work out. There wasn't quite a cultural fit. I was a perfectionist, they were hustlers. I required clarity, they could wing it. I made promises my boss couldn't cash, and he expected me to figure out a way to get away with it. Uncomfortable, and ultimately unprofitable for the both of us. That seems so long ago I can't even remember, without the aid of an old resume, to tell you what happened next. But what I have been doing for the past 6 years has been another adventure altogether.

In that time, I went through an entirely new cycle of Apprenticeship and Journeyman's work in open source, cloud and devops work. Same trajectory, completely new toolset, and under the kind of direction I had never had in my entire career, a truly brilliant technical mind. But that cycle has come to a close and now I am ascending once again to work on a more Lordly level. Plus I've done a bit of Demiurgery. Now people hew to my rules. It's a new rhythm I'm beginning to get used to. I expect to wean myself, in due time, from all roles of ambition and mastery. I do have enough domain knowledge even though the world keeps spinning.

On the horizon are yet another set of interesting technologies associated with distributed non-trusted databases. Yes that old blockchain. I have decided that my angle will be simply that there are centralized fiat trusted databases and decentralized databases. The literature says that they can handle a righteously fast transaction volume. Nobody fully understands what the applications of all that will be. I hope to be able to swoop down from a 10,000 foot level and perch on a reasonably high position of command and control when all this matures. In the meantime, I will continue to be a data architect's best manager. Stay tuned.

DevOps The Hard Way

I'm going to be doing more management and 'glue' business the next year or so. Part of this business is selling and personifying the value of DevOps. Like Cloud, this is something that is insufficiently understood at a deep(er) and nuanced level. So, as is customary, I'm going to tell a story. 

The story, like most of mine, comes from an experience that burned. Something that left scars and had me wondering how people cold get into this mess. And so like the man says, share your scars. The situation was that I was on the very cutting edge of what I could do with Essbase + Essbase Studio. The requirement for drill through was fairly obvious. We all knew the limits of how much data we can squirrel into a multidimensional cube. So I used Essbase Studio to map back to the Oracle DB and bring back some records. Now it turns out that my customers wanted something on the order of 10,000 records in this detail. Well that doesn't seem like much. You could grab 10,000 records across a dozen columns, cut and paste them from one Excel spreadsheet to another right? That should only take a few seconds. Not the drill-through. That data had to come over the network. Well you could copy a spreadsheet with a dozen MB of data from a network drive to your desktop, right? That should only take a minute. Not the drill-through. We had to fulfill a query request from a database. 

My queries were taking 7 minutes and 30 seconds and then dying. 

I had to find out why. Thus began my painful birth of being DevOps. The first thing I had to learn was the difference between a view and a materialized view. Well that wasn't so difficult to learn. But I had always assumed that my DBA was materializing data for me. Well he didn't have enough disk space to do that for ad-hoc queries. So that meant I had to learn the procedure for requesting new disk space from the DBAs. How much did I need? I don't know. A terabyte? Impossible! Impossible? I can go to Best Buy and get a terabyte. Yeah but one live terabyte means four other terabytes according to our backup and DR, and we're at the limit of the current server which means we'd have to get a SAN device and... well how about an NFS drive? Nope. Can't have an NFS drive that would slow down everything I need local storage. Well, we'll get back to you. But how can you be sure that the database is the bottleneck? I don't know.

I had to find out where. What is timing out? Was it the Excel add-in? No. Was it the java middleware? Maybe. Who knows how to read the profile of the java middleware? Well there's no documentation for that, you'll have to call the engineers at Oracle. OK. Open up a service request and get an appointment. Who has access to the middle tier? Get access to the middle tier so you can log on. Oh by the way, the one support engineer is in Mumbai. That means you stay late, past 7pm Pacific time to get your answers, when he's available. OK change the profile, add in this line for the timeout. That didn't work? Oh you have to get the latest patch. Will it work with the version of Essbase Studio we're running here? Oh snap, we're going to have to burn a new version for you, but you're going to have to upgrade your java app server. OK now the explicit timeout is 15 minutes.

Still times out.

I had to find out how. What is the mechanism that creates the time out. Get this new tool called Fiddler, it will help you debug the HTML stream. Debugging HTML streams? Well, maybe it's the size of the download that's stopping things. OK did that. It's not the size. Well the corporate standard timeout is 10 minutes.. What corporate standard? The corporate standard on the firewalls between the users and the data center. Well can we get an exception? Maybe.

So it basically took six weeks for me to deal with the various network engineers, database admins, support staff and their management to prod them all to buy what I was trying to sell, which was the viability of this entire project. My only leverage was that I was consistently riding herd on the problem and I was a very expensive third party contractor. So the project was late and the entire overhead of the difficulty in justifying business as usual in the various departments was the only thing that motivated people to go to extraordinary lengths to solve the problem. Everybody wanted the problem to be somebody else's problem. And until we found out exactly what the problem was, everyone was pointing fingers until the last possible minute. It turned out to be a default in one of the load balancers that everyone assumed was set to 10 minutes, but communicated 7.5 as an override to the other. Those machines required firmware upgrades as well. 

I have been accustomed, throughout my entire career in BI to be responsible for the entire data supply chain. That I could do. But middle-tier service configurations, firewall settings and DR disk availability was all above my pay grade. I was not paid to know and I was too expensive to be paid to learn. In that way, I'm accustomed to being like the wiley developer whose time is too valuable to waste learning these operational details. At the same time, I was equally demanding of all those dependencies. Give me more memory on the app server! Open up the damned ports I want! Get more disk, you lummox! Of course let me not forget the memory constraints on the end user machines. 

All of this was a terrestrial implementation and it had other setbacks too, but it was a fascinating six month engagement. I of course learned a lot about these other systems with respect to how they affected my entire piece of the data warehousing applications. I sensed that I had the capacity to understand, but I'd never remember unless I had some responsibility and permission to make changes. That would be impossible without the cloud. But even when I had the cloud, it was more than just having control of the associated systems but really understanding how they worked. That's a story for another day. What was clear was that it was very difficult to manage all of the departmental areas, and get the priority within those departments (at their various locations) to solve a showstopper problem in this one application. It was 2011 and we were testing the very limits of the IT capabilities of a global corporation. DevOps might be a cool thing to talk about with web startups, IE a DevOps engineer would be cool for your website, but I saw the fundamental management problem that had everything to do with the way multimillion dollar Enterprise applications were built and maintained, and essentially why they were one-shot deals. 

 

The Magic of Blockchains

So I've got the bug. This happens when I have a little downtime in between gigs. 

via www.cubegeek.com

 

Yeah I might as well send some traffic over that way. 

Counting the Dead

  For many many years, I have had an abiding interest in the brutality and lethality of law enforcement. I have a number of experiences...

Today the subject is 'unarmed black men'. There is nothing more that needs to be done because it's not a significant pattern, there is no significant risk. You say 'unarmed black men' as if 'unarmed black men' were a special group at risk, but 20 million unarmed black men are not at risk. If I said 400 unarmed black men were killed by lightning strikes last year, would you consider it necessary to setup a national awareness campaign and change policy? Of course not. Because the number is so small, you cannot extrapolate a pattern.

This is precisely exactly demonstrably the problem with racialist thinking. You take something that happens to an insignificant number of people in reality, and then you apply it to a race of millions. Then you do everything possible to make everybody conscious of your little racial theory. You add racial consciousness by establishing a false causal relationship and substitute prejudicial associations instead of real logic. You turn 'who happen to be' into 'because they are'. Wrong.

Police shootings need vigilance and they get vigilance. Cops don't walk away from fatal shootings without administrative review, and those administrative reviews are under local control. That local control is all that is necessary - anything else takes power away from the people, and subjects local matters to millions of prejudiced eyeballs (through the lens of national media). But your town is not like my town. Every incident requires a case by case examination, in the context of exactly what happened, not according to some racial theory.

There are practical considerations. Consider the fact that there are 12,000 police departments in the US. Our best guess right now is that there are 800 - 1200 bungled arrests that end up fatal. That means in a bad year, the average precinct uses the new system once every 10 years. But we know the distribution is more likely to be the 80/20 rule, IE 20% of the departments are 80% of the problem.

So I know this because

A) my profession is business intelligence. I'm the guy who builds these kinds of databases for a living.

B) my brother is LAPD and I know the kind of paperwork he had to fill out under a similar consent decree.

When you mandate this kind of system it gets real expensive real fast for officers and departments who are not part of the problem. If you've decided that you can't trust the departments, the prosecuting attorneys and the whole system, having an accurate number won't help.

I'm curious to see who is going to pay to hire new FBI agents for this job and how much cooperation they expect to get. They have about 55 field offices, so that means (on average) > 200 departments per field office. Plus the cost of the new system. Heh. That's a contract I'd like to get.

But seriously, for what? If in 2 years we say exactly 1223 deaths, after we've spent a couple million for the new system, and taken the hit off the federal payroll to either dilute the FBI's manpower or increase it with new hires, what is going to restore the public trust when 99.9% is not good enough? Nothing.

What's needed here is a fabulously produced high-tech lynching with wall to wall media coverage in HD and some cops name that becomes as unforgettable as 'OJ Simpson'. So that everybody who doesn't trust the 99.9% can repeat that name for the next 20 years, and justify all their theories.

Why? Because when the Guardian UK put up their site, the facts became too boring to talk about.
(The Counted: people killed by police in the United States in 2015 – interactive) And nobody is talking about Oklahoma and New Mexico, where these statistics are telling us the most shootings per capita are taking place.

People who have volunteered there emotions into this circus will not be satisfied until somebody in uniform is burned at the stake and then beheaded. I expect to see lynching return in my lifetime, but I will have the comfort of knowing that the silent majority will be larger than it ever was. Will that be too small a comfort?  Hard to say. 

 

 

Five Levels of Analytics

As 'Chief Iteration Officer' at Full360 my mantra is 'iterate to quality'. That means preserve what you know and get a little bit better every day. I have always been from the Deming School and don't often advocate the Hammer approach. I like John Boyd's OODA loop and I think it works wonders for business processes. As systems integrators, we don't always get greenfield projects. Most of the time we are given to take over systems that are broken, incomplete, poorly performing or out of date. Just having 'analytics' in place is not sufficient. The general idea is always good, but how well are you doing?

This is a guide to help describe the level of thoroughness of an business intelligence system.

Discovery - Level One

• I can use my system to know what information is available to me.

This is the broadest and most basic level. it means that as a user, you know which systems are available to you, how to access those systems and what you can expect to find in those systems. The cash flow reports are here. The current reports are here. The historical reports are there. The systems that identify operations are there. You can navigate to all of the systems relevant to your area of responsibility; you know their names, functions and update timeliness.

Clarity - Level Two

• The information available to me is clear and unambiguous.

This means that master data issues are resolved. Codes and spellings of descriptions are complete. There is no question that data is in the proper range. It is validated and audited so that there is little doubt that you are dealing with the right information at the right time. You are aware of the units of measurement and these units are converted into standardized, consistent, useable form.

Purpose - Level Three

• I understand how the information relates to the business at hand.

Purpose is achieved when data becomes information and you can describe the business in terms of that information. It might only describe the costs or staffing levels of the business, but it is specifically applicable to the operational function of the particular business unit. You may have unambiguous clarity about 'gallons of gas available' but it serves no purpose to the legal affairs office. What is measured should match what is known about the operations, whether or not the information is managed. Also, proportionality of the numbers themselves are understood at this level. One familiar with the business should know how various measures and metrics relate to each other, if at all.

Predictability - Level Four

• I can determine with precision what specific information describes business performance.

At this level of analytic depth, system users are capable of isolating that information that is a primary indicator of business performance as contrasted to those that might be trailing indicators or otherwise not driving the business. It is the difference between knowing when a small fluctuation or a big fluctuation is material to business performance. It might be something like a regulatory ratio that must be maintained, a quality control statistic for pass/fail of a manufactured lot of goods. It might be a stated performance objective like a tight budget or schedule variance. Industry standard metrics would be here.

Responsibility - Level Five

• I can accurately predict and plan business performance with my system.

The final level of analytic depth is responsibility. At this point reasonable budgets and forecasts are be made on key performance indicators. Variances are taken against actuals and not only business performance, but prediction performance is tracked. A well-managed business is data driven at this level of analysis and planning, and novel KPIs and metrics might be generated whose meaning and value are quickly evident.

Essbase Capacity Planning (1997)

This entry is a pointer to a couple capacity planning documents. The first one I created was back in 1997, which I updated last in 1999. The 1999 Capacity Planning Document is embedded in the extended entry. (Format sucks) The updated version will be linked here.

The Four Pillars of Business Intelligence

BI - What is it?

BI is business intelligence. It is the art and science of getting computers to know and think in ways humans do not do well, but need to know in order to better run their businesses. It involves the capture, management and presentation of structured data for use by decision makers. These data are usually organized into multidimensional hierarchies of key performance indicators. The aims of Business Intelligence systems are decision support and performance management.

Business Intelligence is my preferred term for this activity because it is the broadest, but various aspects of BI have gone as

• Data Warehousing
• Decision Support
• Analytics
• Big Data
• Data Science

The origins of these practices are found in the lessons of Operations Management in the growth of multinational corporations during the 1960s, 70s and 80s. Some thought leaders that led to the creation of this multidisciplinary field include the following:

• Peter Drucker
• Walter Wriston
• Ralph Kimball
• David Liddle

PECCC

The benchmarks for decision support and performance management are as follows:

• Productivity: Increase output.
• Efficiency: Reduce costs.
• Competence: Add capability.
• Confidence: Improve satisfaction.
• Communication: Deliver consistency.

Business Intelligence systems are all about improving one's ability to know, and helping one assess the impact of the changes made because of that knowledge. It is cyclical, so that new discoveries are added and the consequences of those results are measured as well. Business Intelligence systems enable a framework of iteration to quality and maintenance of compliance.

Application Scope

BI systems include a selection of IT tools, technologies and practices arranged for the delivery of applications. Each application should be implemented as an independent project which is determined by the aegis and impact of the key decision makers for the company's operational goal. One should ask, who is responsible for the execution of X, what do they need to know, how often do they need to know it? A proper BI application delivers actionable data. You cannot manage what you cannot measure. A proper BI application delivers the knowledge necessary to improve a specific area of the business.

The Four Pillars

There are four areas of expertise for a BI practitioner.

• Front End
• Back End
• Governance
• Inference

The Back End - Architecture & Process

Back end work is primarily about data management, and data modeling. There are two primary sort of tools one uses in the back end. The first is an ETL tool. The second is a database. ETL stands for extract, transform and load. It is a tool or a suite of technologies used to identify data sources, gather them up, translate them into a standard format and load them into a database. A database is a tool that has a store of data and gives it a particular structure so that it can be queried in part or in its entirety. The database makes the data 'live'. The way the data in the database is stored is called a model. The most important task a BI practitioner can have is designing a proper data model, because this determines the ease and speed with which the data becomes accessible for many years to come. When data stored in a database is not easily or readily available we call it a 'data jail'. All data that serves no purpose to humans is, relatively speaking in a data jail. The job of a BI practitioner is to free data from that jail so it can be part of productive society.

The Front End - Visualization & User Experience

Front end work is all about presentation. It is the arrangement and display of data to the end user in a compelling, accessible manner so that the meaning of the data can be readily communicated. A front end designer is responsible for the user's experience in navigating and interpreting the data. There are a wide number of graphical tools that are used in front ends. They are generally, tables and charts. But they are also alerts. They can be text, graphics, sounds and multimedia. They can be presented online via desktops and mobile devices or offline to paper reports. They can be delivered interactively or according to schedules. The front end designer must know his audience and insure that users do not become confused or frustrated when accessing data. Data presentation must be precise and crisp and encourage habitual consumption.

Governance - Security & Lifecycle

Governance work is about the meta process of business intelligence delivery. It starts with metadata management by considering the provenance and security of the data over the lifespan of the application. Who is authorized to use this data? Who is authorized to change this data? What are alternative sources of the data and how are they reconciled? How long will the data be live, when and where will it be archived? When and how must it be destroyed. What happens when something changes? Governance also can include resolving questions about the cost of maintaining the application and the scope of technologies used to deliver it with regards to upgrades and deprecations.

Inference - Statistics & Practices

Inference is the term I use to describe 'data science'. This is a new aspect of Business Intelligence that manages the interpretive aspects of data itself. It is necessary because the scale of BI application datasets have grown exponentially. So particular care needs be taken when imputations and complex composite metrics are made over these large sets. For certain metrics, direct interpretation is straightforward. How many cars? "159". For others more care must be taken. How much to people like those cars? "3.5". A BI practitioner in this specialization assures that the data is taken only as seriously as it can be, and helps balance decision makers' instincts and understanding against what the computer system is saying. They must understand the business goals and the application of statistical analysis to management. They must always draw the clear distinction between the map and the territory.

All of these areas of expertise must be engaged for a holistic integration of a BI system into a business process of continuous quality improvement. 

Glitching and Learning

My theory of information dynamics says that knowledge cannot be maintained without the expense of energy. I'm finding an interesting inefficiency in my gaming lately. The presence of exploits both expands expertise and decreases propriety.

It is inevitable that as much as I game, that I will be using analogies to that in life. Get over it. Because Lombardi. But just in case you need an intro, here's a thick paragraph about the Destiny Raid:

Destiny Raid
Destiny is a free-roam open sandbox videogame in which the player takes on the avatar of a Guardian. A soldier who runs individual and small cooperative missions against legions of extraterrestrial baddies. The area of operations is the inner planets of the Solar System, Earth, Venus, Mars, Earth's moon, and Mercury. This sort of gameplay (PvE) falls into three categories, Patrol, Strike and Raid. Patrols are random in that there are not specific targets for your activity, although one can select various mercenary missions while on patrol. You can go solo or with up two two buddies as your loot and shoot. Strikes are more crafted adventures with a brief bit of story to get you going - rather like the messages at the beginning of Mission Impossible, you are assigned to take down this and such boss, facing several preliminary battles along the way. The Raids, however are the showpiece and crown jewel of Destiny. On a Raid, you need six players to accomplish tasks that are impossible to guess. Magical things happen that are not obvious, the rules of physics are bent and the team must figure out a way to overcome them. More than just sequences of enemy attacks, but changes in the physical enviroment take place on the way to beat the final boss. There are no instructions for these battles. Teams have to figure out what works. 

Now I'm going to talk about one particular Raid level and the exploit we have been using. It is the second level of the Crota Raid. Interestingly enough, I don't know what it's called. But here's how we play it, and playing it this way made me think about glitching and learning. In case you're not familiar with the term, 'glitching' means taking advantage of a weakness in a systems and short-cutting the process in a way contrary to the way the designers had in mind. For example, if you're hiking down a series of switchbacks, you might literally cut the corners by going down the middle of the hill instead of the gradual left an right on the beaten path. 

As our team enters the second level, we are stand at the top of a set of stairs over looking a pair of plazas. Between the plazas is a cliff. Each plaza is about 200 meters east and west and 75 meters north and south. The gap between the two is about 150 meters. On each plaza are three 10 meter radius circles forming a shallow triangle that points to the other plaza. The top of each triangle is the circle that lies at the foot of an invisible bridge between the two plazas. In the feet of each triangle are tall structures that hover above the center of their respective circle. These, are called 'totems'. So there are four totems, two on each side. The play begins once a player descends the staircase and enters the plaza.

Method One
Behind the near plaza is a large room cut into the face of the mountain. Our method is to hide five players in this room at its extreme rear while one player goes out and excites the enemies. How exactly he does so is a mystery, I've never seen what he does. I'm one of the hidden players. After about 30 seconds, two things happen. First an enemy called a Swordbearer comes out of the hidey room from a small door on the right, below the platforms upon which we are hidden. Then, all of the enemies that have generated out on the plaza dissappear leaving just the Swordbearer. We all then go out to the plaza and attack the Swordbearer who drops the sword. One player is designated to stand at the bridge circle, this materializes the invisible bridge between the two plazas. Two other players are designated to stand in the totem circles. Once gameplay begins, if no players occupy those circles the totems will begin to glow red, and after about 20 seconds destroy the entire team. The trick then is to create the bridge, destroy the Swordbearer, and keep the totems quiet. Then one player can take the sword across the bridge and face another enemy, the Gatekeeper. 

The most difficult part of this part of the game is that usually there are multiple enemies swarming the plaza harassing the players in the totem circles. The difficulty is defeated by the team using the rear platform of the hiding room which, for the purposes of generating enemies, makes the presence of the five hiding players invisible. Destiny levels up difficulty by taking into account the number and strength of the players. Few players, few enemies. More players, more enemies. By hiding, we eliminate the possibility of the game generating more enemies.

Method Two
In this version, all of us go out onto the plaza. One player must be the sort who can regenerate his life after death. We go and thin the enemies a bit. When the Swordbearer comes out we degrade him a small bit, but then all of the players jump off the cliff in close proximity to each other, including the regenerating player. The game begins a countdown clock to resetting the level and just as it hits 1 second, the last player regenerates his life and then restores the lives of the other players. The effect is that the multiple harassing enemies do not regenerate and only the Swordbearer is left. He can then be dispatched in short order and the sword taken across the bridge. 

Both of these methods are effective in eliminating the primary difficulty of defending the totem circles from swarming enemies while trying to kill the Swordbearer, who is more than any single player can defeat in a short period of time. Doing this will creating the bridge adds to the chaos, which is reduced through these glitchy methods.

However the result is that I personally, and certainly many other gamers who have passed through this level have done so without having experienced the game the way on presumes the designers intended. However this is different than a cheat because we were given an environment whose rules were discovered by experimentation, not explicitly laid out by its creators. The aim of the game is to survive and advance. Whatever works, works. Or as the meme says 'Everything bows to success, even grammar'. 

We avoided the difficulty of the challenge of harassment incurred defending the totems, by convincing the game that we didn't exist.

Harassment incurred defending the totems is a particularly apt way to describe a lot of life's challenges present in society every day. It has the virtue of allowing one not particularly skilled at defending totems to survive and advance. One might possibly, through all of the shortcuts, manages to beat Crota, the goal of the Raid. That's a win. The game only polices itself, but there's a meta game as well. How you beat Crota matters in some circles. Ultimately, the most skilled players will figure out how to defend the totems and win, or not waste their time doing so and still win. After all, only multiple wins in the Raid can gain you all the gear randomly distributed at the end of the final boss battle.  

I'm not so much interested in the ethical question here, because that's ultimately longitudinal in the meta game. Either the game creators will fix the exploit - say by resizing the hiding room, or find some way to raise the difficulty in another aspect of the mechanics of the Raid. The designers do indeed update the game to block usage of those exploits they deem to be 'cheese', while leaving others intact. That feedback loop is thus is self-correcting and the game becomes ultimately what the designers intend it to be. Their decision for popularity of a game level, or difficulty is the balance they must strike. But what about the knowledge?

I said at the outset that the presence of exploits increases expertise and decreases propriety. If I beat the plaza level, then I am an expert, whether or not I take the architected path. I will continue to play the way I do and teach others to play the way I do. They in turn will play according to the method they learned. There's an interesting dynamic at work in such cooperative endeavors obviously involving leadership and expertise but also consensus. Everybody on the team doesn't necessarily want to learn so much as win. So it's not so very likely that someone playing with the same circle of friends that team up to raid will learn multiple methods. Only going outside your small circle will you come to better understand the mysteries, and how many of them will be actually doing it 'the hard way' as the designers intended?

The proper way to beat the game involves defending the totems. (Sorry if I can't get over the aptness of the metaphor). And yet when certain methods are employed, that task - which was added for the purpose of making things more challenging (and fun?) - disappears as do the consequent difficulties. The skill of defending totems is lost, and over time their very meaning can be lost. As more experts figure out more alternatives to gaming the game, the rationality of the game design itself can come into question. This happens all the time in Destiny, as in life.

There is a meta game to which we take on various levels of care. The method by which you approach the game is a signal of your skill in the game, which others will interpret as you temporarily collaborate. But glitching itself is a skill which is not only valuable to the player who aims to survive and advance, but to the designer who has an opportunity to eliminate, modify or let the exploit remain. Sometimes you want to play with glitchers, sometimes you don't. 

Three Databases

I just did something remarkably easy that reminds me about why Amazon is winning and will continue to win. I started up three different databases and then threw them away. For anyone who has worked in the Amazon ecosystem, this may seem like just an ordinary thing. Indeed it is, but let me take you back in time.

In 1985, I took my second summer internship at Xerox in El Segundo, CA. I reported to a guy named Jack Starkey who was one of those starched-shirt engineers of the first order. I loved the guy. My assignment was to build a data dictionary for the parts and service database that Xerox used to keep track of the maintenance of their top of the line laser printers. Xerox was considering migrating from an IBM VSAM hierarchical database to something called Focus, a new fangled relational database. My job was to insure that I had all of the definitions correct. I asked Jack if I could use the new Xerox Star Workstation in order to complete my job, which was essentially all about documentation. He agreed. 

My first internship was more interesting because it was more technical. I actually wrote a financial modeling program. But my boss was loathed and feared in that area and a lot of people hoped everything he did would fail. That guy, whose name I actually cannot remember Jim somebody, was a notorious pipe smoker back in those days when you could smoke in the office. He liked something called Amphora Green.  My project did not fail, although there were some interesting twists. My job was specifically to make a realtime pricing model that salesmen could use to develop a quote for customers based upon the way they actually did their electronic printing. At the time, most computer printing came out of printers attached to mainframes and the most popular one was the IBM 3800. But the Xerox printer had duplex and quadplex, meaning it could print on both sides of the same sheet of paper. My program would show the long term economic benefits of using the Xerox tech which often came down to power and supply costs. So I learned 'Total Cost of Ownership' at a fairly young age. The MBA intern with whom I was working had her HP calculator. My code was being held up because she was late in delivering a 'cost matrix' to me. I sat down with her finally to discover that she had just been plugging numbers into a formula run on her HP 12C. The MBAs didn't sit with the engineers, you see, so getting this meeting took weeks. I had to explain to her that this CP/M based computer could actually do that kind of formula calculation. She was shocked that Xerox actually made a computer that could do the same things as an HP 12C. We all learned a little something. I recall later on that a cat named Burkhart, whom I seem to have never forgiven in person, got the chance to present the wonders of my completed program to Xerox folks in London, while I went back to school that September. 

But that was the pace of things in the mid 80s. Three months just to build the equivalent of a DDL script to help move data from one database technology to another. Today I do that in realtime. Another dude I vaguely recall had the radical attitude I might have enjoined were I not so desperate to drive a BMW. He advised me to get all my hacking done before the implementation of ACF2. What we were dealing with was the gap between the time you could get engineers to understand a technology, the time it took them to implement it, the time it was adopted by the business, and the time that an actual payoff could be seen. Then there was the time it took for the capacity of the business to exceed the design limits of the system in place and the time it took for that problem to arise to the level of necessity and a new replacement process begun. 

In the 80s, all of that was glacial. Moore's Law kept us all expectant about the future, mostly in terms of how much of all of the business processes could be captured outside of the pocket calculators of MBAs, but the process of business adaptation as well as the process of re-sizing systems have continued to be slow all the way to the current day. Amazon has emerged to understand these kinds of problems very well because of how e-commerce begat DevOps. And yet the majority of businesses still use 'Enterprise' architecture in their systems implementations. 'Enterprise', for all intents and purposes, was the term used to convince IT buyers that UNIX based servers could handle all of the business once only owned by the sort of mainframe computers that spit data out to Xerox and IBM printers. And then came e-business which introduced the 'n-tier' solutions, multiple tiers required because no single vendor, not even IBM, could provide hardware, networking and software solutions for the new class of applications being envisioned and built.  

I don't have a buzzword we could reliably depend upon to be what this time of transition to cloud architecture will be. 'Post-Enterprise' is all I will hazard. However, what processes can and will be improved is a lot clearer. So I hope to speak to you, my fellows still using the systems designed with on-premise Enterprise class architectures in mind. That's my aim for this year, to help you see what I see and what my company, Full360, can do to help you realize some of the promises of computing that were made a long time ago and have been a long time coming. 

So what I just did, single-handedly, today and yesterday, was build three different database servers Oracle 11g, MySQL and Microsoft SQL Server. They were each 4 core 15GB servers with at least 300GB of SSD. They took about 15 minutes to configure and secure, with automated backups in an alternate data center. I was able to connect to them directly through a secure VPN I had previously setup with my RazorSQL client using JDBC. Today, I'm shutting down the Oracle and MySQL services because my customer only has the MSSQL license. But it was actually fun playing with the alternatives. Fun!

What's going to happen next is that I will start using AWS Config to keep track of all of my compute and networking assets for my customers. I'll be able to tell, at a glance, which server is doing what in which stack in which VPC. I will be able to manage services to an even greater extent for this and my other customers. 

Back in 2001, I had a dream about a company I might work for in the future. I called the company 3DB because as a fundamental competency I wanted people who understood object, relational and multidimensional database technologies. We could then build the kinds of systems I envisioned. Now I work at Full360, but the 3DB concept is reality. We use multiple database technologies in our data management framework called ElasticBI. I didn't envision them running in a cloud architecture, but that is even better. Stay tuned. 

Producers, Consumers & Dynamic Data Warehousing

So we have a bunch of big ideas around here at Full360 and it seems that I have been doing them rather than talking about them. That's what I like about work. Here's what we're doing now.

via www.cubegeek.com

New this year is technical cross-posting. Integrating the free-standing Cubegeek block with everything here at Cobb.

Jeremy Winters at ChefConf 2012

The Logarithmic Shadow

If you've listened to any gangsta rapper for long enough, you know that he will eventually tell you that he could have you killed. Generally, the person who would do the killing would be one of the rapper's former associates, and the price of the deed will be relatively small.

Knowing that there are people who would never kill and that there are others who would kill for cheap, or even free, poses an interesting problem. It is the problem of absolute vs relative value. This is a problem that I have found particularly troublesome in my rise through American society. It cuts several ways.

I characterize this problem as 'the logarithmic shadow', for my lack of the generally accepted term. I observe that people abstract in such a way that small differences are seen as large and large differences are seen as small. The truth of the differences are obscured in shadow, as exemplified by logarithmic scales.

Hacker's 20%
Andrew Hacker, the famous social scientist, took a survey in his book 'Money'. He found that when he asked people of various income levels what their ideal raise would be - a raise they would say could take care of all of their financial needs, the answer was more or less universal. It was 20%. People's expectations are relatively finite.  For the man with $10 shoes, those costing 2 dollars more would be all he desired, but for the man with $200 shoes, he required 40 dollars. 

Look at it the other way. For the man driving a Ford, he would need to get a BMW in order to be complimented. But for the man already driving a BMW, he'd probably need to buy an Aston-Martin to get a similar compliment from his peers. 

The Cocktail Party
A welder is invited to a cocktail party celebrating the opening of the new building he helped construct. He happens to overhear a conversation between an architect and an attorney, both who make a great deal more money. The architect makes an aesthetic statement about the beauty of a particular archway in the lobby of the building. The welder knows that there were many difficult welds used in the steel of that arch. In fact, one of his subordinates injured himself. The attorney takes the architect's statement and elaborates on the beauty. The welder pipes up, smiling with pride, yeah it cost Joe his finger. The conversation halts, abruptly with embarrassment. The subject is changed.

The Cookie vs The Credit Card
A new technology called e-Commerce becomes available so that ordinary people can shop online. In order to accomplish this, a technology called a 'cookie' is stored on the buyer's computer.  It just so happens that a very sophisticated thief can get personal information from you by hacking your computer and accessing this cookie. People are horrified by the possibility. Yet every day these same people speak to travel agents to book their business travel. These travel agents know their favorite airlines, hotels, car rental, and meal preferences. Whenever people go to restaurants, they willingly hand their credit cards over to waitstaff who make much less money.

Flies on the Belly
Marta is a starving sick illiterate child in a poverty stricken country with a population of 2 million.  A healthy altruist comes on television to tell us that for a mere $20 a month, Marta can get food, clothing, shelter and education. Tens of thousands of people respond netting $20 million in contributions. Two years later the same commercial comes on television. Nobody wants to contribute any longer.

--

These are all paradoxes of perception. I could probably structure them more properly were I to spend more time, but they all illustrate to me the difference between effort expended and results achieved, between things of equal import that can generate asymmetric reactions. I am coming to understand very specific kinds of flaws in human perception owing to the fact that I have had a career in translating such perceptions into computer aided systems of representation. A precise translation still often yields misinterpretations. People cheat in over- or underemphasis in order to communication something of value.

What is particularly fascinating in all of this to me are the border conditions of perceptual change and the manner in which shifts are communicated. 

Facebook vs Arpanet
The first time I sent an email, I was still in college. It was through a system called DECnet. I was part of a small project, assisted by a guy named Richard Z. White who was working at Raytheon. We established what was then called NSBENET. It worked for a short period of time, then went unused. Even among college engineering students, it was considered geeky and complicated.  A few years later, in 1985 or 86, I began sending email messages in the current way we do, client server store and forward GUI based email. In fact, I wrote back in 1986 how cool it would be if we could have our computers store and forward digitized music. At the time, CDs were first being sold, and I can remember a time when it was almost impossible to get reggae music on Compact Disk. The social context of spending many hours in a week in front of a computer screen was decidedly declasse. Today, a generation of parents who have probably experienced email for the first time in the late 90s are fretful of how many hours their children spend on Facebook.

--

This is the cross-fertilized field of cognition, economics and philosophy. It is fascinating to me. One last example.

The Ignored Prospectus 
Ever since 2007 behavioral economics has become a dominant theme in discussions of what's going wrong in the market. People who have relied on the 'efficient market hypothesis' have now been faced with evidence that the invisible hand is a lot more clumsy and slow than they ever thought. That's because people, whose behavior ultimately determines the price of things in the market, often misperceive risk. These aggregate misperceptions establish themselves as market prices which are not efficient and they generate bubbles which certain conditions can conspire to sustain. There is probably no better example than Greenspan's indictment of 'irrational exuberance'. If there is, it's probably 'too big to fail'. Both are examples of unsustainable prices for which 'the efficient' market did not correct precisely because the aggregate perception expressed a desire that was contrary to rationality. In other words, if enough people suspend disbelief, you can sustain castles made of sand longer than rationality dictates and 'efficient' markets are not exempt.

So one of the many studies coming out to explain where we went wrong had very definite conclusions about how people perceive risk. You could draw very specific warnings out in the language of any security for sale, and people will generally ignore those expert warnings in favor of their feelings on the matter. It's rather like a reverse transferrance. No matter how big the warning label, once people have decided to buy, they go with their gut.

--

I expect upcoming books by Taleb and Ariely to aid my consideration at length of how people's cognitive errors manifest themselves with regard to how they take up information. I'm particularly interested in this from the perspective of abstracts of big data. There are many phenomena that we have heretofore been physically unable to probe, owing to the massive amounts of data. As our technical ability increases, and more computer mediated interpretations become the norm, what kinds of cognitive errors will we make?

When The Techs Take Over

I have heard it said that there are now as many semiconductor chips on the planet as grains of rice. It is an arresting thought. And as I take my technical interests towards open source software, cloud and devops, I am breathing a whole new gulp of oxygen in what goes on in the state of the art. Take these guys for example. 

There are several ideas behind the speculation I'm about to lay down. The first is based on this reality. The computer sector is outperforming the rest of the economy by leaps and bounds. It's not only doing well, it's doing very well. Somewhere over at TableauPublic, which I can't find and has been making me leave this draft in Park for too long, is a set of charts that show which companies have cash and which have debt. When you look at companies like General Electric it's downright scary. When you look at companies like Apple, they seem to do no wrong. Times change but what if the double dip comes and times get desperate?

Over the long term, there doesn't seem to be any end to the number of excellent and productive ideas that have always been basic to IT. And it gets easier to see them become reality. Back just 10 years ago, I couldn't even imagine today's future and I'm in the industry. Everyone was saying 'the last mile' could not be overcome - that is fiber to the home. Everyone was saying that video on demand was impossible. Now Netflix is bigger than Blockbuster. People were trying and failing. After the dot com bubble, nobody had any idea what to do. The idea the Amazon would do what it has done with EC2 was inconceivable by most of us. When Microsoft was forced to separate IE from Windows and allow other browsers in, they speculated that a company like Netscape would take market share by doing things just through the browser, but nobody dreamed that Google would become what it is. In the year 2000 only the geekiest of us had MP3s and the consensus was that nobody could break the hammerlock of DRM. Now Apple runs a DRM free universe of music on demand. Moore's law has ten more years, at least. That means that the iPhone 7 might have a terabyte on it. 

All of those innovations are extra cool, but the fundamental economics of IT can still be expressed in very simple ways. The key term is 'disintermediation'. What will computer and communications tech allow people to do for themselves that they used to need third parties for? To jump the gun a bit, I will suggest banking, and when that happens ohh doctor!

First off, the fundamental thing that IT gives is the ability to overcome time and distance. It enables human intercommunications on levels never achieved in the history of mankind. In and of itself, this is an economy pulled out of a hat. Without disintermediating planes, trains and automobiles, there are new ways that people interact that make IT a non-zero sum game. Look at a movie from the 80s and find all of the plot holes and crazy situations that could have been obviated by today's cell phone networks and GPS. 

Whenever I hear it said that reading and writing online saves a tree, I also know that it saves oil and electricity. What people tend to forget is that computing owes its very existence to the ability to engineer devices that sense very minute changes in electrical state. IT is energy efficient by definition.

What's happening to crowds these days is something very unusual. People have their own networks, and communicating with them and their paths makes an extraordinary difference in attention and focus. Many people may not quite know what to think about virtual friends, but the opportunity to have multiple online networks is a new social skill. Your telepresence will be a bigger status symbol than your car.  The hardware and networks into your home will be more important than your lawn. For those of us in the business, that future is now. 

Let me ask you. Do you remember the classified ads? 

Newspapers and broadcast TV are being redefined. You might laugh about Hulu today, but people laughed at Tivo five years ago. If you think Apple TV is important now, wait until film is completely passe. What's happening on the edges of digital networks is critical. Devices like the Red camera system and the iPhone are changing everything. Given what I know about the infrastructure that is coming, YouTube and Facebook and eBay are just the beginning. 

I think that in the not too distant future, much of banking is going turned on its head. If you think about banking as a collection of discrete trusted financial transactions, there are many classes of these things that can be disintermediated from the banks of today. It's already happening at Square. And look at what people are doing for themselves at Prosper. 

Back in 1996, I was thinking about the cloud before it had such a name. In my conception, I couldn't imagine that people would trust ISPs to retain their digital assets. So I imagined that only banks would have the financial ability and trust. Now Google does it for free, and Backblaze does it for $50 a year. When you accumulate about 10TB of digital assets, you're going to have to trust people too, and they will be a different set of names. Let us not forget that when I started out in this business, corporate managers would not *think* of using computer systems to transact trusted financial matters. The idea of a manager approving an expense reimbursement without physically touching a receipt was unthinkable in 1990. Now think about Western Union, and money orders. That's a shrinking business. 

In short, there are more and more ways the American economy is going digital, and there are fewer and fewer barriers. Young entrepreneurs are building software systems that disintermediate back office protocols. Every workflow in every business is being optimized by IT - it may be some time before you don't have to go to the DMV to renew your license, but I bet you a team of 10 people could build the system in six months. 

White collar and intellective work is changing. Permanently. That is going to have a tremendous positive effect on the energy efficiency of the American economy. In education, in the professions, in business, finance, banking, insurance, medicine, law, publishing, music, film and all the visual arts.

FUMH

Enderle explains some of the reasons that Mark Hurd is considered a pariah in the industry. Now fled to the barbarian king Ellison, Sam Palmisano of IBM is starting to pitch crap at a weakend HP and Hurd in particular. 

It rankled CEOs from Palmisano to Michael Dell to be compared to a guy who by most measures was slowly killing his company to maximize quarterly returns.  A kind of vampire, Hurd represented the inordinate power financial analysts had and how they were forcing decisions that were crippling long term. However any company that didn’t listen to analysts was pounded in the market with an expensive impact on executive bonuses and longevity.  It was understandably upsetting for Palmisano and others to see Hurd receive massive rewards for sucking the life blood out of HP while their better parenting went largely unnoticed and often unrewarded.

Aha.

And so now some of the reasons why this website exists are becoming clearer. The news there is that Hurd is getting booted off the board of News Corp.

I can tell you that it is music to my ears to hear that this particular kind of exec is getting flamed. HP is one of those great loved companies to engineers, and the very idea of having a finance goober run the joint into the ground just by tweaking numbers to please Wall Street analysts is horrid. So now that this one has outed himself as an asshat. To earn the appellation 'The Turd' takes some doing, even among vulgar grouches. But now the saga has taken root in my attention sphere.

I've always admired HP. They're having it rough up there.

Rights and Wrongs in IT and Software

Ceolas writes about The Industry:

Performance based? Nope, it hasn't been performance based for almost ten years. It's seen a loss of innovation and invention. Microsoft's former VP elaborated the problems within the company in an op-ed in the NY Times that have killed it's future, what he didn't mention, was the fact that Bill Gates has always only been a one trick pony who got lucky, and isn't interested in invention and innovation so much as denying others with the ability, the opportunity to succeed where he has failed. He also didn't mention the fact that the preceding decade has been one where Gates displaced US citizen workers with cheap Indian and other visa workers. Un/under educated, and in no way skilled or talented. Anyone who works in the industry knows that while they might like to act like sharks, they're incompetent and capable of doing little more t han steal the efforts of others. Everyone knows the real story behind the failure of the Indian tablet pc. It was a stolen prototype, one that hadn't been fully developed. India's tech house of cards will fall, because it's based on a lot of hot air and theft. Everything they touch, they foul.

 

A lot of this is right, and I would like to discuss it.

First off, Gates and Microsoft are not the spark of the industry, they are a marketing powerhouse and a very hermetic universe. They are almost incapable of innovation because everything is done the Microsoft way. My experience is that they are simply too thick to be agile and that they spend all of their time selling to mass markets. At Microsoft scale, they are a one size fits all company, and even when they break things up, like the different versions of Windows 7, it's still just too huge. There are very few people who think the way I do who expect much from Microsoft in terms of cutting edge. There are many boring things that Microsoft does well, but few truly exciting things that Microsoft does that win over hearts and minds of techies like me. They have just been too obstructionist too many times and so it is no surprise at all that a company like Google could just come through and steal every bit of glory Microsoft ever had. 

There was a sense two years ago, when I spent a lot of time in Redmond, that MS spends a great deal more time being a defensive company and that their captive audiences really make MS believe that they change the world. Microsoft is starting to show its age. They should voluntarily break themselves up. And they should make a tiny, new OS.

As for the Indian quotient. I have to say that there are just too many flavors to make broad statements, but I know where the dissatisfaction is coming from. I don't blame the Indians so much as I blame corporate IT for outsourcing their own brains. And corporate IT has just discounted itself below the threshold of where people with software skills are willing to work. And this is what leads to the incredibly sclerotic world of enterprise software, where good ideas take forever to materialize and Oracle, SAP, IBM, and a handful of pretty unimaginative business models are entrenched. 

Metathought on Google Wave

I saw something last night from Google that made perfect sense to me. It was that the same two brothers who were the brains behind Google Maps are the brains behind the next generation of email. It's called Google Wave and it is a brilliant blending of instant messaging, email, intelligent conversation threading and blog content management. It might have been more but I fell asleep watching the 80 minute video.

It solves a great number of problems that we emailers have been working our way around for 20 years and promises to put us on the Axiom for 700 years.

I have been considering the nature of a society, or a moment in time and a crevice of society, that celebrates and rewards toolmakers. And I think I have seen the end result. It oddly comes from another show that I watched last night (aside from Wall-E) which was on the Wealth Channel. It turns out that one of the guys that gets on the list for the most expensive cars in the world was showing off his Ferarri 430 Scuidera, his bright yellow Rolls and his Bugatti Veyron. All were customized to excruciating perfection, exactly the way he wanted it done. He kept referring to them as works of art, which they certainly were. This made them one of a kind and useful to nobody else in the world except as the ultimate status symbols, which was why they were on the Wealth Channel in stunning HD. The toolmakers in this case were celebrated by signing their names on the carbon fiber under the hoods next to the prominent steel embossed serial number plates. (His Bugatti was the fifth one imported into the USA). So, essentially and especially given the HD documentary, he has converted these automobiles into instant collectors items. The toolmakers don't have their tool's utility as its own reward, but the customization of it to please a completely other sensibility.

This sensibility has long been in evidence in the computing industry, which has long overproduced any dozen redundant technologies in 8 flavors. It is why the Linux world exists in such variety even though all we ever needed was Red Hat, a viable alternative to BSD, MacOS, Solaris, HPUX, AIX, System V and Windows. Instead there are some 3 or 4 dozen operating systems including the embedded world which is a complete mystery to me, each with dozens of programming languages. So it comes as no surprise that only the juggernaut of Google can do something utilitarian like perfect email once and for all. Further it comes as no surprise that it only takes two or three really smart people to do it.

So what is Web 2.0 and all that social software and clouds and grids all about?

That's a big question for another day, but the short answer is the computer equivalent of suburban sprawl. Nobody, save perhaps the NSA, has sat down and built an actual computer and software system from the ground up for 40 years. There is no planned community. There are only applications and marginal evolutions of applications for an interoperable world that has 60 ways to do the same thing which frustrate us all into looking towards Google to finesse them all back together in unifying ways. Surprise.

But Google is not going to do that, which is why I mentioned the NSA. You see outside of getting applause, stock options and face time at conferences, the rock stars of software development have no transcendent purpose. Only those who have had such purposes have really mattered, and I think we are in short supply these days. So it makes me wonder exactly how Google defines the 'evil' they will not do. It's certainly inclusive of the 'evil' the NSA must do.

From a technological perspective, Google is the embodiment of infinite scalability. Its compute engines are the Saturn Five of the computing world made all the more impressive by their tethering of millions of actual horses to make the horsepower rather than giant, fallible boosters. But I am confounded by the idea that the Rasmussen brothers, the brains behind Google Wave, would be so satisfied with the Google compute environment for what it is that their minds conceive a compute beast with such power could do. It's the old trade off of what kind of millionaire you want to be - the rap star wants $17 worth of respect from 5 million fans. The investment advisor wants $5 million of respect from 17 clients. They have obviously decided to catch the attention of the twittering masses.

So what is the transcendent purpose of making email, instant messaging and blogging incrementally simpler for 50 million Google users? Surely that will not enable one more significant writer on the planet.

This is what I notice every time a new Web 2.0 3.x or whatever program comes out of the software programming world from American rock star programmers, and was at the very beginning of this demo. It's all about 'Hey let's find a new restaurant' or 'Check out my vacation pictures'.

I think I've made my point.

So how about Google Medical Records, bitches?

On Complexity & Emergent Behavior

Riffing off Tim Burke:

There is a fundamental problem with building efficiencies into systems. There are essentially, with regard to white collar productivity, two schools of thought, Deming and Hammer. Deming for constant quality improvement and Hammer for business process re-engineering.

The Deming approach requires a certain understanding of the process in which all workers are engaged. From the standpoint of a software designer, it can all be conceptualized in terms of a workflow. Person A has tasks 1,2,3 Person B with function X has tasks 4,5. Task 4 requires 2, etc on down the line. The willingness of software engineers to absorb the details of workflows in business is highly limited - and the ability of managers of such processes to communicate their complexity and possible efficiencies and chokepoints, etc. is also highly limited. Furthermore the willingness for software engineers and business managers to deal with each other on a long-term basis is slight. Both see their time more profitably spent elsewhere. This is especially costly to the Deming scenario, and so what generally happens is a Hammer implementation.

This does damage to the process by obviating some complexity. So you actually have more complex and new systems shoved into a process which is actually dumbed down because of the communications limitations of managers and engineers. Then what ends up happening is that those employees in the process chain make up the difference by partially working with, and partially working around the new system in ways they have no incentive to explain, and often in ways their managers are unaware of. The outputs of these systems as well as their inputs may be manipulated because those processes to which information technology is applied are rarely end to end systems. So there is a complex adaptive dynamism at work and in the end describing these systems are like describing the flow of molecules of water down a stream. They are chaotic, somewhat random, extraordinarily complex, yet at a macro level comprehensible and predictable.

Once a system becomes predictable at a macro level it gets frozen into place as the practice of the business with all of the inefficiencies and mysteries locked in.

And then the boss changes and all of the internal stresses on the system lose particular incentives.

The Chalkboard

My brief comment in response to the thread.

What strikes me odd about this is that there are thousands of (apparently) companies that keep pointing to youth and (maybe) understanding the technology that connects people together, but they seem to have no respect (or clue) to what people do after they are young. I don’t need another friend and friending as a verb is ridiculous to people who are married with children.

What would be transformational is if someone took another look at the most important things people do in life and built tools around the individual, especially the information intensive individual.

I work in the business intelligence business, and started out with at Xerox long ago under the influence of people like JSB. What I have come to learn is that people interacting with computing systems do indeed come up with verbs, and the most intense interactive experiences end up having their own verbs. Podcasting is an example, as is ‘friending’, ’surfing’. And yet it has only been recently that we have been ‘buying’ or ‘banking’ and obviously nobody has been ‘voting’.

None of these things work as transactions as well as they do as interactions. It’s not just a simple set of verbs but a set of interactions around important functions - this is the approach that works best, when a person’s activities become describable as a workflow. The problem is that we’ve got a bunch of discrete actions in most of our social computing - rather like old video games before the invention of the sandbox, or old top-down programming before simulation loops - that don’t give us the sensation of swimming or anything continuous, like life itself.

I would argue a la Chesterton that we not tear down walls before we understand why they were erected. I’ve enjoyed auditing video lectures of physics ‘at’ Berkeley as much as any student in that classroom. But I doubt many students would be comfortable with my voyeurism - there is a temporal wall in the non-live rebroadcast that gives some propriety, and we all need a certain amount of that.

I would also be wary of giving over to a Friereian view of ‘equality’ in the exchange of information from teacher to student and vice versa. I say so as an old man who is not entirely convinced that all questioning of experience is useful, moral or productive. Pay very close attention to what is given and what is taken, because whatever arrangements we conclude are superior must inevitably be sustained in order for them to be anything more than academic. Keep that in mind especially in view of the factoid about how much laptops cost vis a vis wages in the Third World.

One of the things that was fresh in my mind as I blasted through that comment was this set of improvisations and meditations on the improvement of the Bloomberg Terminal. As you all know I'm very much into Bloomberg these days - information monks from way back, they are, Yoda like in their inscrutable wonkiness. But also I'm thinking of sandbox decision making and how the shape of an environment is determined by the rules many of which are invisible.

We don't often have sandbox-like decision making abilities in our compute environments. They are very discrete. We simply don't have the economics to achieve that yet. Not until the price of sandbox environment development comes way down are we going to be able to break down walls of education and have real simulations of important interactions with realworld implications. I can visualize it all myself - I've been working in the environment forever - but it is only beginning to become economical for SMB much less the general public.

It Seemed Like a Good Idea At the Time

There's something about soccer that best captures the spirit of HD television. But I have to tell you that a lot of sports is looking really good this season. I've been watching and recording the HD broadcasts on Universal HD and they are spectacular. Until I speed them up, and then I realize I don't know what I'm watching.

For a moment this morning I had an insight about machine intelligence from the Culture. I was fast forwarding through a bunch of commercials and then through some of the soccer game, and I realized that I was catching a bit more information on HD than I used to on the standard definition Tivo I used to own. But I was surely at the limits of human comprehensibility - there was no way possible for me to make sense of the soccer game itself. I could only recognize that it was a soccer game and nothing more, even though it was not literally a blur - several images were clear as stills - I couldn't process it with my organic brain as fast as the DVR could play it.

Suddenly I remembered a conversation between ships from 'The Excession' in which they had decided to take some action after a discussion that went for several pages in the book. But one of the ships suggested that before they do, that they ask their passengers what they thought, after all things had changed. One ship objected saying that it had polled the passengers 2 minutes ago and they had assented to go. The other said, well you know how long it takes them to decide things. You have to give them time to think.

A comic routine I heard on XM 150 the other day made a running joke about the common excuse 'It seemed like a good idea at the time'. It is the ultimate excuse. There are a lot of reasons that our human psychology makes the decisions it does, we are subject to certain inevitable biases, not the least of which is that we often make patterns that don't exist. We make errors in cognition due to a lack of time or focus or based on irrelevant data that somehow seems relevant at the time. For example, this very blog piece started off as something about the Olympics, but it is actually more interesting (now) as a thought piece on human vs machine cognition.

Being in the BI community, it is my job to see how machines can help humans make better decisions. The better I understand the business, the better I can tailor the system I'm building to assist the a limited scope of decisions to be made by the audience. When I described my gig to Unk last week, I got down to a specific job that I'm doing for a specific customer. It turned out that he had a similar issue during his days shuffling budgets as an administrator - and so he actually repeated that case back to me flawlessly.

There are lots of reasons we decide poorly, but when we are under time pressure, I think that is the reason we make the most mistakes. I'd make a final observation which is that some folks decide forcefully but then are amended quickly. That tends to be my decision making style - to set up a framework in which a good number of choices can be taken (in a fast OODA loop context) and harness as many observers in the process as possible. The quick decision coupled with authoritarian style can be very effective within organizations, but also dead wrong. In a proper framework, quickly wrong, might not be so bad.

Speed is key.

Korean Beef

According to a quick glance, South Korea does about 78 billion dollars of business with the US annually. However they stopped the importing of American beef five years ago out of fears of mad cow disease. That strikes me as strange. We've never had a problem. Some may say that mad cow is inevitable in all beef supplies, but that doesn't seem a cogent argument against ours. It's about 20 billion worth of business that we've been doing without and in April the Bush Administration signed a deal to restart the flow. That's a lot of beef.

Well it turns out that South Koreans don't trust American beef, no matter what kind of deals our governments have put together. And so thousands filled the streets recently to throw down against riot cops in the streets of Seoul. The Democrats are opposed to the trade agreement between Korea's Lee Myung-bak and GWBush, but it's unclear why. Lee has something like a 25% approval rating in his home country, and there is probably no easier way to excuse any behavior in the US than saying it's the opposite of what GWBush would do. So the South Korean government is caving in by playing on what appears to be a popular idea that cows younger than 30 months old are less susceptible to BSE. Don't ask me about how prion-based infections work, half the websites I looked for on BSE are down (except for PETA's of course).

Bottom line politically is that it comes down to a matter of trust. The number of people who are going to be able to make any actual determination on the quality of American beef is not going to change significantly. The Koreans in the street are going to take it on faith, just as they are taking it on faith that American beef is unsafe. There's an interesting unravelling going on here.

My interest in this is based on my own kind of information theory with regard to the ability of people to establish and maintain policy that is both transparent and well-abstracted. I will follow up in that regard again in speaking about how politics makes people emotionally deranged and affects their willingness to distinguish Natural Truth from Political Truth.

You see there's a bioengineered solution to the problem just at hand. But what Venter understands is that engineered genomics is predictable whereas random DNA fiddling is not. That is precisely the difference between bioengineering and agriculture.

Bloomberg & Government Transparency

NYC Mayor Michael Bloomberg is a pioneer who is at the very beginning of a revolution started simultaneously with Walter Wriston. The second fruits of his pioneering spirit are beginning to bloom. The first fruits were, of course, his fortune in the securities information industry created by the machine and company that bear his name. The second fruits can be seen here in Citywide Performance Management.

Microsoft + Yahoo = Monster Zero

What I learned about Microsoft is that it already has three and a half heads. It's now poised to add a fourth. Since I've been purging myself of MS over the past several weeks I don't have any idea where I put my notes that told me exactly what the acronyms are for the three of Microsoft's 10 billion dollar businesses. But the biggest point, which led to my leaving the Kiretsu was the realization that Microsoft is a technology galaxy of loosely federated product groups. This has specific advantages when it comes to providing a platform, but specific disadvantages when it comes to building product stacks, and product stacks are what sell in the enterprise software space. For my tastes, and for the tastes of every SAP, Oracle and IBM customer under the globe, Microsoft's product strategy doesn't have the requisite focus. I won't belabor the point.

What's shocking about this particular idea is that it is a startling admission that MSN is a failure. I can't imagine anything more or less than a huge, huge culture shock.

What's obvious is that this is an attempt to kill Google, or replicate parts of its business model. But Google's business model is actually besides the point. Advertising revenues through online are really means to a Google end which is to build a single computer for the world and world scalable applications for that computer. Microsoft is just adding functionality to a massive installed base that has never and will never talk to itself the way Google customers talk to themselves. Which is to say Microsoft enables software gadgetry with no overarching design. Adding Yahoo to the gumbo.. well it only makes discrete sense not holistic sense.

So Microsoft is going to grow another head. I think it makes that which is ugly even uglier, and that which is dangerous more dangerous still. It will show Microsoft's ability to fight in yet another domain and it will put another lovely bushel of mulch into the compost, but don't expect all the fermentation to yield anything smooth for another four years.

Cognos To Disappear Inside IBM

I have three gut reactions to the news that Cognos is going to be acquired by IBM for 5 Billion.

1. OK it's official and now there is 'nobody' left standing. 'Nobody' meaning Information Builders, Microstrategy, SAS and the open sourcers. SAS is not going anywhere and was never the same kind of player as everybody else. Microstrategy is probably not an attractive target, nor is IBI. They will not be assimilated and will basically die off, like everybody else.

2. Cognos is going to disappear inside of IBM. It will be absorbed into the Blob of Websphere and will be IBMized into marketing oblivion. Very little of Cognos branding will survive once the product integration is complete. It will find itself barely recognizable amongst other brilliant IBM technologies like Cube Views etc. The good news is that if the DB2 engineers get their hands on it, Report Net will ultimately and finally work as it had been advertised.

3. Microsoft stands alone, again. We (now that I've sold out to the dark side) have a lot to prove in the enterprise space, but what we won't have is a huge product integration story to have to sell. That is actually a big deal in the enterprise space and the FUD works in Microsoft's favor in the short term. Whether or not MS reps can and will capitalize on this moment is anybody's guess. Even though I'm on the MS extended squad, I'm still from Missouri.

Outside of that, I will speculate that this finally represents the top of the bell curve in an industry that has taken too damned long to mature. I walk around with my jaw dropped when I think about what a focused industry might have done with BI had any of these monster companies done this acquisition work in 1999 when they should have. Then again, maybe you're nothing until you're a 5 Billion dollar company as far as the IBMs of the world are concerned.

Which is why everybody ought to be scared as hell of Google. One of these days, the 20%ers are going to recognize what they can do with Gapminder + BigTable. The lightbulb is going to go off in their heads and they're going to see how a bunch of their technologies can be put together into a zero footprint xPM stack. If they do it within 5 years, then they will make all of the big BI acquireres look dumb. What Google doesn't realize is that their ability to structure data from multiple sources is a huge strength considering that they've built an empire with data people have volunteered to them.

What is BI, in the end? It's gobbling up and spitting out very small chunks of data each cell of which has an extraordinary amount of metadata associated with it. Nobody has done that in parallel scale. The entire paradigm is ripe for re-invention. And this time, I hope they put some nicer graphics into it.

More Solutions 2007

Several  impressions and reflections on the last Hyperion conference ever.

MOLAP Wars
The good news is that some of the best Hyperion folks on the planet are still with the company. That is to say some stellar Essbase product managers and developers are on the team and not going away. That includes guys like Marciante, Nader, Hite and Tai as well as very capable new folks I've never seen before. The even better news is that it's Essbase again. Not Hyperion Analysis Services Enterprise 9X or some huge mouthful. So when it gets right down to it, the Essbase we know and love is there in all its beauty.

The bad news is that I didn't have time to sit down with the Essbase guys and talk about futures. It was too crowded. But I did see some slides that said 'Trickle-Feed' and some that said 100s of Gigabytes OK. Since I have heard last year that the aim was for ASO to handle a full set of BSO functionality, eventually, I will have faith. So there is no doubt that Essbase has a future.

In fact, I think the case will be made abundantly clear going forward that the Essbase-centric paradigm of OLAP development will continue into the future. It is the platform for the future of BI. The Oracle-Hyperion people know exactly where the strengths and weaknesses of OLAP are, but Essbase still  has got the advantage. That advantage depends on the combined company's ability to reposition the meaning of BI.

Yeah I said it was murky, but that's because the presence or absence of Essbase is not necessary to make sense of the application stack. It's not until you get into the details of how capable these applications are going to be at the individual level that you begin to recognize how key Essbase is to them. That's why Oracle is challenged to present BI in a whole new context.

The New BI Paradigm
People have been saying 'drill down' and 'slice and dice' for two decades now. And you would think that they would have gotten tired of that. Oracle is going to show them how retarded that is with their new technologies going forward. The reason is that now that System 9 is the big dog on the block, the APIs to Essbase are going to support a class of workflows that have never been so integrated before.

What the Hyperion engineers know, better than perhaps anyone on the planet, is how analytic workflows run. They have been doing complex OLAP for many years. They understand the cyclical nature of OLAP queries and iterative analysis, planning, goal-setting and review. Now this is what they are calling the BI paradigm, and what is new and exciting is that they are going to be exposing more of the data mining features of Essbase (that have been around for years) into the BI paradigm.

Obviously Essbase has always been able to handle multiple scenarios of data. There isn't one real application that we deliver that doesn't have at least three. What's new is that they are going to bring forecasting into the foreground with tools that will allow end users to drill down into a set of products (for example) and mine the past data, as well as the past generated data and automate the process of selecting heuristics, employ them and expose those heuristics for adjustment.

This means in Hyperion Planning, it will automatically fit curves at every level and generate a base forecast for you. It will crank those through and allow you to select for factors that effect the outcome, and recycle. This is way more intense than "take last year's sales number and add 10% and spread it". This is more like figure out which products have the most affinity with this new product we have and predict its sales, calculate the profit and forecast the profitability. Now take into consideration my promotions budget, and reforecast sales volume. Now what's the profitability?  Now lets cut costs here and reallocate to tweak the price against this demand curve. What's profitability now?

That's the new paradigm of BI. It's not just reporting, it's smart extrapolation of financial and operational history and application of data mining for the individual user forecasting. It is a revolution just waiting to happen.
 

Merger News
One of the comments that struck me in elaborating details of the merger was that Oracle was impressed with the amount of work that Hyperion produced per capita. As a relatively small company, the extent to which Hyperion absolutely dominated BPM was stunning to the Oracle folks. In addition, I've heard tell that they are staffing up their pre-sales and post-sales consulting groups. So all that points to more reasons to believe that as a wholly owned subsidiary, Hyperion will not lose, but possibly gain headcount within Oracle.

Crystal Ball
I've been in this business for a long time and even I had a hard time following the presentation by the guy from Decisioneering. What I can tell you is that a whole new era of risk management is coming to be integrated into the planning cycle. That is because unlike the products I've seen before with SAS and SPSS, this stuff is totally integrated with Excel. Excel makes the world go around, and Essbase connects that world with the universe of business data. But I just learned how to do a 10,000 case Monte Carlo Simulation and a 10 factor sensitivity analysis in under an hour. I saw curve fitting and confidence intervals put in place and I saw a 4 year forecast built in front of my eyes that was simply brilliant. And guess what? They did it at Ludicrous Speed.   

MapReduce
By far the coolest thing that I learned at the conference, aside from the fact that I've still got my presentation chops, is that the guys at Tellme are using MapReduce to get massive data into Essbase. What does that mean? It means that I now have a framework for integrating VLDB data into an analytic platform.

Tellme basically generates 100GB of data daily. Huh? What? That's what I said. 100GB daily. it comes from processing 5 million telephone calls which generate on the order of 150 million detail records, PER DAY! They've got something like 30 MySQL master / slave arrangements in a two-tier MapReduce network. They issue a query through this very cheap, open source network of PCs in about 4 minutes. That's right, in 4 minutes they have processed all of the aggregate data they need into an ASO cube. That, my friends is scalability.

SmartSpace
The second coolest thing at this conference was SmartSpace. Basically, this is the future look and feel of BI. All I can say about it is this: Forget portals. Forget Web 2.0. It's all about the widgets. SmartSpace is, make no mistake about it, a killer app for Windows Vista. I could not take my eyes off of the screen. It was absolutely gorgeous. It's a single signon, intergrated IM, multi-canvas workspace for System 9 that deploys sharable widgets for every kind of display and control available to the Hyperion stack. It turns a PC for BI what Photoshop does for a Mac. There are auto-hiding toolbars, floating controls like a calc script initialization widget, for example. I didn't think anything would make me take a second look at all this again, but this is the front-end of the future. Oh by the way, it also works like a thumbdrive. You 'mount' a SmartSpace and double click and you get the hierarchy of System 9 Workspace right in the Windows explorer. I'm in the Beta, buddy.

The Birth of EPMS

The general session of the Hyperion conference here in Orlando is over and ... I like it. This one was light on hype, and pretty much got straight down to business as it was dominated by the presentation of Thomas Kurian who will inherit and run the Hyperion development organization.

First thing first. I've been pretty positive that the merger was a good one, and Hyperion president Godfrey Sullivan communicated that the vision of the two companies was very much aligned. He said that the entire thing went very smoothly and you can be fairly sure that he was straight about it. The whole thing went down in four months and now Hyperion is officially a wholly owned subsidiary of Oracle.

What is very clear here is that Kurian is all business and has a rather deep understanding of the products and their capabilities. But there is a very subtle genius in this acquisition which is going to make a fairly impressive change in the marketplace. But let me give you one piece of background that I discovered yesterday. The Hyperion salesforce is safe. This is a big deal. So it appears to me that Oracle is big enough and healthy enough not to go headlong into 'synergies' and cost cutting and that they probably won't until they have significantly changed the market. Even so there's a whole lot of product to sell, which is the genius. Kurian's product strategy is clear. He's going to a modified Six Sigma business model that boils down to the phrase "Insight into Action". There are going to be vertical suites for all of the major industries, and all of the major products are going to be represented. The secret sauce? Contextual navigation.

You may have heard that before. The whole idea of the Hyperion Portal which died somewhere back in 2000-2002 was all about that. The problem was that even though it was a great idea, nobody with Oracle Financials was going to manage their business from the context of new BI technology. Kurian's emphasis on drilling down to application-level detail and changing assumptions in the ERP and CRM systems shows that this is how customers prefer to manage their operational business. It always has been a difficulty for Hyperion to sell end-to-end, not because of a technical limitation but to a lack of an operational integration story. Those days are over.

It's All System Nine, Baby
Oracle buys System 9 completely. System 9 will be the framework for all of the BPM going forward. It is now recast as Enterprise Performance Management Systems. Before long, everybody will be calling this the Management System. It integrates financial reporting, operational BI and CRM+ERP. The vision is bold, it's sensible, it's workable and it looks damned near inevitable. It's also going to cost a lot of money because all of these systems are going to have a lot of baggage. However, there's a lot of time to work all that out.

But here's my quick sketch on it in terms of emphasis.

Hyperion Winners:

• System 9: New Industry Standard
• Planning: Huge win
• Crystal Ball: Big Win
• Hyperion Scorecard: Win
• Hyperion Strategic Planning: Win
• Hyperion FM: Big Fat Win
• Hyperion MDM: Big Fat Win

Oracle Winners:

• Oracle BI EE: Big Win
• Oracle Alerts: Big Win
• Oracle BI Packaged Apps: New Industry Standard

Muddy:

• Hyperion EIS
• Hyperion Essbase

In the new era that's coming, all of the analytics from every cross-platform thing is going to trickle up through System 9. All that's left is to figure out a new Semantic Layer and Oracle BI Foundation. This is where it gets muddy. Obviously Essbase is the keystone of all of the Hyperion Applications today. But at the enterprise level, you don't care about any of that in terms of presenting the solution to customers. Depending upon how smart Oracle gets in that Semantic Layer (dimensions, hierarchies, metrics, calculations.. etc) pipes through to the BI Foundation can go any which way. In this regard, sure we'll be serving up multidimensional models for analysis and reporting, but they won't necessarily be optimized for Essbase. But that will all be internal to the new development group. I have a lot of confidence that Essbase won't be going away as a product, but in the context of EPMS, it's just a small fraction.

Basically the number of integrated applications available has just skyrocketed. Making all of it work together is going to be a lot quicker and smoother than most people imagine and making sense of the products is going to be less complicated than I thought. The big vision works, the integration is on a fast track, this train is moving.

Why Essbase Looks Muddy
The architectural look of this whole new stack boils down to something fairly straightforward. All of the major Hyperion apps work through a middle tier which is a combination of a relational repository, a Java server and one or more Essbase Cubes. But once the semantic layer is muscled out, you may be able to swap Essbase out for Oracle. That's going to take years, and lots of extentions to OWB, but systems like Star Analytics' SIS could facilitate all that. Now that Oracle owns Essbase, we'll find out price/performance wise if that's a good direction to head, but I'm convinced that the two companies are going with the market leading technologies as a guide. I don't know if Express is behind Oracle BI EE, but Kurian was talking about Essbase as a source on the same level as Hyperion IR and Oracle BI EE. Meaning that pure Essbase apps may be just considered point BI solutions which don't play a major role in the EPMS world going forward.

So I conclude that Oracle is putting behemoth resources behind this. Their combined R&D with Hyperion's, which was no joke, is going to put huge pressures on everybody else in the space. The story already works and already looks architecturally sound. With the combined customer base, the possibilities are very broad and deep.

Fire or Poison?

Deca-BDE is a controversial chemical. According to the SeattlePI:

Stung by the Legislature's groundbreaking ban of a controversial fire retardant, the chemical industry isn't giving up.

Even with Gov. Chris Gregoire's approval all but assured -- she's scheduled to sign the legislation today -- it still didn't extinguish the chemical manufacturers' fight. They've run full-page ads in the state's largest newspapers. They wrote a letter to Gregoire urging a veto. Since 2005, they've spent more than $220,000 here lobbying against a ban.

Why the big spending? Because there is potentially more at stake than prohibitions on flame retardants called PBDEs in Washington furniture, televisions and computers.

A ban here could be the beginning of the end for PBDEs, or polybrominated diphenyl ethers. It could mark a move toward more aggressive protections by local governments -- at a time when the federal government is largely ineffectual in its regulation of long-used but potentially dangerous industrial chemicals.

"The larger story is that there is a highly organized and very well-funded effort to change the way in which chemicals are regulated around the world," said John Kyte, spokesman for the Bromine Science and Environmental Forum, a trade group representing PBDE manufacturers.

One of these days, if a guy like me gets rich in the business, I would have a business model that would put together publicly accessible knowledge bases that would allow environmentalists, industry trade groups, academics, legislators and the general public to look at the same data at the same time and make more scientific conclusions than they are doing now.

My jaundiced eye says this is basically going to come down to some study saying that the increase in the number of cancers in some lab mice goes up x% when they stuff Deca down their throats, and on the other hand we're going to see a repeat of the empty crib commercial because little Johnny's pajamas didn't have fire-retardant. Then somebody else is going to charge that Johnny can't read because the flame retardant in his pajamas is giving him brain damage.

In my business, I make systems that would allow the manufacturer of pajamas (or furniture, or whatever) identify the commodity in his supply chain which is controversial and allow him to swap that out and run a simulation of what a new and improved 'organic' product would cost. I'm talking hours here. With blog-speed, this could become its own marketing campaign. The trick is to make the market that fast.

Even though I'm caustic about environmentalist thinking (mostly because it is ignorant of manufacturing processes and puts an arbitrary infinite cost on those things it wishes to avoid), I have no problem with their end goals. But it should fall to the consumer to have and know the alternatives. I want to be able to go to Dollar Tree and buy pajamas for a buck and know that they are not fire-safe, or that they are fire-safe with DecaBDE for 2 bucks or that they are firesafe with the new post-Washington State controvery Sierra Club approved fire retardant for 3 bucks.

Open the process and break the interest group oligopoly on product R&D. Or sleep naked and buy a fire extinguisher.

Boot Time

I haven't bothered with Windows Vista, and with any luck I won't have to for several years. XP Pro is as good as it gets under Microsoft, and I'm not taking the home network any further along the Microsoft upgrade path. We'll go to Mac first. I refuse DRM, and XP does everything I want.

Over at ZD Adrian Kingsley-Hughes asks why long boot times matter. I essentially get his point, but what he overlooks are the number of software installs, not to mention seemingly random software patches from MSFT that require soft boots. Also, whether we want to admit it or not, there are many companies who require that people turn off their PCs in order to save energy. I know that's dumb and such businesses are on the verge of collapse, but...  But the biggest reason long boot times are annoying is because they don't have to be.

Leaving aside the brilliance of Knoppix and other Linux on a CD rootkits that can embed very cool Windows utilities, we know that there are a whole lot of extra things that go on during the boot cycle depending upon what software you have installed on your machine. Why not just push those back to the post boot phase? When I'm logging in I know that a whole lot of profile driven stuff is loaded onto the machine. How many times have I sat waiting to get a wireless connection in an airport hoping the last few minutes of battery life don't slip away before the hourglass over my toolbar disappears? What's going on here? Something. There could be a class of applications that I want to be able to use right away just after boot, and just like power consumption priorities, I ought to be able to determine readiness priorities.

The Google No Brainer

Arizona State goes from spending 500k/year to zero. Google's threat to Microsoft Exchange is so potent, that it's not even funny. Check out this story:

When Adrian Sannier became the university technology officer at Arizona State University in the fall of 2005, he was handed a simple yet monumental brief by ASU president Michael Crow: make IT the primary driver in Crow's ambitious "New American University" project. The goal is to raise ASU's academic standing while increasing the number of resident students from around 65,000 to 90,000 by the end of this decade.

The first step in the school's new IT strategy was an alliance with Google's Enterprise Solutions division. Last October, ASU became the first large institution to deploy Google Apps, a comprehensive suite of productivity applications that includes e-mail, search, calendars, instant messaging, and even word processing and spreadsheets.

Initially offering new e-mail accounts based on Google's Gmail service (but retaining the "asu.edu" domain) on an opt-in basis, Sannier and his team found that students were making the switch at the rate of around 300 per hour. Today, more than 40,000 ASU students and faculty have made the switch, and he expects to shut down the University's in-house mail servers near the end of this term.

Since the e-mail switch-over, Sannier has been rolling out additional applications including calendar (which users can now share online, a capability the old university system didn't have), IM, and search. Within the next two months he expects to offer personalized home pages as well as online word-processing documents and spreadsheets based on Google Apps.

The cost to ASU: zero.

Ray Ozzie better get busy integrating his Groove stuff into the next gen Microsoft collab-platform. Oh wait, it's done. Who knew? Who'll pay?

 

AS/400 OMG!!

I can remember System 32, OK? Then came System 36 and I think there was a System 38 in there too. But I can really remember when IBM announced the AS/400 as the godbox of all end-user computing. Can you believe that I got certified in LU 6.2?

Because I'm so long in the system tooth. I've had the pleasure of getting data from an AS/400 source and putting it into a client/server interface. You realize of course that people had to be convinced. There are fewer things more aggravating that having to put up with the nonsense of AS/400 gurus as they begin fulminating about the inferiority of every other API and what a waste of time it is to replicate data from the 400 onto another platform. I patiently explain to them that their bosses bosses boss wants historical information charted in color and they give me one of these grimaces usually found on the face of Michael Moore when the security guards escort him from the premises. Yeah but I'm right, they smirk to themselves.

It is at times like these when I think of my girlfriend's sister's boyfriend who married her and moved up to Stockton, California. He was the god or RPG and drove a Mitsubishi Starion. Whoa. OK I exaggerate, it was a Sapporo. But I still couldn't bring myself, ghetto child that I was, to give up life in Los Angeles, even though he had a house and made 36 Thousand Dollars. That was 1980, and I had to hold on to my dreams. I knew RPG, in fact I knew RPG II. It was torture, I couldn't submit to its petty demands, and I wasn't ready to get married.

If you happen to find yourself staring into the wall eyes of an AS/400 guru, do as I do. Count backwards from 10 slowly and tell them how much respect you have for the innovative microcode of the AS/400 instruction set. Then turn around and run like the wind. If you actually have a mandate to work with this dweeb then keep a stiff upper lip and do what you need to do to get your data. Prostrate yourself.

Seriously, you should do whatever possible to get all necessary AS/400 data into SQL Server. You should get a nightly job to push transactions from the 400 onto a separate NT server and then use the ODS on NT.

Generally a customer will have green screens running against the 400. I know there's something of a conceit that the 400s are big hardware, but there really are no 400 databases out there that are too large for MSSQL. The 400 is really all about capturing transactions and they can't afford to run ad-hoc queries.

The best way to manage it is to add a native query job on the 400 side and dump that into a flat file on the 400 in a new outbucket directory. You can then FTP or map to the outbucket and initiate a transfer from the NT machine.

This has three primary advantages.

1. The 400 administrator will have control of the query and when it runs and where the output goes. He can optimize all that on his side. Nothing leaves his control.

2. The only thing that is initiated from the outside is a file transfer. That shouldn't interrupt anything on the 400 side.

3. Once you have the data on NT you can Perl it, DTS it into MSSQL and manipulate it any way you please. You get out of the 400's hair and you're essentially free.

The important thing is to make sure you get everything you need in that one big fat nightly query. Chances are you won't be able to get master data but ask anyway. Your better chance is to run master data from the data itself.

Now go have a chai spice. You win.

More on the Merger

I've had a chance to ponder as well as listen to other folks talk. Here are some of the best ideas of mine and theirs.

1. In the short term, nothing will happen.
We always forget this, and then reality sinks in. Nothing significant is going to happen in the next 4 to 6 months in the product lines. People will be getting over the shock in the marketplace and in the organizations. Associated companies will be getting their bearings and the very few people who knew about this blockbuster of a deal will surprise us again. From what I hear, this secret was kept very well. Although it has been rumored for years (and I wish I could find some of my old comments and speculations on those rumors) it blew everybody away.

2. Current Hyperion customers might upgrade, just to be on the safe side.
If you're sitting on an older version of Hyperion software, chances are you'll upgrade just to buy yourself time. This means busy work for the consulting sector. It's easy if you're already in System 9 of Hyperion, and easy if you're pure Essbase. But if you have a mix of Hyperion products moving all of them up will take a little doing. Still this is the only downside hedging I would expect current Hyperion customers to do.

3. Brio guys aren't breathing hard.
Somebody needs to really sit me down and explain what it is that Business Objects has that Brio doesn't. Whatever BO might have, or Oracle Discoverer might have the Brio folks aren't sweating it. The Brio customers were loyal in their transition to Hyperion and they weren't disappointed. Oracle rebranding doesn't seem to concern them much at all. That's interesting.

4. Product Integration takes time.
Peoplesoft people are still calling Peoplesoft Peoplesoft. So for the time being, it's entirely reasonable to expect that Hyperion will be a 'wholly owned subsidiary' as far as customers are concerned. It's going to take a while for Oracle to learn what Hyperion customers like about Hyperion and what they don't like. So in order to do product integration smartly, they'll most likely run out the current planned release schedule on the Hyperion suite and then start looking to merge. Again, I'm thinking Data Integration Manager (Informatica OEM) and Analytic Integration Server will be the first likely casualties to all of Oracle's data pump tech. Then again, current customers must be served. Perhaps by Solutions 2007 in Orlando in a couple months, we'll hear a few more specifics.

5. IT objections mitigated.
Already, dithering customers have committed to Hyperion based on the fact of the merger. Oracle shops are now open to Hyperion BI. This is bad news for Business Objects seeds in particular. This is one issue that has been the bane of my existence, being one of those rare individuals who thinks both relationally and multidimensionally I have suffered endless tirades about what MOLAP can't do, by people who have no hands-on. When I explain to people that Essbase is a modular database technology that has Hybrid, MOLAP and Aggregate Storage, that it is partitionable both logically and physically, they still find it hard to believe because nobody thinks of Hyperion as a database company. Now instead of showing them case studies with 500 concurrent databases or 50,000 concurrent queries I can just say two words to cease all hostilities. "It's Oracle".

People forget that Hyperion Essbase was Teradata's choice for marting in their Active Data Warehouse strategy. So I will reiterate my trash talk against MSTR. You guys are in for a surprise. The database guy in me is just giggling silly, and I haven't been so pleased since IBM OEM'd Essbase for DB2OLAP.

6. Verticals
Back when I was talking with people who oughta know, the idea for the killer Essbase apps was a certification program. That is to say Essbase had (and still has) the capability of being a brandable kind of analytic engine, as in Analytics powered by Essbase. I was personally involved with the deal that almost got Essbase embedded into Siebel, and I've since heard where the strengths and weaknesses of the chosen technology are. In any case, Hyperion has dreamed of moving towards verticals and now with Oracle's coverage of vertical markets, there are opportunities galore for Essbase to be exactly that. Of course it will require some thoughtful people to make it happen, but the Oracle introduction can be there. The technology is ready. Today.

Oracle + Hyperion = Gorilla

Oracle is buying Hyperion. SAP is buying Pilot. Cognos and Business Objects stand alone.

This is huge. I'm still overwhelmed by the implications.

I don't doubt anything this article says:

Hyperion and its popular Essbase product, has "almost no overlap" with existing Oracle products, Phillips said. But, there is overlap between the PeopleSoft's performance management products Oracle acquired in 2003 and Hyperion's products. The PeopleSoft products will become part of a new enterprise performance management division, Phillips said. Oracle has had a "small" performance management product that "wasn't that pervasive in the market," it considers Hyperion a complementary, best-of-breed vendor, he said.

Instead, Oracle has focused on building out the underlying infrastructure required for BI, he said, and the Hyperion acquisition will give it critical analytical applications to layer on top of its data management and BI technologies. Specifically, the acquisition gives Oracle an enterprise planning system, a financial consolidation product, a powerful OLAP engine and a "dedicated" field sales organization. It also believes the move will help it in its battle's with rival SAP. Many SAP customers use Hyperion, Phillips said, and Oracle is achieving "critical mass" within SAP accounts.

"Now Oracle's Hyperion software will be the lens through which SAP's most important customers view and analyze their underlying SAP ERP data," he said in an earlier statement.

From my perspective as a consultant it's all good. I'm confident that Hyperion's BPM apps are going to go through pretty much as is. It puts the Informatica OEM in a wobbly position, and it will probably delay integration of the more recent product acquisitions like Crystal Ball and Upstream. The integration of Hyperion MDM works technically but it will have to compete with Oracle's Customer Data Hub. I don't know how successful that product is. MDM hasn't sold well on the Hyperion side although everybody who gets it loves it.

There's basically no question that this is excellent news for us core Essbase folks and for those applications based on Essbase. The war between Essbase and Oracle's Express is ages old, and while performance wise, old hackers like myself and those on the Express side could get into lengthy debates, there was no question the Essbase was the more interoperable product. Now that it is plug-compatible with the new Yukon DTS, Oracle will have another way to kick MSFT around.

I say this is really bad news for Microstrategy. When the Oracle technical folks recognize what they'll be able to do with Oracle + Essbase, Microstrategy's whole high end BI story will crumble. In many ways this is a technical match made in heaven.

What's iffy is of course matters of marketing and product synergy. That is to say if I were a product marketing manager at Hyperion, I would get that resume out. Sullivan, in his letter to customers this morning declared in about as obvious as is possible to say that Hyperion sales reps have some security. So that whole organizational end of Hyperion is in jeopardy.

There have been a lot of us who have been worried about Hyperion development, ie without Gersten where would the engineering organization be. I think that Oracle would make a huge mistake to do anything but hang onto a lot of that engineering staff and keep them happy. Obviously System 9 integration will become a priority as product feature planning levels off. The good news is that there are not many major flaws in System 9.

Hyperion's tech support has taken a dive in recent years. The Oracle Developer Network will be a huge boost for that end of the business. If I were smart, I'd probably spin off a training business and start raking in dough for anybody out there who wants to become certified.

There's a lot more to think about, but on the whole there's basically one thing to know. There is now a new gorilla in the BI and BPM space.

The Clocksite

Here's an idea and a term to coin. Clocksite.

A clocksite is a website put together to establish or debunk any economic or social theory that makes predictions about some important occurrence or keeps track of some indicator of public interest.

The term originates from the now familiar practice of billboard advertisements which are calibrated at a certain rate to estimate some important metric of public interest, like the number of smokers who die from lung cancer, the estimated number of dead in war, the number of dollars in the federal deficit or most famously the 'number of minutes to nuclear holocaust'.

A clocksite however, would employ multiple metrics and varying types of analysis and would allow for a more sophisticated and nuanced analysis of the issue at hand.  It might be open for collaboration or some officially sponsored by a partisan group.

As more open source tools become available to present analytic data on websites, we should expect more clocksites to follow the open publishing models that are now being experimented with by the folks at Swivel and ManyEyes. I have been interested in this matter for many years as I have been involved with building such sites with proprietary tools for business.

OODA Diagram

Saas vs Appliances

Don Linstedt has been watching something that I think I should be watching, and that is the emergence of appliances into the enterprise applications space.

Every so often a future suggestion that I've discussed (made by more than just me) actually happens. Now I'm not the kind of guy to normally say "I told you so." However on this occasion, I feel it's important to announce that the market is changing, dramatically - and that software vendors NEED TO TAKE NOTE!!! EAI is now available in plug-and-play appliance format. In this entry we'll discuss what this means, and how it will affect ETL/ELT, EII, and EIEIO (old Mac Donald had an appliance... E-I-E-I-O).

Cast Iron Systems announced in 2006 their EAI appliance. They've made some incredible enhancements, and were recently reviewed again: http://www.infoworld.com/article/07/01/26/05TCcastiron_1.html?APPLICATION+DEVELOPMENT

While the review was neither favorable nor unfavorable, it highlights an important point I've been making about appliances: they will become a combination of hardware and software vendors, specifically ETL/ELT and hardware, EII and hardware, EAI and hardware, and eventually BI and hardware will hook up. Why not? Hardware increases sales, increases price points - at the same time, it can increase ease of use.

He's right. And moreover if these appliances work as advertised, then they will be the counterbalancing force to the ever increasing complexity of tiered services. That means IT can take control where it was losing ground to complexity and simplify at the same time.

My experience is telling me that IT is having increasingly difficult times managing all of the tiers required for the web-centric enterprise. When a repository is not on the same subnet as an application server which is located somewhere else than authentication, stuff gets crazy and nobody wants to debug that. We get a cascade of bouncing servers because the people who installed whatever ear file aren't around any longer or nobody quite knows why some rows are locked in a database whose name doesn't really explain which application is using it. Let's not even get into the prodigious number of bot users which administer tasks for each and every one of the applications dancing in unison. It has been a nightmare, and because of that nightmare I had predicted that more and more companies will find SaaS an increasingly irresistable option.

BUT..

This appliance technology promises plug and play enterprise applications and that's even more irresistable, especially as vendors collaborate. I'm excited.

MDM + CDI Convergence

I haven't been writing so much about technology and my business. That's about to change, because there are actually some very interesting things that are beginning to happen that I may be taking part in. So let's talk shall we?

The first bit of news is that Hyperion has invested into Purisma. Hyperion is the mothership of BI and BPM and it's where I've cast my lot for over a decade starting with Arbor Software in the fall of 1996. I am, by the way, one of those rare creatures I call a 'double DBA'. I understand both relational and multidimensional database structures. My guess is that I could extend that a bit both ways if I were so inclined. Like most old heads, I have some experience in the hierarchical database technologies of yore. And like most new jacks, I have some experience in object models and object databases. Unlike most people I have a whole hodload of experience cobbling enterprise systems together in the real world under the creakiest and most robust IT architectures. I can tell you stories about hybridized VM dispatchers to 64bit parellel databases that would blow your mind and stories about Excel clouds on OS/2 & token-ring that would have you reaching for the Pepto Bismol. Suffice it to say that I've earned my bones as a data architect; it's a hell of a job and I love it. 

Now here's the deal with Purisma. It's got Bob Hagenau. Bob was the guy behind Acta, and Acta was the coolest ETL on the planet a few years back. It got absorbed by Business Objects because Hyperion was slow to the draw and shy of acquisitions. If it's doing nice things today, I have no idea. But it was the key to getting data out of the 'data jail' of ERP systems, especially the high security prison of SAP. As such they were Sapphire certified and even understood Z tables and weirdness in the SAP data model that most ABAP coders mumbled about and SAP tried to hush away. So I have some confidence that Hagenau is not a time-waster and that when it comes to complex data, he'll make some of the jagged edges smooth.

Purisma's business is simple. Master customer data. Hyperion's business is simple, monetize and benchmarkify everything for management. Those are the simple explanations behind the two of the three three letter acronyms that are most significant in this piece. CDI is Customer Data Integration. That's Purisma. BPM is Business Performance Management, that's Hyperion. The killer app between the two will be customer profitability across all channels.

Now there have always been customer profitability models and they're not too hard to build. What's hard is that what your department calls a customer and what marketing calls a customer and what finance calls a customer are three different things. And all of you have incompatible systems that don't let your company decided once and for all what a customer is. In comes the third three letter acronym, MDM. MDM is Master Data Management.

Hyperion purchased a leading MDM vendor a couple years ago. They have a hot product which is now called, drum roll please, Hyperion MDM. What it does is centralize name calling in your organization. Huh? What? Name calling? OK think of the product you are familiar with in your company? What is it called? OK now what's the serial number? What's the product code? What do the engineers who built it call it? Who in marketing changed the name? When? How did they spread the word? What do people in accounting call it? What's the code for it in the accounting system? What about the inventory system? What about when you OEM it to another company? Complex? Well it's your product, you ought to know all of these things. You don't, because you haven't managed your master data. You don't have a chief name caller in charge who has octopus arms that coordinate the names of everything in every system. MDM gives you that ability.

In this regard, CDI is a very particular MDM application. It allows you to understand all the names of all your customers no matter which channel you touch them through. Something in the back of your head says, didn't Siebel promise this years ago with their CRM systems? Sure they did, that was marketing, this is technology. My nickel says Purisma does the job way better than Siebel does, because Siebel is not selling customer data management, they're selling CRM. If you get CRM but not a single view of the customer from Siebel, you don't send it back. If you don't get a single view of the customer from a CDI vendor, you send it back. Different incentives, different results.

So Hyperion, who gets the BPM theory and builds some pretty damned good products, is being smart in seeing the future of CDI and investing in Purisma. But are they hedging against their own MDM product? No. MDM is expensive and takes some doing to implement, it's not easy to sell either. But so long as Hyperion can cover the base of CDI and MDM through whatever agreement they're making with Hagenau and company, they are not going to miss the opportunity to position their company as knowledgeable about the space, and so their money is where their mouth is.

All that remains is for field commanders like myself to staff up some troops and actually battle test these technology weapons against the enemy of corporate information entropy. I'll let you know.

What Happens In Google Stays In Google

Here's what I think about the Google purchase of YouTube. I think that as soon as Google hosts all the YouTube content, there will be no way of knowing the monetization potential of any piece of contested material without Google's participation. In otherwords, everybody who guesses how many hits, and therefore how valuable some piece of content is in terms of eyeballs will have to ask Google and Google would have to be compelled to divulge that information. Then what happens? Google can make a quick calculation of what it's worth, in terms of ad revenue, to deal or no deal. If it's no deal, Google cuts the piece of content loose and its value drops to zero.

The trick is having the entire infrastructure of the Googleplex which would host the valuable content. That makes them the equivalent of the broadcast TV networks, and if entertainment lawyers are dancing naked because they think that YouTube is a litigation free-for all just waiting to happen, they will quickly find out how difficult it is to sue the network.

All of that seems to be a big fuss over content that's primarily only valuable because Google broadcasts it, some very small fraction of which has any commercial value in any other form. Only Google has the free billions of equity required to do any of this so, it's essentially a win situation for them. It seems to me that this is something only Google could do because they're the ones investing in the Googleplex.

Three Hours Flat

One of the reasons I'm resurrecting my blog is because of continuity. I owe continuity to my virtual partners, even if they're tired of hearing me repeat myself. A boring familiar voice is better than silence, and as I've mentioned, I'm occasionally worth reading.

What makes me waste a little time on this blustery Wednesday is the fact that I have learned, within three hours of it becoming 'public', of Google Real Estate. I don't know how many degrees of separation I am from the leaking source - it doesn't really matter; I just find that to be a fairly amazingly short period of time, and I feel fortunate to be in the loop of know.

I've been thinking about explaining the basics of BI to noobs as well as senior managers whose companies have yet to exploit the economies. In answering an imaginary question of where the industry will be in 10 years (actually the iQuestion was 'Where do you see yourself in 10 years?'), I said that there will be four players. They will be Microsoft, Oracle, SAP and ... Google. I'm hoping to see myself with Google, the reason isn't obvious but I have mentioned it before:

Google does not charge the user. All of Google's software and hardware is at Google, behind locked doors, just like the IBM data center. To Google, you are a user, not an owner. You couldn't own the Google software because you would clog up Google's precious computers with your idiot code which couldn't possibly be as efficient and secure as Google's own code. They are masters of the algorithms and queue management. Google releases no product that doesn't scale to infinity. Google never gets a virus. Google is a free IBM datacenter. Google is the opposite of IBM despite all these similarities because its business model does not involve charging the user. Microsoft and IBM are the same because they charge the user. Microsoft makes you configure your machine and calls it freedom.

Furthermore, Google deals in human scale media. Google does not deliver compacted abstracted codes. Google delivers whole books, whole libraries of books. Google delivers satellite images of the whole planet. You are incapable of asking too much of Google, and their orientation is to deliver that all to you for free. Plus they take away all of the headaches that Gates delivered, the world of limitations of PCs. Clunky, insecure software written by anybody.

Google is one of the only computing products that hasn't bloated. Even Linux is running into controversy over kernel bloat. Pigs aren't flying, they're recompiling RPMs on their anti-Microsoft boxes. Same paradigm, same headaches. Free software approaches perfection, but there is no guarantee. Google guarantees, and that is why Google is rich and there are only two or three Linuxes left worth mentioning, all equally user poor.

But Google could charge the corporate user because Google has got what every enterprise desires: fundamentally unlimited scalability. They have the opportunity to license the way they do business and thereby revolutionize the IT business. Google is the ultimate ASP - they always have been, and they've been providing incredibly fast information for years now. Google can simply wait for every Tom, Dick and Ajax to write their little Web 2.0 app, and then determine whether or not it scales into a nice business model. They plug it into the Googleplex and voila.

It is not a stretch for me to see them doing this with 2 million spreadsheets, which is very likely what your corporation's finances are really running on despite MSFT's, Oracle's and SAP's protestations to the contrary.

So here's the question I posed to my BI noob interviewer. 'It takes about 15 minutes for you to find out whenever a car bomb goes off in Baghdad. How long does it take you to learn that your business has lost a customer?'. The answer to this question is generally sub-Google speed, which ultimately is unacceptable. The gap between real-time analytics is not so much technology as it is IT methodologies and captialization. This is something most companies would rather have, but not pay for up front. Some enterprising practitioners (like me) and some Google people could meet one day and come up with an ASP business model for enterprise BI. If and when that happens, MSFT, Oracle and SAP will find that they are a lot slower to the middle-tier webware market than they thought.

I am consistently surprised that people remain skeptical of fat clients. We are seeing 3GHz PCs with 100GB of disk and still people go Web. The hows and whys of that we can discuss later, but it is clearly a fait accompli. Whatever happened to port 80? It's now a thousand different ones for everyone's 'administrative client'. Client server has been swallowed up by very specific clients, ie browsers running interpreters. Whoda thunk? That being the case, applications development companies are essentially web development companies.

I have a hard time believing that people who built WebMethods or Vignette (nothing personal) are going to be developing the applications of the future. The narrow scopes of what IT organizations and (outsourced) consultants build these days unfortunately don't stand a chance against the big huge apps Google rolls out.

One final analogy. Once upon a time, Bloomberg relied on hardware. He had a near monopoly when it came to (bond trading I think). The smarts of that moved to PCs and the second revolution was exemplified in the day-trading bubble. (By the way, once upon a time the same was true of Metaphor DIS, specialty hardware to fat-client software to running on browsers in the BI space). Clearly the next evolution in tiered computing will be making browsers smart enough to do all fat clients do, cleaner. Now it's all about the scalability of the back-end. I have a hard time believing that the same decision makers who are gung-ho about punting to procurement departments when it comes to <strike>outsourcing</strike> staffing IT are going to think twice when offered the opportunity for 2 second response time without capital spending.

Mining the Mind of Steve Krause

Back in the days when I was complaining that 30 year olds were running corporations, the gang of us at Hyperion eCRM had to deal with the equation hype + money - common sense = success. At least that's the way it was in the days of irrational exuberance. As one who always likes to review history in the light of renewed appreciation or scorn, I find it fascinating to find out whatever happened to.. along the lines of my career path and pointed pontifications. It's for that reason that I started my little jag on Xerox History. In the meantime, as the F500 slouches towards real security, BPM and data mining, I have fun digging up data on my own industry.

Steve Krause is my latest find. He puts up a nice practical post on Last.fm and a competitor that I never heard of or paid attention to. It fits rather congruently with my 'Do As I Say' theory. Note that the ultimate judge of the appropriateness is aesthetic consistancy, an entirely human creation.

It also turns out that Krause was a competitor at Personify. It's amazing that they were able to burn through half a billion dollars. If those happy days ever come back, never give a sucker an even break.

Dynamic vs Transactional Engineers

Every once in a while I kick something over the Drezner's site. Especially when he writes on outsourcing which is one of my pet peeves. As a highly technical architecture level consultant that does hands-on work, I really don't blog about it as much as I probably ought to. I've thought that I could run multiple blogs and keep all those worlds from intermixing, but I think I may begin to blur that firewall too. So on the matter of the quality of engineering and technical talent from the emerging second world, I had this to say:

I think that this distinction between dynamic and transactional engineers is very useful and accurately describes what I see in the software industry as well.

Even when chinese and indian programmers are on staff in American companies, there are notable differences. You never quite know what you're getting until you sit folks in a room and start talking about the systems to be built.

Enterprise systems that are to make a difference in the productivity of the target customers are notoriously difficult to assemble, even when using simpler standard technologies that procurement departments are demanding be offshored. That is why companies like Accenture continue to make money in the American and international markets. The skills of management consultants that can do a tight handover to technologists are in high demand, but even so these are applications that tend not to be robust. Anything that takes more than six months to build will suffer from changes in the business environment, turnover in personnel and integration with other systems that themselves are being changed.

Again, the allure of cheap labor in this area is that SQL is SQL. Not necessarily so. It's actually getting more complicated to do these applications properly, primarily because of an attitude the 'best practices' can be built into every application. This means that a lot of abstraction of problems is done, and a number of experts who don't do hands on work are employed. All this is done at the expense of homegrown (meaning inside the client company) experience which is the great hidden expense of outsourced systems.

It comes down, in my view, to a decades old clash between management philosophies. Deming vs Hammer. The Deming method says to evolve the way people are working with technology and business processes. That's highly integrative and evolutionary. It means you have to do a lot of listening and translating. The Hammer schools says, throw out the old and make everyone start from scratch. That's re-engineering. Companies that get re-engineered outsource and offshore better than companies that evolve. Companies that evolve are more productive because the culture of the company teaches everyone what the focus of the business is. They can be more nimble, but there's a steep learning curve.

I'm from the Deming School (an old Xeroid from the McKinsey makeover under David Kearns) and have been in the enterprise sofware business for two decades. The Deming way is harder, and it's often against the best interests of management and technology consulting companies, not to mention software vendors, to evolve a good company to great, but the real loss is that so many American corporations are literally outsourcing their own quality improvement. They don't want to grow their own MBAs, they want somebody else's. This dependency is what both depletes American talent, and keeps consultants like me in new BMWs.

Meanwhile over here at Nissan, things are beginning to get very interesting as deadlines start to loom. While Toyota is poised this year to make 100k more cars than General Motors, I'm finding that the systems that are being built in the wake of Sarb-Ox may have a long-term payoff for American businesses. Because over here in this Japanese company, typical of a large number of companies I've seen, the spaghetti and spontanaity of financial planning and accounting boggles the professional mind. In my close circle, we have a term called 'The Official Ass'. That's where a lot of numbers are pulled from. The number of companies that do real demand planning are few and far between. Demand planning is very difficult to do with most kinds of accounting systems that companies have, and so companies pull numbers out of a collective hole in the ground. That is particularly typical of non-financial companies - that is companies whose business is not primarily the management of money. When your financial staff is considered overhead... well.

It's one thing to understand how to build this database and that database. Sure that stuff can be outsourced. Should it be? It really depends on whether you're building system to be building systems or building them to improve the way a company is run. The latter can only be done by dynamic engineers. I like to think of myself as one of those types, but its not very often that we are called to do all that.

Dynamic vs Transactional Engineers

Every once in a while I kick something over the Drezner's site. Especially when he writes on outsourcing which is one of my pet peeves. As a highly technical architecture level consultant that does hands-on work, I really don't blog about it as much as I probably ought to. I've thought that I could run multiple blogs and keep all those worlds from intermixing, but I think I may begin to blur that firewall too. So on the matter of the quality of engineering and technical talent from the emerging second world, I had this to say:

I think that this distinction between dynamic and transactional engineers is very useful and accurately describes what I see in the software industry as well.

Even when chinese and indian programmers are on staff in American companies, there are notable differences. You never quite know what you're getting until you sit folks in a room and start talking about the systems to be built.

Enterprise systems that are to make a difference in the productivity of the target customers are notoriously difficult to assemble, even when using simpler standard technologies that procurement departments are demanding be offshored. That is why companies like Accenture continue to make money in the American and international markets. The skills of management consultants that can do a tight handover to technologists are in high demand, but even so these are applications that tend not to be robust. Anything that takes more than six months to build will suffer from changes in the business environment, turnover in personnel and integration with other systems that themselves are being changed.

Again, the allure of cheap labor in this area is that SQL is SQL. Not necessarily so. It's actually getting more complicated to do these applications properly, primarily because of an attitude the 'best practices' can be built into every application. This means that a lot of abstraction of problems is done, and a number of experts who don't do hands on work are employed. All this is done at the expense of homegrown (meaning inside the client company) experience which is the great hidden expense of outsourced systems.

It comes down, in my view, to a decades old clash between management philosophies. Deming vs Hammer. The Deming method says to evolve the way people are working with technology and business processes. That's highly integrative and evolutionary. It means you have to do a lot of listening and translating. The Hammer schools says, throw out the old and make everyone start from scratch. That's re-engineering. Companies that get re-engineered outsource and offshore better than companies that evolve. Companies that evolve are more productive because the culture of the company teaches everyone what the focus of the business is. They can be more nimble, but there's a steep learning curve.

I'm from the Deming School (an old Xeroid from the McKinsey makeover under David Kearns) and have been in the enterprise sofware business for two decades. The Deming way is harder, and it's often against the best interests of management and technology consulting companies, not to mention software vendors, to evolve a good company to great, but the real loss is that so many American corporations are literally outsourcing their own quality improvement. They don't want to grow their own MBAs, they want somebody else's. This dependency is what both depletes American talent, and keeps consultants like me in new BMWs.

Meanwhile over here at Nissan, things are beginning to get very interesting as deadlines start to loom. While Toyota is poised this year to make 100k more cars than General Motors, I'm finding that the systems that are being built in the wake of Sarb-Ox may have a long-term payoff for American businesses. Because over here in this Japanese company, typical of a large number of companies I've seen, the spaghetti and spontanaity of financial planning and accounting boggles the professional mind. In my close circle, we have a term called 'The Official Ass'. That's where a lot of numbers are pulled from. The number of companies that do real demand planning are few and far between. Demand planning is very difficult to do with most kinds of accounting systems that companies have, and so companies pull numbers out of a collective hole in the ground. That is particularly typical of non-financial companies - that is companies whose business is not primarily the management of money. When your financial staff is considered overhead... well.

It's one thing to understand how to build this database and that database. Sure that stuff can be outsourced. Should it be? It really depends on whether you're building system to be building systems or building them to improve the way a company is run. The latter can only be done by dynamic engineers. I like to think of myself as one of those types, but its not very often that we are called to do all that.

The Grinchbag Theory

(soon to be translated into Suessese)
The story of the grinchbag is simple. It is an IT parable based the story of the Grinch Who Stole Christmas.

Whoville is where all the users live, and throughout the year they gradually pile up their little Who requirements and expect at the end of the year that the IT Santa Claus will fulfill all their desires.

Now the Grinch may be anybody in this tale, but the point of the grinchbag theory is for whatever reasons, all of the goodies and wishes are accumulated into one giant bag and taken away from Whoville. And whenever you put everything into one grinchbag, it's awfully hard to move all at once.

So if you're holding the grinchbag at the top of the hill and you change your mind for any reason, you're going to have to grow your heart three times and attain superhuman strength to save Christmas. The fact of the matter is that you won't. You can only hope that the Whos will sing Dahoo Dory without their goodies, because the only thing superhuman strength does is save you from being crushed by the grinchbag.

The moral of the story is to spread goodies throughout the year and not try to gather up everything in one grinchbag. The Whos are going to sing with or without you. You really aren't in charge of their spirits, which you can't possibly understand because you don't live in Whoville.

The Grinchbag Theory

(soon to be translated into Suessese)
The story of the grinchbag is simple. It is an IT parable based the story of the Grinch Who Stole Christmas.

Whoville is where all the users live, and throughout the year they gradually pile up their little Who requirements and expect at the end of the year that the IT Santa Claus will fulfill all their desires.

Now the Grinch may be anybody in this tale, but the point of the grinchbag theory is for whatever reasons, all of the goodies and wishes are accumulated into one giant bag and taken away from Whoville. And whenever you put everything into one grinchbag, it's awfully hard to move all at once.

So if you're holding the grinchbag at the top of the hill and you change your mind for any reason, you're going to have to grow your heart three times and attain superhuman strength to save Christmas. The fact of the matter is that you won't. You can only hope that the Whos will sing Dahoo Dory without their goodies, because the only thing superhuman strength does is save you from being crushed by the grinchbag.

The moral of the story is to spread goodies throughout the year and not try to gather up everything in one grinchbag. The Whos are going to sing with or without you. You really aren't in charge of their spirits, which you can't possibly understand because you don't live in Whoville.

OLAP Blog Integration

I'm also about to crank up the volume on getting some BI & OLAP bloggers coordinated. I think it's sad to see a bunch of stragglers out there with no trackbacks or comments. I'm going to play cat herder and see what I can come up with.

Part of the problem is that there isn't a good back and forth with industry experts. Rumor has it, however, that John Kopke is a blast in front of customers. I know he understands the technology and I used to work with him at Pilot Software back in the days. And yet there are a lot of people trembling with fear at guys like Nigel Pendse. Now there's a showdown I'd like the blogosphere to witness.

I've been neglecting the dialog because I've been downmarket for a while and basically nobody's talking because smaller companies don't have time to listen. But now that I've come back to the state of the art and I see what Hyperion has been doing in terms of rolling out new products and upping the ante and expanding the scope of BI'able applications, there's plenty to say. The question is, who's saying it? I certainly would have been, but that's not what Cubegeek turned out to be. Let's give it a shot in this brave new world.

UPDATE
Introducing The Cubegeek Blog

OLAP Blog Integration

I'm also about to crank up the volume on getting some BI & OLAP bloggers coordinated. I think it's sad to see a bunch of stragglers out there with no trackbacks or comments. I'm going to play cat herder and see what I can come up with.

Part of the problem is that there isn't a good back and forth with industry experts. Rumor has it, however, that John Kopke is a blast in front of customers. I know he understands the technology and I used to work with him at Pilot Software back in the days. And yet there are a lot of people trembling with fear at guys like Nigel Pendse. Now there's a showdown I'd like the blogosphere to witness.

I've been neglecting the dialog because I've been downmarket for a while and basically nobody's talking because smaller companies don't have time to listen. But now that I've come back to the state of the art and I see what Hyperion has been doing in terms of rolling out new products and upping the ante and expanding the scope of BI'able applications, there's plenty to say. The question is, who's saying it? I certainly would have been, but that's not what Cubegeek turned out to be. Let's give it a shot in this brave new world.

UPDATE
Introducing The Cubegeek Blog

Master Data Management

One of the most exciting pieces of software to come down the pike in many years is one picked up in a recent acquisition by Hyperion Solutions. It's one of the reasons I have to be fairly jazzed about the kinds of systems I'll be able to build in the coming months. Formerly called Razza, it's Master Data Manager.

If you had asked me a month ago what was the best way to make money in the Enterprise Computing business, I would have told you Master Data Management. I wouldn't have used that precise term, I would have probably said something like this:

One of the biggest problems for me, in building systems with the tools I have is always the political problem of getting all the people talking the same language. A significant reason why DW initiatives fail is because the metadata is all over the place and everybody spends too much time chasing the data down rather than analyzing it. All I need are my tools (speaking of Essbase outlines) and then I get functional people and technical people speaking the same language, because everybody can see how the numbers and entities roll up. The reason Informatica is making all kinds of money in this space is because they promise to solve this problem.

Well here's what IDC says.

Master data management is a challenging, long-standing problem. But recent attention to business performance management and compliance represent a new opportunity to deal with the issue in a way that can improve both information accuracy and organizational agility.

With Hyperion's MDM, I believe the problem has been solved. As soon as I get a copy I'll get deep into the details, but basically this is a collaborative tool that will allow enterprises to manage all of their dimensions, whether they change slowly or quickly, back through history.

Imagining the worst spaghetti possible, a partial migration between ERP systems without the benefit of ETL, a MDM Server would get everyone on the same page. How many times have I had people squawk about the complexity of Peoplesoft Trees and complain that their reporting systems use one drill down and their interal reporting systems use another and that the Business Objects Universe was painstakingly coded with another? And how many times have I had to be the one to reverse engineer all that rot and put into my systems? Too many to count.

I'm going to have a field day with this tool. Believe that.

Master Data Management

One of the most exciting pieces of software to come down the pike in many years is one picked up in a recent acquisition by Hyperion Solutions. It's one of the reasons I have to be fairly jazzed about the kinds of systems I'll be able to build in the coming months. Formerly called Razza, it's Master Data Manager.

If you had asked me a month ago what was the best way to make money in the Enterprise Computing business, I would have told you Master Data Management. I wouldn't have used that precise term, I would have probably said something like this:

One of the biggest problems for me, in building systems with the tools I have is always the political problem of getting all the people talking the same language. A significant reason why DW initiatives fail is because the metadata is all over the place and everybody spends too much time chasing the data down rather than analyzing it. All I need are my tools (speaking of Essbase outlines) and then I get functional people and technical people speaking the same language, because everybody can see how the numbers and entities roll up. The reason Informatica is making all kinds of money in this space is because they promise to solve this problem.

Well here's what IDC says.

Master data management is a challenging, long-standing problem. But recent attention to business performance management and compliance represent a new opportunity to deal with the issue in a way that can improve both information accuracy and organizational agility.

With Hyperion's MDM, I believe the problem has been solved. As soon as I get a copy I'll get deep into the details, but basically this is a collaborative tool that will allow enterprises to manage all of their dimensions, whether they change slowly or quickly, back through history.

Imagining the worst spaghetti possible, a partial migration between ERP systems without the benefit of ETL, a MDM Server would get everyone on the same page. How many times have I had people squawk about the complexity of Peoplesoft Trees and complain that their reporting systems use one drill down and their interal reporting systems use another and that the Business Objects Universe was painstakingly coded with another? And how many times have I had to be the one to reverse engineer all that rot and put into my systems? Too many to count.

I'm going to have a field day with this tool. Believe that.

Tableau Software: Visualization Comes to BI

As part and parcel of my boredom with politics and lack of inspiring things like juicy battles overseas with which to occupy my mind, I will indulge a bit of the geekside. Something tells me I ought to do so elsewhere, in fact I probably will although I really am divided about all that with regard to implementation etc. I like to keep my politics and my profession in separate parts of the plate.

Anyway the reason for the excitement at the current moment is my discovery of Tableau Software. This is the sweetest front-end I've seen since I first laid eyes on Wired for OLAP. It's a pure visualization tool that has about as much wow factor as anything in BI. It could singlehandedly destroy the concept of prefab dashboards, however chances are that's not going to happen for some time.

What's best about it is that it is pretty simple to use, give dramatic presentations which are very rich and informative and it uses a the single best realtime fat client on the planet as its backbone, Microsoft Excel. Tableau hasn't wasted a bunch of time and energy on the n-tier fantasy, instead they have made use of both corporate inertia and Moore's Law. Guess what ladies and gentlemen, PCs can handle big fat clients, and this one delivers.

What I'm doing these days is ramping up my own data warehouse on the home network, which has about 9 boxes right now in various states of function. I've managed to keep various scraps of master data from a zillion clients over the years and I have a fake data generator that I built from scratch. So there's about to be a world of research I'm going to be able to accomplish here at Lab 107, (a wholly owned subsidiary of Metro Decisions).

Hell, I might even snarf some interesting data from blogs and other public data sources just to show off. Stay tuned, the geekfest is about to begin.

Tableau Software: Visualization Comes to BI

As part and parcel of my boredom with politics and lack of inspiring things like juicy battles overseas with which to occupy my mind, I will indulge a bit of the geekside. Something tells me I ought to do so elsewhere, in fact I probably will although I really am divided about all that with regard to implementation etc. I like to keep my politics and my profession in separate parts of the plate.

Anyway the reason for the excitement at the current moment is my discovery of Tableau Software. This is the sweetest front-end I've seen since I first laid eyes on Wired for OLAP. It's a pure visualization tool that has about as much wow factor as anything in BI. It could singlehandedly destroy the concept of prefab dashboards, however chances are that's not going to happen for some time.

What's best about it is that it is pretty simple to use, give dramatic presentations which are very rich and informative and it uses a the single best realtime fat client on the planet as its backbone, Microsoft Excel. Tableau hasn't wasted a bunch of time and energy on the n-tier fantasy, instead they have made use of both corporate inertia and Moore's Law. Guess what ladies and gentlemen, PCs can handle big fat clients, and this one delivers.

What I'm doing these days is ramping up my own data warehouse on the home network, which has about 9 boxes right now in various states of function. I've managed to keep various scraps of master data from a zillion clients over the years and I have a fake data generator that I built from scratch. So there's about to be a world of research I'm going to be able to accomplish here at Lab 107, (a wholly owned subsidiary of Metro Decisions).

Hell, I might even snarf some interesting data from blogs and other public data sources just to show off. Stay tuned, the geekfest is about to begin.