Cobb

The Cost of the Answer

If you want to know what I believe, it is that the answer is out there. You just don't want to figure it out.

I had an interesting experience the other day over at Quora. Quora is a general question and answer forum that uses some decently new technology that helps their interface evolve. It's a kind a combination between an interactive Wikipedia,  a dumbed down StackOverflow, and a polite Reddit. It can be both fascinating and unnerving. I spend about an hour a day answering some decent questions that are posed and several more minutes gaping incredulously at the vapidity of the rest.

Anyway I decided to ask a question pointing to something specific about the statistical juju behind all of the silly 'What kind of X are you?' surveys that I get from time to time. How do they correlate your answers to 10 questions into a prescription that says 'Aries' or 'Libertarian'? I guess that it's something related to cluster analysis, but I wanted pointers to code or whatever real statisticians call it. In case you're wondering, I want to get started building the Lorite Interrogator. Well the memorable answer that I got was something to the effect of "Well I could give you an answer but before I do I need to know if you're capable of understanding it. Please give me your background."  That was an intriguing response, but somewhat nervy too. But I wanted to know badly enough to respond in full. In the end, the person who asked for my vitae just gave a thumbs up to that response, but never answered the question. So I was basically stuck with nothing better than Wikipedia. Quora fail.

The failure gave me something to think about - which was that by describing my own qualifications to understand a sophisticated response, I has essentially proven to myself that I have no excuses not to take a graduate level course. After all, I wanted a pointer to a book and/or an algorithm in the first place. I want to know about statistics what experts know. As the idea has been on my mind of late, I wandered over to the UCLA Extension online. It turns out that the books required for their $700 course costs about $400. 

It's going to cost me $1100 and three months to know what I really want to know. Now that I'm aware of the costs, I'm not sure I want to know that badly. Actually, I do want to know that badly, I just don't have the time or the cash at the moment. 

What I believe is that everybody makes economic decisions like this all of the time, particularly about how much they believe they need to know. Most of the time, I think people are lazy, and laziness can be a virtue. One wants to follow an 80/20 rule - that's certainly why I asked at Quora in the first place, considering that I have the email of Emanual Derman. 

The implications of this, to me, are clear. Most people would rather ask their way up a presumed chain of knowledge for free. The free ask is 'just above my head' or 'just enough knowledge to be dangerous'. In other words, there is a presumption that there is a cheat, a cheap and easy epiphany, just a question or two away, and this assumption,  exacerbated by the ease of using the Internet, makes the cost of a complete answer, ie real education, seem higher than it actually is. People can and will use socialized 'uncommon' knowledge more readily than they will pay for expert and complete knowledge.

There's the fundament. Where is the tipping point when colleges are already too expensive? 

Essbase Capacity Planning (1997)

This entry is a pointer to a couple capacity planning documents. The first one I created was back in 1997, which I updated last in 1999. The 1999 Capacity Planning Document is embedded in the extended entry. (Format sucks) The updated version will be linked here.

Driving Around the ExxonMobile Refinery in Torrance

The following is my route around the site. My realtime audio recording is here:

Download TorranceExplosion  (yeah I know I said the wrong date at the end. Today is the 18th not the 17th.) I drove clockwise around the refinery. My route is the yellow line. The large damaged building is the red rectangle. The police checkpoints are the blue squares. 

 

 Here's the flare you can see as normal traffic is driving by about a quarter mile west from the checkpoint at Del Amo and Maple:

This is the best picture I have seen illustrating the damage done to that large building. It's an angle looking northwest from about Del Amo & Crenshaw. That's a big building. Credit to SoCalPix

 

 

Actually, this one is even better and shows that the explosion went clearly through the entire structure.  As you can see, this building is a good 12 stories high. The unit is an electrostatic precipitator.  My guess is that it is a wet setup owing to the steam that is visible. It's basically a very large air pollution device. It conforms with my experience of smelling hydrogen sulfide on that stretch of Crenshaw. 

The Four Pillars of Business Intelligence

BI - What is it?

BI is business intelligence. It is the art and science of getting computers to know and think in ways humans do not do well, but need to know in order to better run their businesses. It involves the capture, management and presentation of structured data for use by decision makers. These data are usually organized into multidimensional hierarchies of key performance indicators. The aims of Business Intelligence systems are decision support and performance management.

Business Intelligence is my preferred term for this activity because it is the broadest, but various aspects of BI have gone as

• Data Warehousing
• Decision Support
• Analytics
• Big Data
• Data Science

The origins of these practices are found in the lessons of Operations Management in the growth of multinational corporations during the 1960s, 70s and 80s. Some thought leaders that led to the creation of this multidisciplinary field include the following:

• Peter Drucker
• Walter Wriston
• Ralph Kimball
• David Liddle

PECCC

The benchmarks for decision support and performance management are as follows:

• Productivity: Increase output.
• Efficiency: Reduce costs.
• Competence: Add capability.
• Confidence: Improve satisfaction.
• Communication: Deliver consistency.

Business Intelligence systems are all about improving one's ability to know, and helping one assess the impact of the changes made because of that knowledge. It is cyclical, so that new discoveries are added and the consequences of those results are measured as well. Business Intelligence systems enable a framework of iteration to quality and maintenance of compliance.

Application Scope

BI systems include a selection of IT tools, technologies and practices arranged for the delivery of applications. Each application should be implemented as an independent project which is determined by the aegis and impact of the key decision makers for the company's operational goal. One should ask, who is responsible for the execution of X, what do they need to know, how often do they need to know it? A proper BI application delivers actionable data. You cannot manage what you cannot measure. A proper BI application delivers the knowledge necessary to improve a specific area of the business.

The Four Pillars

There are four areas of expertise for a BI practitioner.

• Front End
• Back End
• Governance
• Inference

The Back End - Architecture & Process

Back end work is primarily about data management, and data modeling. There are two primary sort of tools one uses in the back end. The first is an ETL tool. The second is a database. ETL stands for extract, transform and load. It is a tool or a suite of technologies used to identify data sources, gather them up, translate them into a standard format and load them into a database. A database is a tool that has a store of data and gives it a particular structure so that it can be queried in part or in its entirety. The database makes the data 'live'. The way the data in the database is stored is called a model. The most important task a BI practitioner can have is designing a proper data model, because this determines the ease and speed with which the data becomes accessible for many years to come. When data stored in a database is not easily or readily available we call it a 'data jail'. All data that serves no purpose to humans is, relatively speaking in a data jail. The job of a BI practitioner is to free data from that jail so it can be part of productive society.

The Front End - Visualization & User Experience

Front end work is all about presentation. It is the arrangement and display of data to the end user in a compelling, accessible manner so that the meaning of the data can be readily communicated. A front end designer is responsible for the user's experience in navigating and interpreting the data. There are a wide number of graphical tools that are used in front ends. They are generally, tables and charts. But they are also alerts. They can be text, graphics, sounds and multimedia. They can be presented online via desktops and mobile devices or offline to paper reports. They can be delivered interactively or according to schedules. The front end designer must know his audience and insure that users do not become confused or frustrated when accessing data. Data presentation must be precise and crisp and encourage habitual consumption.

Governance - Security & Lifecycle

Governance work is about the meta process of business intelligence delivery. It starts with metadata management by considering the provenance and security of the data over the lifespan of the application. Who is authorized to use this data? Who is authorized to change this data? What are alternative sources of the data and how are they reconciled? How long will the data be live, when and where will it be archived? When and how must it be destroyed. What happens when something changes? Governance also can include resolving questions about the cost of maintaining the application and the scope of technologies used to deliver it with regards to upgrades and deprecations.

Inference - Statistics & Practices

Inference is the term I use to describe 'data science'. This is a new aspect of Business Intelligence that manages the interpretive aspects of data itself. It is necessary because the scale of BI application datasets have grown exponentially. So particular care needs be taken when imputations and complex composite metrics are made over these large sets. For certain metrics, direct interpretation is straightforward. How many cars? "159". For others more care must be taken. How much to people like those cars? "3.5". A BI practitioner in this specialization assures that the data is taken only as seriously as it can be, and helps balance decision makers' instincts and understanding against what the computer system is saying. They must understand the business goals and the application of statistical analysis to management. They must always draw the clear distinction between the map and the territory.

All of these areas of expertise must be engaged for a holistic integration of a BI system into a business process of continuous quality improvement.