Cubegeek

Complimentary Combinations

I have just completed some work with a rather massive amount of data, quite enough to say that I have some experience with big data. There are several things I have learned.

Data Management Matters
As a priority in dealing with big data, the less you touch it, the better off you are. Which means, unfortunately, that an iterative approach does not work very well. You need to know exactly how to deal with your data and you need to optimize the processes before you commit to them. A significant number of assumptions one normally takes for granted can no longer be taken. For example, if you have two or three ways to determine how to eliminate duplicate records within your data, you probably need to find *the* most efficient method. That means watching explain plans. That means timing a shell script vs a Ruby script. That means working processes 'close to the iron' rather than remotely.

Sensor Data
It's rather obvious in retrospect but when you are dealing with 'sensor data' most of it is numbers. That means there are not a  lot of attributes and keys for you to deal with, and compression is not going to go as far as you normally think. Therefore the grain of interest for your end-user queries is of paramount importance.  Your time dimension means everything. Know the time scope of your queries and partition your data appropriately.

Postgres Rules
I have been playing with Postgres for about two years now, and there have been a lot of theoretical benefits for doing so, not the least of which is that it is rather standard and has decent free tools. Now that I do so much more work with Vertica and Redshift, Postgres is my best friend. And I suspect it will remain that way as I start playing with VoltDB.  Postgres on my Mac is everything that was once Access, Excel and MSSQL Express. And 95% of my code is portable. It makes a huge difference.

Rubular Expressionism
Everything I need in an ETL, I hand build with Bash, Ruby and Regex. My partner in this exercise in crime is Rubular.com It is the singular most irreplaceable tool in my kit. I now have a large string of parsing templates in Ruby, many of which simply require just the right regular expression. But let me go into detail for a moment.

Digital Industrialism
I am convinced that the Digital Renaissaence is just about over. There are a lot of innovations left in our imagination, but the fundamental leap forward in handiwork is just about complete. In a way, we coders are all like stonemasons in a world increasing dominated by poured concrete. There are two ways to look at this, and I look at it, despite my proclivities towards hand-building, as a good thing. On the one hand, you don't build cathedrals with poured concrete, on the other hand, you do build giant factories and skyscrapers. Cathedrals are spectacular and fascinating. Skyscrapers and factories are mundane and boring. But the work that runs the world happens in the utility buildings. You may be inspired to pray for safety in the cathedral, but you build airbags in the factory. 

So now that our tools are all grown up, we are moving from the Digital Renaissance to the Digital Industrial Revolution. I'd like to think of myself (with any luck) as a digital industrialist. This would be in contrast to a Digital Impressario or a Digital Tourguide. I have to say that at long last I look at Social Media as an evolutionary dead end. (Another Story)

What I mean by having my tools all grown up is precisely what I've been talking about with database interoperability and multiple database systems. We at Full360 are very serious about building open platform systems in which we ingest data into massive storage, manage data and security with key-value store databases, process transactions with tech like VoltDB and serve big data analytics with Redshift and Vertica. Is that wild or what?

By the way we are looking to be on the verge of hacking virtual platforms as well. We have already seen AMIs working in VirtualBox and VMWare. It's only a matter of time before we can spin up databases of platforms for databases. We are deeply into that kind of customization. So yeah, that's another one of my old geek dreams coming true - what I used to call '3DB systems', then thinking of multidimensional, relational and object oriented databases in one system. (The impetus for this dream was a lack of a CORBA toolset for requirements we got from I2 Technologies 15 years ago - prior to the release of the Java API for custom Essbase functions, but I digress). The dream was downgraded to a desire to get 'double DBAs' working for Arberion...

So I want to be a digital industrialist now that I'm all grown up and that means realizing the dreams of 'operational business intelligence' or in other words, dealing with the larger scale structured transactional data previously off limits to non-shardable analytic database engines. OMG it's like rural electrification. There is so much data we can now deal with. Not long ago we in the industry talked about 'subject specific data warehouses', now we're talking about multiples of that at a price mid-market companies can handle.

For me, I've always preferred working with aerospace and manufacturing companies. Why? Because when you have tooled up a factory, you already know what data matters and the analytic value of it (if you could only capture and use it all).  So now we can put the equivalent of a mainframe on the factory floor, in the engineering labs, and connect marketing and finance people too. Awesome.

Mobile Alerts
The more I live with my iPhone, the more I realize that all I really need when it comes to business mobile computing, are alerts. So I'm going to go out and find something that delivers me IOS badges. Something a bit more specific and sophisticated than Twitter, but not much.   

What's Next?
Aviation.  

Market Triangulation

I have this really difficult time writing about my industry and I hope to resolve it sometime this year. In the meantime I do attend a lot of Meetups here in LA to try and catch the flavor of a distribution of souls who operate where they do. It's rather tough to do all this working at home since most of my career I've had sales and marketing folks arranging those sorts of meetings for me. So these days I'm feeling the odd pain of not having a sales and marketing department and an army of fellow employees with new acronyms and strategies on their lips after three days of team building at the big hotel and convention center...

What I do have is the rare opportunity to meet practitioners out here on what seems to be the edge of technologies and practices the market may or may not come to grips with in the next few years. That also being in the relatively new field for me of Open Source makes it doubly complicated. I mean, everybody can know the same tools, have the same capabilities. Scary. Weird. So I'm always a bit hesitant to talk about exactly what it is I do, what it is I want to do, and I tend to stick to what I think is worth doing. That generally makes me sound like an idiot because I take clues from so many places. Cool when you're already a millionaire, silly when you have a day job. Nevertheless I'm a shameless data guy and I like talking with data guys. I kind of assume that everyone can build what I would like to see built and what's all keeping us from it is ... well the same lack of appropriately funded sales and marketing people.

Tonight was a big meet in West LA sponsored by an outfit I happen to think a great deal of - that being Factual, where all the men have long hair. OK 75% of the men have long hair. I do trust my gut, and my gut tells me they're going to do very well in the long term. I had that same gut feeling about Informatica and Comscore in my previous lives. Then again, I was wrong about Acta, or did they get acquired by SAP? I forget. I have said it before, if I had to work any other place it would be there or Evernote or SpaceX. Jeez. I have expensive taste. There is actually one other place that ranks close to my current paradise and that is in the CUSP program at NYU directed by none other than Steven E. Koonin, one of the T50.

For those of you who don't know, I have decided that there are probably only 50 people alive in the world whose books and speeches are worth reading and hearing. I call them the T50 among whom are Koonin, Nassim Taleb, Clay Shirky, Thomas PM Barnett, Neal Stephenson, David P Goldman and Danny Hillis. Hmm. I've got to put up a T50 page. 

Anyway, in all things data, I'm trying to find a secure way to share everything I know without giving away my lunch, and quite frankly I don't have the math, the money or the market share to be too loose-lipped. So I find it a precarious business, and I'm not quite secure enough. Still, I know what I like and what I like is what I am exemplifying in a couple semi-open projects. H/T goes to someone who will probably never read Cubegeek about staking out territory whether or not you can defend your claim. 

I am compelled by my history and my interests to value highly that data which might serve to inform the public about the real statistics behind every claim putatively made for the public benefit. The example I gave tonight was the price of shrimp in the Gulf of Mexico. And I am interested to see that the next generation of journalists will participate in the curation of open source data on some platform and standard that I have something to do with. That is the aim of Project Longitude. If you recall the transfixing tragedy that was the failure of the Deepwater Horizon and all of the conspiracy theories that arose, you may recall the arguments in support or against the seemingly arbitrary figure the President sought in damages against BP, the owner (?) operator(?) of the underwater gusher.  One of these was the matter of the depletion of wildlife and economic impact to the region. I happen to think there ought to be a goodly number of people who would like to curate that information who might communicate with commodities traders. At some point, especially with journalists, information that is no longer breaking news loses its value, and so could be open sourced. The problem is making a good format. I would like to create a market for 'antique' data and make some hub for structured time series data on a relatively low cost, on demand cloud platform. 

Right now, there is structured data of high value currently churning through corporate America. I know, that's how I paid for everything I own. I have been fortunate enough to start in the market before it got specialized. So I have done backends, front-ends, sales, tech support, consulting and management all around corporate financial data. This has allowed me the dubious distinction of being able to see that whole big picture from experience, rather than shutup and be a deeply experienced Java programmer. (yea I saw that coming too.. before they even had Swing). But now the price of much of data care and feeding is making a lot more data reasonably available to a lot more people, and I've packed my Conestoga and hitched it to what I hope to be a bull market. 

We'll see. 

I talked to (that) Randal Schwartz about my old old idea of XRepublic and tried not to bore him with the details. As I explained it, I realized that I'm a bit more cynical about the prospects myself. This prepared me for his response which was that we need a better quality of citizen before a better set of tools will produce a better quality of politics. I agree and disagree. We had a good laugh about a parliamentarian bot, or something programmed to autoboot defiers of Godwin's Law, or at least I thought it was funny enough to mention here. But I do believe that certain behaviors can be created with the proper UX. I mean, if Farmville doesn't make one good actual farmers, the virtual farms do actually grow and they do represent the results of your efforts. Good democratic practices can be gamified even if the result isn't real democracy. The experience has to be more instructive than what currently exists, exactly nothing. But sadly, it reminds me that I had a much less pleasant conversation on the same topic with Dave Winer 6 years ago. Maybe that's my clue that this may not be a good idea. Yet.

Most edifying by far were conversations with Lars and Jason about the shared pain of data management in AWS. I've been going through a bit of that, and I think the ideas I've been kicking around for a nicely kitted out VM or appliance spec will find a lot of uptake.  That one is Project Kleiglight, now internal to Full360 as I push for it. In the meantime, I suspect that there are a number of smarter ways for me to accomplish a bit of that without springing for DMExpress from Syncsort. Or maybe not. But this goes back to thinking about the dimensions of the product / service offering and interpreting what Factual and Infochimps are doing in their architectures that might have already covered that ground.  TBD. 

All told it was a great Meetup at Steingarten, and cheap parking too.

Project Longitude

I'm going to do my piece for improving debate in the US by designing a standard for longitudinal data in public debates. I'm not going to over-smith the words right now, I just want to get out to a quick start. I'm hoping to gather collaborators that will assist in telling me what they need from nice, quick statistical data of the various sorts we want to open source.

The intro is here.

The first basic idea is to provide a kind of package that contains data, schema and validation dropped into a S3 repo. Think of it as Github for data.

What I've Been Up To

What tumultuous times!

I've got to say it has been really weird looking at the empty space that is Cubegeek for the past 60 days or so. But one of my goals for the year is actually going to be convergence. I've finally put so many things in philosophical order in my life, at long last, that I don't feel the necessity to separate all of my avocations from my vocations. Plus I worked out a good deal with my own skills, my intentions and the good folks at Dreamhost. But there's a lot of ground to cover.

The huge news, of course, is Amazon Redshift which throws a big 500 pound gorilla wrench in everybody's business model. A number of pundits have pooh-poohed the whole thing but I have to tell you, this is a major part of the future, whether you like it or not. Redshift is Moore's Law for databases. It's impossible to ignore. Quite frankly I'm not even sure how I can deal with the fact of its existence, because basically somebody with a reasonable amount of skills can put together a data warehouse, quickly.  The upshot is that a lot of consulting can be done at home, the way I do it, and a lot of cheap - even throwaway DWs can be built. This has scary implications for the quality of said DWs and nobody knows exactly what sections of the market Redshift will come to dominate, but I can tell you this, our friends in the database world are defacating building materials. 

I have been working with Redshift for several weeks now and its strengths are many. Primarily, I'm all focused on its elascticity and its price. Additionally I like that I can script everything at the API level. I haven't done all that yet, but I know that I can. It is lacking some nice developer tools at the Toad level, and if I were one of the guys at Panic Software, I'd make sure that is my next project. As much as I love the Bootstrap web interface that Amazon has got running, nothing beats a finely honed fat client. Anyway, the biggest strength of Redshift right now is its ability to load data from S3, and we're thinking up some techniques and product designs that are going to take advantage of that. So check back with me in six months and ask about Project Kleiglight. In the meantime, we are learning by doing in Redshift. 

Here's my first opinion. Everybody who is using MySQL or MSSQL should migrate to Redshift as soon as they think they're ready for more performance. Period. Whatever market that is, I'll take it. 

Here's my second opinion. Teradata is toast. 

--

My Ruby-fu is up marginally. I picked up the Nokogiri gem and am now working a bit smarter with File. I've done some nice integration with standard unix command and also with loggers. So I would call myself competent with XML, YAML and JSON. I still haven't swung back to improve my Cucumber but am plenty comfortable with rspec. I'm working on a utility gem of my own for some text manipulation stuff that I do all the time. Next I'm going to play with the parallel gem to see how I can scale up certain ops.

I'm lagging on my seal book - the OReilly on Exploring with R, and I'm finally getting rid of the paranoia that sent me wheeling two months ago. Nevertheless, I still read Darkside and attend a couple security hacker meetups. 

I've seriously upped my Chef game in the past couple months. Working on our elasticPM code has gotten me fairly deep into the implementation end of orchestration. It is now clear to me that much of what we have been doing is so utterly advanced - we've been on the edge in many ways of what Chef can do with Windows, and our unorthodox approach has been what has been making Chef's learning curve more difficult than what I expected. In addition, working more with Vagrant has improved my capabilities with virtualization. The next version of Vagrant is going to be awesome, I hear.  So I've got about a dozen VMs here on my Mac. As they become migratable into AWS AMIs it's going to be awesome.

Speaking of which, I did get a chance to migrate an AMI across a region via the (2 month) old way of moving core snapshots. So before I could write code to automate that (but I've been busy) Amazon introduced a way to do it directly. So I haven't done it the newest way, but there's one more barrier to internationalization knocked over. 

You have to realize that these days I consider myself to be something of an IT guy in the biggest IT shop on the planet, which is AWS. The new architecture is improving every month. More on this separately. 

---

The reintegration project starts with me getting into a couple , three web architectures. I've gotten the static blog thing worked out with Jekyll and Octopress. So I'll probly migrate all this Cubegeek stuff under the single new site. But I really have to get this Node.js and Rails thing knocked out so I can speak that language to customers as well. You see a lot of our business comes from people with low resistance to moving their assets to the cloud - since a lot of them have used colocation before. I'm going to try to head a lot of them off at the pass since Amazon has DynamoDB, RDS, Redshift and Hadoop, three of which I've had my hands on. So a lot of the confusion I used to have over MongoDB, Cassandra, Riak, CouchDB and SOLR, I no longer have. I just ape the party line and say go Dynamo. 

--

Vertica has been very very good to me. So my take on this in splitting the difference goes something like this. Redshift is for when you have invested *some* time into your DW and you want something low maintenance. Vertica is for when you need to tune the crap out of your system and you want near-realtime stuff. Basically, Vertica has all the bells and whistles for extreme computing. Redshift is more like MSSQL to Vertica's Oracle. Sorry, I hate analogies too, but that's about as close as I want to get to a hardball assessment in this post. I've played with a lot of databases in my time and I love Essbase and Vertica for the same reasons - their internals are beautiful and they enable an entirely new class of computing. However, I like Redshift for the same reason I like MSSQL, simplicity and elegance - except I know Redshift has a lot more upside than MSSQL. 

I have worked with just about every major database technology going back to something called BCC out of Utah. Right now is the golden age, because today we have all the major technologies  available in stacks that can be built on Amazon. It a very exciting time to be a data architect. 

Related articles

Redshift Performance & Cost

DynamoDB - Price Reduction and New Reserved Capacity Model

IP Overload

This is the second time I've heard ideas to the effect that the very low level protocols of routing are not appropriate to the way we are actually using it today. The first time was from Bob Cringely - codename 'Bufferbloat'.

Security Weekend

It's the first week of the month and so I have a number of tasks to do, but it's also a quarter and a year so the administrative piles up. And of course my system crashed. There is so much to relate since my last entry over here.

I have some technical goals for the year, many of which have to do with hacking and security. Essentially, I have put together a vision of the future which involves my working in several capacities beyond what I am doing now. I expect that my current company will grow with the idea. So here's the idea. My nickel says that five years from now, there will be a global systems integrator specializing in building 'enterprise' applications and migrating current such technologies to the Amazon Cloud. It will be the equivalent of the Cambridge Technology Partners of the 90s. In other words, really sharp, really reputable and really profitable. And shortly thereafter it will morph into something massive and more mediocre, like Avenade or the old Andersen Consulting. In other words, cloud computing is on its way to becoming standardized and cheap (first with AWS) and then big CIOs will take the leap. Developers with API programming skills, network guys with security skills, database guys with infrastructure skills, these will be the core movers and shakers in global enablement. I'm doubling down on the cloud by incrementally adding to my security props. Why?

Because *this* year I'm going to be doing instant data warehouses, limited only by the reluctance of people to give up the data. So I'm going to have to get fairly familiar with what attackers do and what processes insure data integrity. So when we go big at Full360 I will be well along the full enterprise migration practice path.  

So I am spending a lot more time with technical meetup groups. 

The first discipline that I will have to master is the end to end usage of VMs. At Full360 we have created a gem that takes some generic images and makes them usable locally through VMWare Fusion and/or VirtualBox - Vagrant. So a little ubiquity of VMs is a good start as we can run our Chef configurations on them locally or remotely hosted on Amazon. The next step is getting some portability with these VMs from local, to local region to international region. The recent addition of an EBS Snapshot Migration capability to the AWS bag of tricks (not quite standardized into the awscli or into right_aws) allows me to conjure up a VM on my laptop, configure it locally (free) then formalize the Chef recipe, recreate the same thing on a micro instance in an AWS US region (free) then move it over into the EU or Australia or elsewhere for basically the cost of data transfer. 

I have yet to get some objective statistics on exactly how much of a difference it makes to run queries against a local vs international region (I suspect it's not a whole lot depending on the heaviness of the query) but it's a strong selling point to be able to move a customer stack to and fro. Not only that but it will make cross-regional applications something else we can do, another level of DR.

This weekend I got a head full of security and paranoia, which is my general heading for understanding the evil that men do. I ran into a guy who sits in NOCs and figures out where the bad guys are likely to hit. His hardnosed attitude was a refreshing wakeup call, and forthrightly more useful than any number of Hak5 videos. Not saying that Hak5 isn't useful, they very much are, but only when you talk to a security professional can you get an idea of what can't be said nicely on a freely available YouTube. A lot. What M reminded me was that in the military, a 'threat' is always considered a possibility because a threat assessment deals with enemy capacity regardless of intention or liklihood. So threat assessment is very different from risk management. Risk management is guessing what can hurt you if you do X. Threat assessment is dealing with what your worst enemy can do to you no matter what you do. Managing risk is small peanuts compared to handling threat contingencies. Or to put it more crudely, risk management is guessing which skirt is a bit too short to wear to the party. Threat contingency planning is practicing plunging the knife into the throat of the rapist when he's on top of you. 

So part of the security business talks about stuff that sounds like FUD, until you see it. That's what new in my world this week. I saw it. I watched a tool snatch a password out of memory in 2 seconds flat, in plaintext. I saw a tool that.. well to save a lot of noise, I saw a small fraction of the Backtrack Linux suite in action.  It's faster than Alka-Seltzer. And even if you think you are an uninteresting target (which I probably am considering the people who may have (deniably) already hacked me), seeing these tools being worked by a pro is kind of awesome scary. 

I also got a taste of what goes on in dox attacks, which is a kind of second stage mayhem that may be put upon you once your box has been owned. Or, maybe you don't have to have had your network penetrated - there's plenty enough free and easy information to get on you via various services, and a world of not so free information that can be gotten for the price of a little deception. 

In all of these matters, what's fascinating are the tools. Tools like Pipl, Nesus, Intellus, Freemind, Armitage/metasploit, nmap, Maltego, KnowEm, Ancestry, Expose, Monav2.. and a bunch of others flavors of kiddie candy that I'm a little silly about at the moment.

I've also had the decent fortune to do more administrative tape-ape kind of work of the sort I guess most college kids get nowadays, but I never did. Don't ask me why my college had NOS and little to no UNIX that I ever saw, but that was the 80s. I always thought that admin type duties were grunt work, but I'm seeing the use of grunting. It's no longer enough for me to know all those two letter commands, I've got to see what they do in odd combinations. So I got a pretty thorough hash through ssh this past week, learning just a bit more than I think you can learn from a book. So now a bit of that tunelling and reverse shell stuff makes a bit more sense - I can review it now. 

The school of hard knocks continues. What fun!

Mou

Everyday I find another reason to celebrate that I am living in the future. Today I heard about Markdown for the fifth or sixth time, it clicked in my head and I started using it. Markdown is RTF for the web.

If you don't remember RTF, it was the predominating word processing format before the great Microsoft Word - Wordperfect wars of the 80s and 90s resulted in the triumph of Microsoft Word. RTF has always been great, but there has been no need to use it considering the ubiquity of Word and the periodic need to use tables and pictures. 

The web has changed all that with HTML and with blogging in particular. I've been blabbing and blogging for about a decade now and 99% of what I need to do is handled with this little short two level toolbar that's above the small text window in my browser. I'm talking about the Typepad editor. Well, you take some combination of that and a wiki editor and you have what Markdown is and does. It's perfectly lightweight for the overwhelming majority of writing I need to do, especially considering that I pretty much want to get everything out onto the web. Everything else that I want more permanently, I print *from* the web into a PDF and send it off to Evernote.

But I can tell you this. After more than 20 years on the web, I need embedded graphics and fancy formats about 4% of the time, if that. And the best fancy graphics are put together as an embedded graphic that speaks for itself, regardless of where some layout genius decides to put it. If you're like me, you love Evernote Clearly and other tools that de-slickify web pages and gives me the text I need in a clear, readable format. 

Titles, headers, bolds, links, bullets, blockquotes, italics and the occasional table, strikethrough and graphic. That's what I need. That's what Markdown delivers. I'm sold.

It turns out that for Mac, Mou is the prizewinner. Perfect, simple, effective. This is where everything goes from now on. Let somebody clever turn my Markdown into HTML5 of XML stuff. Let some CMS put together the CSS template and fancify. I've got my new editor. 

The Meeker Presentation

2012 KPCB Internet Trends Year-End Update from Kleiner Perkins Caufield & Byers

Because it's information you have just got to know.

The Answer for Apple Television

OK. I know what it is. Call me intuitive or foolish but here's my bet. My bet is that the revolution of television is going to be what the revolution of iTunes was. iTunes was basically digital distribution of every profitable song ever recorded that might sustain market interest today. iTunes for TV is the same thing. 

Get it?

It's all about getting the Jetsons and the Flintstones and every other property into the Apple distribution, and ultimately superdistribution. Tim Cook is going to come up with the market, soon to be joined by everybody else (meaning Amazon) that is going to allow Apple product users the ability to turn up the Wayback Machine and find their 32 favorite episodes of I Love Lucy. He's going to make an offer that the heirs and scions of Desilu Productions cannot refuse. It's a big data solution to the last holdout, which is the gigantic library of syndicated television. And he's going to show them how it has been done with music, including AC/DC and The Beatles and it has made those rights holders more money than they could have ever expected through a planet full of Tower Records stores. Apple has the platform, now Apple TV is all about managing the content. 

That's what Apple TV will be. 

The actual hardware is basically already done. There's not much that needs to be done to take that tiny thing that is the actual product called Apple TV right now and embed it into any set. In fact, Apple's probably spending lawyer time trying to keep Samsung from doing that right now. What Apple can do is negotiate the rights for the back ends.

Abulafia

I have an abiding passion for imaginative fiction, but not of the sort in which an individual's perspective is involved. But the closest thing I got to that was perfectly done by Umberto Eco in Foucault's Pendulum. Now that I think about it, I have forgotten Eco, and the last book I read by him was Baudolino. Since then, nothing. But in FP, there was a conspiracy of individual perspective to invent a bogus world and that conspiracy began on a misfunctioning computer: Abulafia.

'Technical fascination' is my way of evading the term 'geek', because there is something deep within me that is unsettled by the idea of being a 52 year old 'geek'. So I have abandoned the term 'geekery' and am likely at some point to sink the ship of Cubegeek. But I can't resist the urge to ask that geeky question, have you ever named your computer 'Abulafia'. I have, standing at the nexis between computer geek and literature nerd.  Heavens to mergatroid what is wrong with our culture?

It turns out that I am about to enter the days of perfection where the tools of AWS have begun to resemble the inner workings of the Krell Machine. I try not to be excitable and paranoid, buy make no mistake, Amazon has built 'mydatawarehouse.com'. In fact, they have built my super grid compute resource, and few of us have even begun to think big.

This week in Vegas, of course, re:Invent took place for the first time. Full360 was mentioned amongst the most glitterait of ISVs. And we have launched our new website as well. I'm starting to feel the hotness. 

Again. Bezos is absolutely right. I'm not quite sure these days if I find him or Elon Musk most admirable.  Bezos is a deep contributing member of the Long Now, and he understands the future very well as he demonstrated in his chat with Vogels. And in that regard, Bezos is the Sam Walton of our generation. Chances are that if I don't get rich, then it will be the Blue Origin spaceship I ride in, not the Virgin Galactic or the SpaceX. 

Here is the thing that strikes me, apropos Abulafia. There is an Abulafia wiki which seems to be just an index into a crowdsourced set of literary subroutines, although I can't figure out how to join that crowd (Bad UI! Bad UI!), it is an idea I have as part of something I'd been cooking up. My thing is called enumbra and it's basically an enumeration routine. It's a tool for what I call random dimensioning - which is essentially an iterative chunking process for handling large numbers of things.  This concept goes back to tabletop computing - which I have recently been thinking about in the context of building the perfect office, which is my enduring long-term wish. It has been about two years since I last talked about that, and now I think I'm reasonably competent in Ruby to build it.  It also ties in with Bucketmaster, which I'm sure I first thought about 12 years ago. I hope I've described all that well enough.

But I'm really drawn to the idea that there are naming paradigms that work well with each of our brains, and it's something that we can do to assist in analogous thinking. There seem to be a lot of possibilities for recategorizing  - taking off on tagging and the ontology problem. What if, for example, our whole way of naming animals and plants are incorrectly biased to the peculiarities of discoverers? I mean how much time is wasted when we think of - for specious example - "Fleming's Sparrow" aka the Red Throated Sparrow, and then it's Latin Name and what it's called in the Orkney Islands? What if you could crowdsource tag all birds? Now what if you had expert curators re-tag all birds? Now what if you had an enumbra type system re-tag and reclassify all birds? What if there were a most appropriate name for everything based upon the actual things that make it most outstanding? This is a kind of tagging problem and it needs a combination of human and non-human intelligences to do it. 

I bring this up because these days I am thinking about what it means to classify all sorts of data that have never really been organized, and some that are at risk of being permanently dysorganized, dysfunctionally organized according to the institutional biases of those we leave by default to do the organizing for us. Somebody mentioned this is a video I watched last night. Google is interested in personalizing my search. I don't want to know what people Google thinks are 'like me'  think is the best pizza. I want the opnion of the best person who knows pizza. I don't want all of my information to be served to me in a way that makes my search self-serving. I want to explore to find out things I don't know, which means I have to use some terms that are not completely familiar to me, or even anonymize the objects whose metrics and behaviors I find notable. 

I know this sounds all very abstract, but then so would be the opinions of someone in the business intelligence world who appreciates spycraft and understands semiotics. We have every opportunity to go off the deep-end to madness in our abstraction business.  How we structure up and index our data is important; we need to be able to do it iteratively from the perspective of many measures.