For a long time, I was an Essbase guy. Like since before `my.yahoo.com`. I'm writing about how I came to understand the importants of DevOps as a kind of testimony about a culmination of the lessons of my prior positions. If you don't know, Essbase was voted one of the top 10 compute technologies of the 2000s by Information Age. And I quote:
The multi-dimensional database technology that put online analytical processing (OLAP) on the business intelligence map. Developed by Arbor Software (now part of Hyperion Solutions), it spurred the creation of scores of rival OLAP products – and billions of OLAP cubes.
There were two essential qualities to Essbase implementations that spoiled me and gave me a particular set of instincts that serve me now.
Essbase was important, and it was disruptive.
Essbase was always in production.
Let's talk about the second thing first. Essbase was built with the understanding that business rules were always going to be changing as business changed. So Essbase employed a semantic layer that was able to be edited in real time and then applied to its data models immediately. While it took some expertise to understand the batch time implications of any particular code, Essbase never supported a kind of migration framework to move its metadata from DEV to QA to PROD. This annoyed a lot of people, especially those fun folks who were the DBAs and ERP managers whose data was upstream from our Essbase cubes. Moreover, Essbase itself made the making of multidimensional models easier and cheaper than those built with traditional older tech. So the very idea of having multiple business models was part and parcel of this. So: multiple olap cubes, deployable in production with each of those cubes with multiple scenario dimensions and the facility supporting dynamic definitions of data-driven business rules. It was, quite frankly, more flexibility than most finance and planning organizations could wrap their heads around.
These days we call that kind of adaptability in production systems 'Continuous Deployment', which is a DevOps term. Now what Essbase did not have was Continuous Integration with systems outside of its aegis. But it did have tight semantic integration with its 'Essbase Ready' clients. Which meant that if master data changed in the database model, no additional changes had to be made to the reporting components. I am not aware of any next-gen databases that do that. That ability was part and parcel of the Essbase server engineers designing an API for their clients rather than just a pipe of text.
This combination of deployability and tight integration allowed us Essbase guys to grab data, get it into the model quickly and evolve the model right in front of our customers. That was why Essbase saved so many lost data warehouse projects. We surfaced changes quickly. We allowed our customers to explore data immediately and made multiple looks of flowing data happen on a daily basis, by design.
Essbase's importance and disruptive nature may sound subjective but it was not. I had the privilege of working for a hot Valley company that proved that it could IPO and make money. So when people decided to buy our database, it was a big deal and a high priority project. More often than not we were swapping out systems to the benefit of the financial guys and amidst the howls of IT. So when we in the field service group were assigned to get it up and running, the hot visibility was on us. Essbase lived up to the hype. In 3 years I never lost a trial.
The point is that when companies decided to buy Essbase, they went all in. It wasn't an experiment. It had to replace the company's financial reporting - all of it, or it was a failure. Essbase was originally not scalable to retail solutions, and that's one reason competing companies like Microstrategy were able to survive. But for the right fit, our customers swore by it. It changed the way people looked at IT because it just worked and it eliminated a large number of problems. These benefits were clearly presented up front before companies made their purchasing decisions. They bought the technology with the understanding that it would change their behavior, make business processes faster and improve transparency. It wasn't about having Essbase because all the cool kids had Essbase. It wasn't a market sweeping phenomenon. After all, we were competing in a crowded market. But for people who could see how this technology could improve their business, the way was clear.
I cannot say that what I learned technically using Essbase or any of the BI tools themselves helped me understand the broad variety of open source technologies that are associated with DevOps. Those were subjects that required their own mastery. But these two phenomenon having to do with the effect of the right technology on the business were instructive in my understanding of the conflicts and benefits that would arise upon implementation. I now know the effects of these changes in the organization and what kind of management approaches are necessary. In short, I have the experience of breaking ITIL standards and old habits of getting things done in medium and large organizations. Needless to say, you cannot simply drop technology into production and expect behaviors to change, but when you understand how functional behaviors need to change for the good of the business, it goes a long way in gaining acceptance to radical tech and process change.
I'm going to be doing more management and 'glue' business the next year or so. Part of this business is selling and personifying the value of DevOps. Like Cloud, this is something that is insufficiently understood at a deep(er) and nuanced level. So, as is customary, I'm going to tell a story.
The story, like most of mine, comes from an experience that burned. Something that left scars and had me wondering how people cold get into this mess. And so like the man says, share your scars. The situation was that I was on the very cutting edge of what I could do with Essbase + Essbase Studio. The requirement for drill through was fairly obvious. We all knew the limits of how much data we can squirrel into a multidimensional cube. So I used Essbase Studio to map back to the Oracle DB and bring back some records. Now it turns out that my customers wanted something on the order of 10,000 records in this detail. Well that doesn't seem like much. You could grab 10,000 records across a dozen columns, cut and paste them from one Excel spreadsheet to another right? That should only take a few seconds. Not the drill-through. That data had to come over the network. Well you could copy a spreadsheet with a dozen MB of data from a network drive to your desktop, right? That should only take a minute. Not the drill-through. We had to fulfill a query request from a database.
My queries were taking 7 minutes and 30 seconds and then dying.
I had to find out why. Thus began my painful birth of being DevOps. The first thing I had to learn was the difference between a view and a materialized view. Well that wasn't so difficult to learn. But I had always assumed that my DBA was materializing data for me. Well he didn't have enough disk space to do that for ad-hoc queries. So that meant I had to learn the procedure for requesting new disk space from the DBAs. How much did I need? I don't know. A terabyte? Impossible! Impossible? I can go to Best Buy and get a terabyte. Yeah but one live terabyte means four other terabytes according to our backup and DR, and we're at the limit of the current server which means we'd have to get a SAN device and... well how about an NFS drive? Nope. Can't have an NFS drive that would slow down everything I need local storage. Well, we'll get back to you. But how can you be sure that the database is the bottleneck? I don't know.
I had to find out where. What is timing out? Was it the Excel add-in? No. Was it the java middleware? Maybe. Who knows how to read the profile of the java middleware? Well there's no documentation for that, you'll have to call the engineers at Oracle. OK. Open up a service request and get an appointment. Who has access to the middle tier? Get access to the middle tier so you can log on. Oh by the way, the one support engineer is in Mumbai. That means you stay late, past 7pm Pacific time to get your answers, when he's available. OK change the profile, add in this line for the timeout. That didn't work? Oh you have to get the latest patch. Will it work with the version of Essbase Studio we're running here? Oh snap, we're going to have to burn a new version for you, but you're going to have to upgrade your java app server. OK now the explicit timeout is 15 minutes.
Still times out.
I had to find out how. What is the mechanism that creates the time out. Get this new tool called Fiddler, it will help you debug the HTML stream. Debugging HTML streams? Well, maybe it's the size of the download that's stopping things. OK did that. It's not the size. Well the corporate standard timeout is 10 minutes.. What corporate standard? The corporate standard on the firewalls between the users and the data center. Well can we get an exception? Maybe.
So it basically took six weeks for me to deal with the various network engineers, database admins, support staff and their management to prod them all to buy what I was trying to sell, which was the viability of this entire project. My only leverage was that I was consistently riding herd on the problem and I was a very expensive third party contractor. So the project was late and the entire overhead of the difficulty in justifying business as usual in the various departments was the only thing that motivated people to go to extraordinary lengths to solve the problem. Everybody wanted the problem to be somebody else's problem. And until we found out exactly what the problem was, everyone was pointing fingers until the last possible minute. It turned out to be a default in one of the load balancers that everyone assumed was set to 10 minutes, but communicated 7.5 as an override to the other. Those machines required firmware upgrades as well.
I have been accustomed, throughout my entire career in BI to be responsible for the entire data supply chain. That I could do. But middle-tier service configurations, firewall settings and DR disk availability was all above my pay grade. I was not paid to know and I was too expensive to be paid to learn. In that way, I'm accustomed to being like the wiley developer whose time is too valuable to waste learning these operational details. At the same time, I was equally demanding of all those dependencies. Give me more memory on the app server! Open up the damned ports I want! Get more disk, you lummox! Of course let me not forget the memory constraints on the end user machines.
All of this was a terrestrial implementation and it had other setbacks too, but it was a fascinating six month engagement. I of course learned a lot about these other systems with respect to how they affected my entire piece of the data warehousing applications. I sensed that I had the capacity to understand, but I'd never remember unless I had some responsibility and permission to make changes. That would be impossible without the cloud. But even when I had the cloud, it was more than just having control of the associated systems but really understanding how they worked. That's a story for another day. What was clear was that it was very difficult to manage all of the departmental areas, and get the priority within those departments (at their various locations) to solve a showstopper problem in this one application. It was 2011 and we were testing the very limits of the IT capabilities of a global corporation. DevOps might be a cool thing to talk about with web startups, IE a DevOps engineer would be cool for your website, but I saw the fundamental management problem that had everything to do with the way multimillion dollar Enterprise applications were built and maintained, and essentially why they were one-shot deals.
I noticed some folks reading through the Redshift category and noticed that I haven't written anything new for a while. So here's what's new:
We see that Redshift has improved its vacuum capabilities and added more functionality all around. It has improved its performance all around, but it hasn't changed its overall performance characteristics. Redshift is not fundamentally different after two years. It still behaves like Redshift when compared to Vertica, the other MPP columnar database we support at Full360.
We have been able to learn quite a bit more about tuning Redshift. Full360 will be offering this service soon. It's called Upshift and it is surely the most comprehensive performance evaluation available in the industry.
I have to qualify all of this by saying that I personally work a lot more with Vertica than I do with Redshift. These two products may seem very similar but the details are often overwhelming. Fortunately we are developing a methodology that expresses the rules for optimization very well. So while the characterization I've made still holds true, there will be a growing number of exceptions and interesting circumstances. I call Vertica a magical sword. It is powerful, precise and it sharpens itself. I can cut intricate and delicate patterns, and chop hundreds of large heads. I call Redshift an ogre's club. It is massively powerful, brain dead simple to use and relatively inexpensive. So you basically have to look at your application and know whether or not it is a job for a club or a sword. Our methodology will tell you exactly which, but like I said, the devil is in the details and we are wrangling dozens of demons.
The good news is that both products are improving at a good clip. Still, I confess I'm paying more attention to Vertica. I'm really impressed with the overview I got yesterday on Vertica 8. They've optimized some of their geospatial algorithms. They've incorporated several ML features directly into the core product. They've dramatically improved their integration with Hadoop, Spark and Kafka. They're claiming to perform 160% of Impala. So that's superb. Most importantly there is enthusiasm for the creation of the new company, which is less like Vertica getting sold to Microfocus and more like a rebirth of Microfocus itself who is, by the way, the owner of Suse Linux. The Vertica guys are thrilled that they'll be working for a software company. That means the upcoming integration with S3 is serious as is their priority on cloud implementations. All good.
I grabbed up a modest 7 TB of 7200 RPM Hitachis. I'm thinking about reformating them to btrfs but I think ext4 will suffice for the moment. I can always change; that's the point. It took me a minute to figure out what I was doing wrong with my RAID controller, but nicely it turned out to be a physical problem. Evidently there's a SATA to SAS converter for my drive sleeves. So I seated them as if I had the converters, which I didn't. Easy fix. The PERC 6/E worked flawlessly.
I need to redo everything so that I can stop using LVM, or maybe bring half of those TBs onboard. Dunno. The LVM is fine for me but not for Vertica 8, which is the point of this playground for the moment. Since there's a chance I can bring all Vertica into a Docker container and still get decent single node performance, we'll see how that works. I already know how to run the JMeter tests for that. Should be interesting. So yeah I dumped Proxmox. Who needs it? I'm not going to keep the 2950 up for long periods of time, even though I'm used to the noise. It's going to be strictly about the database which doesn't need to run.
In the meantime I'm going to finish loading up the Hashi suite of goodies. I have consul. Next is vault.
Eyebrows ask why? The answer is nebulous, but having something to do with a desire to know and do for myself. If I was more of a car guy, this would be the Jaguar XKE I'm slowly building in my garage. So the challenge continues.
So far I've gotten a sweet Dell PowerEdge 2950. I picked up for a modest $165. Not bad for two processors, four cores each running 2.66GHz. 16GB Ram, redundant power supply and six drive bays. It's loud and obnoxious which is the worst part and it doesn't boot very well from USB. Fortunately I haven't upgraded my Mac Mini and so will leave it so that I can continue to burn DVDs. Last weekend I made a successful load of Proxmox 4.3 VE. It recognized everything and I was able to admin remotely. My first VM was CentOS7 and that was also troublesome getting going, but worth it. Next I installed Kali2, but that install failed. I'll give it another shot locally on VMWare on my main workstation to see how it should work. I have gotten the live version working a couple times - enough to check out the MATE desktop, which is fair. I ran a couple utils but they seemed to just sit there. Next time.
Today I flipped on the MD1000. There are no drives in the bays but that will come in January. I am quite pleased to know that the Dell PERC 6/E adapter is the one that I need to install in order to get all that working properly. I found a new one on Amazon for $85 bucks. The MD1000 is relatively quiet as compared to the PowerEdge. So I'm thinking that I may just hook it up to a quiet tower as a permanent storage solution I can keep spinning - or not.
The next step is putting together a VPN and a firewall that I can configure myself. That will harden up the homebase a little. In due time I'd like to host something more secure. I should learn the appropriate terms. I have a lot of network stuff to learn, and I expect that learning it the hard way on old hardware will make it very easy for me to understand what AWS makes easy for the world. It's a bit surprising to be on the thin side of my expertise, but that's the discipline. In a few months it will be even more fun.
Edison is already telling me that I spend way more money per month than my neighbors on power. Yeah well so long as I don't crash the circuit, I don't care.
The best thing about being in IT is also the worst. Software works and you have lots of choices. So now I've just had some great fun with more operations responsibilities than I've had in a very long time. In order to accomplish this new set of challenges I've had quite a few overlapping options, and so wandering a bit away from my personal specialties, I'm now working with Hashicorp's tools.
Consul Consul is something on the order of a 100Kb executable that is an agent that works as a client and a server in a swarm. It does everything in a single process easily tracked in ps and in the task manager. Unfortunately there's not a binary for Beaglebone Debian, but I'm sure that's coming. Using version 0.7.0, I have floated an army of them out in my own home net. So now I know every time my machine goes down. I haven't hooked it up to SNS so I do have to check first, but it's pretty cool.
A couple weeks ago, something profound occurred to me about enterprise software. I realized that when it is priced by server or by processor, that it's a ripoff. When it is priced by data usage, it's a bargain. You are paying for the potential of being able to use 8 cores even when you are not using them. Admittedly this is rather easy to see if you have experience working in the cloud and then take a turn doing on-premise practices, but it seemed rather profound to me. Imagine, if this is not obvious to you, that you pay a standard or premium license fee to your mobile phone carrier based upon whether you are using a brand new smartphone or an old one. Right. It makes no sense to pay money for anything but the minutes you use on the phone. Mobile phone billing is done right. You pay for minutes. All software could be that way, so that you're not paying for servers, but for functions that spring to life do your bidding, charge you a fractional penny and then die.
This week my boss sent me a link to a guy named swardley who likes ducks. It turns out that this character is quite capable of blowing my tiny mind. He's done it twice already, and so now I have the burden of a speck of enlightenment. It is an enlightenment which is commensurate with my understanding of (Wladawsky-Berger 1999) n-tier computing and then (Vogels 2008) horizontal scaling. So since I've been doing cloud for six years, I now understand what's happening next. God save us all.
Tying software development to economics and cost accounting has long been the stuff of magic, SWAG and charlatans. At least that's what it seemed like for most of my career. But I think Simon Wardley has the solution. He has outlined an ecosystem and a framework for understanding how one can iterate (captive) algorithms towards mutual value for the developers and the customers. He calls it FinDev.
Like most useful thinking on the progressive edge of IT, one must assume AWS. That is to say, very little of what exists in the world that is not part of AWS's ecosystem can be thought of as having great potential in the future of computing. AWS is beyond doing things very well, they are evolving at a monstrous pace and at enormous scale. As an aside, I asked some Agilists last night at El Torito why Amazon manages its businesses so well. He said it's because Amazon is a collection of relatively small businesses that work on a common billing system, and that's what keeps it simple to manage. They are not only eating, but profiting from their own dogfood, which is the fully meticulous tracking of compute resource costing, and now with Lambda, down to the function. All the hardware is a sunk cost. Cloud computing is a utility. What matters now is billing by the function. Simply assume the cloud. It's already done.
So there is a scary aspect to this which is something we all should have been afraid of all the time. It is what happens to craft when things become industrialized. Your personal touch matters less in a market which is defined towards optimization, cost-cutting and efficiency. Nobody cares how you show off the horse, we're all driving cars now. Nobody cares about your budget system, we're all using SAP now. Nobody cares about your fat client, we're all using browsers now. What's coming is a COTS revolution in which your college professor optimized the Towers of Hanoi solver and now owns the moneyTicker on the algo. In a global library of cloud interoperable functions, the scope of what you get to work on gets narrower and narrower. The good news is that we are 20 years away from lockdown. The V8 of compute engine economies is invented. Say hello to the next 50 years. You Wankels don't stand a chance. When I was an undergrad, I used to think of software as the same thing as law. There are lots and lots of lawyers but only a few legislators. The assumption was that the best lawyers at some point got to legislate and the rest just interpreted and borrowed citations for the benefit of those who never read the law. I believe there will be some measure of stare decisis in the new FinDev ecosystem.
So the future belongs to engineers who really know their customer's needs. The economy of FinDev provides value to developers and customers only to the extent that something can be built (at scale in the cloud) that customers want to use. When you charge by the use, that's a different business model than anything we've seen. Chances are it will be disruptive because it will go after captive inefficiently spent money. But there's greenfield out there too. More hopefully, there are new places computing can and will go once we wean ourselves from the economics of capacity planning, system depreciation, outsourced consulting and all that. I think AWS will be capable enough to handle global innovation in this regard; they're certainly leading. Now is the time to work our way towards best practices, evolving towards the revolution.
In Martin Cruz Smith's Arkady Renko series, the protagonist, Renko informally adopts an orphan who is a chess genius. Playing at the genius level, the kid doesn't require a board or pieces. He, and those like him, can just recite moves. He has a virtual queen and doesn't even need hardware. If you're thinking about physically moving pieces, you're not playing chess.
It's probably the most famous classified operation of the US Security agencies known to man. At the moment the details are hazy to me, though known to be sinister and sneaky. Of course that's what intelligence services do. So what else is new? Well, what's new is that Black Mirror is showing some of the possibly dystopian consequences of what we know these new technologies might unleash on society given the new ubiquity of powerful IT tools and techniques. We're kind of sliding into it sideways without, I think, an adequate amount of useful terminology in the public mind.
So let me slide sideways a minute and go back to 9/11. When that happened, I was hanging out with a group called Brainstorms, and one of the members of the group was connected to the intelligence community. We asked this dude to help us understand what we have now come to know as 'asymmetrical warfare' and it was very difficult to get straight answers. That was a combination of his commitment to secrecy and our relative ignorance of the context of the intel warrior's job. Well one of the things that is very difficult to grasp is how many tens of thousands of programmers there are who now understand and have access to the kind of technology the intelligence services deploy. The asymmetrical warfare of small, agile coders with excellent software is on the other foot, as it were. The short term advantage is with the little guy.
So it occurs to me that as I get better at securing data for my clients, I should add a level of abstraction. Especially when it comes to describing best practices, it's very useful over the long term to talk about the customer in a way that isolates them. So I'm going to help anonymize my customers by giving their projects codenames. Nothing particulary spooky but spookish nonetheless.
"How many programmers does it take to change a lightbulb?" "None. That's a hardware problem" -- old joke
What a lovely thing is work. Putting your brain to somebody else's business and being paid to do so is the desire of every prisoner of the mind. And thus I have been reinvigorated from an unexpected corner. Operations. Hard drives. Routers. /opt. I have just recently discovered the joy of Linux server administration. I have to admit it's a lot more fun when you have three dozen very large servers to administer and everyone around you is quite comfortable that you have root. But this was the joyful gap that I didn't realize existed in all the years I endeavored to pull teeth in corporate IT and get the environment my brilliant developments deserved. I am three degrees of separation from end users, and for the first time in memory, I don't particularly mind it at all. I'm working for the machines.
What's doubly ironic about this is that I made a fairly loud exit from the world of Enterprise six years ago in abject frustration, looking desperately to find my place among open source and the clouds. Now, I want nothing more than to build my own datacenter in my garage. I want a DIY cloud, which is to say a big rack of hardware and some virtualization software. What would be lacking from a real cloud would be a real API, but that will be me. I'm the API of my own cloud and the captain of my soul. I think that making this move towards the machine is the fulfillment of yet another yearning on par with my move to stoicism. I had only thought I was dealing with reality in politics, but what I actually hungered for was the history of political philosophy. Similarly what I want in computing is access to the evolution of hardware and networks. Applications and clicks are as ephemeral as slogans and votes. Everybody does it without thought, what really counts is infrastructure.
Only I see where the whole cloud infrastructure game is going. It's going to oligopoly, with the fourth wildcard. Between the intellectual, legal and industrial capture of Google, Amazon and Microsoft is that nasty fourth thing called the Dark Web, and its next monstrous intervention will be the zero-trust tier of blockchain computing. If you don't know, then consider the history of Bitmain, and know that a man named McAfee is still alive. There is a future out there populated with hackers and systems that are much more immune from the sorts of ordinary disasters and bugs that destroy the lives of Enterprise IT jocks and their sheepy user base. There are two kinds of systems in the world, those who have survived hurricanes of DDOS attacks and those who are still wearing bunny slippers when it sprinkles outside. They don't know what a torrent is. They may not ever learn until it's too late. For those of us who will populate masts we would rather not be lashed to in the coming storms of cyberwar, the hedge is personal hardware and power, private networks and circles of trust. Or so it seems to me right now.
It turns out that grepping log files and shooting pistols are both loads of fun, and skills that require years of practice. But they're also fun. Shooting the zombies of tomorrows apocalypses will not fall merely to the burnt out crusaders longing like for the halcyon fields of home and bipolar yearning for honorable death. It will be fun.
So the first thing that I've done is resurrect a couple old machines from the scrap heap. My old MacBook Pro is now an Ubuntu machine. There's nothing in the Mac world that I need on that machine when I think about it, and bloat has really taken over. It's getting rather obvious and tiresome, Apple. Cut it out. That and an old 386 Dell will host lightweight stuff as yet to be determined. But right now they're running Consul and Zerotier (more on them later). Next I want to get a physical rack and a discarded Dell MD1000. That, I will fill with 7200 RPM terabyte hard drives and lower my cost of S3 storage, which I can and probably will just move all to Glacier. My S3 bill just peaked over $40 a month. No can do. Since I have gotten the new 256GB iPhone, there is essentially no reason for me to have that extra duplicate copy of my 100k pictures on S3. And I'm pretty sure that I have everything Flikr on my own drives too.
The big deal and central object of The Wall is my investment in Vertica and reference data. So I'll have a relatively high powered database server running so that I can practice spinning up entire orchestrated things around referenceable datasets. IE. click on this package and get a full 'enterprise' quality query space, stuff that almost nobody does. I expect to get under the community wire with open source tools, and I am especially looking forward to using Fugue as its applicable to instant-up a data reference stack. The collaboration between AWS and VMWare will help a lot, as will a lot of Hak5 reruns and r/homelab.