Aside – Indeterminate measurements and false hope

 Aside, General Technology, General thoughts  2 Responses »
Feb 012012
 

Percentage Complete

I’d like to suggest that we should specify that “percentage complete” estimates – be they progress bars or sliders or any other representation, visual or textual, need a defined unit of measurement to them.

And we should define that unit of measurement as a maybe.

That is, if a piece of software reports that it is 98% complete at something, that’s 98 maybes out of a 100.

I perhaps, should mention, that I’m not thinking of NetWorker when I make this case. Indeed, it’s actually springing from spending 4+ hours one day monitoring a backup job from one of NetWorker’s competitors. A backup job that for the entire duration was at … 99% complete.

You see, in a lot of software, progress indicators just aren’t accurate. This lead to the term “Microsoft minute”, for instance, to describe the interminable reality bending specification of time remaining on file copies in Microsoft operating systems. Equally we can say the same thing of software installers; an installer may report that it’s 95% complete with 1 minute remaining for anywhere between 15 seconds and 2 hours – or more. It’s not just difficult to give an upper ceiling, it’s indeterminate.

I believe that software which can’t measure its progress with sufficient accuracy shouldn’t give an actual percentage complete status or time to complete status without explicitly stating it as being an estimate. To fail to do so is an act of deceit to the user.

I would also argue that no software can measure its process with sufficient accuracy, and thus all software should provide completion status as an estimate rather than a hard fact. After all:

  • Software cannot guarantee against making a blocking IO call
  • Software cannot guarantee that the operating system will not take resources away from it
  • Software cannot guarantee that a physical fault will not take resources away from it

In a real-time and fault-tolerant system, there is a much higher degree of potential accuracy. Outside of that – in regular software (commercial or enterprise), and on regular hardware/operating systems, the potential for interruption (and therefore, inaccuracy) is too great.

I don’t personally think it’s going to hurt interface designers to clearly state whenever a completion estimate is given that it’s an estimate. Of course, some users won’t necessarily notice it, and others will ignore it – but by blatantly saying it, they’re not implicitly raising false hope by citing an indeterminate measurement as accurate.

 Posted by at 20:23  Tagged with:

Data Awareness Distribution in the Enterprise

 Architecture, Backup theory, General Technology, General thoughts  No Responses »
Jan 272012
 

Continuing on my post relating to dark data last week, I want to spend a little more about data awareness classification and distribution within an enterprise environment.

Dark data isn’t the end of the story, and it’s time to introduce the entire family of data-awareness concepts. These are:

  • Data – This is both the core data managed and protected by IT, and all other data throughout the enterprise which is:
    • Known about – The business is aware of it;
    • Managed – This data falls under the purview of a team in terms of storage administration (ILM);
    • Protected – This data falls under the purview of a team in terms of backup and recovery (ILP).
  • Dark Data – To quote the previous article, “all those bits and pieces of data you’ve got floating around in your environment that aren’t fully accounted for”.
  • Grey Data – Grey data is previously discovered dark data for which no decision has been made as yet in relation to its management or protection. That is, it’s now known about, but has not been assigned any policy or tier in either ILM or ILP.
  • Utility Data – This is data which is subsequently classified out of grey data state into a state where the data is known to have value, but is not either managed or protected, because it can be recreated. It could be that the decision is made that the cost (in time) of recreating the data is less expensive than the cost (both in literal dollars and in staff-activity time) of managing and protecting it.
  • Noise – This isn’t really data at all, but are all the “bits” (no pun intended) that are left which are neither grey data, data or utility data. In essence, this is irrelevant data, which someone or some group may be keeping for unnecessary reasons, and in actual fact should be considered eligible for either deletion or archival and deletion.

The distribution of data by awareness within the enterprise may resemble something along the following lines:

Data Awareness Percentage Distribution

That is, ideally the largest percentage of data should be regular data which is known, managed and protected. In all likelihood for most organisations, the next biggest percentage of data is going to be dark data – the data that hasn’t been discovered yet. Ideally however, after regular and dark data have been removed from the distribution, there should be at most 20% of data left, and this should be broken up such that at least half of that remaining data is utility data, with the last 10% split evenly between grey data and noise.

The logical implications of this layout should be reasonably straight forward:

  1. At all times the majority of data within an organisation should be known, managed and protected.
  2. It should be expected that at least 20% of the data within an organisation is undiscovered, or decentralised.
  3. Once data is discovered, it should exist in a ‘grey’ state for a very short period of time; ideally it should be reclassified as soon as possible into data, utility data or noise. In particular, data left in a grey state for an extended period of time represents just as dangerous a potential data loss situation as dark data.

It should be noted that regular data, even in this awareness classification scheme, will still be subject to regular data lifecycle decisions (archive, tiering, deletion, etc.) In that sense, primary data eligible for deletion isn’t really noise, because it’s previously been managed and protected; noise really is ex dark-data that will end up being deleted, either as an explicit decision, or due to a failure at some future point after the decision to classify it as ‘noise’, having never been managed or protected in a centralised, coordinated manner.

Equally, utility data won’t refer to say, Q/A or test databases that replicate the content of production databases. These types of databases will again have fallen under the standard data umbrella in that there will have been information lifecycle management and protection policies established for them, regardless of what those policies actually were.

If we bring this back to roles, then it’s clear that a pivotal role of both the DPAs (Data Protection Advocates) and the IPAC (Information Protection Advisory Council) within an organisation should be the rapid coordination of classification of dark data as it is discovered into one of the data, utility data or noise states.

 Posted by at 16:40  Tagged with: , , , , , , ,

Clouds and bandaids

 General Technology, General thoughts  No Responses »
Nov 092011
 

I’m going to stir the pot a bit here and suggest that a reasonable percentage of businesses considering cloud deployments have their heads in the sand over where their problems really lay.

In short, any company that sees cloud – public cloud – as a solution to becoming more flexible and cost competitive don’t actually understand the real problem they’re facing.

That problem? Business/IT alignment.

One of the oft-touted advantages of public cloud is that it’s about enabling businesses to react more quickly to changing IT requirements than their own IT departments do.

How many people can honestly look at that description and think that the solution is to cut off the IT limb from the business body?

The solution – the real solution – is to address whatever divide there is between the business and IT. And have no doubt – if blame is to be laid it will likely need to be laid with equal measure at both the feet of IT and the rest of the business.

Shifting to public cloud instead says that the business itself is unwilling to seriously look at its own function, behaviour and attitudes and address real issues. It’s the business hoping for an “abracadabra” solution rather than working on a real solution.

It’s like having an arterial bleed, and sticking a bandaid on it.

 Posted by at 17:22  Tagged with: , ,

Questions about “big data”

 Aside, General Technology, General thoughts  No Responses »
Nov 062011
 

I’ve been watching the “big data” discussion happen in a variety of circles, with a slightly cynical concern that this may be like Cloud 2.0 – another sad meme for technology that’s already been in use for some time, but with an excuse to slap a 30% mark up on it.

So, the simple question really is this – is “big data” a legitimate or an illegitimate problem?

By legitimate – is it a problem which truly exists in and of itself? Has data growth in places hit a sufficiently exponential curve that existing technology and approaches can’t keep up …

OR

… is it an illegitimate problem, in that it speaks of (a) a dumbing down of computer science which has resulted in a lack of developmental foresight into problems which we’ve seen coming for some time and/or (b) a failure of IT companies (from base component manufacturers through to vendors across the board) failing to sufficiently innovate?

For me, the jury is still out, and I’ll use a simple example as to why. I deal with big data regularly – since “big data” is defined as being anything outside of a normal technical scope, if I get say, a 20 GB log file from a customer that I have to analyse, none of my standard tools assist with this. So instead, I have to start working on pattern analysis – rather than trying to extract what may be key terms or manually read the file, I’ll skim through it – I’ll literally start by “cat”ting the file and just letting it stream in front of me. At that level, if the software has been written correctly, you’ll notice oddities in the logs that start you pointing to the area you have to delve into. You can then refine the skimming, and eventually drill down to the point where you actually just analyse a very small fragment of the file.

So I look at big data and think – is this a problem caused by a lack of AI being applied to standard data processing techniques? Of admitting – we need to build a level of heuristic decision making into standard products so they can scale up to deal with ever increasing data sets? That the solution is more intelligence and self-management capabilities in the software and hardware? And equally, of developers failing to produce systems that generate data in such a way that it’s susceptible to automated types of pattern analysis?

Of course, this is, to a good degree, what people are talking about when they’re talking about big data.

But why? Do we gain any better management and analysis by cleaving “data” and “big data” into two separate categories?

Or is this a self-fulfilling meme that came out as a result of poor approaches to information science?

 Posted by at 12:18  Tagged with:

Where RIM went wrong

 Architecture, Aside, Backup theory, General Technology, General thoughts  No Responses »
Oct 142011
 

(Quick note: I posted this on my personal blog – insufficient coffees thus far this morning, and decided to repost here.)

In case it’s not been immediately obvious to anyone, I’ve done some simple diagrams to explain where RIM went wrong in this catastrophic outage they’ve been suffering.

You see, most companies implement what we call redundant infrastructure. In systems that require high availability, this is often accomplished with something as simple as clustered (either LAN or WAN) hardware and communications. Sometimes it’s designed that each component runs at the same time, sharing the load, but if one fails, the other one takes over and runs all the load. In simple terms, it looks like this:

Active/Active Cluster

That all makes sense, right?

Unfortunately, RIM seemed more focused on having failover capabilities for upper level management, so it instead clustered its’ CEOs:

Active/Active CEOs

The supposed theory behind this is that the two CEOs, working in an active/active arrangement, could handle load better and get the job done better than a single CEO – and provide resiliency!

 

Unfortunately though, the hardware resiliency wasn’t as up to scratch, and when it started to fail, RIM started having a catastrophic outage.

 

Now, you may have expected at that point for the active/active CEO cluster to step in and help. Unfortunately though, they’ve barely been heard from. So, in cluster terms, we have to assume a sort of reversed split-brain situation has occurred, where both components of the cluster think the other component is still running:

RIM-splitbrain

And there you have it – why RIM is having their current outage.

 

It’s also a lesson for all you other companies out there: you need fault tolerant infrastructure as well as CEOs.

 Posted by at 06:16  Tagged with: , , , , , ,

Auditing should be done only by the experts

 Backup theory, General Technology, General thoughts, NetWorker  No Responses »
Jul 282011
 

I’ve said it before – auditing should only be done by the experts. I first realised this when a security auditor from one of the (then) “Big 5″ accounting companies audited the Solaris servers I was administering 11 years ago. Having checked /etc/passwd, the auditor noted in the report:

All user passwords are set to *, which is highly insecure and should be addressed immediately to ensure continued security compliance.

The fallout from that was briefly atrocious, and resolved only by convincing a manager to try to log onto a list of user accounts using * as the password.

It appears that there’s still room for security auditors who don’t really understand security, as evidenced by “Our security auditor is an idiot, how do I give him the information he wants? – Server Fault“. The system administrator was told he had to handover the following as part of the audit:

  • A list of current usernames and plain-text passwords for all user accounts on all servers
  • A list of all password changes for the past six months, again in plain-text
  • A list of “every file added to the server from remote devices” in the past six months
  • The public and private keys of any SSH keys
  • An email sent to him every time a user changes their password, containing the plain text password

Up until this point, I thought that it would be impossible for anyone to have an experience to trump my “all user passwords are set to *” experience.

It turns out I was wrong.

What’s this got to do with backups, I hear you ask?

Well, everything. If your company is getting in auditors who aren’t subject matter experts (or at least product experts), then your audit isn’t worth the paper it’s written on. Maybe you’ll get a compliance rubber stamp. Maybe you won’t. But it won’t make one iota of difference as to whether there’s been any valid checking of your environment.

Please, ensure that if you want your backups audited you ask some experts in. Knowing the sorts of prices the “big” auditing companies charge, it’ll likely not only cost you less, but actually give you more!

 Posted by at 09:58  Tagged with: , , ,

Arrays, auto hot-spot migration and application performance tuning

 Architecture, General Technology, General thoughts  No Responses »
Jul 212011
 

I’m not a storage person, as I’ve been at pains to highlight in the past. My personal focus is at all times ILP, not ILM, and so I don’t get all giddy about array speeds and feeds, or anything along those lines.

Of course, if someone were to touch base with me tomorrow and offer me a free 10TB SSD array that I could fit under my desk, my opinion would change.

Queue the chirping crickets.

But seriously, in my “lay technical” view of arrays, I do have this theory and the problems introduced by hot spot migration, and I’m going to throw the theory out there with my reasoning.

First, the background:

  1. When I was taught to program, the credo was “optimise, optimise, optimise”. With limited memory and CPU functionality, we didn’t have the luxury to do lazy programming.
  2. With the staggering increase in processor speeds and memory, many programmers have lost focus on optimisation.
  3. Many second-rate applications can be deemed as such not by pure bugginess, but a distinct lack of optimisation.
  4. The transition from Leopard to Snow Leopard was a perfect example of the impacts of optimisation – the upgrade was about optimisation, not about major new features. And it made a huge difference.
And now, a classic example:
  1. In my first job, I was a system administrator for a very customised SAP system running on Tru64.
  2. Initially the system ran really smoothly all through the week.
  3. Over the 2-3 years I was administering, rumbling slowly developed that on Friday the system would get slower and slower.
  4. This always happened while people were entering their timesheets.
  5. Eventually, as part of Y2K remediation, someone took a look at the SQL commands used for timesheets, and noticed that someone had written a really bad query years ago which basically started by selecting all time sheet entries by all employees, then narrowing down. (Your classic problem of having an SQL query select the wrong results first.)
  6. This was fixed.
  7. System performance leapt through the roof.
  8. Users congratulated everyone on the fantastic “upgrade” that was done.
So, here’s my concern:
  1. For most applications, even complex ones these days, performance will be first IO bound before they become CPU or memory bound.
  2. Hot spot migration to faster media will mask, but not solve performance problems such as those described above.
  3. An application administrator (e.g., DBA) trying to solve application performance will find it challenging to resolve it around hot spot migration, particularly if they run multiple attempts to resolve the problem.
The problem, in short, is two-fold:
  1. First, hot spot migration will mask the problem.
  2. Second, hot spot migration will make problem debugging and resolution more problematic.
Clearly, there’s solutions to this. As someone said to me by reply today – a lot of what we do in IT already introduces these problems. It’s why, for instance, I’d never configure a NetWorker storage node as a virtual machine, because it’s using shared resources for performance. It’s why for instance, I’m always reluctant to use blades in the same situation. The solution, I think, is to to always be mindful of the following:
  1. Hot spot migration, while fantastic for handling load spikes, masquerades rather than solves application architecture/design issues.
  2. Hot spot migration, if supported by the array, but unknown by the application administrator, at best makes analysis and rectification extremely challenging, and at worst may actually make it impossible.
  3. It will always be important to have the option of turning off hot spot migration for deep analysis and debugging.
At least, that’s what I think. What do you think?
 Posted by at 18:39  Tagged with: , , , , , , , , ,

How indispensable is your company?

 Aside, General Technology, General thoughts  5 Responses »
Jul 172011
 

A wise man once said in a meeting:

If you want to see how indispensable you are, stick your finger in a glass of water and measure the size of the hole left when you pull it back out.

This week I’ve been reflecting a lot on that statement given the radical licensing changes that have originated out of VMware for vSphere 5.

I want to reflect on the background to the “are they right or are they wrong” argument here – I think every business is entitled to make a fair and reasonable profit. I also say this as an outsider – my area of interest remains backup and recovery, not virtualisation. In short, for me, virtualisation is a tool, a means to an end – it’s a butler, not the work.

So I think I can look at this as someone who is exposed to the business of virtualisation, but isn’t directly bound by it.

For any company that sells software rather than hardware, there are going to be times when licensing is re-evaluated and new cost models are developed. NetWorker for years had a licensing model that was growing in complexity. Over the last few years EMC has been working at simplifying that, with the most recent change being the capacity licensing. This hasn’t been a big hit because it’s more aimed at people who can’t quite step up to the enterprise license, rather than the average business, but it’s still a step in the right direction, and a portent of things to come.

VMware has clearly hit the point where they’re having to say to the market, “the way we’ve previously been pricing this is no longer sustainable”.

As has been so often the case within the IT industry over the past 20 years, pricing has raced to the bottom, and once it’s hit the bottom, there’s a need for an adjustment. I do partly blame Microsoft on this front – they’re renown for dropping their pricing pants in order to smack around the competition. That’s not a healthy business model.

Much is premised around a false sense of entitlement. “Someone produces X so I should get X for as cheap a price as possible”. It’s the logic of the IT industry, it seems. Yet let’s look at say, the car industry as a comparison. That business model – “get customers by giving it to them as cheap as possible” almost wiped out the US car industry. It was reported, for instance, that between the rebates and the discounts on offer by 2008, some US car companies were losing up to $500 per vehicle sold.

Selling volume at discount is fine.

Selling volume at loss isn’t.

VMware are by no means indispensable in the IT industry. The pricing model change will undoubtedly drive some companies to consider the alternatives out there – Hypervisor, Xen and Parallels, for instance.

But I think we, as an industry, have to take some responsibility here – we have to accept our part that this is a mea culpa of sorts: we’ve allowed the “race to the bottom” pricing model to become too pervasive, and are now getting to reap the rewards of that.

 Posted by at 08:49  Tagged with: , , ,

Technology is not the solution

 Architecture, General Technology, General thoughts  No Responses »
Jun 192011
 

Earlier in the year, I wrote a post, “Technology is rarely the issue“. In that post, I said:

As techos though, let’s be honest. The technology is rarely the issue. Or to be more accurate, if there’s an issue, technology is the tip of the iceberg – the visible tip. And using the iceberg analogy, you know I mean that technology is rarely going to be the majority of the issue.

Now it’s time for the follow-up.

In that article, I was effectively talking about specific situations – e.g., when someone says to you, “product X is crap; we’ve had it for Y months and it still doesn’t work properly”. While sometimes it will mean that product X is bad, it usually means that the wrong product was purchased, or there wasn’t enough training, or it’s being misused.

In this post, I want to turn from the specific, to the generic, and suggest that it is rarely going to be the case moving forward that technology is the solution. It will be part of the solution, but we are moving out of a situation where a single piece of technology is the entire solution to a problem. In fact I’d suggest that it was rarely the case anyway, but we must be more aware as technology continues to get more powerful – it’s not a magic bullet.

For this reason, the core emerging technology – that we must continue to demand from vendors, and continue to support the development of – is interoperability. This may come from open standards, or it may come from virtualisation, but it has to become core to all future technology.

Why?

We no longer have the luxury of mass swapping and changing of technology. Martin Glasborow, AKA Storagebod, wrote in “Migration is a way of life“:

One of the things which is daunting is the sheer amount of data that we are beginning to ingest and the fact that we are currently looking at a ‘grow forever’ archive; everything we ingest, we will keep forever.

Even though we are less than two years into the project, we are already thinking about the next refresh of technology. And what is really daunting is that with our data growth; once we start refreshing, I suspect that we will never stop.

Not only will we be storing petabytes of new content every year; we will be moving even more old content between technologies every year. We are already looking at moving many hundreds of terabytes into the full production system without impacting operations and with little to no downtime.

While Martin’s organisation is undoubtedly at the “big data” end of town, it reflects a growing problem for many organisations – the shrinking grace period. Previously we had scenarios where capital expenditure periods of say, 3 years worth of equipment purchase, would have short implementation periods, followed by long-term controlled and pre-allocated growth periods, followed by the final preparatory process leading into the next CapEx cycle.

This is increasingly becoming a luxury. As data growth continues, regardless of whether that data is hosted locally or externally, mass migration projects will become a thing of the past. It’s not possible to stop a business long enough to do a migration. They have to run seamlessly and synchronously in the background, transparent to users and the business, and the only way this will happen is via interoperability.

The two methods to achieve this are either compatible APIs/protocols, and virtualisation. In the cloud space for instance, whatever brick level storage is chosen, only a fool would deploy their business storage on just a single cloud provider. So you need two different providers, and you need to be able to interface with the same storage at both providers without every access step being an “If writing to Cloud X, this way, else that way.”

For locally accessible storage, virtualisation is critical – not just at the OS layer, but also at the storage layer. That way, it doesn’t matter whether you’re currently buying vendor X, Y or Z arrays and storage – and which ones are currently active. It should all be transparent to the business.

This is why technology is not the solution. Or rather, specific technology is not the solution. It’s the application of technology, and the interoperability of currently deployed technologies that will be the solution every time.

If you’re not thinking along these lines, you’re still staring into the past.

 Posted by at 19:57  Tagged with: , , ,

Adherents to the Repeated Meme

 Architecture, General Technology, General thoughts  1 Response »
Jun 102011
 

How often have you heard these two memes?

“Tape Sucks”

“Tape is dead”

Oh it just goes on and on, and on and on and on. One might think that I’m having a dig at EMC and Data Domain here – particularly in light of my response on another topic’s comment thread here. And while some folk at EMC and Data Domain would technically be in my sights on this post, there’ll equally be folks from NetApp and a plethora of other vendors who think that tape is dead. So I’m not so much picking on any company, just the meme itself.

It’s the same story, over and over again. Some new whiz-bang product comes out, and people jump onto the “tape is dead” bandwagon again. Only like a really bad villain in a superhero movie, tape just won’t die. It has more lives than every cat in the world combined.

Sure, its use has evolved over time. I’m the first to admit that. When I first started in backup myself, the notion of backing up to disk was a complete anathema. After all, I had to beg, borrow, plead and promise long on-call shifts just to get a couple of extra 2GB spindles for my backup server to handle indices and temp space. Why would I have been so crazy as to backup to such an expensive medium? Tape, on the other hand, was much cheaper.

Over time disk became cheaper and had higher capacities, but it still isn’t as cheap or as high capacity as tape over the long haul. Where it exceeds tape every time is on the economics of access. You need that data back straight away? Then it needs to come back from disk, not tape. There’s no load times, etc., when it comes to disk.

And so over time as disk became cheaper, we (the industry) evolved backups to use tape as secondary, long term or high capacity storage. Backup to disk, keep the most frequently recovered backups on that medium (i.e., the most recent), and keep copies on tape. As space fills, we shift those older backups off to tape, and keep using disk for the high frequency recoveries. Disk also smooths out those pesky shoe-shining issues we see in highly varied streaming speeds to tape, too.

So it’s a win-win solution, and it’s going to stay that way for some time to come. Tape may have evolved, but it’s still better, cheaper, and more reliable for longer term storage. Curtis Preston has an excellent summary of this point here, for what it’s worth.

Will the “tape is dead” people come around to reality? Probably not. Adherents to the repeated meme don’t always give up so easily. After all, there’s even people who still believe in a flat earth.

 Posted by at 14:17  Tagged with: ,
 Older Entries
© 2012 The NetWorker Blog Suffusion theme by Sayontan Sinha