I’d like to suggest that we should specify that “percentage complete” estimates – be they progress bars or sliders or any other representation, visual or textual, need a defined unit of measurement to them.

And we should define that unit of measurement as a maybe.

That is, if a piece of software reports that it is 98% complete at something, that’s 98 maybes out of a 100.

I perhaps, should mention, that I’m not thinking of NetWorker when I make this case. Indeed, it’s actually springing from spending 4+ hours one day monitoring a backup job from one of NetWorker’s competitors. A backup job that for the entire duration was at … 99% complete.

You see, in a lot of software, progress indicators just aren’t accurate. This lead to the term “Microsoft minute”, for instance, to describe the interminable reality bending specification of time remaining on file copies in Microsoft operating systems. Equally we can say the same thing of software installers; an installer may report that it’s 95% complete with 1 minute remaining for anywhere between 15 seconds and 2 hours – or more. It’s not just difficult to give an upper ceiling, it’s indeterminate.

I believe that software which can’t measure its progress with sufficient accuracy shouldn’t give an actual percentage complete status or time to complete status without explicitly stating it as being an estimate. To fail to do so is an act of deceit to the user.

I would also argue that no software can measure its process with sufficient accuracy, and thus all software should provide completion status as an estimate rather than a hard fact. After all:

• Software cannot guarantee against making a blocking IO call
• Software cannot guarantee that the operating system will not take resources away from it
• Software cannot guarantee that a physical fault will not take resources away from it

In a real-time and fault-tolerant system, there is a much higher degree of potential accuracy. Outside of that – in regular software (commercial or enterprise), and on regular hardware/operating systems, the potential for interruption (and therefore, inaccuracy) is too great.

I don’t personally think it’s going to hurt interface designers to clearly state whenever a completion estimate is given that it’s an estimate. Of course, some users won’t necessarily notice it, and others will ignore it – but by blatantly saying it, they’re not implicitly raising false hope by citing an indeterminate measurement as accurate.

Share this using...

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

Dark data isn’t the end of the story, and it’s time to introduce the entire family of data-awareness concepts. These are:

• Data – This is both the core data managed and protected by IT, and all other data throughout the enterprise which is:
• Known about – The business is aware of it;
• Managed – This data falls under the purview of a team in terms of storage administration (ILM);
• Protected – This data falls under the purview of a team in terms of backup and recovery (ILP).
• Dark Data – To quote the previous article, “all those bits and pieces of data you’ve got floating around in your environment that aren’t fully accounted for”.
• Grey Data – Grey data is previously discovered dark data for which no decision has been made as yet in relation to its management or protection. That is, it’s now known about, but has not been assigned any policy or tier in either ILM or ILP.
• Utility Data – This is data which is subsequently classified out of grey data state into a state where the data is known to have value, but is not either managed or protected, because it can be recreated. It could be that the decision is made that the cost (in time) of recreating the data is less expensive than the cost (both in literal dollars and in staff-activity time) of managing and protecting it.
• Noise – This isn’t really data at all, but are all the “bits” (no pun intended) that are left which are neither grey data, data or utility data. In essence, this is irrelevant data, which someone or some group may be keeping for unnecessary reasons, and in actual fact should be considered eligible for either deletion or archival and deletion.

The distribution of data by awareness within the enterprise may resemble something along the following lines:

That is, ideally the largest percentage of data should be regular data which is known, managed and protected. In all likelihood for most organisations, the next biggest percentage of data is going to be dark data – the data that hasn’t been discovered yet. Ideally however, after regular and dark data have been removed from the distribution, there should be at most 20% of data left, and this should be broken up such that at least half of that remaining data is utility data, with the last 10% split evenly between grey data and noise.

The logical implications of this layout should be reasonably straight forward:

1. At all times the majority of data within an organisation should be known, managed and protected.
2. It should be expected that at least 20% of the data within an organisation is undiscovered, or decentralised.
3. Once data is discovered, it should exist in a ‘grey’ state for a very short period of time; ideally it should be reclassified as soon as possible into data, utility data or noise. In particular, data left in a grey state for an extended period of time represents just as dangerous a potential data loss situation as dark data.

It should be noted that regular data, even in this awareness classification scheme, will still be subject to regular data lifecycle decisions (archive, tiering, deletion, etc.) In that sense, primary data eligible for deletion isn’t really noise, because it’s previously been managed and protected; noise really is ex dark-data that will end up being deleted, either as an explicit decision, or due to a failure at some future point after the decision to classify it as ‘noise’, having never been managed or protected in a centralised, coordinated manner.

Equally, utility data won’t refer to say, Q/A or test databases that replicate the content of production databases. These types of databases will again have fallen under the standard data umbrella in that there will have been information lifecycle management and protection policies established for them, regardless of what those policies actually were.

If we bring this back to roles, then it’s clear that a pivotal role of both the DPAs (Data Protection Advocates) and the IPAC (Information Protection Advisory Council) within an organisation should be the rapid coordination of classification of dark data as it is discovered into one of the data, utility data or noise states.

Share this using...

Related posts:

1. Dark Data We’ve all heard the term Big Data - it’s something the...
2. The 7 Challenges of Data Protection It struck me recently while working on a report that...
3. But where does the DPA fit in? In yesterday’s post, I suggested that it was time for...

Related posts brought to you by Yet Another Related Posts Plugin.

In short, any company that sees cloud – public cloud – as a solution to becoming more flexible and cost competitive don’t actually understand the real problem they’re facing.

That problem? Business/IT alignment.

One of the oft-touted advantages of public cloud is that it’s about enabling businesses to react more quickly to changing IT requirements than their own IT departments do.

How many people can honestly look at that description and think that the solution is to cut off the IT limb from the business body?

The solution – the real solution – is to address whatever divide there is between the business and IT. And have no doubt – if blame is to be laid it will likely need to be laid with equal measure at both the feet of IT and the rest of the business.

Shifting to public cloud instead says that the business itself is unwilling to seriously look at its own function, behaviour and attitudes and address real issues. It’s the business hoping for an “abracadabra” solution rather than working on a real solution.

It’s like having an arterial bleed, and sticking a bandaid on it.

Share this using...

Related posts:

1. Technology is not the solution Earlier in the year, I wrote a post, “Technology is...

Related posts brought to you by Yet Another Related Posts Plugin.

So, the simple question really is this – is “big data” a legitimate or an illegitimate problem?

By legitimate – is it a problem which truly exists in and of itself? Has data growth in places hit a sufficiently exponential curve that existing technology and approaches can’t keep up …

OR

… is it an illegitimate problem, in that it speaks of (a) a dumbing down of computer science which has resulted in a lack of developmental foresight into problems which we’ve seen coming for some time and/or (b) a failure of IT companies (from base component manufacturers through to vendors across the board) failing to sufficiently innovate?

For me, the jury is still out, and I’ll use a simple example as to why. I deal with big data regularly – since “big data” is defined as being anything outside of a normal technical scope, if I get say, a 20 GB log file from a customer that I have to analyse, none of my standard tools assist with this. So instead, I have to start working on pattern analysis – rather than trying to extract what may be key terms or manually read the file, I’ll skim through it – I’ll literally start by “cat”ting the file and just letting it stream in front of me. At that level, if the software has been written correctly, you’ll notice oddities in the logs that start you pointing to the area you have to delve into. You can then refine the skimming, and eventually drill down to the point where you actually just analyse a very small fragment of the file.

So I look at big data and think – is this a problem caused by a lack of AI being applied to standard data processing techniques? Of admitting – we need to build a level of heuristic decision making into standard products so they can scale up to deal with ever increasing data sets? That the solution is more intelligence and self-management capabilities in the software and hardware? And equally, of developers failing to produce systems that generate data in such a way that it’s susceptible to automated types of pattern analysis?

Of course, this is, to a good degree, what people are talking about when they’re talking about big data.

But why? Do we gain any better management and analysis by cleaving “data” and “big data” into two separate categories?

Or is this a self-fulfilling meme that came out as a result of poor approaches to information science?

Share this using...

Related posts:

1. 5 questions for Drobo I love my Drobo. I have a 4-drive FW800/USB-2 unit,...
2. The 7 Challenges of Data Protection It struck me recently while working on a report that...

Related posts brought to you by Yet Another Related Posts Plugin.

In case it’s not been immediately obvious to anyone, I’ve done some simple diagrams to explain where RIM went wrong in this catastrophic outage they’ve been suffering.

You see, most companies implement what we call redundant infrastructure. In systems that require high availability, this is often accomplished with something as simple as clustered (either LAN or WAN) hardware and communications. Sometimes it’s designed that each component runs at the same time, sharing the load, but if one fails, the other one takes over and runs all the load. In simple terms, it looks like this:

That all makes sense, right?

Unfortunately, RIM seemed more focused on having failover capabilities for upper level management, so it instead clustered its’ CEOs:

The supposed theory behind this is that the two CEOs, working in an active/active arrangement, could handle load better and get the job done better than a single CEO – and provide resiliency!

Unfortunately though, the hardware resiliency wasn’t as up to scratch, and when it started to fail, RIM started having a catastrophic outage.

Now, you may have expected at that point for the active/active CEO cluster to step in and help. Unfortunately though, they’ve barely been heard from. So, in cluster terms, we have to assume a sort of reversed split-brain situation has occurred, where both components of the cluster think the other component is still running:

And there you have it – why RIM is having their current outage.

It’s also a lesson for all you other companies out there: you need fault tolerant infrastructure as well as CEOs.

Share this using...

Related posts:

1. This is wrong In “Distribute.IT reveals shared server data loss – News –...
2. If you wouldn’t drink it, don’t cook with it… There’s a pertinent adage in cooking when it comes to...

Related posts brought to you by Yet Another Related Posts Plugin.

If you wouldn’t drink it, don’t cook with it.

It’s simple: if you don’t like the taste of it in a glass, what makes you think you’ll like the taste of food you’ve added it to?

There are two similar rules for backup, and they’re particularly important when it comes time to do those periodic hardware refreshes in your environment:

If it’s not good enough to run production, don’t use it for DR.

If it’s not good enough to run production, don’t use it for backup.

The way in which both of these come into play is quite simple:

1. If it’s not good enough to run production, don’t use it for DR. I’ve seen companies have a hardware refresh cycle of “move production equipment to DR, buy new production equipment”. However, invariably that equipment is being pulled out of production because it’s either lacking in capacity, or lacking in performance. That equipment is then going to be replaced with new equipment with planned usage time of (typically) 2-3 years. So let’s assume you get a year down the track – your in-use storage capacity has gone up, your processing load has increased, then there’s a major production fault and you have to failover to DR. At which point, you’re trying to run your production environment on something that was sized to max out 12 months ago. Chances of it adequately running production? Minimal.
2. If it’s not good enough to run production, don’t use it for backup. Another common mistake is a situation whereby say, a storage array is pulled out of production and replaced with a new, faster array with more capacity. People invariably hate to see things go to waste, so someone suggests “let’s use the old array as {backup to disk | VTL | etc}”. Again, sounds simple enough on the face of it, except the equipment was either lacking in performance, or lacking in capacity. If it was lacking in performance, you’re putting it into a situation where you’re going to be copying off something that is purchased, on the outset, to be significantly faster than it. It’s similar with capacity – you’re going to be trying to backup a very large bucket to a much smaller bucket.

Whether your company likes the idea of it or not, backup and disaster recovery are not areas that should be assigned “hand me downs” by the rest of the business. They require their own capital budget, and a planning that allows for the following two factors:

1. Performance should at least match the throughput on offer from production;
2. It should exceed your production capacity.

If either of these conditions are not met, your strategy is insufficient.

Share this using...

Related posts:

1. 7 common problems with deduplication In an earlier article, I suggested some space management techniques...

Related posts brought to you by Yet Another Related Posts Plugin.

That’s something I’ll blog about here later, but it led to another discussion, which effectively came down to that fear that many people in IT, and in fact, business overall, seem to have towards DBAs.

The fear is sometimes so much that it’s a wonder cubicle maps don’t look something like this:

As a consultant, I’ve gone to many environments – and in my previous work as a system administrator, I dealt with a variety of situations, and in my time I’ve come across my fair share of database administrators.

As I mentioned in my book, DBAs have a duty of care towards the databases they’re responsible for, and it’s fair to say that in 99.99% of cases the DBAs that I’ve encountered have been passionately cognisant of that duty of care, and have taken it very, very seriously.

But it’s time to call a spade a spade, and also acknowledge that maybe up to half of the time, the DBAs at sites are viewed with fear, as if there’s a dragon walking around the hallway. There’s some common stereotypes: volatile tempers, intransigence, inflexibility and, well, blunt. In actual fact, there are people of this personality type regularly scattered across all of IT, regardless of business function, but for some reason, we seem to notice it most in DBAs. (Maybe that’s because they tend to also be so highly passionate about what they do.)

So why do people get away with that kind of volatile behaviour? Because the business lets them be that way.

This is a classic management problem, but it ends up reflecting poorly on IT. I think this partly stems from the origin of most IT managers. Particularly at the team leader level, and their immediate superiors, management have been pushed up out of technical roles into management roles. In most businesses, this happens because of a few key reasons:

• the person is technically competent enough to mentor new staff
• the person is able to be organised
• the person is able to get along with colleagues

Those qualities alone don’t make someone a manager. Managers also have to deal with conflict resolution, and people who have come up from a purely technical role in IT into management because of those qualities won’t necessarily have conflict resolution skills.

If you have staff on site who either have anger management issues, or are strongly confrontational, but management who aren’t equipped to work in conflict resolution, you have a problem brewing that will be obvious to anyone who walks onto your site. If you have to, at the end of a meeting, pull someone aside and apologise for the behaviour of someone else at the meeting, then it’s obvious there’s a problem that needs to be solved.

It’s time we start taming dragons in IT. Of course, this isn’t just about DBAs – that was just a way of kick starting this discussion. I’ve equally seen people with those personality traits in storage, in virtualisation, in backup, in email, in general system administration. We all have. If you’re still reading this, there’s a high degree that you’re not one of those people, by the way. (If you are one of those people, you’re likely either already deleting this blog from your bookmarks, or penning a strongly worded comment!)

No business should be ‘afraid’ of its staff; furthermore, everyone should remember the old adage:

If you want to know how irreplaceable you are, stick your finger in a glass of water and measure the size of the hole that you leave behind.

Just because someone is good at what they do shouldn’t excuse poor behaviour. I’ve seen environments where that happens – most notably at stockbroking companies. In those companies, the traders who are making good money for the company get away with almost anything. One stockbroking firm I used to work for maintained detailed logs of people who downloaded pornography at work. At the start of 2000, some traders were downloading over 1GB a month of porn, at work, and not getting punished. Why? Because they made the company money. Anyone who made that list who wasn’t a trader though … heaven help them. It was hypocrisy exemplified.

Poor behaviour is poor behaviour – and just because someone is damn good at what they do, or someone works on something that is damn important to the company doesn’t mean they should be allowed to run rough-shod over other staff.

The problem when you have dragons in the environment is that they’re usually highly resistant to change. There may be very valid business reasons on why something should be done, but if the dragon (sometimes literally ROARS) “NO!”, then everyone pales back and whispers “OK, please don’t eat us!” and lets the dragon go back to sleep. And while the dragon sleep, the business atrophies.

It’s time we start tearing up all those cubicle maps that have “Here be dragons!” on them, regardless of what job the dragon does.

Share this using...

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

All user passwords are set to *, which is highly insecure and should be addressed immediately to ensure continued security compliance.

The fallout from that was briefly atrocious, and resolved only by convincing a manager to try to log onto a list of user accounts using * as the password.

It appears that there’s still room for security auditors who don’t really understand security, as evidenced by “Our security auditor is an idiot, how do I give him the information he wants? – Server Fault“. The system administrator was told he had to handover the following as part of the audit:

• A list of current usernames and plain-text passwords for all user accounts on all servers
• A list of all password changes for the past six months, again in plain-text
• A list of “every file added to the server from remote devices” in the past six months
• The public and private keys of any SSH keys
• An email sent to him every time a user changes their password, containing the plain text password

Up until this point, I thought that it would be impossible for anyone to have an experience to trump my “all user passwords are set to *” experience.

It turns out I was wrong.

What’s this got to do with backups, I hear you ask?

Well, everything. If your company is getting in auditors who aren’t subject matter experts (or at least product experts), then your audit isn’t worth the paper it’s written on. Maybe you’ll get a compliance rubber stamp. Maybe you won’t. But it won’t make one iota of difference as to whether there’s been any valid checking of your environment.

Please, ensure that if you want your backups audited you ask some experts in. Knowing the sorts of prices the “big” auditing companies charge, it’ll likely not only cost you less, but actually give you more!

Share this using...

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

Of course, if someone were to touch base with me tomorrow and offer me a free 10TB SSD array that I could fit under my desk, my opinion would change.

Queue the chirping crickets.

But seriously, in my “lay technical” view of arrays, I do have this theory and the problems introduced by hot spot migration, and I’m going to throw the theory out there with my reasoning.

First, the background:

1. When I was taught to program, the credo was “optimise, optimise, optimise”. With limited memory and CPU functionality, we didn’t have the luxury to do lazy programming.
2. With the staggering increase in processor speeds and memory, many programmers have lost focus on optimisation.
3. Many second-rate applications can be deemed as such not by pure bugginess, but a distinct lack of optimisation.
4. The transition from Leopard to Snow Leopard was a perfect example of the impacts of optimisation – the upgrade was about optimisation, not about major new features. And it made a huge difference.

Share this using...

Related posts:

1. Well, it performs – or performance? We are approaching the point where it would be conceivable...
2. Technology is not the solution Earlier in the year, I wrote a post, “Technology is...

Related posts brought to you by Yet Another Related Posts Plugin.

If you want to see how indispensable you are, stick your finger in a glass of water and measure the size of the hole left when you pull it back out.

This week I’ve been reflecting a lot on that statement given the radical licensing changes that have originated out of VMware for vSphere 5.

I want to reflect on the background to the “are they right or are they wrong” argument here – I think every business is entitled to make a fair and reasonable profit. I also say this as an outsider – my area of interest remains backup and recovery, not virtualisation. In short, for me, virtualisation is a tool, a means to an end – it’s a butler, not the work.

So I think I can look at this as someone who is exposed to the business of virtualisation, but isn’t directly bound by it.

For any company that sells software rather than hardware, there are going to be times when licensing is re-evaluated and new cost models are developed. NetWorker for years had a licensing model that was growing in complexity. Over the last few years EMC has been working at simplifying that, with the most recent change being the capacity licensing. This hasn’t been a big hit because it’s more aimed at people who can’t quite step up to the enterprise license, rather than the average business, but it’s still a step in the right direction, and a portent of things to come.

VMware has clearly hit the point where they’re having to say to the market, “the way we’ve previously been pricing this is no longer sustainable”.

As has been so often the case within the IT industry over the past 20 years, pricing has raced to the bottom, and once it’s hit the bottom, there’s a need for an adjustment. I do partly blame Microsoft on this front – they’re renown for dropping their pricing pants in order to smack around the competition. That’s not a healthy business model.

Much is premised around a false sense of entitlement. “Someone produces X so I should get X for as cheap a price as possible”. It’s the logic of the IT industry, it seems. Yet let’s look at say, the car industry as a comparison. That business model – “get customers by giving it to them as cheap as possible” almost wiped out the US car industry. It was reported, for instance, that between the rebates and the discounts on offer by 2008, some US car companies were losing up to $500 per vehicle sold.

Selling volume at discount is fine.

Selling volume at loss isn’t.

VMware are by no means indispensable in the IT industry. The pricing model change will undoubtedly drive some companies to consider the alternatives out there – Hypervisor, Xen and Parallels, for instance.

But I think we, as an industry, have to take some responsibility here – we have to accept our part that this is a mea culpa of sorts: we’ve allowed the “race to the bottom” pricing model to become too pervasive, and are now getting to reap the rewards of that.

Share this using...

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.