Talking about Ransomware

 Architecture, Backup theory, General thoughts, Recovery, Security  Comments Off on Talking about Ransomware
Sep 062017
 

The “Wannacry” Ransomware strike saw a particularly large number of systems infected and garnered a great deal of media attention.

Ransomware Image

As you’d expect, many companies discussed ransomware and their solutions for it. There was also backlash from many quarters suggesting people were using a ransomware attack to unethically spruik their solutions. It almost seems to be the IT equivalent of calling lawyers “ambulance chasers”.

We are (albeit briefly, I am sure), between major ransomware outbreaks. So, logically that’ll mean it’s OK to talk about ransomware.

Now, there’s a few things to note about ransomware and defending against it. It’s not as simplistic as “I only have to do X and I’ll solve the problem”. It’s a multi-layered issue requiring user education, appropriate systems patching, appropriate security, appropriate data protection, and so on.

Focusing even on data protection, that’s a multi-layered approach as well. In order to have a data protection environment that can assuredly protect you from ransomware, you need to do the basics, such as operating system level protection for backup servers, storage nodes, etc. That’s just the beginning. The next step is making sure your backup environment itself follows appropriate security protocols. That’s something I’ve been banging on about for several years now. That’s not the full picture though. Once you’ve got operating systems and backup systems secured via best practices, you need to then look at hardening your backup environment. There’s a difference between standard security processes and hardened security processes, and if you’re worried about ransomware this is something you should be thinking about doing. Then, of course, if you really want to ensure you can recover your most critical data from a serious hactivism and ransomware (or outright data destruction) breach, you need to look at IRS as well.

But let’s step back, because I think it’s important to make a point here about when we can talk about ransomware.

I’ve worked in data protection my entire professional career. (Even when I was a system administrator for the first four years of it, I was the primary backup administrator as well. It’s always been a focus.)

If there’s one thing I’ve observed in my career in data protection is that having a “head in the sand” approach to data loss risk is a lamentably common thing. Even in 2017 I’m still hearing things like “We can’t back this environment up because the project which spun it up didn’t budget for backup”, and “We’ll worry about backup later”. Not to mention the old chestnut, “it’s out of warranty so we’ll do an Icarus support contract“.

Now the flipside of the above paragraph is this: if things go wrong in any of those situations, suddenly there’s a very real interest in talking about options to prevent a future issue.

It may be a career limiting move to say this, but I’m not in sales to make sales. I’m in sales to positively change things for my customers. I want to help customers resolve problems, and deliver better outcomes to their users. I’ve been doing data protection for over 20 years. The only reason someone stays in data protection that long is because they’re passionate about it, and the reason we’re passionate about it is because we are fundamentally averse to data loss.

So why do we want to talk about defending against or recovering from ransomware during a ransomware outbreak? It’s simple. At the point of a ransomware outbreak, there’s a few things we can be sure of:

  • Business attention is focused on ransomware
  • People are talking about ransomware
  • People are being directly impacted by ransomware

This isn’t ambulance chasing. This is about making the best of a bad situation – I don’t want businesses to lose data, or have it encrypted and see them have to pay a ransom to get it back – but if they are in that situation, I want them to know there are techniques and options to prevent it from striking them again. And at that point in time – during a ransomware attack – people are interested in understanding how to stop it from happening again.

Now, we have to still be considerate in how we discuss such situations. That’s a given. But it doesn’t mean the discussion can’t be had.

To me this is also an ethical consideration. Too often the focus on ethics in professional IT is around the basics: don’t break the law (note: law ≠ ethics), don’t be sexist, don’t be discriminatory, etc. That’s not really a focus on ethics, but a focus on professional conduct. Focusing on professional conduct is good, but there must also be a focus on the ethical obligations of protecting data. It’s my belief that if we fail to make the best of a bad situation to get an important message of data protection across, we’re failing our ethical obligations as data protection professionals.

Of course, in an ideal world, we’d never need to discuss how to mitigate or recover from a ransomware outbreak during said outbreak, because everyone would already be protected. But harking back to an earlier point, I’m still being told production systems were installed without consideration for data protection, so I think we’re a long way from that point.

So I’ll keep talking about protecting data from all sorts of loss situations, including ransomware, and I’ll keep having those discussions before, during and after ransomware outbreaks. That’s my job, and that’s my passion: data protection. It’s not gloating, it’s not ambulance chasing, it’s let’s make sure this doesn’t happen again.


On another note, sales are really great for my book, Data Protection: Ensuring Data Availability, released earlier this year. I have to admit, I may have squealed a little when I got my first royalty statement. So, if you’ve already purchased my book: you have my sincere thanks. If you’ve not, that means you’re missing out on an epic story of protecting data in the face of amazing odds. So check it out, it’s in eBook or Paperback format on Amazon (prior link), or if you’d prefer to, you can buy direct from the publisher. And thanks again for being such an awesome reader.

Ransomware is a fact of life

 Data loss, Security  Comments Off on Ransomware is a fact of life
Feb 012017
 

The NetWorker usage survey for 2016 has just finished. One of the questions I asked in this most recent survey was as follows:

Has your business been struck by ransomware or other data destructive attacks in the past year?

(_) Yes

(_) No

(_) Don’t know

(_) Prefer not to say

With the survey closed, I wanted to take a sneak peek at the answer to this question.

Ransomware, as many of you would know, is the term coined for viruses and other attacks that lead data erased or encrypted, with prompts to pay a ‘ransom’ in order to get the money back. Some businesses may choose to pay the ransom, others choose not to. If you’ve got a good data protection scheme you can save yourself from a lot of ransomware situations, but the looming threat – which is something that has already occurred in some instances – is ransomware combined with systems penetration, resulting in backup servers being deliberately compromised and data-destructive attacks happening on primary data. I gave an example of EMC’s solution to that sort of devastating 1-2 punch attack last November.

Ransomware is not going away. We recently saw massive numbers of MongoDB databases being attacked, and law enforcement agencies are considering it a growing threat and a billion dollar a year or more industry for the attackers.

So what’s the story then with NetWorker users and ransomware? There were 159 respondents to the 2016 NetWorker usage survey, and the answer breakdown was as follows:

  • No – 48.43%
  • Don’t know – 11.32%
  • Prefer not to say – 9.43%
  • Yes – 30.82%

An August 2016 article in the Guardian suggested that up to 40% of businesses had been hit by ransomware, and by the end of 2016 other polls were suggesting the number was edging towards 50%.

Ransomware Percentages

I’m going to go out on a limb and suggest that at least 50% of respondents who answered “Prefer not to say” were probably saying it because it’s happened and they don’t want to mention it. (It’s understandable, and very common.) I’ll also go out on a limb and suggest that at least a third of respondents who answered “Don’t know” probably had been but it might have been resolved through primary storage or other recovery options that left individual respondents unaware.

At the very base numbers though, almost 31% of respondents knew they definitely had been hit by ransomware or other data-destructive attacks, and with those extrapolations above we might be forgiven for believing that the number was closer to 38.9%.

The Guardian article was based on a survey of Fortune 500 senior IT executives, and ransomware at its most efficacious is targeted and combined with other social engineering techniques such as spear phishing, so it’s no wonder the “big” companies report high numbers of incidents – they’re getting targeted more deliberately. The respondents on the NetWorker survey however came from all geographies and all sizes, ranging from a few clients to thousands or more.

Bear in mind that being hit by ransomware is not a case of “lightning never strikes twice”. At a briefing I went to in the USA last year, we were told that one business alone had been hit by 270+ cases of ransomware since the start of the year. Anecdotally, those customers of mine even who mention having been hit by ransomware talk about it in terms of multiple situations, not just a single one.

Now as much as ever before, we need robust data protection, and air-gapped data protection for sensitive data – the Isolated Recovery Site (IRS) is something you’ll hear more of as ransomware gets more prevalent.

NetWorker users have spoken – ransomware is a real and tangible threat to businesses around the world.

I’ll be aiming to have the full report published by mid-February, and I’ll contact the winner of the prize at that time too.

Basics – Configuring a reports-only user

 NetWorker, Security  Comments Off on Basics – Configuring a reports-only user
May 252015
 

Something that’s come up a few times in the last year for me has been a situation where a NetWorker user has wanted to allow someone to access NetWorker Management Console for the purpose of running reports, but not allow them any administrative access to NetWorker.

It turns out it’s very easy to achieve this, and you actually have a couple of options on the level of NetWorker access they’ll get.

Let’s look first at the minimum requirements – defining a reports only user.

To do that, you first go into NetWorker Management Console as an administrative user, and go across to the Setup pane.

You’ll then create a new user account:

New User Account in NMC

Within the Create User dialog, be certain to only select Console User as the role:

NMC new user dialog

At this point, you’ve successfully created a user account that can run NMC reports, but can’t administer the NetWorker server.

However, you’re then faced with a decision. Do you want a reports-only user that can “look but don’t touch”, or do you want a reports-only user that can’t view any of the NetWorker configuration (or at least, anything other than can be ascertained by the reports themselves)?

If you want your reports user to be able to run reports and you’re not fussed about the user being able to view the majority of your NetWorker configuration, you’re done at this point. If however your organisation has a higher security focus, you may need to look at adjusting the basic Users NetWorker user group. If you’re familiar with it, you’ll know this has the following configuration:NetWorker Users Usergroup

This usergroup in the default configuration allows any user in the NetWorker datazone to:

  • Monitor NetWorker
  • Recover Local Data
  • Backup Local Data

The key there is any user*@*. Normally you want this to be set to *@*, but if you’re a particularly security focused organisation you might want to tighten this down to only those users and system accounts authorised to perform recoveries. The same principle applies here. Let’s say I didn’t want the reports user to see any of the NetWorker configuration, but I did want any root, system or pmdg user in the environment to still have that basic functionality. I could change the Users usergroup to the following:

Modified NetWorker Users usergroup

With this usergroup modified, logging in as the reports user will show a very blank NMC monitoring tab:

NMC-monitoring reports user

Similarly, the client list (as an example) will be quite empty too:

NMC-config reports user

Now, it’s worth mentioning there are is a key caveat you should consider here – some modules may be designed in anticipation that the executing user for the backup or recovery (usually an application user with sufficient privileges) will at least be a member of the Users usergroup. So if you tighten the security against your reports user to this level, you’ll need to be prepared to increase the steps in your application onboarding processes to ensure those accounts are added to an appropriate usergroup (or a new usergroup).

But in terms of creating a reports user that’s not privileged to control NetWorker, it’s as easy as the steps above.

Mar 022015
 

Lock

Backup security is – or rather should be a big requirement in any organisation. Sadly we still see periodic examples of where organisations fail to fully comprehend the severity of a backup breach. In a worst case scenario, a backup breach involving physical theft might be the equivalent of someone having permanent and unchecked access to a snapshot of your entire network.

There are two distinct aspects to backup security to consider:

  • Physical
  • Electronic

For each type of backup security, we need to consider two key areas:

  • At rest
  • In transit

This usually leads businesses to start backup security planning by considerations such as:

  • Do we encrypt backup media?
  • Do we used security guards for movement of backup media?
  • Are on-disk backups encrypted?

Oddly enough, there’s a bigger gorilla in the room for backup security that is less often thought of: your backups are only as secure as the quality of and your/their adherence to your security policies.

A long time ago in a state far, far away, a colleague was meeting with a system administrator in the offices of an environmental organisation. She needed to ensure the security restrictions for system access could be drastically lowered from the default install criteria. “Everyone here is an anti-authority hippy” she said (or words to that effect), “If we give them hard passwords they’ll just write the in permanent marker on their monitors.”

The solution was to compromise to a mid-point of security vs ease-of-access.

These days few organisations would yield to their users disdain for authority so readily, but it serves to highlight that a system is only as secure as you choose to make it. A backup environment does not sit in isolation – it resides on the hosts it is being used to protect (in some form or another), and it will hae a host-based presence within your network at some point. If someone can breach that security and get onto one of thoe hosts, there’s a good chance a significant aspect of your backup security protocols have been breached as well.

That’s why all backup security has to start at a level outside the backup environment … rather, it requires consideration at all layers. It doesn’t start with the complexity of the password required to access an administrator interface, and nor does it end with enabling data-at-rest encryption. So if you’re reading this thinking your backups are reasonably secure but your organisation only has mediocre access restrictions to get onto the network, you may have closed the gates after the horse has bolted.

Records retention and NMC

 Basics, Best Practice, Security  Comments Off on Records retention and NMC
Dec 102014
 

For those of us who have been using NetWorker for a very long time, we can remember back to when the NetWorker Management Console didn’t exist. If you wanted reports in those days, you wrote them yourself, either by parsing your savegroup completion results, processing the NetWorker daemon.log, or interrogating mminfo.

Over time since its introduction, NMC has evolved in functionality and usefulness. These days there are still some things that I find easier to do on the command line, but more often than not I find myself reaching for NMC for various administrative functions. Reporting is one of those.


(Just a quick interrupt. The NetWorker Usage Survey is happening again. Every year I ask readers to participate and tell me a bit about their environment. It’s short – I promise! – you only need around 5 minutes to answer the questions. When you’re finished reading this article, I’d really appreciate if you could jump over and do the survey.) 


There’s a wealth of reports in NMC, but some of the ones I find particularly useful often end up being:

  • User auditing
  • Success/failure results and percentages
  • Backup volume over time
  • Deduplication statistics

In order to get maximum use out of those, you want to make sure those details are kept for as long as you need them. In newer versions of NetWorker, if you go to the Enterprise Console and check out the Reports menu, you’ll see an option labelled “Data Retention”, and the default values are as follows:

default NMC data retention values

Those values are OK for using NMC reporting just for casual checking, but if you’re intending to perform longer-term checking, reporting or compliance based auditing, you might want to extend those values somewhat. Based on conversations with a couple of colleagues, I’m inclined to extend everything except for the Completion Message section to at 3 years in sites where longer-term compliance and auditing reporting is required. The completion messages are generally a little bigger in scope, and I’d be inclined to limit those to 3 months at the most. So that means the resulting fields would look like:

alternate NMC data retention values

Ultimately the values you set in the NMC Reports Data Retention area should be specific to the requirements of your business, but be certain to check them out and tweak the defaults as necessary to align them with your needs.


(Hey, now you’ve finished reading this article, just a friendly reminder: The NetWorker Usage Survey is happening again. Every year I ask readers to participate and tell me a bit about their environment. It’s short – I promise! – you only need around 5 minutes to answer the questions. When you’re finished reading this article, I’d really appreciate if you could jump over and do the survey.)


 

Oct 062014
 

As I mentioned in an earlier article, Data Domain OS 5.5 introduced a new feature, in-flight encryption. This applies to Data Domain Boost connections, and requires v3 of the DDBoost Libraries, which are thankfully found in NetWorker 8.2.

Currently you won’t find the controls for in-flight encryption within NetWorker – it’s something that needs to be enabled or disabled from the Data Domain itself. Thankfully, it’s relatively trivial to do:

DDBoost In-flight encryption configuration

Sample in-flight encryption configuration

There’s two different commands that have been used here:

# ddboost clients show config

That command lists the current configuration setting for in-flight encryption. The second command I used was to enable in-flight encryption for a host, viz.:

# ddboost clients add centaur encryption-strength high

As is the case with any Data Domain command, you can get a full list of the options for a command by typing help command – in this case, help ddboost clients. A basic dump of the options can be triggered by an incomplete command, too:

All in-flight encryption options

All in-flight encryption options

Once in-flight encryption has been enabled, all you have to do is ensure the client direct option is enabled for the client. So long as the client can reach the Data Domain directly via the Boost device names in NetWorker (and the client is running NetWorker 8.2), you’ll get encryption of data during transit.

Those who have been using NetWorker for a while will know that there are other options for triggering encrypted backups, but of course such options are incompatible with successful deduplicated storage on a Data Domain – this method still allows for distributed stream processing and effective deduplication, but keeps the data secure during transit.

As you’d expect, the in-flight encryption works for recoveries as well as backup. Using tcpdump on the Data Domain I captured a couple of recoveries, one with encryption disabled, and one with encryption enabled. The files I recovered were those you’ll typically find in /usr/share/doc on a Linux system, and the results look quite different when viewed in Wireshark.

First, the unencrypted data:

Unencrypted DDBoost Backup Content

Unencrypted DDBoost Backup Content

With in-flight encryption, the traffic looks considerably different:

Encrypted DDBoost Backup Content

Encrypted DDBoost Backup Content

That’s all there is to it – it’s simple and straight forward to enable (and yes, as you may have guessed from the very first screen-capture, you can indeed use wild-cards in client definitions).

External NetWorker Authentication without AD

 NetWorker, Security  Comments Off on External NetWorker Authentication without AD
Aug 182014
 

One of the least used features in NetWorker is the option for external authentication of user accounts for use with NMC. This is normally discussed in the context of integrating NMC authentication into an Active Directory environment, but in theory, other LDAP v3 compliant directory services are compatible.

So over the weekend, I gave myself two goals: learn enough on the Ukulele to be able to play a song my boyfriend would recognise and integrate a lab NetWorker environment with the directory services provided by my OS X Server (10.9).

Surprisingly, I managed both – though perhaps unsurprisingly, the NMC/LDAP authentication was the trickier goal to get sorted out.

The first step I followed was to create a new group in LDAP called ‘nsradmin’, and placed into that group the user accounts that I wanted to be able to administer the NetWorker server. With that done, I switched back to NMC:

External authentication 1

From within NMC’s main window, go to Setup > Configure Login Authentication… and choose to configure an external repository, as shown below:

External Authentication 2

My external repository is pretty basic; as a home server, it’s a fairly flat structure, so the configured repository resembled the following:

External Authentication 3

In the distinguished name, I referenced the full DN to the directory administrator. This is normally undesirable; a preferred option would be to configure another directory user that has appropriate read permissions but limited to no modification permissions. I didn’t feel like diving into that level of control within LDAP and it was only a lab server so I plunged ahead with the actual directory administrator.

The user and group search path are both straight forward:

  • User Search Path: cn=users,dc=miranda,dc=turbamentis,dc=int
  • Group Search Path: cn=groups,dc=miranda,dc=turbamentis,dc=int

For Apple’s directory services, you need to modify most of the options in the Advanced field, viz:

  • User ID Attribute becomes ‘uid’ for non-AD servers
  • User Object Class is ‘apple-user’
  • Group Object Class is ‘apple-group’
  • Group Member Attribute is ‘memberUid’

For what it’s worth, I confirmed those settings by using the ldapsearch tool on the directory server:

ldapsearch -LLL -h miranda.turbamentis.int -b "cn=users,dc=miranda,dc=turbamentis,dc=int" -D "uid=diradmin,cn=users,dc=miranda,dc=turbamentis,dc=int" -W
...snip...
dn: uid=services,cn=users,dc=miranda,dc=turbamentis,dc=int
uid: services
uidNumber: ...
homeDirectory: /Users/services
cn: Services User
sn: User
loginShell: /bin/bash
givenName: Services
objectClass: person
objectClass: inetOrgPerson
objectClass: organizationalPerson
objectClass: posixAccount
objectClass: shadowAccount
objectClass: top
objectClass: extensibleObject
objectClass: apple-user
# ldapsearch -LLL -h miranda.turbamentis.int -b "cn=Groups,dc=miranda,dc=turbamentis,dc=int" -D "uid=diradmin,cn=users,dc=miranda,dc=turbamentis,dc=int" -W
...snip...
dn: cn=nsradmin,cn=groups,dc=miranda,dc=turbamentis,dc=int
objectClass: top
objectClass: posixGroup
objectClass: extensibleObject
objectClass: apple-group
apple-group-realname: nsradmin
cn: nsradmin
apple-ownerguid: ...
apple-generateduid: ...
gidNumber: ...
apple-group-memberguid: ...
apple-group-memberguid: ...
memberUid: pmdg
memberUid: services

If you’re encountering issues with the configuration (and more importantly, subsequent testing), I’d recommend setting the LDAP Debug Level to 1 so that you can see what sort of LDAP searches NMC is performing – these can be seen from the gstd.raw file in the NetWorker Management Console logs directory. If you’re not sure whether you’ve got all the details correct, by the way, just hit the Next> button … you can’t progress to the next screen unless NMC can successfully query the list of groups and names based on the details you’ve entered.

Clicking next, you’ll be prompted to confirm which users and roles will have ‘Console Security Administrator Role’ – this is a critical field; this defines the users who can re-invoke this form after the switchover to external authentication has happened:

External Authentication 4

Make sure there’s at least one actual user account defined in there. This is where I came-a-cropper the first few times – I assumed I could just use the group in there and it would be sufficient. (I’d need to go back and check against an Active Directory associated NMC server to confirm whether it’s any different there as I can’t recall off-hand.)

Click Next> again once you’ve populated that – again, NMC will query and confirm the validity of the entered details before it lets you progress:

External Authentication 5

You’ll then be prompted to confirm which servers you want to distribute the authority file to – in my case, since GST services are running on the same host as the backup server itself, it’s two instances of the same server, NetWorker and NMC. The distribution should log as follows:

External Authentication 6

Click Finish, but whatever you do, don’t yet exit NMC. There’s a few more bits and pieces you need to do. Specifically, you have to do the following:

  1. Add at least the referenced security console administrator user (from above) as a user in NMC, assigning the user all security roles.
  2. Equally, add that user (e.g., user=pmdg,host=hostName) to the NetWorker Application Administration list (within the NetWorker Administration console).
  3. Test the login of that user using another browser or RDP session. Once you exit the console session you’ve been using, internally defined accounts will be disabled. (In fact, they actually already are, but because this session remains authenticated while you remain connected.)

In my testing, I found that (at least with OSX 10.9 Server LDAP), I couldn’t successfully define administrative NetWorker control via the External Roles field in the User Groups list, viz.:

External Authentication 7

That is, it wasn’t sufficient to define ‘group=nsradmin’ or an external role of ‘nsradmin’ to grant NetWorker administrative rights to anyone in that external group. (I suspect as much as anything that this is a peculiarity between the operation of OS X 10.9 Server directory services and NMC than a failing in NMC itself.)

Even with the slightly less integrated approach, where administrative accounts will need to be named individually within the User group for NetWorker, there are still definite advantages of external authentication integration:

  1. Reducing number of passwords you have to remember in your overall environment
  2. Auditor satisfaction that an account disabled in directory services will be disabled from NMC access
  3. Auditor satisfaction of named user account tracking (rather than local-to-NMC and possibly generic accounts) in NMC

In case you’re wondering – if someone with a directory account tries to log in and there hasn’t been an account defined in NMC, NMC will automatically create the account, but not assign any privileges to it. This allows a previously authenticated administrator to quickly edit the privileges.

One final note – if you do happen to mess up the authentication process and can’t log in, the short-term solution is quite straight forward:

  • Stop the NetWorker Management Console services
  • On the NMC server, touch under the Management Console ‘cst’ directory a file called ‘authoverride’.
  • Restart the NMC services
  • Log in as administrator
  • Either switch back to local authentication, or adjust the external roles/etc as appropriate
  • Stop NMC services
  • Remove the ‘authoverride’ file
  • Restart the NMC services
  • Verify it’s working

Keeping all that in mind, it’s relatively straight-forward to jump into the realm of external user authentication with NMC – and that procedure above is your get-out-of-gaol card if for some reason, your directory services goes down.

NetWorker Tunnels

 NetWorker, Security  Comments Off on NetWorker Tunnels
Apr 012013
 

Tunnel

NetWorker and firewalls has always been a bit of a challenging combination. It’s become increasingly simplified over time – to the point where even a network luddite such as myself can readily configure ports access across a firewall – so long as the firewall administrators or interface are cooperative.

But the rub has always been the need for multiple ports. Many firewall administrators would like to only have to open one port across a DMZ. This is a feature some competing products have cited as an advantage in secure environments over NetWorker and to be honest, in situations where port minimisation is a key required feature, it’s been difficult to argue against that.

A new feature in NetWorker 8 that I hadn’t noticed before however is a new option – tunnelling. As per typical network tunnels, the scenario available to NetWorker now is specifying a single IP address and port number on either side of the DMZ to pass all traffic through.

This functionality designates a communications proxy on either side of the firewall – a new NetWorker daemon, nsrtund, comes into play, and access is configured via aliases and the server network interface option within clients. Currently this looks to be restricted in operating systems … the tunnel proxy and the NetWorker server need to be any of the following:

  • Solaris/Sparc (10 or 11)
  • Solaris/AMD (10 or 11)
  • RHEL on x86/x64;
  • SLES on x86/x64.

I’m hoping Windows gets added to that mix soon – we know Windows represents a substantial aspect of NetWorker server deployments these days.

I’ve not yet had a chance to run up a configuration to test tunnelling, but the documentation looks comprehensive, and if you’re interested in using it yourself, you’ll find it in the Technical Notes section on the support.emc.com website.

(Oh, and while I’m at it – kudos to EMC for renaming their documentation files to sensible names reflective of the title and content, rather than the old standard of part numbers. It’s great to see technology companies adjusting things to suit customers better.)

 

Feb 222011
 

I’ve been involved with an increasing number of NetWorker 7.6 SP1 configurations on Windows 2008 R2, and I’m not sure whether what I’ve encountered is specific to Windows 2008 R2 or just a general deficiency in the NetWorker installer’s firewall configuration process. Either way, since it caused some challenges for me, I wanted to note down the issues I’ve observed.

First, the firewall configuration is only applied to the “Public” profile. This is OK for single-interface servers, but if your system has multiple interfaces, it isn’t sufficient – you need to edit the rules to apply to all three of “Domain”, “Private” and “Public”:

Firewall configuration 1

The next issues encountered were relating to tape libraries on storage nodes. In particular, it appeared that the default automatic NetWorker firewall configuration on at least Windows 2008 R2 didn’t add support for the nsrmmgd or nsrlcpd daemons to communicate.

To create these rules:

  • On the server:
    • Copied two of the existing rules – one for TCP, one for UDP – and updated the “Programs and Services” pane to reference X:pathtobinnsrmmgd.exe.
  • On each storage node:
    • Copied two of the existing rules – one for TCP, one for UDP – and updated the “Programs and Services” pane to reference X:pathtobinnsrlcpd.exe.

With these sets of changes in play, NetWorker has behaved a lot more normally.

(Obviously, any firewall changes you make in your environment should be considered against site requirements.)

All your data are belong to us

 Backup theory, NetWorker, Security  Comments Off on All your data are belong to us
Feb 022010
 

There’s a simple rule to remember when it comes to removable media handling (both within backups, and generally within IT) – if you don’t know where your media is, you can’t be certain someone hasn’t misappropriated it.

Taking this further, if you can’t be sure of the security of your backup media, you can’t be sure of the security of your backups; and if you can’t be sure of the security of your backups, you can’t be sure of your security of your data.

So, how can you be certain of the security of your media, and therefore your backups and data?

Here’s a few guidelines:

  • Always use reputable media handling companies. This is for a two-fold requirement. First, you want to make sure that the company that handles and stores your media knows how to treat it carefully. That means correct handling procedures, storage in appropriate environmental conditions, and storage in a location that is unlikely to be affected by disasters that could affect your datacentre. The second part of the requirement is knowing that the media is always secure. This means signed, authorised access, a known reputation for security, audited processes and (preferably) premises that you can periodically visit to confirm security levels.
  • Store media securely on-site too. It is far from the case that media can only be stolen when off-site or travelling to/from site. Indeed, some of Australia’s biggest media losses have occurred on-site due to poor media handling security. (I seriously doubt Australia is unique in this). Tapes shouldn’t be kept insecurely anywhere on-site. When being transported from the computer room to on-site storage, they should be securely monitored at all times. When readying for transport off-site, they should be kept under lock and key, or kept in a secure location. And when at-rest on-site, they should also be kept under lock and key.
  • Media encryption. For a long time media encryption has been available only to the high end of enterprise backup. However, with tape technologies such as LTO-4 incorporating hardware encryption, any company using removable media in their backup environment should either:
    • Already be using media encryption, or
    • Be actively planning moving to media encryption, or
    • If nothing else, use NetWorker’s software encryption on critically sensitive data if the business is too small to afford hardware-encryption devices. This means taking a hit on backup performance, but as the old saying goes, you can’t have your cake and eat it too. I.e., there’s always a cost to encryption.
  • Secure key management. Media encryption doesn’t mean a thing if you’re not using some form of secure key management. Discuss and plan backup key management with your corporate security policy makers.
  • Have established, immutable processes for the recall of media. Media that has been sent to offsite storage should either be returned under specific, agreed circumstances. That may be a fixed rotation policy normally, with provisions for recall for recoveries with specific authorisation. Make sure that authorisation process is locked down with your media offsite vendor so that social engineering attacks can’t be employed (particularly when it comes to ex-employees).
  • Use strong password management for backup server access. As I’ve previously discussed, your entire backup environment is only as secure as your backup server. This places a special responsibility on backup and system administrators to ensure that the backup environment is highly secure.

Of course, there’s more to backup systems security than the above, but I wanted to focus primarily on physical security considerations for removable media, which for a lot of sites represent the weakest link in the security of the backup environment (and by extension, a significantly weak link in the security of the company’s IT systems and data as well).

If you fail to focus on removable media security, you potentially leave your company open to data loss.

%d bloggers like this: