Auditing should be done only by the experts

 Backup theory, General Technology, General thoughts, NetWorker  No Responses »
Jul 282011
 

I’ve said it before – auditing should only be done by the experts. I first realised this when a security auditor from one of the (then) “Big 5″ accounting companies audited the Solaris servers I was administering 11 years ago. Having checked /etc/passwd, the auditor noted in the report:

All user passwords are set to *, which is highly insecure and should be addressed immediately to ensure continued security compliance.

The fallout from that was briefly atrocious, and resolved only by convincing a manager to try to log onto a list of user accounts using * as the password.

It appears that there’s still room for security auditors who don’t really understand security, as evidenced by “Our security auditor is an idiot, how do I give him the information he wants? – Server Fault“. The system administrator was told he had to handover the following as part of the audit:

  • A list of current usernames and plain-text passwords for all user accounts on all servers
  • A list of all password changes for the past six months, again in plain-text
  • A list of “every file added to the server from remote devices” in the past six months
  • The public and private keys of any SSH keys
  • An email sent to him every time a user changes their password, containing the plain text password

Up until this point, I thought that it would be impossible for anyone to have an experience to trump my “all user passwords are set to *” experience.

It turns out I was wrong.

What’s this got to do with backups, I hear you ask?

Well, everything. If your company is getting in auditors who aren’t subject matter experts (or at least product experts), then your audit isn’t worth the paper it’s written on. Maybe you’ll get a compliance rubber stamp. Maybe you won’t. But it won’t make one iota of difference as to whether there’s been any valid checking of your environment.

Please, ensure that if you want your backups audited you ask some experts in. Knowing the sorts of prices the “big” auditing companies charge, it’ll likely not only cost you less, but actually give you more!

 Posted by at 09:58  Tagged with: , , ,

All your data are belong to us

 Backup theory, NetWorker, Security  No Responses »
Feb 022010
 

There’s a simple rule to remember when it comes to removable media handling (both within backups, and generally within IT) – if you don’t know where your media is, you can’t be certain someone hasn’t misappropriated it.

Taking this further, if you can’t be sure of the security of your backup media, you can’t be sure of the security of your backups; and if you can’t be sure of the security of your backups, you can’t be sure of your security of your data.

So, how can you be certain of the security of your media, and therefore your backups and data?

Here’s a few guidelines:

  • Always use reputable media handling companies. This is for a two-fold requirement. First, you want to make sure that the company that handles and stores your media knows how to treat it carefully. That means correct handling procedures, storage in appropriate environmental conditions, and storage in a location that is unlikely to be affected by disasters that could affect your datacentre. The second part of the requirement is knowing that the media is always secure. This means signed, authorised access, a known reputation for security, audited processes and (preferably) premises that you can periodically visit to confirm security levels.
  • Store media securely on-site too. It is far from the case that media can only be stolen when off-site or travelling to/from site. Indeed, some of Australia’s biggest media losses have occurred on-site due to poor media handling security. (I seriously doubt Australia is unique in this). Tapes shouldn’t be kept insecurely anywhere on-site. When being transported from the computer room to on-site storage, they should be securely monitored at all times. When readying for transport off-site, they should be kept under lock and key, or kept in a secure location. And when at-rest on-site, they should also be kept under lock and key.
  • Media encryption. For a long time media encryption has been available only to the high end of enterprise backup. However, with tape technologies such as LTO-4 incorporating hardware encryption, any company using removable media in their backup environment should either:
    • Already be using media encryption, or
    • Be actively planning moving to media encryption, or
    • If nothing else, use NetWorker’s software encryption on critically sensitive data if the business is too small to afford hardware-encryption devices. This means taking a hit on backup performance, but as the old saying goes, you can’t have your cake and eat it too. I.e., there’s always a cost to encryption.
  • Secure key management. Media encryption doesn’t mean a thing if you’re not using some form of secure key management. Discuss and plan backup key management with your corporate security policy makers.
  • Have established, immutable processes for the recall of media. Media that has been sent to offsite storage should either be returned under specific, agreed circumstances. That may be a fixed rotation policy normally, with provisions for recall for recoveries with specific authorisation. Make sure that authorisation process is locked down with your media offsite vendor so that social engineering attacks can’t be employed (particularly when it comes to ex-employees).
  • Use strong password management for backup server access. As I’ve previously discussed, your entire backup environment is only as secure as your backup server. This places a special responsibility on backup and system administrators to ensure that the backup environment is highly secure.

Of course, there’s more to backup systems security than the above, but I wanted to focus primarily on physical security considerations for removable media, which for a lot of sites represent the weakest link in the security of the backup environment (and by extension, a significantly weak link in the security of the company’s IT systems and data as well).

If you fail to focus on removable media security, you potentially leave your company open to data loss.

 Posted by at 20:06  Tagged with: , , , , ,

Enhancing NetWorker Security: A theoretical architecture

 Architecture, Backup theory, NetWorker, Security  3 Responses »
Nov 182009
 

It’s fair to say that no one backup product can be all things to all people. More generally, it’s fair to say that no product can be all things to all people.

Security has had a somewhat interesting past in NetWorker; much of the attention to security for a lot of the time has been to with (a) defining administrators, (b) ensuring clients are who they say they are and (c) providing access controls for directed recoveries.

There’s a bunch of areas though that have remained somewhat lacking in NetWorker for security. Not 100% lacking, just not complete. For instance, user accounts that are accessed for the purposes of module backup and recovery frequently need higher levels of authority than standard users. Equally so, some sites want their <X> admins to be able to control as much as possible of the <X> backups, but not to be able to have any administrator privileges over the <Y> backups. I’d like to propose an idea that, if implemented, would both improve security and make NetWorker more flexible.

The change would be to allow the definition of administrator zones. An “administrator zone” would be a subset of a datazone. It would consist of:

  1. User groups:
    • A nominated “administrator” user group.
    • A nominated “user” user group.
    • Any other number of nominated groups with intermediate privileges.
  2. A collection of the following:
    • Clients
    • Groups
    • Pools
    • Devices
    • Schedules
    • Policies
    • Directives
    • etc

These obviously would still be accessible in the global datazone for anyone who is a datazone administrator. Conceptually, this would look like the following:

Datazone with subset "administrator" zonesThe first thing this should point out to you is that administrator zones could, if desired, overlap. For instance, in the above diagram we have:

  1. Minor overlap between Windows and Unix admin zones (e.g., they might both have administrative rights over tape libraries).
  2. Overlap between Unix and Oracle admin zones.
  3. Overlap between Windows and Oracle admin zones.
  4. Overlap between Windows and Exchange admin zones.
  5. Overlap between Windows and MSSQL admin zones.

Notably though, the DMZ Admin zone indicates that you can have some zones that have no overlap/commonality with other zones.

There’d need to be a few rules established in order to make this work. These would be:

  1. Only the global datazone can support “<x>@*” user or group definitions in a user group.
  2. If there is overlap between two zones, then the user will inherit the rights of the highest authority they belong to. I.e., if a user is editing a shared feature between the Windows and Unix admin zones, and is declared an admin in the Unix zone, but only an end-user in the Windows zone, then the user will edit that shared feature with the rights of an admin.
  3. Similarly to the above, if there’s overlap between privileges at the global datazone level and a local administrator zone, the highest privileges will “win” for the local resource.
  4. Resources can only be created and deleted by someone with data zone administrator privileges.
  5. Updates for resources that are shared between multiple administrator zones need to be “approved” by an administrator from each administrator zone that overlaps or a datazone administrator.

Would this be perfect? Not entirely – for instance, it would still require a datazone administrator to create the resources that are then allocated to an administrator zone for control. However, this would prevent a situation occurring where an unprivileged user with “create” options could go ahead and create resources they wouldn’t have authority over. Equally, in an environment that permits overlapping zones, it’s not appropriate for someone from one administrator zone to delete a resource shared by multiple administrator zones. Thus, for safety’s sake, administrator zones should only concern themselves with updating existing resources.

How would the approval process work for edits of resources that are shared by overlapping zones? To start with, the resource that has been updated would continue to function “as is”, and a “copy” would be created (think of it as a temporary resource), with a notification used to trigger a message to the datazone administrators and the other, overlapping administrators. Once the appropriate approval has been done (e.g., an “edit” process in the temporary resource), then the original resource would be overwritten with the temporary resource, and the temporary resource removed.

So what sort of extra resources would we need to establish this? Well, we’ve already got user groups, which is a starting point. The next step is to define an “admin zone” resource, which has fields for:

  1. Administrator user group.
  2. Standard user group.
  3. “Other” user groups.
  4. Clients
  5. Groups
  6. Pools
  7. Schedules
  8. Policies
  9. Directives
  10. Probes
  11. Lockboxes
  12. Notifications
  13. Labels
  14. Staging Policies
  15. Devices
  16. Autochangers
  17. etc.

In fact, pretty much every resource except for the server resource itself, and licenses, should be eligible for inclusion into a localised admin group. In it’s most basic, you might expect to see the following:

nsradmin> print type: NSR admin zone; name: Oracle


               type: NSR admin zone;


               name: Oracle;


     administrators: Oracle Admins;


              users: Oracle All Users;


  other user groups: ;


            clients: delphi, pythia;


             groups: Daily Oracle FS, Monthly Oracle FS,


                     Daily Oracle DB, Monthly Oracle DB;


              pools: ;


          schedules: Daily Oracle, Monthly Oracle;


           policies: Oracle Daily, Oracle Monthly;


         directives: pythia exclude oracle, delphi exclude oracle;


...etc...

To date, NetWorker’s administration focus has been far more global. If you’re an administrator, you can do anything to any resource. If you’re a user, you can’t do much with any resource. If you’ve been given a subset of privileges, you can use those privileges against all resources touched by those privileges.

An architecture that worked along these lines would allow for much more flexibility in terms of partial administrative privileges in NetWorker – zones of resources and local administrators for those resources would allow for more granular control of configuration and backup functionality, while still keeping NetWorker configuration maintained at the central server.

 Posted by at 05:39  Tagged with: , , , ,

What's wrong with the NMC installation process?

 NetWorker, Security  No Responses »
Aug 172009
 

There is, in my opinion, an unpleasant security hole in the NMC installation/configuration process.

The security hole is simple: it does not prompt for the administrator password on installation. This is inappropriate for a data protection product, and I think it’s something that EMC should fix.

The NMC installation process is slightly different depending on whether you’re working with 7.5.x or 7.4.x and lower.

For 7.4.x and lower, the process works as follows:

  • Install NetWorker management console.
  • (On Unix platforms, manually run the /opt/lgtonmc/bin/nmc_config file to initialise the configuration.)
  • Launch NMC.
  • Use the default username/password until you get around to changing the password.

For 7.5.x and higher installations, the process works as follows:

  • Install NetWorker management console.
  • First person to logon gets to set the administrator password.

In both instances, this represents a clear security threat to the environment, particularly when installing NetWorker on the backup server or another host that already has administrator access to the datazone, and needs to be managed carefully. Two clear options, depending on the level of trust you have within your environment are:

  • Use firewall/network security configuration options to restrict access to the NMC console port (9000) to a single, known and trusted host, until you are able to log on and change the password.

or

  • Be prepared to log onto NMC as soon as the installation (or for Unix, installation/configuration) is complete and trust that you “get there first”.

In reality, the second option would not be declared secure by any security expert, but for small environments where the trust level is high, it may be acceptable for local security policies.

The real solution though is simple: EMC must change the NMC installation process to force the input of a secure administrator password at install time. That way, by the time the daemons are first started, they are already secured.

 Posted by at 07:55  Tagged with: , , , ,

Are you Monitoring RAP?

 NetWorker, Security  No Responses »
Jul 032009
 

Introduced in NetWorker v7 was a feature called “Monitor RAP”. There’s two unfortunate aspects to this setting in the NetWorker server resource (“NSR”):

  • It is not enabled by default;
  • The setting name obfuscates the purpose of the setting for most users.

Personally, I would have preferred when this option was made available that it was called “Audit Changes”, and that it was enabled by default.

In the NetWorker management console, with diagnostic mode enabled, this option is available in the first tab of the NetWorker server properties:

Monitor RAP setting

Monitor RAP setting

With this setting enabled, NetWorker maintains a new log, rap.log, within the logs directory. This tracks changes that are made to the NetWorker configuration, as they are made.

Here’s an example:

06/30/2009 06:49:19 PM MONITOR_RAP: preston@archon CHANGED 'NSR group'
resource, Staging Development:
 autostart: Enabled;
 autostart: Disabled;

This tells us that at 18:49:19 on 30 June 2009, user ‘preston’ on host ‘archon’ changed the group ‘Staging Development’, changing ‘autostart’ from ‘Enabled’ to ‘Disabled’.

This means that from the NetWorker level*, you can easily keep track of who does what to the NetWorker configuration. Interestingly, you can also use this information to also track self-changes to the system – i.e., where NetWorker updates its own configuration. As an example, if you use a license manager, then whenever NetWorker updates/checks its licenses against the license server, you’ll get entries in the logs such as:

06/30/2009 05:10:00 PM MONITOR_RAP: root@nox CHANGED 'NSR license' resource,
Autochanger Module, 40 slots/40:
 checksum: \
"'j&P_-QFc]]D~GIQ\\_[q-jMFx;ajW%U~\\1^UvCm`dJEwg/
T#XGMpaVYet\\l]M\\w\\{QQ\\\\\
WuR]\"Ax*@^XX'[ZAG388M)I6fvztWrC9q\"G3PeML!wl
{P6L0]JU9a9[{WYZ";
 checksum: \
"Z$EP_Jts]gYNZV_ATn]AZSxcTorL#f2(\"8/PTcIYec}K+}
e_GGsJ$6IC)QLTz\\_aS?[|lc^\\Z\
rN9}A~L@}?b^Vlud-e:SD+Js<U]T!eXR\"y/,bQHO
0_CShOlw?U1h\\g?";

Using the Monitor RAP setting allows you to easily monitor changes to the NetWorker configuration, and I believe that every NetWorker site should always have this setting enabled.

–
* Auditing is also available within NMC. For maximum auditing, I always recommend that both options be used.

 Posted by at 16:38  Tagged with: , ,

Your datazone is only as secure as your NetWorker server

 NetWorker, Policies, Security  1 Response »
Jun 262009
 

A topic I discuss in my book that’s worth touching on here is that of datazone security.

Backup is one of those enterprise components that touches on a vast amount of infrastructure; so much so that it’s usually one of those most broadest reaching pieces of software within an environment. As such, the temptation is always there to make it “as easy as possible” to configure. Unfortunately this sometimes leads to making it too easy to configure. By too easy, I mean insecure.

Regardless of the “hassle” that it creates, a backup server must be highly secured. Or to be perhaps even blunter – the entire security of everything backed up by your backup server depends on the security of your backup server. Having an insecure NetWorker server, on the other hand, is like handing over the keys to your datacentre, as well as having the administrator/root password for every server stuck to each machine.

Thinking of it that way, do you really want the administrator list on your backup server to include say, any of the following?

  • *@*
  • *@<host>
  • <user>*@

If your answer is yes, then you’re wrong*.

However, datazone security isn’t only about the administrator list (though that forms an important part). At bare minimum, your datazone should have the following security requirements:

  1. No wild-cards shall be permitted in administrator user list definitions (server, NMC).
  2. No client shall have an empty servers file (client).
  3. No wild-cards shall be permitted in remote access user list definitions (client resources).

Note: With the advent of lockboxes in version 7.5, security options increase – it’s possible, for instance, to have passwords for application modules stored in such a way that only the application module for the designated host can retrieve the password.

–
* I do make allowance for some extreme recovery issues that have temporarily required users to enter wild-card administrators temporarily where it was not possible to wait for a bug fix.

 Posted by at 17:27  Tagged with: , , , ,

Basics – Using datazone encryption with NetWorker

 Basics, NetWorker  4 Responses »
Mar 222009
 

I’m not fond of software encryption (or compression, for that matter). Particularly in a 24×7 enterprise environment, clients (i.e., production servers) have better things to be doing than doing on-the-fly software encryption or compression. In these environments, hardware encryption routers should be the product of choice for achieving totally secure backups. Such devices also have advantages in terms of key management – much more flexible, scalable and appropriate for role based data access.

That being said, in smaller environments, or environments where servers are relatively idle overnight, NetWorker’s datazone encryption can be sufficient to achieve a reasonable modicum of backup protection with minimum effort – and most importantly, cost.

To get started using NetWorker datazone encryption, you first need to assign a pass phrase. This is done in the NetWorker server properties (typically accessed within NMC):

datazone-encryptionWith the pass phrase in place, you can then configure directives within NetWorker to make use of AES 256 bit encryption. However! As soon as you turn encryption on, you lose all potential for hardware based compression for your media. Why? Quite simply, compression is about finding patterns in data and reducing all the matching patterns to a single reference point; however, encryption is all about eliminating patterns, making the data appear completely random.

Thus, if you want to still get some measure of compression, you should, when using this method, employ software based compression in your directive as well.

Thus, a base directive might look like the following:

<< / >>
+compressasm: .
+aes: .

This will apply compression first to all files encountered, then once the file has been compressed, it will be encrypted. A side benefit of this is that by compressing first, you reduce the amount of data to be encrypted*.

So long as the datazone pass phrase is stored in the server, encryption will occur, and no password will be required to recover the data. Remember, this style of encryption, using a single pass-phrase, isn’t about being able to restrict whom within the datazone can recover the data, but instead it’s about keeping the data stored on-tape (which is potentially off-site, or otherwise at higher risk of theft), from being recovered.

[Edit, 2009-08-15]

It’s been pointed out to me that you can’t compress + encrypt at the client side. Indeed, I’ve now found the part in the administration guide that explicitly says this. What is extremely disappointing about this is that NetWorker actually doesn’t warn you that it’s not going to compress + encrypt! To me, that’s a security issue.

So, for the examples above, forget about enabling client side compression as well as encryption – you can have one or the other, but not both.

–
* In the same way that ice-cream that’s 99% fat free, but 87% sugar is a “benefit”.

 Posted by at 17:03  Tagged with: , , ,
© 2012 The NetWorker Blog Suffusion theme by Sayontan Sinha