Basics – Configuring a reports-only user

 NetWorker, Security  Comments Off on Basics – Configuring a reports-only user
May 252015
 

Something that’s come up a few times in the last year for me has been a situation where a NetWorker user has wanted to allow someone to access NetWorker Management Console for the purpose of running reports, but not allow them any administrative access to NetWorker.

It turns out it’s very easy to achieve this, and you actually have a couple of options on the level of NetWorker access they’ll get.

Let’s look first at the minimum requirements – defining a reports only user.

To do that, you first go into NetWorker Management Console as an administrative user, and go across to the Setup pane.

You’ll then create a new user account:

New User Account in NMC

Within the Create User dialog, be certain to only select Console User as the role:

NMC new user dialog

At this point, you’ve successfully created a user account that can run NMC reports, but can’t administer the NetWorker server.

However, you’re then faced with a decision. Do you want a reports-only user that can “look but don’t touch”, or do you want a reports-only user that can’t view any of the NetWorker configuration (or at least, anything other than can be ascertained by the reports themselves)?

If you want your reports user to be able to run reports and you’re not fussed about the user being able to view the majority of your NetWorker configuration, you’re done at this point. If however your organisation has a higher security focus, you may need to look at adjusting the basic Users NetWorker user group. If you’re familiar with it, you’ll know this has the following configuration:NetWorker Users Usergroup

This usergroup in the default configuration allows any user in the NetWorker datazone to:

  • Monitor NetWorker
  • Recover Local Data
  • Backup Local Data

The key there is any user*@*. Normally you want this to be set to *@*, but if you’re a particularly security focused organisation you might want to tighten this down to only those users and system accounts authorised to perform recoveries. The same principle applies here. Let’s say I didn’t want the reports user to see any of the NetWorker configuration, but I did want any root, system or pmdg user in the environment to still have that basic functionality. I could change the Users usergroup to the following:

Modified NetWorker Users usergroup

With this usergroup modified, logging in as the reports user will show a very blank NMC monitoring tab:

NMC-monitoring reports user

Similarly, the client list (as an example) will be quite empty too:

NMC-config reports user

Now, it’s worth mentioning there are is a key caveat you should consider here – some modules may be designed in anticipation that the executing user for the backup or recovery (usually an application user with sufficient privileges) will at least be a member of the Users usergroup. So if you tighten the security against your reports user to this level, you’ll need to be prepared to increase the steps in your application onboarding processes to ensure those accounts are added to an appropriate usergroup (or a new usergroup).

But in terms of creating a reports user that’s not privileged to control NetWorker, it’s as easy as the steps above.

Mar 022015
 

Lock

Backup security is – or rather should be a big requirement in any organisation. Sadly we still see periodic examples of where organisations fail to fully comprehend the severity of a backup breach. In a worst case scenario, a backup breach involving physical theft might be the equivalent of someone having permanent and unchecked access to a snapshot of your entire network.

There are two distinct aspects to backup security to consider:

  • Physical
  • Electronic

For each type of backup security, we need to consider two key areas:

  • At rest
  • In transit

This usually leads businesses to start backup security planning by considerations such as:

  • Do we encrypt backup media?
  • Do we used security guards for movement of backup media?
  • Are on-disk backups encrypted?

Oddly enough, there’s a bigger gorilla in the room for backup security that is less often thought of: your backups are only as secure as the quality of and your/their adherence to your security policies.

A long time ago in a state far, far away, a colleague was meeting with a system administrator in the offices of an environmental organisation. She needed to ensure the security restrictions for system access could be drastically lowered from the default install criteria. “Everyone here is an anti-authority hippy” she said (or words to that effect), “If we give them hard passwords they’ll just write the in permanent marker on their monitors.”

The solution was to compromise to a mid-point of security vs ease-of-access.

These days few organisations would yield to their users disdain for authority so readily, but it serves to highlight that a system is only as secure as you choose to make it. A backup environment does not sit in isolation – it resides on the hosts it is being used to protect (in some form or another), and it will hae a host-based presence within your network at some point. If someone can breach that security and get onto one of thoe hosts, there’s a good chance a significant aspect of your backup security protocols have been breached as well.

That’s why all backup security has to start at a level outside the backup environment … rather, it requires consideration at all layers. It doesn’t start with the complexity of the password required to access an administrator interface, and nor does it end with enabling data-at-rest encryption. So if you’re reading this thinking your backups are reasonably secure but your organisation only has mediocre access restrictions to get onto the network, you may have closed the gates after the horse has bolted.

Oct 062014
 

As I mentioned in an earlier article, Data Domain OS 5.5 introduced a new feature, in-flight encryption. This applies to Data Domain Boost connections, and requires v3 of the DDBoost Libraries, which are thankfully found in NetWorker 8.2.

Currently you won’t find the controls for in-flight encryption within NetWorker – it’s something that needs to be enabled or disabled from the Data Domain itself. Thankfully, it’s relatively trivial to do:

DDBoost In-flight encryption configuration

Sample in-flight encryption configuration

There’s two different commands that have been used here:

# ddboost clients show config

That command lists the current configuration setting for in-flight encryption. The second command I used was to enable in-flight encryption for a host, viz.:

# ddboost clients add centaur encryption-strength high

As is the case with any Data Domain command, you can get a full list of the options for a command by typing help command – in this case, help ddboost clients. A basic dump of the options can be triggered by an incomplete command, too:

All in-flight encryption options

All in-flight encryption options

Once in-flight encryption has been enabled, all you have to do is ensure the client direct option is enabled for the client. So long as the client can reach the Data Domain directly via the Boost device names in NetWorker (and the client is running NetWorker 8.2), you’ll get encryption of data during transit.

Those who have been using NetWorker for a while will know that there are other options for triggering encrypted backups, but of course such options are incompatible with successful deduplicated storage on a Data Domain – this method still allows for distributed stream processing and effective deduplication, but keeps the data secure during transit.

As you’d expect, the in-flight encryption works for recoveries as well as backup. Using tcpdump on the Data Domain I captured a couple of recoveries, one with encryption disabled, and one with encryption enabled. The files I recovered were those you’ll typically find in /usr/share/doc on a Linux system, and the results look quite different when viewed in Wireshark.

First, the unencrypted data:

Unencrypted DDBoost Backup Content

Unencrypted DDBoost Backup Content

With in-flight encryption, the traffic looks considerably different:

Encrypted DDBoost Backup Content

Encrypted DDBoost Backup Content

That’s all there is to it – it’s simple and straight forward to enable (and yes, as you may have guessed from the very first screen-capture, you can indeed use wild-cards in client definitions).

Architectural implications of in-flight Data Domain Boost encryption

 Architecture, Data Domain, NetWorker  Comments Off on Architectural implications of in-flight Data Domain Boost encryption
Aug 252014
 

Between the Boost libraries included in NetWorker 8.2, and the in-flight encryption functionality added in Data Domain OS 5.5, it’s now possible to configure Boost-encrypted backups – and there’ll be quite a few sites with rigorous security requirements for which that’s a highly desirable function.

But it’s important to keep in mind that there are some architectural considerations to consider in this scenario, and the most important ones are about when the backup is encrypted.

Currently in NetWorker 8.2, there are no options for controlling the selection of  in-flight encryption. This is done entirely at the Data Domain by configuring client groups. In essence, you can configure collections of clients at the Data Domain to either have encryption of either:

  • none
  • medium
  • high

This is accomplished through using the command:

# ddboost clients add client-list [encryption-strength {medium|high}]

and:

# ddboost clients modify client-list [encryption-strength {none|medium|high}]

The astute reader will immediately identify the key restrictions this places: because all the control is being done from the Data Domain, it applies only to client backups that interface directly with the Data Domain.

For conventional backups, this means that Boost in-flight encryption will only apply to client-direct backups, i.e.:

Client Direct Data Flow

In this scenario, with data flowing directly between the NetWorker client and the Data Domain server, in-flight encryption is available. However, when a storage node is inserted into the data flow, that changes:

Traditional client data flow

 

In this scenario, the traffic sent between the client and the storage node does not utilise the Boost libraries, and thus flows unencrypted. The data is subsequently encrypted in transit between the storage node and the Data Domain, but by this point an organisation requiring in-flight encryption of backup traffic is already likely remiss in its obligations.

There are other scenarios where in-flight encryption is feasible:

  • VBA with ‘hotadd’ transport mode – this connects the ESX datastores directly to the VBA appliance (or its proxies) for the purposes of backup, keeping the initial data traffic within primary storage. The traffic sent from the VBA appliance or proxies, utilising Boost, gets encrypted as required. However, those Data Domain ddboost commands for enabling encryption will apply to the VBA appliance and its proxies – not individual virtual machines. Thus, encryption will be for all virtual machines backed up by the appliances (or none), and nothing in-between.
  • VADP on an 8.2 storage node using ‘SAN’, ‘hotadd’ or ‘NBDSSL’ transport modes:
    • ‘SAN’ transport mode is for physical storage nodes where the ESX LUNs are fibre-channel connected to the storage node; thus, data transport to the VADP proxy will be over fibre-channel, and the IP traffic will be encrypted via Boost between the storage node and the Data Domain;
    • ‘hotadd’ – same as VBA;
    • ‘NBDSSL’ – in this mode, the data is sent from the ESX server to the VADP proxy via IP, but in encrypted traffic; Boost level encryption subsequently takes over for transmission from the VADP proxy to the Data Domain.
  • An up-to-date Data Domain Boost plugin (e.g., Oracle RMAN Plugin for Data Domain) – though technically, this is fully outside of NetWorker.

Boost based in-flight encryption is a great addition in a NetWorker environment, but it’s important to keep in mind the restrictions regarding when it will apply.

NetWorker Tunnels

 NetWorker, Security  Comments Off on NetWorker Tunnels
Apr 012013
 

Tunnel

NetWorker and firewalls has always been a bit of a challenging combination. It’s become increasingly simplified over time – to the point where even a network luddite such as myself can readily configure ports access across a firewall – so long as the firewall administrators or interface are cooperative.

But the rub has always been the need for multiple ports. Many firewall administrators would like to only have to open one port across a DMZ. This is a feature some competing products have cited as an advantage in secure environments over NetWorker and to be honest, in situations where port minimisation is a key required feature, it’s been difficult to argue against that.

A new feature in NetWorker 8 that I hadn’t noticed before however is a new option – tunnelling. As per typical network tunnels, the scenario available to NetWorker now is specifying a single IP address and port number on either side of the DMZ to pass all traffic through.

This functionality designates a communications proxy on either side of the firewall – a new NetWorker daemon, nsrtund, comes into play, and access is configured via aliases and the server network interface option within clients. Currently this looks to be restricted in operating systems … the tunnel proxy and the NetWorker server need to be any of the following:

  • Solaris/Sparc (10 or 11)
  • Solaris/AMD (10 or 11)
  • RHEL on x86/x64;
  • SLES on x86/x64.

I’m hoping Windows gets added to that mix soon – we know Windows represents a substantial aspect of NetWorker server deployments these days.

I’ve not yet had a chance to run up a configuration to test tunnelling, but the documentation looks comprehensive, and if you’re interested in using it yourself, you’ll find it in the Technical Notes section on the support.emc.com website.

(Oh, and while I’m at it – kudos to EMC for renaming their documentation files to sensible names reflective of the title and content, rather than the old standard of part numbers. It’s great to see technology companies adjusting things to suit customers better.)

 

Auditing should be done only by the experts

 Backup theory, General Technology, General thoughts, NetWorker  Comments Off on Auditing should be done only by the experts
Jul 282011
 

I’ve said it before – auditing should only be done by the experts. I first realised this when a security auditor from one of the (then) “Big 5” accounting companies audited the Solaris servers I was administering 11 years ago. Having checked /etc/passwd, the auditor noted in the report:

All user passwords are set to *, which is highly insecure and should be addressed immediately to ensure continued security compliance.

The fallout from that was briefly atrocious, and resolved only by convincing a manager to try to log onto a list of user accounts using * as the password.

It appears that there’s still room for security auditors who don’t really understand security, as evidenced by “Our security auditor is an idiot, how do I give him the information he wants? – Server Fault“. The system administrator was told he had to handover the following as part of the audit:

  • A list of current usernames and plain-text passwords for all user accounts on all servers
  • A list of all password changes for the past six months, again in plain-text
  • A list of “every file added to the server from remote devices” in the past six months
  • The public and private keys of any SSH keys
  • An email sent to him every time a user changes their password, containing the plain text password

Up until this point, I thought that it would be impossible for anyone to have an experience to trump my “all user passwords are set to *” experience.

It turns out I was wrong.

What’s this got to do with backups, I hear you ask?

Well, everything. If your company is getting in auditors who aren’t subject matter experts (or at least product experts), then your audit isn’t worth the paper it’s written on. Maybe you’ll get a compliance rubber stamp. Maybe you won’t. But it won’t make one iota of difference as to whether there’s been any valid checking of your environment.

Please, ensure that if you want your backups audited you ask some experts in. Knowing the sorts of prices the “big” auditing companies charge, it’ll likely not only cost you less, but actually give you more!

All your data are belong to us

 Backup theory, NetWorker, Security  Comments Off on All your data are belong to us
Feb 022010
 

There’s a simple rule to remember when it comes to removable media handling (both within backups, and generally within IT) – if you don’t know where your media is, you can’t be certain someone hasn’t misappropriated it.

Taking this further, if you can’t be sure of the security of your backup media, you can’t be sure of the security of your backups; and if you can’t be sure of the security of your backups, you can’t be sure of your security of your data.

So, how can you be certain of the security of your media, and therefore your backups and data?

Here’s a few guidelines:

  • Always use reputable media handling companies. This is for a two-fold requirement. First, you want to make sure that the company that handles and stores your media knows how to treat it carefully. That means correct handling procedures, storage in appropriate environmental conditions, and storage in a location that is unlikely to be affected by disasters that could affect your datacentre. The second part of the requirement is knowing that the media is always secure. This means signed, authorised access, a known reputation for security, audited processes and (preferably) premises that you can periodically visit to confirm security levels.
  • Store media securely on-site too. It is far from the case that media can only be stolen when off-site or travelling to/from site. Indeed, some of Australia’s biggest media losses have occurred on-site due to poor media handling security. (I seriously doubt Australia is unique in this). Tapes shouldn’t be kept insecurely anywhere on-site. When being transported from the computer room to on-site storage, they should be securely monitored at all times. When readying for transport off-site, they should be kept under lock and key, or kept in a secure location. And when at-rest on-site, they should also be kept under lock and key.
  • Media encryption. For a long time media encryption has been available only to the high end of enterprise backup. However, with tape technologies such as LTO-4 incorporating hardware encryption, any company using removable media in their backup environment should either:
    • Already be using media encryption, or
    • Be actively planning moving to media encryption, or
    • If nothing else, use NetWorker’s software encryption on critically sensitive data if the business is too small to afford hardware-encryption devices. This means taking a hit on backup performance, but as the old saying goes, you can’t have your cake and eat it too. I.e., there’s always a cost to encryption.
  • Secure key management. Media encryption doesn’t mean a thing if you’re not using some form of secure key management. Discuss and plan backup key management with your corporate security policy makers.
  • Have established, immutable processes for the recall of media. Media that has been sent to offsite storage should either be returned under specific, agreed circumstances. That may be a fixed rotation policy normally, with provisions for recall for recoveries with specific authorisation. Make sure that authorisation process is locked down with your media offsite vendor so that social engineering attacks can’t be employed (particularly when it comes to ex-employees).
  • Use strong password management for backup server access. As I’ve previously discussed, your entire backup environment is only as secure as your backup server. This places a special responsibility on backup and system administrators to ensure that the backup environment is highly secure.

Of course, there’s more to backup systems security than the above, but I wanted to focus primarily on physical security considerations for removable media, which for a lot of sites represent the weakest link in the security of the backup environment (and by extension, a significantly weak link in the security of the company’s IT systems and data as well).

If you fail to focus on removable media security, you potentially leave your company open to data loss.

Nov 182009
 

It’s fair to say that no one backup product can be all things to all people. More generally, it’s fair to say that no product can be all things to all people.

Security has had a somewhat interesting past in NetWorker; much of the attention to security for a lot of the time has been to with (a) defining administrators, (b) ensuring clients are who they say they are and (c) providing access controls for directed recoveries.

There’s a bunch of areas though that have remained somewhat lacking in NetWorker for security. Not 100% lacking, just not complete. For instance, user accounts that are accessed for the purposes of module backup and recovery frequently need higher levels of authority than standard users. Equally so, some sites want their <X> admins to be able to control as much as possible of the <X> backups, but not to be able to have any administrator privileges over the <Y> backups. I’d like to propose an idea that, if implemented, would both improve security and make NetWorker more flexible.

The change would be to allow the definition of administrator zones. An “administrator zone” would be a subset of a datazone. It would consist of:

  1. User groups:
    • A nominated “administrator” user group.
    • A nominated “user” user group.
    • Any other number of nominated groups with intermediate privileges.
  2. A collection of the following:
    • Clients
    • Groups
    • Pools
    • Devices
    • Schedules
    • Policies
    • Directives
    • etc

These obviously would still be accessible in the global datazone for anyone who is a datazone administrator. Conceptually, this would look like the following:

Datazone with subset "administrator" zonesThe first thing this should point out to you is that administrator zones could, if desired, overlap. For instance, in the above diagram we have:

  1. Minor overlap between Windows and Unix admin zones (e.g., they might both have administrative rights over tape libraries).
  2. Overlap between Unix and Oracle admin zones.
  3. Overlap between Windows and Oracle admin zones.
  4. Overlap between Windows and Exchange admin zones.
  5. Overlap between Windows and MSSQL admin zones.

Notably though, the DMZ Admin zone indicates that you can have some zones that have no overlap/commonality with other zones.

There’d need to be a few rules established in order to make this work. These would be:

  1. Only the global datazone can support “<x>@*” user or group definitions in a user group.
  2. If there is overlap between two zones, then the user will inherit the rights of the highest authority they belong to. I.e., if a user is editing a shared feature between the Windows and Unix admin zones, and is declared an admin in the Unix zone, but only an end-user in the Windows zone, then the user will edit that shared feature with the rights of an admin.
  3. Similarly to the above, if there’s overlap between privileges at the global datazone level and a local administrator zone, the highest privileges will “win” for the local resource.
  4. Resources can only be created and deleted by someone with data zone administrator privileges.
  5. Updates for resources that are shared between multiple administrator zones need to be “approved” by an administrator from each administrator zone that overlaps or a datazone administrator.

Would this be perfect? Not entirely – for instance, it would still require a datazone administrator to create the resources that are then allocated to an administrator zone for control. However, this would prevent a situation occurring where an unprivileged user with “create” options could go ahead and create resources they wouldn’t have authority over. Equally, in an environment that permits overlapping zones, it’s not appropriate for someone from one administrator zone to delete a resource shared by multiple administrator zones. Thus, for safety’s sake, administrator zones should only concern themselves with updating existing resources.

How would the approval process work for edits of resources that are shared by overlapping zones? To start with, the resource that has been updated would continue to function “as is”, and a “copy” would be created (think of it as a temporary resource), with a notification used to trigger a message to the datazone administrators and the other, overlapping administrators. Once the appropriate approval has been done (e.g., an “edit” process in the temporary resource), then the original resource would be overwritten with the temporary resource, and the temporary resource removed.

So what sort of extra resources would we need to establish this? Well, we’ve already got user groups, which is a starting point. The next step is to define an “admin zone” resource, which has fields for:

  1. Administrator user group.
  2. Standard user group.
  3. “Other” user groups.
  4. Clients
  5. Groups
  6. Pools
  7. Schedules
  8. Policies
  9. Directives
  10. Probes
  11. Lockboxes
  12. Notifications
  13. Labels
  14. Staging Policies
  15. Devices
  16. Autochangers
  17. etc.

In fact, pretty much every resource except for the server resource itself, and licenses, should be eligible for inclusion into a localised admin group. In it’s most basic, you might expect to see the following:

nsradmin> print type: NSR admin zone; name: Oracle
type: NSR admin zone;
name: Oracle;
administrators: Oracle Admins;
users: Oracle All Users;
other user groups: ;
clients: delphi, pythia;
groups: Daily Oracle FS, Monthly Oracle FS,
Daily Oracle DB, Monthly Oracle DB;
pools: ;
schedules: Daily Oracle, Monthly Oracle;
policies: Oracle Daily, Oracle Monthly;
directives: pythia exclude oracle, delphi exclude oracle;
...etc...

To date, NetWorker’s administration focus has been far more global. If you’re an administrator, you can do anything to any resource. If you’re a user, you can’t do much with any resource. If you’ve been given a subset of privileges, you can use those privileges against all resources touched by those privileges.

An architecture that worked along these lines would allow for much more flexibility in terms of partial administrative privileges in NetWorker – zones of resources and local administrators for those resources would allow for more granular control of configuration and backup functionality, while still keeping NetWorker configuration maintained at the central server.

What’s wrong with the NMC installation process?

 NetWorker, Security  Comments Off on What’s wrong with the NMC installation process?
Aug 172009
 

There is, in my opinion, an unpleasant security hole in the NMC installation/configuration process.

The security hole is simple: it does not prompt for the administrator password on installation. This is inappropriate for a data protection product, and I think it’s something that EMC should fix.

The NMC installation process is slightly different depending on whether you’re working with 7.5.x or 7.4.x and lower.

For 7.4.x and lower, the process works as follows:

  • Install NetWorker management console.
  • (On Unix platforms, manually run the /opt/lgtonmc/bin/nmc_config file to initialise the configuration.)
  • Launch NMC.
  • Use the default username/password until you get around to changing the password.

For 7.5.x and higher installations, the process works as follows:

  • Install NetWorker management console.
  • First person to logon gets to set the administrator password.

In both instances, this represents a clear security threat to the environment, particularly when installing NetWorker on the backup server or another host that already has administrator access to the datazone, and needs to be managed carefully. Two clear options, depending on the level of trust you have within your environment are:

  • Use firewall/network security configuration options to restrict access to the NMC console port (9000) to a single, known and trusted host, until you are able to log on and change the password.

or

  • Be prepared to log onto NMC as soon as the installation (or for Unix, installation/configuration) is complete and trust that you “get there first”.

In reality, the second option would not be declared secure by any security expert, but for small environments where the trust level is high, it may be acceptable for local security policies.

The real solution though is simple: EMC must change the NMC installation process to force the input of a secure administrator password at install time. That way, by the time the daemons are first started, they are already secured.

Are you Monitoring RAP?

 NetWorker, Security  Comments Off on Are you Monitoring RAP?
Jul 032009
 

Introduced in NetWorker v7 was a feature called “Monitor RAP”. There’s two unfortunate aspects to this setting in the NetWorker server resource (“NSR”):

  • It is not enabled by default;
  • The setting name obfuscates the purpose of the setting for most users.

Personally, I would have preferred when this option was made available that it was called “Audit Changes”, and that it was enabled by default.

In the NetWorker management console, with diagnostic mode enabled, this option is available in the first tab of the NetWorker server properties:

Monitor RAP setting

Monitor RAP setting

With this setting enabled, NetWorker maintains a new log, rap.log, within the logs directory. This tracks changes that are made to the NetWorker configuration, as they are made.

Here’s an example:

06/30/2009 06:49:19 PM MONITOR_RAP: preston@archon CHANGED 'NSR group'
resource, Staging Development:
 autostart: Enabled;
 autostart: Disabled;

This tells us that at 18:49:19 on 30 June 2009, user ‘preston’ on host ‘archon’ changed the group ‘Staging Development’, changing ‘autostart’ from ‘Enabled’ to ‘Disabled’.

This means that from the NetWorker level*, you can easily keep track of who does what to the NetWorker configuration. Interestingly, you can also use this information to also track self-changes to the system – i.e., where NetWorker updates its own configuration. As an example, if you use a license manager, then whenever NetWorker updates/checks its licenses against the license server, you’ll get entries in the logs such as:

06/30/2009 05:10:00 PM MONITOR_RAP: root@nox CHANGED 'NSR license' resource,
Autochanger Module, 40 slots/40:
 checksum: 
"'j&P_-QFc]]D~GIQ\_[q-jMFx;ajW%U~\1^UvCm`dJEwg/
T#XGMpaVYet\l]M\w\{QQ\\
WuR]"Ax*@^XX'[ZAG388M)I6fvztWrC9q"G3PeML!wl
{P6L0]JU9a9[{WYZ";
 checksum: 
"Z$EP_Jts]gYNZV_ATn]AZSxcTorL#f2("8/PTcIYec}K+}
e_GGsJ$6IC)QLTz\_aS?[|lc^\Z
rN9}A~L@}?b^Vlud-e:SD+Js<U]T!eXR"y/,bQHO
0_CShOlw?U1h\g?";

Using the Monitor RAP setting allows you to easily monitor changes to the NetWorker configuration, and I believe that every NetWorker site should always have this setting enabled.


* Auditing is also available within NMC. For maximum auditing, I always recommend that both options be used.

%d bloggers like this: