Taking a second look at domain controller virtualization

http://searchwindowsserver.techtarget.com/tip/0,289483,sid68_gci1515899,00.html?track=NL-1112&ad=773321&asrc=EM_NLT_11921211&uid=5609825

Taking a second look at domain controller virtualization

A while back I wrote an article about the then controversial issue of virtualizing domain controllers (DCs). The big issue at the time was Microsoft's aversion to providing support for DCs hosted on virtual machines for anything other than its own virtualization software, Virtual Server 2005. Back then, Microsoft required admins to reproduce their DCs on physical hardware – something that is pretty tough to do with domain controllers.

With the entire IT industry going fast and furious down the virtualization path and so much new technology available, it seems appropriate to take another look at the issue of domain controller virtualization.

The first thing to understand about virtualizing domain controllers is that you can't treat a DC as you would any other server; it isn't any other server. A domain controller is a key security component in your infrastructure, and it simply plays by different rules than a member server running applications.

Of course, you could make the argument that there is really no "one size fits all" method for determining how a server can be virtualized. For instance, you wouldn't use the same criteria for virtualizing Exchange Server that you would for SQL Server. Servers use different resources in different ways, and they have different backup requirements and recovery methods. Domain controllers are different for similar reasons.

Understanding the issues

The biggest argument to be made against virtualizing domain controllers involves security and administration. Large and medium-size companies will usually have different admins for Active Directory (AD) and server virtualization. Simply stated, you might not want the administrator for the virtual server hosts to have AD domain admin rights and vice-versa.

In the Active Directory world, it's a best practice to limit domain admin rights to those who are highly skilled and trusted. If these folks don't have admin rights to the virtual server host that the domain controller VMs run on, it may prevent them from a timely restore of the DC. For instance, if the domain controller goes down, you must have access to the virtualization software's console to start it up or configure it. This right may be limited to virtualization admins.

There are many examples of crossing over security responsibilities, but you get the idea -- implementing a domain controller as a virtual machine creates additional complexity.

Another key component of virtualizing domain controllers involves backups. Backups for other types of virtual servers can be accomplished by backing up VHD files, making snapshots of virtual host machines, or taking and restoring images. These methods, however, don't work when it comes to backing up a domain controller.

Microsoft Knowledge Base article 888794 refers to this issue as USN rollback, which is used intentionally for authoritative restores. This works when the domain controller is properly backed up, resetting the invocation ID. When the host machine creates and restores a virtual DC from a snapshot, however, it will not reset the invocation ID. Therefore this older set of data is not updated and the problem domain controller is unable to tell other DCs that it is out of date in relation to them, thus preventing replication.

This is very difficult to detect, as there are no events or errors that say "We can't replicate because you have executed USN rollback without resetting the invocation ID." The problem leads to inconsistencies such as certain objects being on some domain controllers and not others. The only fix for this is to demote the problem DC and re-promote it.

Don't confuse this process with doing Volume Shadow Copy Service (VSS) backups within the VM. I like Active Directory expert Sean Deuby's statement that you should never do anything to a virtual DC that the DC itself – and the directory service – isn't aware of. For instance, copying or exporting a virtual machine's VHD is done on the host while the VM itself is unaware of what is going on. I saw one scenario where there were two virtual domain controllers on different hosts. As a backup strategy, each night the admins would export the virtual machine from each DC and store it on the other. This is in violation of Deuby's rule described above. Does the guest machine – the DC itself – know about this export and save? Of course not; therefore, it should not be done. Instead, just back up the VM from within the VM and you'll be fine.

Time synchronization is also an issue. System time in the virtual domain controller should not be tied to the host. The host is not a DC, so why should it participate in the time services infrastructure? Only domain controllers' sync times are used as time servers. The primary domain controller (PDC) is the authoritative time server for the domain, so synching the host's time service to drive the domain controller's time would circumvent the root DC being authoritative for the domain. You would not sync a physical DC from two separate time sources.

The domain controller is a foundational element of the domain structure and it must stay running or be able to boot up immediately if it does go down. It isn't a big problem to pause a virtual DC for a short period of time, just like it isn't a problem to shut a physical DC off for a while. As long as the downtime is less than the tombstone lifetime period, it will properly replicate when it comes online and get itself up to date. Yes, there will be complaints in the event log, but this can be done for maintenance, reconfiguring resource settings for the VM and more, just as you would shut it down for maintenance with a physical computer.

Workarounds for domain controller virtualization

In his blog, Microsoft's Ben Armstrong offers some interesting options for resolving the security and administration issues noted above, including how to make the host a standalone server and creating a separate domain for the host. In the end, he suggests putting the host in the domain with the virtual domain controller. He also makes some other excellent recommendations for virtual DCs:

1. Configure virtual DCs to start when the host (parent) machine starts. Remember, the host machine needs maintenance and updates, so that will add to the possibility of taking the virtual DC offline. This will make the domain controller start up without manual intervention.
2. Other virtual machines on the host that are not DCs should be configured to have a delayed start time, giving boot priority to the virtual DCs. This provides resources for the virtual DC that will help facilitate a faster boot.
3. Configure the virtual domain controllers to shut down without saving state if the host shuts down.
4. Domain admins should have a way to access the host machine and the virtualization software's console in case the DC fails to start. This could be accomplished via an operator who is physically near the host as well and can start it manually without needing Active Directory privileges.

I would also add the following suggestions:

5. Don't put all of your virtual DCs on one physical host. I know of one admin who had two domain controllers and built them as VMs on one host machine. The host had a disk go bad and he lost both DCs, making recovery limited or non-existent. It's better to spread them out among multiple hosts for each domain.
6. New advancements in virtualization technology will make domain controller virtualization more attractive. Hyper-V for Windows Server 2008 R2 SP1 includes support for dynamic memory and better connection support for DCs. Hot-add memory and storage in Hyper-V R2 allows for expansion of resources without requiring downtime.
7. Use common sense -- avoid single points of failure and make sure your backup and recovery plan works. Also, monitor performance to make sure the domain controller is handling the load, just as you would for a physical server.

I've heard some discussion about the wisdom of putting all domain controllers on virtual machines, or maybe leaving just one or two as physical machines. I have talked to one admin who said, due to budget restraints, he was forced to virtualize all of his DCs. He has run them all on virtual machines for several months and (so far) has not experienced any trouble. I think most people want to hold on to physical machines simply because they are comfortable with them.

So the takeaway here is that it is possible to successfully virtualize domain controllers, but you must follow the best practices noted here and in the Microsoft references to ensure success.

Perhaps someday we will feel comfortable in a completely virtualized Active Directory environment.