In Part 1, I covered configuring the Hyper-V Failover Cluster and the Scale-Out File Server solution. In Part two, I will cover:

• Creating the file shares in the Scale-Out File Server
• Creating a virtual machine to use the SMB3.0 shares in the Scale-Out File Server
• Verifying we can Live Migrate the virtual machines in the Hyper-V Failover Cluster

Creating the File Share

Execute the following steps to create a file share in the Scale-Out File Server

1. In Failover Cluster Manager, right-click on the Scale-Out File Server role in the center pane and choose Add File Share. This starts the New Share Wizard
2. In the Select Profile screen, choose SMB Share - Applications and click Next
3. For the Share Location, choose one of the CSV Volumes and click Next
4. Provide a Share Name, verify the path information and click Next
5. In the Other Settings screen, Enable Continuous Availability is checked by default. Click Next
   Note: Some selections are greyed-out. This is because they are not supported for this share profile in a Failover Cluster
6. In the Permissions screen, click Customize Permissions. In the Advanced Security Settings screen, note the default NTFS and Share permissions and then proceed to add the Hyper-V Failover Cluster Nodes Computer Accounts to the NTFS permissions for the share and ensure they have Full Control. If the permissions listing does not include the cluster administrator(s), add it and give the account (or Security Group) Full Control. Click Apply when finished

Complete configuring the file shares.

As a test, connect to each of the shares from the Hyper-V Failover Cluster and verify you can write to each location before proceeding to the next step.

Creating a Virtual Machine to use an SMB 3.0 Share

Execute the following steps to create a new virtual machine

1. On one of the nodes in the Hyper-V Cluster, open Failover Cluster Manager
2. In the left-hand pane, click on Roles and then in the right-hand Actions pane click on Virtual Machines and choose New Virtual Machine
3. Choose one of the cluster nodes to be the target for the virtual machine and click OK
4. This starts the New Virtual Machine Wizard. Review the Before You Begin screen and click Next
5. In the Specify Name and Location screen, provide a name for the virtual machine and enter an UNC path to a share on the Scale-Out File Server and then click Next
   
   
   
6. Configure memory settings and click Next
7. Configure network settings and click Next
8. In the Connect Virtual Hard Disk screen, make a selection and click Next
9. Review the Summary screen and click Finish
10. Verify the process completes successfully and click Finish

Testing Live Migration

Once all the virtual machines are created, you may want to test Live Migration. Depending on how many simultaneous live migrations you want to support, you may have to modify the Live Migration settings on each of the Hyper-V Failover Cluster nodes. The default is to allow two simultaneous live migrations. Here is a little PowerShell script you can run to take care of the settings for all the nodes in the cluster -

In my cluster, I have all the virtual machines running on the same node -

I will use a new feature in Windows Server 2012 Failover Clusters, multi-select, and select all of the virtual machines and live migrate them to another node in the cluster -

Since there are only four virtual machines and the maximum number of live migrations is equal to six, all will migrate.

If I were to rerun my script and make a change back to two, then two migrations will be queued until at least one of the in progress migrations completes.

You can use the Get-SmbSession PowerShell cmdlet on any node in the Scale-Out File Server to determine the number of sessions. For illustration purposes, I have all virtual machines running on the same Hyper-V Failover Cluster node (Fabrikam-N21) and the CSV volumes are running on the same node in the Scale-Out File Server (Fabrikam-N1) -

Distributing the virtual machines across the multi-node Hyper-V Failover Cluster (Fabrilam-N21, Fabrikam-N22, and Fabrikam-N23) is reflected on the Scale-Out File Server -

Finally, I re-distribute the CSV volumes across the Scale-Out File Server nodes as shown here -

This is reflected in the Get-SmbSession PowerShell cmdlet output -

Thanks, and come back again soon.

Chuck Timon
Senior Support Escalation Engineer
Microsoft Enterprise Platforms Support
High Availability\Virtualization Team

On Tuesday, Jaunary 8, the below recommended fix was released and available on Windows Update for Windows .NET Framework 4.5.

2750149
An update is available for the .NET Framework 4.5 in Windows 8, Windows RT and Windows Server 2012
http://support.microsoft.com/kb/2750149/EN-US

When installing this update to a Windows 2012 Cluster Server, you will receive the below error when you select Roles or Nodes from within Failover Cluster Manager.

A weak event was created and it lives on the wrong object, there is a very high chance this will fail, please review and make changes on your code to prevent this issue

You can still manage your Cluster from a node that does not have this fix or through Powershell.  The recommendation at this time is to remove and not install this fix. 

Microsoft is aware of the issue.  Once the cause has been identified and a resolution available, this blog will be updated to reflect the resolution.

==========
UPDATE - January 23, 2013
==========

This issue has been resolved.  After installing the KB2750149 patch for .NET Framework, please also install the below patch.

KB2803748
Failover Cluster Management snap-in crashes after you install update 2750149 on a Windows Server 2012-based failover cluster
http://support.microsoft.com/kb/2803748

John Marlin
Senior Support Escalation Engineer
Microsoft Enterprise Platforms Support

Hello everyone. Today’s post is going to cover the steps needed to follow should you ever have to remove a ‘Physical Disk’ resource from a clustered service or application where that disk is configured as a mount point. Removing a disk from a group may be needed if the application no longer requires the storage and there’s a need to utilize the disk in some other group or decommission it entirely.

First, I wanted to talk a little about dependencies and their function. Resource dependencies are created between one or more resources in a cluster group in order to determine the order in which resources in the group are taken online/offline. Take for instance a SQL Server resource that’s dependent on the ‘Physical Disk’ resources where SQL’s data is stored. A dependency should be established so that the SQL Server resource is dependent on the disk resource. This dependency will make sure the disk comes online first, and then SQL Server. Same thing when taking those things offline, except in reverse. The SQL Server resource will come offline before the disk comes offline. Obviously, we would not want to have SQL Server attempt to start until all the disks it was using was online first.

Once resources in a cluster group are linked with dependencies, you have to be careful when deleting resources out of a group. If you don’t remove dependencies properly, you may end up inadvertently removing other resources as well.

In this example, I have two resources in a cluster group, Resource A and Resource B. I establish dependencies between them so that Resource B is dependent on Resource B.

This means that when the group is brought online, Resource B will not be brought online until Resource A is online.

Here is what the dependency report looks like.

Now that these two resources are linked, you have to be careful when deleting these resources. If I were to delete Resource A from the group without first removing the dependency, BOTH resources will get removed.

At this point, a pop-up will appear warning that a removal of this resource could affect applications using this resource.

If you click ‘Yes’, both resources get removed.

Lesson learned. If the resource you are deleting is dependent on any other resources, remove the dependency first.

Now, we get to the main point of this post. The above process is fine for deleting resources from a cluster group unless the resource you are deleting is configured as a “Physical Disk’ resource, and it a mount point disk. The process differs slightly and you must follow this process or you could find yourself unintentionally moving every resource in the group into ‘Available Storage’.

First, lets cover proper way to remove a mount point disk from a cluster group. In this example, I have a plain File Server group with a Network Name, IP Address, File Server, and three disks. A root disk (Disk X:) and two mount points using folders on the root of X: called X:\MountPointA and X:\MountPointB.

Since I don’t have any shares located on X:\MountPointB, I want to remove that disk so I can use it in some other application. The FIRST thing I need to do is take the resource offline.

Then I can right-click the resource and click ‘Remove from GroupName’

When you remove a ‘Physical Disk’ resource from a cluster group, it doesn’t actually remove the cluster resource altogether, it moves the disk resource into the ‘Available Storage’ group.  This is so that you can reallocate the resource to another group if needed.

As you can see, the resource now shows in ‘Available Storage’

At this point, you can remove the mount point configuration, or change it to a lettered drive so you can use for some other application.

Now let’s go over what can happen if you don’t take the mount point offline before removing it. The main reason in going over this is to show you how to recover so that there’s no adverse impact to the Cluster.

In this example, I am removing the same resource from the File Server group following the same process above WITHOUT taking it offline first. First, I verified there were no dependencies on the ‘DiskX:\MountPointB’ resource.

Now here’s where it gets fun. After I attempted to remove the mount point, ALL of my resources disappear from the group. ??????

Time to panic? No, all is not lost. What happened is that because we had a mount point configured, and a mount point is not usable unless there’s a root disk, ALL of the resources moved to ‘Available Storage’ because the rest of the resources DO have dependencies.

It may appear all of the resources disappeared. Because in the UI, we only show ‘Physical Disk’ resources, if any other resources get put in that group, they don’t show up in the UI. However, if we run a command line to display all resources and their groups, we can see that the resources are still there.

To get the resources back into the right group, just move the disks back to the original File Server group. Right click the disk, More actions, and select Move this resource to another service or application. The same dependency tree will cause the resources to move back.

Now we have all our resources back and we can follow the correct process of taking the mount point disk offline BEFORE removing it.

Cheers!

Jeff Hughes
Senior Support Escalation Engineer
Microsoft Enterprise Platforms Support

The IT Pro Evangelism team, Microsoft Learning and the Microsoft Virtual Academy are pleased to announce the next Jump Start courses!

Introduction to Hyper-V

Thursday, January 24th

This one-day live event is designed for IT Pros experienced in virtualization (i.e., VMware) but in need of learning how to leverage Hyper-V to perform essential tasks in the Windows Server 2012 platform. Introduction to Microsoft Virtualization, Hyper-V Infrastructure, Hyper-V Networking, Hyper-V Storage, Hyper-V Management, Hyper-V High Availability and Live Migration, Integration with System Center 2012 Virtual Machine Manager, and Integration with Other System Center 2012 Components

COURSE OUTLINE

(01) Introduction to Microsoft Virtualization
(02) Hyper-V Infrastructure
(03) Hyper-V Networking
(04) Hyper-V Storage

**Meal Break**

(05) Hyper-V Management
(06) Hyper-V High Availability and Live Migration
(07) Integration with System Center 2012 Virtual Machine Manager
(08) Integration with Other System Center 2012 Components

Microsoft Virtualization for VMware Professionals

Wednesday, January 30th.

This course is designed for VMware professionals looking to get up-to-speed with how Microsoft virtualization and Windows Server 2012 Hyper-V works and compares with VMware vSphere 5.1.  Introduction & Scalability, Storage & Resource Management, Multi-tenancy & Flexibility, Backup & High-Availability, Introduction & Overview of System Center 2012, Application Management, Cloud on your Terms, and Foundation, Hybrid Clouds & Costs.

COURSE OUTLINE

AM | Windows Server 2012 Hyper-V vs. VMware vSphere 5.1
(01) Introduction & Scalability
(02) Storage & Resource Management
(03) Multi-tenancy & Flexibility
(04) Backup & High-Availability

**Meal Break**

PM | System Center 2012 SP1 vs. VMware’s Private Cloud
(05) Introduction & Overview of System Center 2012
(06) Application Management
(07) Cloud on your Terms
(08) Foundation, Hybrid Clouds & Costs

These live online events are designed for IT Pros that are new to virtualization, or experienced in other hypervisors (such as VMware or Citrix) and want to learn about Windows Server 2012 Hyper-V. 

Join Microsoft & VMware virtualization experts Symon Perriman, Jeff Woolsey and Matt McSpirit in a demo-rich learning experience with a live Q&A for a full days of training from 8am to 5pm.

These events are FREE and open to the PUBLIC so please register and spread the word for Introduction to Hyper-V andMicrosoft Virtualization for VMware Professionals today!

Join us again in late February for our Microsoft Tools for VMware Migration & Integration Jump Start.

I often get asked the question by customers on how to find the newly released hotfixes for Cluster and Hyper-V or the customer wants to know what hotfixes are out there. We have 3 Wiki pages that will make life easier for you when looking for hotfixes for Windows Server 2008 R2 Cluster and Hyper-V. You can also setup an RSS Subscription to the Wiki page so you can the updated Wiki page in your favorite RSS app or Outlook.

Hyper-V: Update List for Windows Server 2008 R2 (en-US)

http://social.technet.microsoft.com/wiki/contents/articles/1349.hyper-v-update-list-for-windows-server-2008-r2-en-us.aspx

List of Cluster Hotfixes for Windows Server 2008 R2

http://social.technet.microsoft.com/wiki/contents/articles/2008.list-of-cluster-hotfixes-for-windows-server-2008-r2.aspx

List of Failover Cluster Packages Updated After the Release of Windows Server 2008 R2 Service Pack 1 (SP1)

http://social.technet.microsoft.com/wiki/contents/articles/3153.list-of-failover-cluster-packages-updated-after-the-release-of-windows-server-2008-r2-service-pack-1-sp1.aspx

Steven Graves
Support Escalation Engineer
Microsoft Enterprise Platform Support
High Availability Team

The Microsoft Virtual Academy training portal offers quick and easy access to presentations, webcasts, whitepapers and other technical content about Microsoft’s Cloud technologies.  It is public, free and you learn at your own pace while earning awards and recognitions, and connecting with over 1 million other IT Pros.

Microsoft Virtualization for VMware Professionals Jump Start

This course is designed for VMware professionals looking to get up-to-speed with how Microsoft virtualization and Windows Server 2012 Hyper-V works and compares with VMware vSphere 5.1. This one-day event will feature Microsoft Technical Evangelist Symon Perriman and Technical Product Manager Matt McSpirit (both VMware Certified Professionals) delivering an engaging, demo-rich, live learning experience.

Microsoft Virtualization for VMware Professionals agenda includes:

1. Introduction & Scalability
2. Storage & Resource Management
3. Security, Multi-tenancy & Flexibility
4. High-Availability & Resiliency
5. System Center 2012 Introduction & Overview
6. Application Management
7. Cross-Platform Management
8. Foundation, Hybrid Clouds & Costs

This training is available now for free and can be downloaded for viewing at a later time, if desired.

In a previous blog, it was discussed on where you can get a list of the Failover Clustering and Hyper-V hotfixes for Windows Server 2008 R2.

Looking for Windows Server 2008 R2 Cluster and Hyper-V Hotfixes?
http://blogs.technet.com/b/askcore/archive/2013/02/15/looking-for-windows-server-2008-r2-cluster-and-hyper-v-hotfixes.aspx

In this blog, we give the links to newly released hotfixes for Failover Clustering and Hyper-V so you know what hotfixes are out there. There are two Wiki pages that will make life easier for you when looking for hotfixes for Windows Server 2012 Clustering and Hyper-V.

Hyper-V: Update List for Windows Server 2012
http://social.technet.microsoft.com/wiki/contents/articles/15576.hyper-v-update-list-for-windows-server-2012.aspx

List of Failover Cluster Hotfixes for Windows Server 2012
http://social.technet.microsoft.com/wiki/contents/articles/15577.list-of-failover-cluster-hotfixes-for-windows-server-2012.aspx

These Wiki pages are updated on a regular basis.  You can setup an RSS Subscription to the Wiki page so you can the updated Wiki page in your favorite RSS app or Outlook.

John Marlin
Senior Support Escalation Engineer
Microsoft Enterprise Platforms Support

The IT Pro Evangelism team, Microsoft Learning and the Microsoft Virtual Academy are pleased to announce the next FREE& PUBLIC event Live Q&A: Introduction to Hyper-V on Wednesday April 3^rd, from 8:30 am – 10:00am PST with virtualization experts Jeff Woolsey (Principal Program Manager) & Symon Perriman (Senior Technical Evangelist).

Ask your customers to join this live online event designed for IT professionals that have questions about Microsoft virtualization and want to learn about Windows Server 2012 Hyper-V.  Register here: http://aka.ms/MVAf-HyperV.  If you cannot make the live event, sign up anyway so you can receive a notification when the recording is published on the Microsoft Virtual Academy.

Topics and demos may include:

• Introduction to Microsoft Virtualization
• Hyper-V Infrastructure
• Hyper-V Networking
• Hyper-V Storage
• Hyper-V Management
• Hyper-V High Availability and Live Migration
• Integration with System Center 2012 Virtual Machine Manager
• Integration with Other System Center 2012 Components

Tweet: Ask us your questions about #Windows#Sever 2012 #HyperV! Register for this live free public Q&A event on April 3rd: http://aka.ms/MVAf-HyperV

Also check out our recent full day training Microsoft Virtualization for VMware Professionals Jump Start which is now available on the Microsoft Virtual Academy.

I recently encountered an issue involving the failure of a new Network Name resource to come online. Doing some investigation I found a number of instances where this has been encountered, with different resolutions provided.  Since no root cause was defined, fellow Directory Services Engineer Robert Williams and I set out to determine the cause. 

You’ll know you’ve encountered this issue if you create a new Network Name resource and it fails to online with the following errors:

In the System Event log you will see a Failover Cluster event 1194:

Log Name:      System

Source:        Microsoft-Windows-FailoverClustering

Date:          3/27/2013 1:19:07 PM

Event ID:      1194

Task Category: Network Name Resource

Level:         Error

Keywords:     

User:          SYSTEM

Computer:     ComputerName

Description:

Cluster network name resource 'ComputerName' failed to create its associated computer object in domain 'DomainName' for the following reason: Unable to obtain access to Computer Object in DS.

The text for the associated error code is: Access is denied.

Please work with your domain administrator to ensure that:

• The cluster identity 'CNO' can create computer objects. By default all computer objects are created in the 'Computers' container; consult the domain administrator if this location has been changed.
• The quota for computer objects has not been reached.
• If there is an existing computer object, verify the Cluster Identity 'CNO' has 'Full Control' permission to that computer object using the Active Directory Users and Computers tool.

In the Cluster log you will see the following entries:

00000ea4.000012b0::2013/03/25-16:55:04.113 ERR   [RES] Network Name < NetworkName>: Failed to obtain access to computer account < AccountName>, status 80070005

00000ea4.000012b0::2013/03/25-16:55:04.128 ERR   [RHS] Online for resource <NetworkName> failed.

Note:   To generate a Cluster log, run the following command from an administrators command prompt. The Cluster.log file will be generated in the c:\windows\cluster\reports directory.  The entry will be in the Cluster log on the Node where the online attempt failed.

Cluster log /gen

We determined that the root cause of the issue is due to the removal of NT AUTHORITY\Authenticated Users from the local Users group.  Note below that it is present by default:

The best solution is to add back NT AUTHORITY\Authenticated to the local Users Group. This will require a reboot for the change to take effect.  If your security team is unwilling to do this, you can disable the following two Security policies and refresh the policy by running gpupdate /force:

Network access: Do not allow anonymous enumeration of SAM accounts

Network access: Do not allow anonymous enumeration of SAM accounts and shares

You will have to determine which of these two options best fits the security requirements for your environment.   It may be a good option to create a separate Organizational Unit (OU) for your Cluster servers.  This will allow you to affect the preferred change to the limited subset of servers. 

Steven Andress

Senior Support Escalation Engineer

Microsoft Customer Support & Services

Greetings AskCore fans.  I thought I would start creating a blog for a subset of the videos/sessions available at each of the conferences that Microsoft has throughout the year.  These sessions will be available and will deal with topics that are discussed here on the AskCore Blog.  There are numerous other sessions if you want to get into System Center products, Azure, SQL Server, Exchange, etc.  They are just to numerous to list here. 

All sessions at our conferences are approximately 1:15 long and are presented by either Program Managers, Product Managers, Technical Evangelists, Support Escalation Engineers, etc.  Sit back and enjoy the sessions at your leisure.  Most all sessions can be viewed in either MP4 and WMV.  You can also download the sessions and PowerPoint presentations from the deliveries to be viewed at a later time or on some other device.

If you have attended any of these conferences, our many thanks are given.  If you haven't, you should look into going to one as it is a good way of networking with others and have a good time while getting up to speed on all the new technologies.

So, without further adieu, first up:  MMS 2013.

=====================

The Microsoft Management Summit (MMS) brings together the brightest IT professionals from around the world to increase their technical expertise through an intensive week of training led by experts in desktop, device management, datacenter, and cloud technologies.  This is held in Las Vegas every year and is a good time held by all.

MMS 101: Conquering the Summit
http://channel9.msdn.com/Events/MMS/2013/MMS101
If this is your first time attending the Microsoft Management Summit, then you don't want to miss this session! You'll learn priceless tips and tricks to maximize your investment in MMS2013. We'll give you the inside scoop on all the sessions, labs and Expo vendors. If that's not enough, we'll get you networked with other attendees and alumni from your area. Finally, we'll answer all your questions so that you have everything you need to conquer the Summit!

Windows Server 2012 in 60 Minutes
http://channel9.msdn.com/Events/MMS/2013/WS-B326

Introduction to Failover Clustering with Windows Server 2012
http://channel9.msdn.com/Events/MMS/2013/WS-B317

What's New in Windows Server 2012 Hyper–V
http://channel9.msdn.com/Events/MMS/2013/WS-B330

Availability Strategies for a Resilient Private Cloud
http://channel9.msdn.com/Events/MMS/2013/WS-B302

A Geek's Guide to USMT 5.0
http://channel9.msdn.com/Events/MMS/2013/DC-B301

Demonstrations of Assessment and Deployment Kit Tools
http://channel9.msdn.com/Events/MMS/2013/DC-B302

Advanced Microsoft Deployment Toolkit 2012 Update 1 Customizations
http://channel9.msdn.com/Events/MMS/2013/DC-B303

Implementing the Windows To Go Concept in an Enterprise Environment
http://channel9.msdn.com/Events/MMS/2013/DC-B304

Deploying Windows 8 Using Lite Touch
http://channel9.msdn.com/Events/MMS/2013/DC-B308

Choosing the Right OS Deployment Tool
http://channel9.msdn.com/Events/MMS/2013/DC-B309

Windows Sysinternals Primer
http://channel9.msdn.com/Events/MMS/2013/DC-B311

What's New with Windows 8 Bitlocker and Microsoft BitLocker Administration and Management 2.0
http://channel9.msdn.com/Events/MMS/2013/DC-B312

Real World Windows 8 Deployment with MDT 2012 Update 1
http://channel9.msdn.com/Events/MMS/2013/DC-B316

Deploying Windows To Go in the Real World
http://channel9.msdn.com/Events/MMS/2013/DC-B317

What's New in Windows 8 Deployment
http://channel9.msdn.com/Events/MMS/2013/DC-B318

Making PC Recovery Easier with the Microsoft Diagnostics and Recovery Toolset
http://channel9.msdn.com/Events/MMS/2013/DC-B321

Competitive Advantages of Hyper-V over VMware vSphere
http://channel9.msdn.com/Events/MMS/2013/WS-B201

File Storage Strategies for Private Cloud
http://channel9.msdn.com/Events/MMS/2013/WS-B309

How to Design and Configure Networking in VMM and HyperV (Part 1 of 2)
http://channel9.msdn.com/Events/MMS/2013/WS-B312

How to Design and Configure Networking in VMM and HyperV (Part 2 of 2)
http://channel9.msdn.com/Events/MMS/2013/WS-B313

Switching to Hyper–V: Migrating from VMware
http://channel9.msdn.com/Events/MMS/2013/WS-B325

Sit back, relax, grab some popcorn, and enjoy the sessions.

John Marlin
Senior Support Escalation Engineer
Microsoft Enterprise Platforms Support

A few months ago we held the annual 2013 Microsoft Management Summit in Las Vegas. As in years past, the event sold out quickly and it was a very busy week. To everyone that attended, our sincere thanks.  As a recap, the below blog gives you the list of available sessions online to view that deal with topics covered on the Core Blog site.

Sessions from MMS 2013 Now Available
http://blogs.technet.com/b/askcore/archive/2013/04/23/sessions-from-mms-2013-now-available.aspx

As usual, the hands-on labs and instructor-led labs continue to be some of the most popular offerings at MMS. MMS Labs offer folks the opportunity to kick the tires on a wide array of Microsoft technologies and products. As usual the lines started early. For the fourth year in a row, all of the MMS Labs were 100% virtualized using Windows Server Hyper-V and managed via System Center by our partners at XB Velocity and using HP servers and storage. Of course, this year we upgraded to the latest version so everything was running on a Microsoft Cloud powered by Windows Server 2012 Hyper-V and System Center 2012 SP1.

(BTW, Microsoft blogged about this topic in the past years, if you’re interested, the links are here and here.)  Before I jump into the Microsoft Private Cloud, let me provide some context about the labs themselves.

What is a MMS Hand On Lab?

One of the reasons the MMS Hands on Labs are so popular is because it’s a firsthand opportunity to evaluate and work with Windows Server and System Center in a variety of scenarios at your own pace. Here’s a picture of some of the lab stations…

With the hands on labs, we’ve done all the work to create these scenarios based on your areas of interest. So, what does one of these labs look like on the backend? Let’s be clear, none of these labs are a single VM. That’s easy. Been there, done that. When you sit down and request a specific lab, the cloud infrastructure provisions the lab on highly available infrastructure and deploys services that can be anywhere from 4 – 12 virtual machines in your lab in seconds. There are over 650 different lab stations and we have to account for all types of deployment scenarios. For example,

1. In the first scenario, all users sit down at 8 am and provision exactly the same lab. Or,
2. In the second scenario, all users sit down at 8 am and provision unique, different labs. Or,
3. In the third scenario, all users sit down at 8 am and provision a mix of everything

The lab then starts each lab in a few seconds. Let’s take a closer look at what some of the labs look like in terms of VM deployment.

MMS Lab Examples

Let’s start off with a relatively simple lab. This first lab is a Service Delivery and Automation lab. This lab uses:

• Four virtual machines
• 16 virtual processors
• 15 GB of memory total
• 280 GB of storage
• 2 virtual networks

…and here’s what each virtual machine is running…

Interested in creating virtualizing applications to deploy to your desktops, tablets, Remote Desktop Sessions? This next lab is a Microsoft Application Virtualization (App-V) 5.0 Overview lab. This lab uses:

Seven virtual machines

• 14 virtual processors
• 16 GB of memory total
• 192 GB of storage
• 2 virtual networks

How about configuring a web farm for multi-tenant applications? Here’s the lab which uses:

• Six virtual machines
• 24 virtual processors
• 16 GB of memory total
• 190 GB of storage
• 2 virtual networks

Ever wanted to enable secure remote access with RemoteApp, DirectAccess and Dynamic Access Control? Here’s the lab you’re looking for. This lab uses:

Seven virtual machines

• 28 virtual processors
• 18 GB of memory total
• 190 GB of storage
• 2 virtual networks

Again, these are just a few of the dozens of labs ready for you at the hands on labs.

MMS 2013 Private Cloud: The Hardware

BTW, before I get to the specifics, let me point out that this Microsoft/HP Private Cloud Solution is an orderable solution available today...

Compute. Like last year, we used two HP BladeSystem c7000s for compute for the cloud infrastructure. Each c7000 had 16 nodes and this year we to upgraded to the latest BL460c Generation 8 Blades. All 32 blades were then clustered to create a 32 node Hyper-V cluster. Each blade was configured with:

• Two sockets with 8 cores per socket and thus 16 cores. Symmetric Multi-Threading was enabled and thus we had a total of 32 logical processors per blade.
• 256 GB of memory per blade with Hyper-V Dynamic Memory enabled
• 2 local disks 300 GB SAS mirrored for OS Boot per blade
• HP I/O Accelerator cards (either 768 GB or 1.2 TB) per blade

Storage. This year we wanted to have a storage backend that could take advantage of the latest storage advancements in Windows Server 2012 (such as Offloaded Data Transfer and SMI-S) so we decided to go with a 3Par StoreServ P10800 storage solution. The storage was configured as a 4 node, scale-out solution using 8 Gb fibre channel and configured with Multi-Path IO and two 16 port FC switches for redundancy. There was a total of 153.6 TB of storage configured with:

• 64 x 200 GB SSD disks
• 128 x 600 GB 15k FC disks
• 32 x 2 TB 7200k RPM SAS

As you can see, the 3Par includes SSD, 15k and 7200k disks. This is so the 3Par can provide automated storage tiering with HP’s Adaptive Optimization. With storage tiering, this ensures the most frequently used storage (the hot blocks) reside in the fastest possible storage tier whether that’s RAM, SSD, 15k or 7200k disks respectively. With storage tiering you can mix and match storage types to find the right balance of capacity and IOPs for you. In short, storage tiering rocks with Hyper-V. From a storage provisioning perspective, both SCVMM and the 3Par storage both support standards based storage management through SMI-S so the provisioning of the 3Par storage was done through System Center Virtual Machine Manager. Very cool.

Networking. From a networking perspective, the solution used VirtualConnect FlexFabric 10Gb/E and everything was teamed using Windows Server 2012 NIC Teaming. Once the network traffic was aggregated in software via teaming, that capacity was carved up in software.

Time for the Pictures…

Here’s a picture of the racks powering all of the MMS 2013 Labs. The two racks on the left with the yellow signs are the 3Par storage while the two racks on the right contain all of the compute nodes (32 blades) and management nodes (a two node System Center 2012 SP1 cluster). What you don’t see are the crowds gathered around pointing, snapping pictures, and gazing longingly…

MMS 2013: Management with System Center. Naturally, the MMS team used System Center to manage all the labs, specifically Operations Manager, Virtual Machine Manager, Orchestrator, Configuration Manager, and Service Manager. System Center 2012 SP1 was completely virtualized running on Hyper-V and was running on a small two node cluster using DL360 Generation 8 rackmount servers.

Operations Manager was used to monitor the health and performance of all the Hyper-V labs running Windows and Linux. Yes, I said Linux. Linux runs great on Hyper-V (it has for many years now) and System Center manages Linux very well… J To monitor health proactively, we used the ProLiant and BladeSystem Management Packs for System Center Operations Manager. The HP Management Packs expose the native management capabilities through Operations Manager such as:

• Monitor, view, and get alerts for HP servers and blade enclosures
• Directly launch iLO Advanced or SMH for remote management
• Graphical View of all of the nodes via Operations Manager

In addition, 3Par has management packs that plug right into System Center, so Operations Manager was used to manage the 3Par storage as well…

…having System Center integration with the 3Par storage came in handy when one of the drives died and Operations Manager was able to pinpoint exactly what disk failed and in what chassis…

Of course, everything in this Private Cloud solution is fully redundant so we didn’t even notice the disk failure for some time…

In terms of managing the overall solution, here’s a view of some of the real time monitoring we were displaying and where many folks just sat and watched.

Virtual Machine Manager was used to provision and manage the entire virtualized lab delivery infrastructure and monitor and report on all the virtual machines in the system. In addition, HP has written a Virtual Machine Manager plug-in so you can view the HP Fabric from within System Center Virtual Machine Manager. Check this out:

It should go without saying that to support a lab of this scale and with only a few minutes between the end of one lab and the beginning of the next, automation is a key precept. The Hands on Lab team was positively gushing about PowerShell. “In the past, when we needed to provide additional integration it was a challenge. WMI was there, but the learning curve for WMI is steep and we’re system administrators. With PowerShell built-into WS2012, we EASILY created solutions and plugged into Orchestrator. It was a huge time saver.”

MMS 2013: Pushing the limit…

As you may know, Windows Server 2012 Hyper-V supports up to 64 nodes and 8,000 virtual machines in a cluster. Well, we have a history for pushing the envelope with this gear and this year was no different. At the very end of the show, the team fired up as many virtual machines to see how high we could go. (These were all lightly loaded as we didn’t have the time to do much more…) On Friday, the team fired up 8,312 virtual machines (~260 VMs per blade) running on a 32 node cluster. Each blade has 256 GB of memory each and we kept turning on VMs until all the memory was consumed.

MMS 2013: More data…

• Over the course of the week, over 48,000 virtual machines were provisioned. This is ~8,000 more than last year. Here’s a quick chart. Please note that Friday is just a half day…

• Average CPU Utilization across the entire pool of servers during labs hovered around 15%. Peaks were recorded a few times at ~20%. In short, even with thousands of Hyper-V VMs running on a 32 node cluster, we were barely taxing this well architected and balanced system.
• While each blade was populated with 256 GB, they weren’t maxed. Each blade can take up to 384 GB.
• Storage Admins: Disk queues for each of the hosts largely remained at 1.0 (1.0 is nirvana). When 3200 VMs were deployed simultaneously, the disk queue peaked at 1.3. Read that again. Show your storage admins. (No, those aren’t typos.)
• The HP I/O Accelerators used were the 768 GB version and 1.2 TB versions. The only reason we used a mix of different sizes because that’s what we had available.
• All I/O was configured for HA and redundancy.
  • Network adapters were teamed with Windows Server 2012 NIC Teaming
  • Storage was fibre channel and was configured with Active-Active Windows Server Multi-Path I/O (MPIO). None of it was needed, but it was all configured, tested and working perfectly.
• During one of the busiest days at MMS 2013 with over 3500 VMs running simultaneously, this configuration wasn’t even breathing hard. It’s truly a sight to behold and a testament to how well this Microsoft/HP Private Cloud Solution delivers.

From a management perspective, System Center was the heart of the system providing health monitoring, ensuring consistent hardware configuration and providing the automation that makes a lab this complex successful. At its peak, with over 3500 virtual machines running, you simply can’t work at this scale without pervasive automation.

From a hardware standpoint, the HP BladeSystem and 3Par storage are simply exceptional. Even at peak load running 3500+ virtual machines, we weren’t taxing the system. Not even close. Furthermore, the fact that the HP BladeSystem and 3Par storage integrate with Operations Manager, Configuration Manager and Virtual Machine Manager provides incredible cohesion between systems management and hardware. When a disk unexpectedly died, we were notified and knew exactly where to look. From a performance perspective, the solution provides a comprehensive way to view the entire stack. From System Center we can monitor compute, storage, virtualization and most importantly the workloads running within the VMs. This is probably a good time for a reminder…

If you’re creating a virtualization or cloud infrastructure, the best platform for Microsoft Dynamics, Microsoft Exchange, Microsoft Lync, Microsoft SharePoint and Microsoft SQL Server is Microsoft Windows Server with Microsoft Hyper-V managed by Microsoft System Center. This is the best tested, best performing, most scalable solution and is supported end to end by Microsoft.

One More Thing...

Finally, we’ve been talking about Windows Server and System Center as part of our Microsoft Private Cloud Solution. I’d also like to point out that Windows Server 2012 Hyper-V is the same rock-solid, high performing and scalable hypervisor we use to power Windows Azure too.

Read that again.

That’s right. Windows Azure is powered by Windows Server 2012 Hyper-V. See you at TechEd.

P.S. Hope to see you at the Hands on Lab at TechEd!

More pictures below…

Here’s a close up of one of the racks. This rack has one of the c7000 chassis with 16 nodes for Hyper-V. It also includes the two managements heads clustered used for System Center. At the bottom of the rack are the Uninterruptible Power Supplies.

…and here’s the back of one of the racks that held a c7000…

HP knew there was going to be a lot of interest, so they created full size cardboard replicas diagraming the hardware in use.

…and here’s one more…

John Marlin
Senior Support Escalation Engineer
Microsoft Enterprise Platforms Support

Today’s blog is a listing of our favorite videos and presentations that recently occurred at TechEd 2013.  I have gone through them and picked what we think are the most important to the technologies our team supports.

1. Deployment/Client
2. Windows Server 2012
3. Windows Server 2012 R2
4. Azure
5. For Fun

For a full list of the presentations take a look at http://channel9.msdn.com/Events/TechEd/NorthAmerica/2013

Deployment/Client

· Building the Perfect Windows 8 Image:  http://channel9.msdn.com/Events/TechEd/NorthAmerica/2013/WCA-B305

· Replacing BIOS with UEFI Deployment:  http://channel9.msdn.com/Events/TechEd/NorthAmerica/2013/WCA-B337

· Advanced Microsoft Deployment Toolkit 2012 Customizations: Dueling MDT Enhancements:  http://channel9.msdn.com/Events/TechEd/NorthAmerica/2013/WCA-B302

· Windows in the Enterprise: http://channel9.msdn.com/Events/TechEd/NorthAmerica/2013/FDN07

· What’s new with Microsoft Deployment Toolkit 2012 Update 1:  http://channel9.msdn.com/Events/TechEd/NorthAmerica/2013/WCA-B347

· Windows Store Apps: Enterprise LOB App Deployment Scenarios:  http://channel9.msdn.com/Events/TechEd/NorthAmerica/2013/WCA-B358

· Everything You Ever Wanted to Know about Windows, But Were Afraid to Ask:  http://channel9.msdn.com/Events/TechEd/NorthAmerica/2013/WCA-B201

· Windows RT in the Enterprise:  http://channel9.msdn.com/Events/TechEd/NorthAmerica/2013/WCA-B357

· What's New in Windows 8 Deployment:  http://channel9.msdn.com/Events/TechEd/NorthAmerica/2013/WCA-B349

· Windows 8 Deployment Using Microsoft Deployment Toolkit 2012 Update 1 and Microsoft System Center..:  http://channel9.msdn.com/Events/TechEd/NorthAmerica/2013/WCA-H209

· Microsoft Diagnostics and Recovery Toolset (DaRT):  http://channel9.msdn.com/Events/TechEd/NorthAmerica/2013/WCA-H206

Windows Server 2012

· Overview of Windows Server 2012 Hyper-V:  http://channel9.msdn.com/Events/TechEd/NorthAmerica/2013/MDC-B338

· Cluster in a Box in 2013: How Real Customers Are Making Their Businesses Highly-Available with Windows Server 2012:  http://channel9.msdn.com/Events/TechEd/NorthAmerica/2013/MDC-B336

· Windows Server 2012 Hyper-V Storage Performance:  http://channel9.msdn.com/Events/TechEd/NorthAmerica/2013/MDC-B345

· The Top Ten New Features You Really Care about in Windows Server 2012:  http://channel9.msdn.com/Events/TechEd/NorthAmerica/2013/MDC-B207

· Failover Cluster Networking Essentials:  http://channel9.msdn.com/Events/TechEd/NorthAmerica/2013/MDC-B337

· Windows Server 2012 Deployment and Ongoing Management: Why Server Core Is Right for You:  http://channel9.msdn.com/Events/TechEd/NorthAmerica/2013/MDC-B339

· Windows Server 2012 Failover Cluster Disaster Recovery: Everything You Need to Know to Prepare and Successfully Recover:  http://channel9.msdn.com/Events/TechEd/NorthAmerica/2013/MDC-B355

· Practical Implementation of Windows Server 2012 Storage Technologies:  http://channel9.msdn.com/Events/TechEd/NorthAmerica/2013/MDC-B391

Windows Server 2012 R2

· Introduction to Windows Server 2012 R2:  http://channel9.msdn.com/Events/TechEd/NorthAmerica/2013/MDC-B205

· Windows Server 2012 R2 Clustering:  http://channel9.msdn.com/Events/TechEd/NorthAmerica/2013/C9-22

· Deploying Windows Server 2012 R2 File Services for Exceptional $/IOPS:  http://channel9.msdn.com/Events/TechEd/NorthAmerica/2013/MDC-B217

· Continuous Availability: Deploying and Managing Clusters Using Windows Server 2012 R2:  http://channel9.msdn.com/Events/TechEd/NorthAmerica/2013/MDC-B305

· Storage and Availability Improvements in Windows Server 2012 R2:  http://channel9.msdn.com/Events/TechEd/NorthAmerica/2013/MDC-B333

· Hyper-V – What’s New in Windows Server 2012 R2:  http://channel9.msdn.com/Events/TechEd/NorthAmerica/2013/MDC-B330

· Storage Spaces: What’s New in Windows Server 2012 R2:  http://channel9.msdn.com/Events/TechEd/NorthAmerica/2013/MDC-B218

· Deep Dive on Hyper-V Network Virtualization in Windows Server 2012 R2:  http://channel9.msdn.com/Events/TechEd/NorthAmerica/2013/MDC-B380

Azure

· Crash Course on Automating Deployments in Windows Azure Virtual Machines. How and Which Tools?:  http://channel9.msdn.com/Events/TechEd/NorthAmerica/2013/MDC-B405

· Take Control of the Cloud with the Windows Azure PowerShell Cmdlets:  http://channel9.msdn.com/Events/TechEd/NorthAmerica/2013/WAD-B305

· Windows Azure Internals:  http://channel9.msdn.com/Events/TechEd/NorthAmerica/2013/WAD-B402

· Getting the Most out of Windows Azure Storage:  http://channel9.msdn.com/Events/TechEd/NorthAmerica/2013/WAD-B406

· Infrastructure Services on Windows Azure: Virtual Machines and Virtual Networks with Mark Russinovich:  http://channel9.msdn.com/Events/TechEd/NorthAmerica/2013/MDC-B212

· Best Practices from Real Customers: Deploying to Windows Azure Infrastructure Services (IaaS):  http://channel9.msdn.com/Events/TechEd/NorthAmerica/2013/MDC-B361

For FunJ

· Sysinternals Primer: TechEd 2013 Edition:  http://channel9.msdn.com/Events/TechEd/NorthAmerica/2013/ATC-B313

· Case of the Unexplained 2013: Windows Troubleshooting with Mark Russinovich:  http://channel9.msdn.com/Events/TechEd/NorthAmerica/2013/WCA-B306

Scott McArthur
Senior Support Escalation Engineer
Microsoft Support

One question I get asked a lot working with our customers is how to failover the ‘Cluster Group’ and ‘Available Storage’ groups?

Under the covers, those two groups are still just groups like any other you’d see under ‘Services and Applications’ in Failover Cluster Manager. We just didn’t expose through the GUI, a way to fail those groups between nodes when you want to move say ‘Available Storage’ to a node you are working on so you can copy data to a cluster shared disk.

The resources in the ‘Cluster Group’ are located in the middle pane in Failover Cluster Manager when you highlight the name of the cluster underneath ‘Failover Cluster Manager’ in the left-hand pane.

Note in the above screenshot, in the top ‘Summary of Cluster’ box, the ‘Current Host Server: shows as JH-W2K8-R2SP1-2. That’s the node that currently owns the ‘Cluster Group’ or ‘Core Cluster Resources.

The below screenshot shows the ‘Available Storage’ group. This is a group containing cluster shared disks that have not yet been added to a ‘Services and applications’ group.

Note also that the current owner of the ‘Available Storage’ group shows as JH-W2K8-R2SP1-2.

So, there are two ways we can move either the ‘Cluster Group’ or the ‘Available Storage’ groups between nodes of the cluster.

The first way is with the cluster PowerShell cmdlet Move-ClusterGroup

Move-ClusterGroup “Cluster Group”

or

Move-ClusterGroup “Available Storage”

The second way is a cluster.exe command line

cluster group “Cluster Group” /move

or

cluster group “Available Storage” /move

If you have more than two nodes in the cluster, you can specify the destination node with the following syntax:

PowerShell:

move-ClusterGroup –node NodeName

where NodeName is where you want to move the group.

Command Line:

cluster group “Cluster Group” /Move:NodeName

where NodeName is where you want to move the group.

Jeff Hughes
Senior Support Escalation Engineer
Microsoft Enterprise Platforms Support

Several months ago, I posted a blog on adding a new disk to an existing cluster. Another question we get asked a lot is “How do I replace a disk?”

In this blog, I’ll walkthrough the process of replacing a 1GB disk with a 2GB disk. This process is similar to how you could go about doing a SAN migration where you are replacing all of your shared disks with storage from a new SAN.

The preferred way of getting a larger cluster disk is to use the built in capability of most SANs to dynamically expand a LUN then use an OS utility like DiskPart or Disk Manager to extend the size of the disk. If that’s not feasible or you simply want to replace a LUN with a larger one or as I mentioned, as part of a SAN migration, this process works well.

The first thing we need to do is present your new disk to the cluster. The nuts and bolts of how to do that are outside the scope of this post so just ask your SAN administrator for a new LUN and present it to all nodes of the cluster. Since by default in Server 2008, we leave new LUNs offline, there’s no risk in presenting a new LUN to all nodes at the same time. In the below figure is what Disk Manager would look like after my new disk had been presented.

Figure 1.

Note how the new disk ‘Disk 9’ is in an ‘Offline’ state. In order to prepare it to be the replacement disk for an existing disk, we need to do the following.

• Online the disk
• Initialize the disk (MBR or GPT)
• Create a new volume
• Format as either FAT32 or NTFS

Note: You do NOT need to assign the new drive a drive letter during the format process.

Figure 2.

Figure 2 now shows ‘Disk 9’ as Online and formatted with an NTFS partition. At this point, we can now go into Failover Cluster Manager to complete the rest of the replacement.

The screenshot below shows a File Server group with ‘Cluster Disk X:’ of size 1GB. This is the disk that I am going to replace with the new 2GB disk from above.

Figure 3.

Failover Cluster Manager has a built in ‘repair’ functionality that allows replacing a failed disk with a new disk. Since we’re not really replacing a failed disk but a working one, we need to put that disk into an 'offline' state so that the ‘repair’ function will be enabled.

Figure 4.

Now right-click the disk resource, ‘More actions…’, ‘Repair’. This will launch the ‘Repair a Disk Resource’ window.

Figure 5.

Figure 5 shows the disk that we presented and created in Figure 2. Select that disk, click [OK]/

Now bring the resource online. You’ll see in Figure 8. that the disk now shows as 2GB. We essentially swapped one disk for another without having to worry about resource dependencies. If the drive letter needs to be changed to match the old drive letter, do so now.

Figure 6.

So now that we’ve replaced the 1GB disk with the 2GB disk, what happened to the old disk? When you used the ‘Repair’ function, the old disk got removed from under the control of the cluster. The final step in the replacement is to bring the old disk back into the cluster so that we can bring it online and move the data from the old disk to the new.

To add the disk back in, from Failover Cluster Manager, go to the ‘Storage’ group. In the right-most column, in the ‘Actions’ pane, click on ‘Add a disk’

Figure 7.

Figure 8 shows the disk we just removed from the cluster. Select this disk, click [OK]

Figure 8.

This disk now shows up in ‘Available Storage’. Figure 12.

Figure 9..

The final steps in the replacement are to assign this disk a drive letter so that it’s exposed to the OS to get your data moved from the old disk to the new.

Figure 10.

Now that ‘Cluster Disk 7’ (the old disk) shows as online and has a drive letter (D:) , you can use your favorite data copy method to move the data from the old disk to the new disk. If you are no longer going to use the old LUN, you can simply delete this resource from Failover Cluster Manager and unpresent that LUN from all nodes of the cluster. That finishes up the clean-up process. You can also just leave the disk in ‘Available Storage’, format it, and have it ready for some other ‘Service or application’ cluster group to use in the future.

Hope you find this blog useful especially for those SAN migrations.

Jeff Hughes
Senior Support Escalation Engineer
Microsoft Enterprise Platforms Support

Greetings Cluster fans,

John Marlin back for another go at it.  I wanted to write something about what we have been seeing involving the use of Cluster Shared Volumes (CSV) and File Server resources on the same Cluster.

There have been multiple instances we have seen regarding a Stop 0x00000050 on Cluster Servers that point to CSVFILTER.SYS as being the culprit. 

CSVFILTER.SYS is a filter driver used by Failover Clustering to filter metadata I/O writes to a Cluster Shared Volume.  If there is a metadata write from a node that“owns” or is the coordinator node, it allows the direct I/O write.  If the node is not the coordinator, CSVFILTER.SYS redirects the I/O over the network to the node that is the coordinator.

Since it is a filter driver, it will attach itself to all drives in the Cluster.  The stop error occurs because CSVFILTER sees SMB I/O that it does not want to see.

These are three different scenarios where you can get a Stop 0x00000050 error.

1.       A Cluster that has File Server resources only (no Hyper-V VMs) with Cluster Shared Volumes enabled.

2.       File Server resources or shares that are located on the Cluster Shared Volumes.

3.       A Cluster that has both Hyper-V VMs on Cluster Shared Volumes and File Server resources on non-CSV drives.

Scenario 1

==========

A Cluster that has File Server resources only (no Hyper-V VMs) with Cluster Shared Volumes enabled. 

When you are in this scenario, there is no need for Cluster Shared Volumes to be enabled.  To resolve this, you should disable CSV so that CSVFILTER.SYS is no longer in play. 

To do this, run Powershell from the Administrative Tools with this command:

get-cluster | %{$_.EnableSharedVolumes="Disabled"}

This will disable Cluster Shared Volumes and you will no longer receive the stop errors.  In this type of configuration, there is no need for the enabling of Cluster Shared Volumes as they are not being used anyway.

Scenario 2

==========

File Server resources or shares that are located on the Cluster Shared Volumes.

When you enable Cluster Shared volumes, you will receive this dialog box:

As it states, you do not want any kind of user or application data on these volumes.  Key point in the box above is “may result in unpredictable behavior, including data corruption or data loss” and we all know that data integrity needs to be there.

So if you are keeping user or application data on a CSV drive, get it off or bad things can happen. This is not a valid or supported configuration.

Scenario 3

==========

A Cluster that has both Hyper-V VMs on Cluster Shared Volumes and File Server resources on non-CSV drives.

In this configuration, you have all the highly available virtual machines on CSV drives and separate groups for File Servers on non-CSV drives.  As mentioned at the beginning of this, CSVFILTER.SYS is attaching itself to all drives, including these non-CSV drives.  This is where you would need the workaround and there are two options to consider.

The first is to create a virtual machine that is the File Server resource and shares.  Add this VM into the Cluster on the drive that you can convert to Cluster Shared Volume.  This one would take some work and a little bit of time to do.

The second option is to detach CSVFILTER.SYS from the non-CSV drives.  This one is the easiest and quickest to do, but it is a little kludgy. For example, say your non-CSV was the Z: drive.  To detach it, the command would be:

Fltmc detach csvfilter z:

This would remove CSVFILTER.SYS as a filter on the drive.  The caveat to this is that if you restart the Cluster Service, reboot the machine, or simply move the group to another node, CSVFILTER.SYS may attach itself again. 

To get around this, you would want to create a batch file with the above command and place it on the Z: drive.  You would need to create a Generic Application resource with this batch file. You would then want to have the File Server resources depend on this Generic Application Resource and the Generic Application resource depend on the Drive Z: resource.  This way, no matter what happens, the disk comes online, CSVFILTER is told to attach, the File Server resources do what they do.

No more stop errors. Is it kludgy?  Yes.  Does it do the job?  Yes.

Microsoft is looking into this further.  There are no guarantees that a fix will be created at this point.  For now, we must utilize the workarounds mentioned above.

Happy Clustering !!

John Marlin

Senior Support Escalation Engineer

Microsoft Enterprise Platforms Support

There is an issue with Cluster Shared Volumes and McAfee VirusScan Enterprise that I wanted to pass along.  When installing McAfee VSE 8.7 Patch 5 or 8.8 Patch 1, the CSV drives will go into redirected mode and will not go out of it.

The reason for this is that the McAfee filter driver (mfehidk.sys) is using decimal points in the altitude to help in identifying upgrade scenarios for their product.  The Cluster CSV filter only accepts whole numbers and puts the drives in redirected access mode when it sees this decimal value.

When seeing this, if you run FLTMC from an administrative command prompt, you may see something similar too:

C:\> fltmc

Filter Name    Num Instances      Altitude    Frame
------------------------------------------------------
CSVFilter            2            404900        0
mfehidk                           329998.99   <Legacy>
mfehidk              2            321300.00     0

If you were to generate a Cluster Log, you would see the below identifying that it cannot read the altitude value properly.

INFO [DCM] FsFilterCanUseDirectIO is called for \\?\Volume{188c44f1-9cd0-11df-926b-a4ca2baf36ff}\
ERR  mscs::FilterSnooper::CanUseDirectIO: BadFormat(5917)' because of 'non-digit found'
INFO [DCM] PostOnline. CanUseDirectIO for C2V1 => false

McAfee has released the following document giving a temporary workaround.

Cluster Shared Volumes (CSV) status becomes Online (Redirected access)
https://kc.mcafee.com/corporate/index?page=content&id=KB73596

Microsoft is aware of the problem and currently working on a fix.  When this fix is available, this will be updated and a new KB Article will be created with the fix.

John Marlin
Senior Support Escalation Engineer
Microsoft Enterprise Platforms Support

My name is Sean Dwyer and I am a Support Escalation Engineer with the Microsoft CORE team.

I’d like to share a quick tip for handling Windows Server Cluster administrators.

There may come a time, for whatever reason, that a Cluster managed volume is flagged as dirty and you will see an event ID message indicating that CHKDSK needs to run against the volume.  Just for a little background, the NTFS File System is monitoring the drive/partition at all times.  If it detects corruption, it will flip a bit on the volume and mark it as dirty.  During the online process of a Clustered drive, it will check for the existance of this bit and spawn CHKDSK if it sees it.  You can check, at any time, to see if a volume it is dirty with the CHKNTFS command.

C:\> chkntfs z:

The type of the file system is NTFS.

Z: is not dirty.

C:\> chkntfs z:

The type of the file system is NTFS.

Z: is dirty.

In a best case scenario, you can take the volume out of production, run CHKDSK on the volume if needed (refer to: http://technet.microsoft.com/en-us/library/cc772587.aspx, and then put the volume back into production.

In most situations though, the volume that needs attention is a heavily utilized production volume and will be extremely disruptive to have the volume offline for any length of time.

For example, a recent case I was involved with had a 14Tb* (see note 1 below) volume that was being flagged for CHKDSK to run on it about once a month. The volume had about 9tb of data on it. Apart from the concern of why the volume was continually being flagged as corrupt, the length of time that CHKDSK took to run on the volume was extremely painful for the customer’s business. When it ran initially, it took roughly 80 hours to complete a run on the volume.

It may be necessary to temporarily configure a problem volume to block CHKDSK from running against it while troubleshooting continues to determine why the volume is being flagged for CHKDSK to run.

I stress the word temporary here.

Turning off the health monitoring tool for the file system as a permanent solution could only lead to more downtime in the future.  You may end up on the phone with one of the File Systems experts on my team, such as Robert Mitchell.

Ok – so let’s talk specifics about temporarily blocking CHKDSK from doing work on a Cluster volume.

Say we have determined that we need to suspend CHKDSK from running on a problem volume. For you old school Cluster admins, the first command parameter that probably jumps to mind is SKIPCHKDSK.

This works just fine for Windows 2003 Server Clusters, but will NOT work for Windows 2008 and 2008R2 Failover Clusters.

If SKIPCHKDSK is used for a Clustered volume, it will be ignored when the disk is next brought online and CHKDSK will be run. In a situation where the volume is 18tb, the volume will remain unavailable for use until CHKDSK finishes* (See note 2 below).

The correct way to configure a volume to block CHKDSK from running on it, is to use the DiskRunChkdsk parameter.  Keep in mind that these two parameters we are discussing only apply to the Cluster environment.  If the machine is restarted, the OS may prompt for CHKDSK to run on the affected volumes.

For information on how to configure the OS to ignore the dirty bit, refer to:

KB158675

How to Cancel CHKDSK After It Has Been Scheduled

http://support.microsoft.com/default.aspx?scid=kb;EN-US;158675

Before walking through an example of setting the DiskRunChkdsk parameter, I first must expain what the values mean.  In Windows 2003 Server Clusters, the SKIPCHKDSK parameter was either 0x0 (disabled) or 0x1 (enabled).  In Windows 2008 and 2008R2 Failover Clusters, there are different settings and what it is checking varies.

DiskRunChkDsk <0x0>: This is the default setting for all Failover Clusters. This policy will check the volume to see if the dirty bit is set and it will perform a Normal check of the file system. The Normal check is similar to running the DIR command at the root. If the dirty bit is set or if the Normal check returns a STATUS_FILE_CORRUPT_ERROR or STATUS_DISK_CORRUPT_ERROR, CHKDSK with be started in Verbose mode (Chkdsk /x /f).

DiskRunChkDsk <0x1>: This setting will check the volume to see if the dirty bit is set and it will perform a Verbose check. A verbose check will scan the volume by traversing from the volume root and checking all the files) of the file system. If the dirty bit is set or if the Verbose check returns a STATUS_FILE_CORRUPT_ERROR, CHKDSK with be started in normal mode (Chkdsk /x /f).

DiskRunChkDsk <0x2>: This setting will run CHKDSK in Verbose mode (Chkdsk /x /f) on the volume every time it is mounted.

DiskRunChkDsk <0x3>: This setting will check the volume to see if the dirty bit is set and it will perform a Normal check of the file system. The Normal check is similar to running the DIR command at the root. If the dirty bit is set or if the Normal check returns a STATUS_DISK_CORRUPT_ERROR, CHKDSK will be started in Verbose mode (Chkdsk /x /f), otherwise CHKDSK will be started in read only mode (Chkdsk without any switches).

DiskRunChkDsk <0x4>: This setting doesn’t perform any checks at all.

DiskRunChkDsk <0x5>: This setting will check the volume to see if the dirty bit is set and it will perform a Verbose check (scan the volume by traversing from the volume root and checking all the files) of the file system. If a problem is found, CHKDSK will not be started and the volume will not be brought online.

So now that we know what the varies switches do, to have CHKDSK never run during an online operation of the disk, we want to set DiskRunChkdsk to 0x4.

Here are the steps you can run through to accomplish this task.

Step 1: Determine the resource name as seen by Cluster

Step 2: Open either an Administrative command prompt or Windows Powershell Modules and run the command:

C:\> cluster res "Cluster Disk 8" /priv DiskRunChkdsk=4

or

PS C:\> Get-ClusterResource "Cluster Disk 8" | Set-ClusterParameter DiskRunChkdsk 4

Note: For the setting to WORK, the disk must be brought offline and back online.  Otherwise, it is simply stored until the next time it is taken offline and back online.

Step 4: Bring the disk offline, then online again.

Step 5: Verify the setting is applied

or

PS C:\> Get-ClusterResource "Cluster Disk 8" | Get-ClusterParameter DiskRunChkdsk

Object            Name             Value

------            ----             -----

Cluster Disk 8    DiskRunChkDsk    4 

Step 6: Actively start troubleshooting what could cause the volume to end up flagged dirty and needing CHKDSK.

Footnotes:

Note 1: It’s not suggested to run with volumes this large. In my experience once they exceed 2tb in size, they rapidly become an administrative liability, especially in a situation where CHKDSK has to run against the volume. We strongly suggest that mount points be used to carve up larger volumes like this, into more administratively friendly chunks. CHKDSK runs against mount points just fine, too.

Note 2: While it’s not recommended to interrupt CHKDSK while it’s running, an admin is not locked into having to let CHKDSK finish once it starts. The process can be terminated if absolutely required. However, we cannot guarantee that the end result will be positive. If the process is interrupted during the “magic moment” when CHKDSK is making changes, the results may be worse than the initial reason for the volume being flagged as corrupt.

Additional reading material related to the components and tools mentioned in this post:

KB947021

How to configure volume mount points on a server cluster in Windows Server 2008

http://support.microsoft.com/default.aspx?scid=kb;EN-US;947021

The shared disk on Windows Server 2008 cluster fails to come online

http://support.microsoft.com/default.aspx?scid=kb;en-US;2517696

FSUTIL utility; marking a volume dirty for testing

http://technet.microsoft.com/en-us/library/bb490641.aspx

In summary; try to keep your production volumes’ size under control, be aware that command line switches may not persist through all versions of a product, and continue being successful with Windows Server 2008!

I hope this post has been helpful!

Sean Dwyer

Support Escalation Engineer

Windows CORE Team

A fix for the above titled problem has been released.  If you are running into this problem, please downlaod and install the following fix on all Clusters running McAfee and wanting the updates they provide.

2674551
Redirected mode is enabled unexpectedly in a Cluster Shared Volume when you are running a third-party application in a Windows Server 2008 R2-based cluster
http://support.microsoft.com/default.aspx?scid=kb;EN-US;2674551

===============

Below is the information from the original post:

There is an issue with Cluster Shared Volumes and McAfee VirusScan Enterprise that I wanted to pass along. When installing McAfee VSE 8.7 Patch 5 or 8.8 Patch 1, the CSV drives will go into redirected mode and will not go out of it.

The reason for this is that the McAfee filter driver (mfehidk.sys) is using decimal points in the altitude to help in identifying upgrade scenarios for their product. The Cluster CSV filter only accepts whole numbers and puts the drives in redirected access mode when it sees this decimal value.

When seeing this, if you run FLTMC from an administrative command prompt, you may see something similar too:

C:\> fltmc

Filter Name   Num Instances   Altitude   Frame
------------------------------------------------------
CSVFilter          2          404900       0
mfehidk                       329998.99  <Legacy>
mfehidk            2          321300.00    0

If you were to generate a Cluster Log, you would see the below identifying that it cannot read the altitude value properly.

INFO [DCM] FsFilterCanUseDirectIO is called for \\?\Volume{188c44f1-9cd0-11df-926b-a4ca2baf36ff}\
ERR mscs::FilterSnooper::CanUseDirectIO: BadFormat(5917)' because of 'non-digit found'
INFO [DCM] PostOnline. CanUseDirectIO for C2V1 => false

McAfee has released the following document giving a temporary workaround.

Cluster Shared Volumes (CSV) status becomes Online (Redirected access)
https://kc.mcafee.com/corporate/index?page=content&id=KB73596

Microsoft is aware of the problem and currently working on a fix. When this fix is available, this will be updated and a new KB Article will be created with the fix.

John Marlin
Senior Support Escalation Engineer
Microsoft Enterprise Platforms Support

Hello, my name is Steven Graves and I am a Support Escalation Engineer (SEE) in the Platforms Support group here at Microsoft. One of the technologies I support is Windows Server Failover Clustering. I’d like to take a couple of minutes to share some information on an issue I previously worked on. The customer wanted to create an Exchange 2010 DAG, which would be the first Windows Server 2008 R2 cluster in their environment and they were having issues bringing the CNO online after the cluster was created. The customer domain was originally 2003 and they had to add a 2008 R2 DC and update the schema in order to install Exchange 2010 DAG.

For starters, since I knew the CNO was not coming online after creating the cluster I had the customer destroy the cluster, pre-staged a new computer object for the CNO then created a new cluster based on the name of the new CNO. After the cluster was created I noticed that the computer object was still disabled in AD and the following error message in the cluster log.

00000e80.00000d5c::2012/03/14-16:07:33.149 INFO [RES] Network Name <Cluster Name>: Trying to find computer account W2K8R2Cluster object GUID(cae8b3dcc60aa040bbcef250634427bb) on any available domain controller.
00000e80.00000d5c::2012/03/14-16:07:33.306 WARN [RES] Network Name <Cluster Name>: Search for existing computer account failed. status 8007052E
00000e80.00000d5c::2012/03/14-16:07:33.352 WARN [RES] Network Name <Cluster Name>: Couldn't get information from DC \\Info-dc3.infoimage.com. status 5
00000e80.00000d5c::2012/03/14-16:07:33.352 INFO [RES] Network Name <Cluster Name>: Trying to find object cae8b3dcc60aa040bbcef250634427bb on a PDC.
00000e80.00000d5c::2012/03/14-16:07:33.462 WARN [RES] Network Name <Cluster Name>: Couldn't get information about PDC. status 5
00000e80.00000d5c::2012/03/14-16:07:33.462 INFO [RES] Network Name <Cluster Name>: Unable to find object cae8b3dcc60aa040bbcef250634427bb on a PDC.
00000e80.00000d5c::2012/03/14-16:07:33.462 INFO [RES] Network Name <Cluster Name>: GetComputerObjectViaGUIDEx() failed, Status 8007052E.

‘Access is denied’

In the System Event log you will see an ID 1207 that should be in synch with the time in the cluster log. The main thing to focus on is the “Unable to get the Computer Object using GUID”.

Log Name: System
Source: Microsoft-Windows-FailoverClustering
Date: 3/14/2012 9:07:10 AM
Event ID: 1207
Task Category: Network Name Resource
Level: Error
Keywords:
User: SYSTEM
Computer: W2K8R2Cluster.Corp.com
Description:
Cluster network name resource 'Cluster Name' cannot be brought online. The computer object associated with the resource could not be updated in domain 'infoimage.com' for the following reason:
Unable to get Computer Object using GUID.
The text for the associated error code is: Logon failure: unknown user name or bad password.

At this point, I’m pretty convinced there are some issues with the GPOs on the domain controllers but I still need to do my due diligence in troubleshooting the issue with the Cluster Network Name in a failed state.

Since I pre-staged the CNO and it was still disabled after creating a new cluster, this gave me more evidence indicating an issue with the DC. I created a new OU and blocked inheritance in order to prevent any GPOs from being applied to the Node(s). I refreshed the GPO’s on the Node(s), confirmed there are no GPOs applied by running Gpresult /V from an Administrative CMD Prompt, but the Cluster Network Name still fails to come online. I’m convinced there is some issue with GPO’s on the DC but I’m not sure where to start looking.

Next, I verified the permissions in AD on the CNO and, to be on the safe, I granted the CNO Full Control to the object and also confirmed that the CNO has the correct permissions to the OU(READ permissions on the OU should be sufficient rights to access the OU and get to the computer object). Despite this, the Cluster Network Name failed to come online.

I moved on to check the DNS Host A record for the CNO not really thinking this is the issue but more or less making sure everything is in order. I came to find out a Host A record was not created for the Cluster Network Name because they do not have Dynamic Updates enable for DNS. I created the Host A record and checked off “Allow any authenticated user to update the DNS records with the same owner name.” I already knew the node was able to resolve the DC from the warnings in the cluster log but couldn’t get information from DC \\W2K8R2-DC.Corp.com. So it was not a name resolution issue trying to access the DC.

At this point, I have gone through all the normal troubleshooting steps that generally resolve the ID 1207 and the CNO in a failed state from the cluster perspective. Now it’s time to engage Directory Services to take a deeper look at the DC configuration. After some time reviewing the Domain Controller configuration and GPOs the DS engineer narrowed it down to permission issues in the “Access this computer from the network” policy. The default permissions are pictured below.

Access this computer from the network - This user right determines which users and groups are allowed to connect to the computer over the network. Since "Everyone" and "Authenticated Users" were missing from the settings, this meant that no computer would be able to access the domain controller.

Picture above shows the default permissions for the Access this computer from the network policy

The DS engineer modified the “Access this computer from the network” policy in the Default Domain Controllers policy by adding Authenticated Users, refreshed GPOs by running GPUpdate /force, ran RSOP.msc to confirm the GPO is applied, and the CNO came online.

Steven Graves
Support Escalation Engineer
Microsoft Enterprise Platforms Support