Search results for 'app:weblogs' matching tag 'Clustering'

Adjusting Exchange 2010 DAG Failover in High Latency Networks

Anonymous — Tue, 29 Nov 2011 06:00:00 GMT

Some Exchange 2010 DAG implementations have DAG members that are separated by high latency WAN networks. In these networks you may find that the increased latency causes unexpected DAG failovers or failures.

This is especially likely to happen with a two node DAG with a File Share Witness (FSW). When network latency increases to the point that the cluster heartbeat threshold is reached, the node furthest from the FSW will go offline. The node that's in the same LAN as the FSW will stay online because it maintains quorum.

There are two properties that specify cluster health, as measured in heartbeats.

CrossSubnetDelay specifies the heartbeat interval (in milliseconds) between subnets. The default is 1000 milliseconds (1 second).
CrossSubnetThreshold specifies how many heartbeats can be missed between subnets before cluster failover (or failure) occurs. The default is 5 heartbeats.

With the default settings, any WAN latency that causes 5 missed heartbeats over 5 seconds causes the cluster to fail. This matches the settings used by the SameSubnetDelay and SameSubnetThreshold properties.

If WAN latency causes unexpected cluster failover or failures, adjust the CrossSubnetDelay value to its maximum value of 4000 milliseconds (4 seconds) and the CrossSubnetThreshold property to its maximum value of 10, With these settings the cluster will not failover or fail until there are 10 missed heartbeats, 4 seconds apart (40 seconds).

This is accomplished from Powershell by doing the following:

From one of the DAG members open the Windows Powershell Modules in Administrative Tools. This will launch Powershell and import all the Windows Powershell modules for installed features, including the new Windows Failover Cluster module.
Run the following one-liner to configure the maximum values:

$cluster = Get-Cluster; $cluster.CrossSubnetThreshold = 10; $cluster.CrossSubnetDelay = 4000

Check your settings using the following cmdlet:

Get-Cluster | fl *

Since cluster properties are instantly replicated between all nodes in the cluster, this only needs to be configured from one node in the DAG. The changes go into effect immediately and there is no need to restart services or the server.

Note that you can configure the same properties using cluster.exe, but I'm using Powershell here because cluster.exe is deprecated in Windows Server 2008 R2.

Exchange DAG Cluster Service Terminated with Error 7024

Anonymous — Tue, 25 Oct 2011 05:00:00 GMT

I ran into an interesting issue at a client site yesterday on an Exchange 2010 SP1 DAG member. One DAG member's databases would not mount (even Public Folders) and the Cluster service would not start. The DAG is configured as a three member stretched DAG, with two nodes in the main site and another in the DR site.

The event IDs logged were:

Log Name:      System
Source:        Service Control Manager
Date:          10/23/2011 12:07:44 AM
Event ID:      7024
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      EX03.domain.local
Description:
The Cluster Service service terminated with service-specific error Log service encountered an invalid log block..

Log Name:      System
Source:        Microsoft-Windows-FailoverClustering
Date:          10/23/2011 12:00:43 AM
Event ID:      1177
Task Category: Quorum Manager
Level:         Critical
Keywords:
User:          SYSTEM
Computer:      EX03.domain.local
Description:
The Cluster service is shutting down because quorum was lost. This could be due to the loss of network connectivity between some or all nodes in the cluster, or a failover of the witness disk.
Run the Validate a Configuration wizard to check your network configuration. If the condition persists, check for hardware or software errors related to the network adapter. Also check for failures in any other network components to which the node is connected such as hubs, switches, or bridges.

Log Name:      System
Source:        Microsoft-Windows-FailoverClustering
Date:          10/23/2011 12:00:43 AM
Event ID:      1135
Task Category: Node Mgr
Level:         Critical
Keywords:
User:          SYSTEM
Computer:      EX03.domain.local
Description:
Cluster node 'EX02' was removed from the active failover cluster membership. The Cluster service on this node may have stopped. This could also be due to the node having lost communication with other active nodes in the failover cluster. Run the Validate a Configuration wizard to check your network configuration. If the condition persists, check for hardware or software errors related to the network adapters on this node. Also check for failures in any other network components to which the node is connected such as hubs, switches, or bridges.

Log Name:      System
Source:        Microsoft-Windows-FailoverClustering
Date:          10/23/2011 12:00:43 AM
Event ID:      1135
Task Category: Node Mgr
Level:         Critical
Keywords:
User:          SYSTEM
Computer:      EX03.domain.local
Description:
Cluster node 'EX01' was removed from the active failover cluster membership. The Cluster service on this node may have stopped. This could also be due to the node having lost communication with other active nodes in the failover cluster. Run the Validate a Configuration wizard to check your network configuration. If the condition persists, check for hardware or software errors related to the network adapters on this node. Also check for failures in any other network components to which the node is connected such as hubs, switches, or bridges.

The cluster failed when two of the three DAG members (EX01 and EX02) went offline at 12:00:43 AM due to a network failure in the active site. For some reason, this corrupted the CLUSDB.BLF file on the member in the DR site which prevented that node from coming online when the network came back up. CLUSDB.BLF is a CLFS Base Log File used by the cluster service which contains metadata that is used to manage access to the log data.

To correct the problem, navigate to the %WINDIR%\Cluster folder and rename CLUSDB.BLF to CLUSDB.BLF.OLD. Then restart the Exchange server. The cluster service will generate a new CLUSDB.BLF file on restart. The cluster service will be able to start and the databases will mount.

Winner of the contest

Anonymous — Thu, 05 May 2011 05:00:00 GMT

Dear Readers,

I am very please to announce winner of the contest about describing best HA & DR solution. The contest was successful, there are about 5 entries I have get from various sources like linkedin and email, I would like to thank all the participant and readers. The winner is Dave Thomas.

My apologies, I suppose to declare winner on 25th April but could not able to make it.

Describe HA and DR strategy and win

Anonymous — Sun, 06 Mar 2011 06:00:00 GMT

Dear Readers,

I am very please to made an announcement about the contest. As you aware that I have wrote a book on SQL Server 2008 High Availability which was published on 24th January 2008 by Packt Publication. Now you have a chance to win a subscription worth £150 of digital library at Packt Publication, here is the information.

Describe the best HA+DR solution you have designed or worked upon, especially in case of physically dispersed location, and the reason why?

The best answer will win a subscription of digital library worth £150 at Packt Library, so what are you waiting for!!!

The best answer will be judged by me and Satya Shyam K Jayanty whom I admired as Guru.

Who this book is for?

This book is written for the System Administrator, experienced SQL Developers who want to learn about the topic – SQL Server High Availability, Aspiring DBAs.

That means, this book is having a step-by-step instructions, pre-requisite with plenty snapshots to get you through the installation of SQL Server High Availability options like Cluster, Replications ( Snapshot, Transactional, Peer-2-Pee and Merge), Log Shipping and Database Mirroring.

I have tried to include external reference for further study on that particular topic if you wish to read some advance information, I have also include few common issues and how to resolve them for every topic i.e. How to troubleshoot common issues for Clustering, Replication, Log Shipping and Database Mirroring.

Here is the detailed index for every chapter of this book at - http://www.sql-server-citation.com/2011/01/sql-server-2008-high-availability-book.html

Where to buy?

One can purchase my book SQL Server 2008 High Availability from Packt Publication in Paper and eBook format at https://www.packtpub.com/microsoft-sql-server-2008-high-availability/book or one can purchase this from Amazon at http://www.amazon.com/Microsoft-Server-2008-High-Availability/dp/1849681228/ref=sr_1_1?ie=UTF8&s=books&qid=1299390127&sr=8-1

Regards

-- Hemantgiri S. Goswami | www.sql-server-citation.com

SQL Server 2008 High Availability – A Book

Anonymous — Tue, 25 Jan 2011 06:00:00 GMT

I have no words to explain my excitement and happiness to share news with you all - my book on SQL Server 2008 High Availability is published!!

I have started authoring this book in May 2010 and as the time passes and gradually I move to the final chapter of the book, I was feeling like aha finally it is done!!! – Do you think so, it was this easy?

No, every publication house has their own set of rules to be followed while authoring a book and a process to be followed before the chapter or entire book is ready to be published i.e. 1^st edit, 2^nd edit of the chapter until final edit and the Packt Publication is no exception.

Since this is the first time I was authoring book I was having little or no idea about authoring a book. The editors from Packt Publication helped me a lot by helping me time to time with proper answers to my queries. I must admit that I could not be able to complete this book without support and encouragement from my family including my 2 year old daughter Dhruti.

I hope that you all will enjoy reading and learning from this book as I did while authoring and contributed my little part for the growth of my beloved SQL Server J

Let me tell you about this book:

The term High Availability means that the servers or systems that host or run the business-critical applications should be highly available 24 X 7. As the word it-self defines how important it is to make these applications and data available for end-users as well as business users, if this data is not available for a short time, it will be a big problem for both sets of users. Imagine a bank spread across the country and having a huge customer base. One fine day, their server crashes! If the bank relies only on backups, then it might end up losing approximately 15 to 30 minutes of data, depending on the backup strategy. Now the HA options related to SQL Server such as clustering, replication, log shipping, and database mirroring will help overcome this situation.

In this book I have tried to cover every single piece of information that might requires for installing and configuring SQL Server HA option like Clustering, Replication, Log Shipping and Database Mirroring, below are the chapters:

Chapter 1: Understanding Windows Domains, Domain Users, and SQL Server Security
Chapter 2: Implementing Clustering
Chapter 3: Snapshot Replication
Chapter 4: Transactional Replication
Chapter 5: Merge Replication
Chapter 6: Peer-to-Peer Replication
Chapter 7: Log Shipping
Chapter 8: Database Mirroring
Appendix A: Troubleshooting
Appendix B: External References

If you are interested to read more about the detailed / contents of each chapter you may find them here https://www.packtpub.com/toc/microsoft-sql-server-2008-high-availability-table-contents

And, here is the link for the sample chapter https://www.packtpub.com/sites/default/files/1223-chapter-1-understanding-windows-domains-domain.pdf

Regards

Hemantgiri S. Goswami

Server Health Checks

BrianM — Fri, 29 Jan 2010 06:00:00 GMT

I’d like to share some of the things I look at while do a health check on a server. Its funny how few resources there are out there on the Internet. I believe people keep this kind of stuff to them self because they are scared they are going to miss something and they will never live it down. My response to that is, So What! Heck, I don’t claim to know it all but why not share what I do know and maybe others can share via the Comments!!!

When I’m troubleshooting I like to compartmentalize what I'm looking for. With that my health checks are set up the same way. I also believe health checks are quick snapshots of the health of a server. Sure there are tools that you can use to analyze systems further but in this case we are doing a quick health check. Not all of these need to be done but some should, you get to decide.

CPU

Occasional high CPU spikes are ok as long as you are aware of the process causing this. A server should maintain 80% CPU utilization for an extended period of time. If it does it may be time to upgrade. Its a good idea to keep Task Manager open during the duration of your troubleshooting to see trends.

Check CPU Usage

Open Task Manager

Check the Processes tab, ensure there are no processes consuming excessive CPU

Check the Performance tab, ensure there are no single CPU’s that have excessive CPU usage

Check CPU HW

Open Device Manager (right click computer –> Manage)

Ensure that no CPU’s have red X or yellow ! underneath the Processors

Processes

This is one area that you may not want to do for quick health checks but is something you should be familiar with. Task Manager only gives you basic info on processes and you will find that you may need to dig a bit deeper. For that I recommend Process Monitor from the great SysInternal tools. Process Explorer can also be used. In fact download and play with all these tools…they will save your bacon, I guarantee it.

In-Depth Check
SysInternals:

Copy Process Monitor locally, then launch it.

Analyze each process and watch what operations open the reg keys, file etc.

Copy Process Explorer locally, then launch it.

Analyze each process based upon the number of threads, handles, loaded DLL’s, etc.

Two great webcasts can be viewed here to see these types of tools in action.

Memory

General rule of thumb is to make sure the general memory utilization does not exceed 80%within a given period of time.

Check Memory Availability

Open Task Manager
Select the Performance tab
Look at the Physical memory box, and multiply the total memory by .2
If the total available memory is less than this number then the box is currently utilizing more than 80 percent of the memory.

Current utilization by process

Select the Process tab

Check the ‘show processes from all users’ box in the bottom left corner

Click the column header ‘Mem Usage’ to sort the processes by memory utilization, highest to lowest. This will help you determine what processes are currently utilizing the memory on the box and can help you narrow your search for memory intensive processes.

Network

Check NIC HW

Verify both ends of the network cable are securely seated in the port

On the back of the server verify you have a green blinking link light on the NIC port

Verify NIC HW is working properly by using Device Manager and ensure the active NICs are showing green

Verify gateway, IP, subnet mask, DNS, DNS suffixes, etc. are properly configured.

If everything is properly configured and HW is working, you should be able to get a ping response from the gateway.

Check Network Connections
Here are some other checks you should perform to ensure proper network connectivity:

ipconfig /all will display all you TCP/IP settings including you MAC address

ipconfig /flushdns will flush your dns resolver cache

ipconfig/displaydns will display what is in your dns name cache

Netstat -an command will show all the connections & ports from a machine

Nbtstat command will show net bios tcp/ip connection stats

Tracert <IP or DNS Name> command will show you the path the packet takes, the routers, and the response time for each hop.

pathping <IP or DNS Name> command combines ping and tracert to the 100th degree. It pings each hop 100 times and is great for testing wan connectivity

Disk Space

All kinds of bad stuff can happen when your disk space is filling up. The best way to alleviate this is to write a script to notify you when you reach a certain threshold. In a future post I'll share a method for you to do just that…however if there is a problem and you need to perform a health check then here is how you check the space the old fashion way.

To check disk space manually:

Right Click on My Computer

Select Manage

Select Disk Management

Validate each disk more than 10 percent free space

Event Logs

Event logs can reveal a more historical perspective on what is going on with the system and applications. Things to look for when troubleshooting event logs is to query either the system or the application logs and look for the presence of events that have a timestamp near the time of the issue you are troubleshooting.

Events have 3 categories in the event viewer:

Informational: Noted with a white icon and letter ‘i’. Successful operations are logged as informational. Usually not used in troubleshooting problems or failures

Warning: Noted with a yellow icon and exclamation point. These usually are looked up as they serve as predictive future failure indicators, such as disk space running low, dhcp ip address lease renewal failures, etc.

Error: Noted with a red circle icon and ‘x’. These are indications that something has failed outright and are a good starting point for troubleshooting.

When looking at event logs, use the information to determine the following:

Is the incident tied to a particular time or outage incident?

Is this a one-off, or has this particular error occurred multiple times in the past?

Does this error appear on other systems or is it unique to the system that has failed?

Also make sure you take a look at eventcombmt from Microsoft. This tool allows you to search the logs of multiple machines. The benefit to this is to see if a specific error or warning message is also occurring on other systems. This can help rule out issues.

Services

Troubleshooting services should be limited to the specific that is affected by the problem being troubleshot. Each server will have specific services varying upon the types of applications running. You should document how your servers services are configured to and compare that to the server in question to see if anything is not configured correctly.

Cluster

Servers that host applications and services that require high availability should be clustered so that if one node fails the other can pick up the workload. Clustered servers need the same type of health checks as stand-alone systems except you will want to check on the health of the cluster.

Check Cluster Resource Status

Open Cluster Administrator: Log onto server, select Start –> Run –> cluadmin

Check the Resources and ensure all are Online

If Cluster Administrator does not open, ensure that the Cluster Service is running on the node.

Cluster resource status can also be checked from a remote server. From a command prompt, just type - cluster res <cluster name>

Client Side Health

Right click on My Computer, select Manage

Open Device Manage

Drill down to SCSI and RAID Controllers, verify that the HBA HW is visible and does not show any errors

If it does not show up in Device Manager, you may need to re-scan for the HW, re-seat the fiber card, or re-install the driver.

If the HBA is showing healthy in Device Manager, open the tool that you use to view configuration and settings for the fiber card and verify there aren’t any transmit/receive errors on link statistics or counters

Switch Health

Make sure fiber is properly connected to each switch

Make sure switch has no errors

If you’re using zoning verify it is properly configured

Check Fiber and SAN Connectivity

Log onto san appliance and verify that the SAN is in general good health and no major errors are present for the controllers, loops, switches, or ports.

Ensure that the LUNs are presented to the servers in the cluster

NLBS

Some applications will require you to spread the load across multiple servers. Web servers are a very popular choice to network load balance. As with clusters we will need to check the status of the load balancing.

Check NLBS Status CMD Line

From a command prompt on the local system, run ‘wlbs query’. This will give you the convergence status of the local node with the nlbs cluster.

Other useful NLBS commands: wlbs stop (stops nlbs), wlbs start (starts nlbs), wlbs drainstop (drains node)

Check NLBS Configurations

Open up the network properties –> Network Load Balancing, right click & select Properties

On the Cluster Parameters tab, verify that the IP address is configured for the shared NLBS IP and that the subnet mask, domain, and operation mode are configured correct1y.

On the Host Paramters tab, make sure each node of the cluster has a unique host identifier. Also verify the IP and subnet mask are configured for the local values.

Also make sure that your switch has a static ARP entry if using multi-cast NLBS. The entry should be that of the virtual MAC of the cluster. To get the virtual MAC of the cluster, you can run the following command: WLBS IP2MAC <virtual IP address>

Name Resolution

To healthcheck name resolution, open a command prompt and enter the following

nslookup <servername>

Verify that the servername is correctly entered in DNS

If a record does not show up in the DNS query, or maps to a different name, perform a reverse lookup by IP address to see what name is associated with the IP address * nslookup <IP address>

If no name shows up associated with the IP address, log into the domain controller and check the DNS records for this particular name/ip address

From a Domain Controller go to start–>run–>dnsmgmt.msc

Expand the Forward Lookup Zones

Expand the zone for you primary zone that holds the records for the system/s you are troubleshooting

Validate that the record exists. If it does not exist manually enter the record name and IP address by right clicking on this same zone,

Select new host (a)

Enter the name and IP address

Check the box next to Create associated pointer (PTR) record

Click add Host

Additionally log back into the node that you manually entered the record for and ensure that DNS is registering in DNS

Right click on the My Network Places icon on the desktop and select Properties

Double click on the primary adapter

Select properties

Highlight internet protocol (TCP/IP) and select properties

Validate the IP addresses of the DNS servers are correct

Select Advanced

Select DNS tab

Make sure the box is checked next to Register this connection’s address in DNS

As I wrap this up I realize there is so much more that can be done. Each application type of server needs its own set off health checks. For example web servers, terminal servers and database servers. Remember this is just the baseline for each server and that other components can and should be layered on top of it. Again I would love to hear from others so please feel free to add you comments below.

Almost after 1 year, I'm back with a new beginning :)

Nirmal — Thu, 29 Oct 2009 05:00:00 GMT

Recently, wrote an article on Hyper-V Quick Migration.

Increasing the Availability of Virtualised Applications and Services

http://www.simple-talk.com/sysadmin/virtualization/increasing-the-availability-of-virtualised-applications-and-services/

Build an effiective Disaster Recovery solution for SQL Server Reporting Services

Anonymous — Sat, 24 Oct 2009 05:00:00 GMT

As a Systems Administrator/DBA or even as an user have you ever realised "What is your data worth to you?" If so then an important aspect of data recovery solution is a required step, whatever is the application or system you use data access methods including a disaster recovery plan is essential to ensure your data’s security after a catastrophic event. It is incredibly expensive to retrieve data from failed hard drives, and not always possible. You may expect or no doubt that this blog states on...(read more)

Another Gmail Outage

Anonymous — Thu, 24 Sep 2009 05:00:00 GMT

Clustering Validation Check Fails: Failed to prepare storage for testing on node <fqdn> status 87

jeffl — Sun, 13 Sep 2009 05:00:00 GMT

Clustering can be a fun task especially when you are using Hyper-V Server 2008 R2 in the days before it launches. Strange errors can frustrate even the brave at heart.

The "failed to prepare storage for testing on node..." error results from bit lock creating a partition in the first 100 MB of your primary drive and then not setting a drive letter.

Given the command line interface for Hyper-V Server 2008 R2, I had to assign the drive letter by using the command lines.

From the command Prompt:

type diskpart

type select volume 1

type assign

Re-run the cluster validation tool.