BizTalk Server Disaster Recovery

I have started putting together steps required for BizTalk Server Disaster Recovery. I will keep this updated. In case if you need any help and support in Setting up DR you can get in touch with me.

1. Setting up BizTalk Server at DR Site
1. Configure these BizTalk Server run-time servers using the BizTalk Configuration Wizard to join them to the production BizTalk group. When configuring the BizTalk Server run-time servers at the disaster recovery site (including the disaster recovery Enterprise Single Sign-On Master Secret server) select No to the question “Is this the master secret server?” and select “Join” to the question “Do you want to create or join a BizTalk Server group?”
2. After configuring the BizTalk Server run-time disaster recovery servers, create BizTalk host instances on the disaster recovery site that correspond to the production site host instances, but do not start these host instances. For example, if the production site has three Hosts Send, Receive, and Orchestration with instances on server1, server2, and server3, create three corresponding host instances on DRserver1, DRserver2, DRserver3.
3. All BizTalk Server-related Windows services such as the BizTalk Host Instance and Rules Engine Service at the disaster recovery site should be set to “disabled” in the Services Manager to prevent the disaster recovery site from performing any processing.
2. Setting up SQL Server
1. Create a set of SQL Server database instances in the disaster recovery site. To ensure that the disaster recovery SQL Server database instances can provide the same level of performance as the production SQL Server database instances, the disaster recovery SQL Server database instances should be configured with similar hardware and number of physical computers running SQL Server.
2. Configure log shipping for each production SQL Server database instance to apply to a corresponding SQL Server database instance at the disaster recovery site.
3. Make sure drive letter(s) on the production site where the database files are stored match the drive letter(s) at the disaster recovery site where the database files are restored. So if the SQL Server database file group is located on G:\data in production, there must be a G:\data directory on the destination (DR) server or the restore will fail.
4. Create SQL Server security logins for the disaster recovery site that correspond to the production site so that in the event that a failover to the disaster recovery site is required, all required security logins are present on the destination system.
5. Once installation of the disaster recovery SQL Server instances is completed, perform a full backup of master and msdb databases so that a clean system can be restored in the event that a switch to the disaster recovery site fails.
6. Set up log shipping for SQL Server. Refer Appendix 2.
3. BizTalk Application Backup and SQL Server backup
1. Always keep the latest back up of the applications
2. Make sure the SQL backup jobs are running and we have the latest backup
3. Keep the backup of machine.config, registry settings
4. Keep the MSMQ configuration backup
4. Backing up IIS – Use the backup and restore scripts.
5. IIS certificates – install IIS 6.0 Resource Kit.
6. Recovering from a Disaster
1. Restoring the BizTalk Group
1. Stop all the processing(Refer appendix 1)
2. Restore SQL Server and BizTalk applications http://msdn.microsoft.com/en-us/library/cc296741(BTS.10).aspx and http://msdn.microsoft.com/en-us/library/aa546753.aspx
1. Restore SQL Server from Log Shipping – Refer Appendix 3
2. Restore the Master Secret (http://msdn.microsoft.com/en-us/library/cc296749(BTS.10).aspx)
3. Restoring SQL Server Jobs http://msdn.microsoft.com/en-us/library/ee406928.aspx
4. Restore BAM database by using the BAM backup.
5. Resolve all the incomplete BAM activity
3. IIS and certificates restore
1. IIS restore – use IISRestore batch file to restore the IIS.
2. IIsCertDeploy.vbs will be located in %programfiles%\IIS Resources\IIsCertDeploy.  Use below query to restore the certificates.

iiscertdeploy.vbs -e d:\webbackups\certs\site1.pfx
i w3svc/1 -p site1pwd -q On

4. Recovering the Runtime Computers
1. Update Runtime Computers http://msdn.microsoft.com/en-us/library/cc296621(BTS.10).aspx
2. If receive locations and send ports are not affected by the loss of the production site, it may not be necessary to reconfigure the application with disaster recovery-specific locations. http://msdn.microsoft.com/en-us/library/cc296699(BTS.10).aspx
5. Troubleshooting Log Shipping http://msdn.microsoft.com/en-us/library/cc296675(BTS.10).aspx

Appendix 1 – Stopping BizTalk Server Processing

To stop application processing on the source system, ensure that no connections are open between the production BizTalk Server runtime computers and the SQL Server computers that house the BizTalk Server databases. Follow these steps to stop application processing on the production BizTalk Server runtime computers:

1. Stop the IIS so that no new request should come to Service Bus.
2. Disable all receive locations on the BizTalk Server computers in the BizTalk group. Make a note of all receive locations that are disabled so that these receive locations can be re-enabled later. This will stop BizTalk Server from processing incoming messages.
3. Stop all host instances from running on the BizTalk Server computers in the group. This can be done from the BizTalk Server Administration console. Make a note of all the host instances that were stopped so that these host instances can be restarted later.
4. Stop all SQL Server Agent jobs related to BizTalk Server on the SQL Server computers that house BizTalk Server databases.
5. Stop any other BizTalk Server services in Services Manager that may be running on the BizTalk Server computers in the group, for example, the Enterprise Single Sign-On Service and the Rule Engine Update Service. Make a note of the services that are stopped so that they can be restarted later.
6. Close all applications that connect to the SQL Server computers that house BizTalk Server databases. This includes instances of the BizTalk Server Administration console, Visual Studio 2010 and any other installed BizTalk applications.
7. Verify that there is no database activity generated by BizTalk Server. Use SQL Server Enterprise Manager or SQL Server 2005 Management Studio to see what processes are connected to the SQL Server computers that house BizTalk Server databases. This can be done by clicking to expand Management, Current Activity in SQL Server Enterprise Manager or by clicking to expand Management and Activity Monitor in SQL Server 2005 Management Studio. Then click to select Process Info. Alternatively use the system stored procedures sp_who or sp_who2 to identify any open connections to the SQL Server computers that house BizTalk Server databases. If there are any processes connected, locate them and terminate them normally; or as a last resort, right-click each process in the Process Info pane in SQL Server Enterprise Manager or SQL Server 2005 Management Studio and click Kill process to terminate the connection.
8. Additional database processing may be occurring in application databases. If these databases will be restored, ensure that all processing is stopped.

Appendix 2 – Log Shipping

1. Log in as a member of the BizTalk Server Administrators group to perform this procedure. You must have the same version of SQL Server on both the source and destination systems. SQL Server must be installed in the same relative location on both the source and destination systems.
2. The directory for SQL transaction log (.LDF files) on the source system must also exist on the destination system. If this directory is not on the destination system, create the directory with the same name and permissions as on the source system.
3. Open the SQL Server management studio.
4. Connect to appropriate SQL server
5. Run below script on master database –  
1. C\Program Files\Microsoft BizTalk Server 2009\Schema\LogShipping_Destination_Schema.sql
2. C\Program Files\Microsoft BizTalk Server 2010\Schema\LogShipping_Destination_Logic.sql
3. Ad Hoc Distributed Queries on destination system by using below query
SQL

sp_configure 'show advanced options', 1;
RECONFIGURE;
sp_configure 'Ad Hoc Distributed Queries', 1;
RECONFIGURE;
GO
SELECT a.*
FROM OPENROWSET('SQLNCLI', 'Server=Seattle1;Trusted_Connection=yes;',
    'SELECT GroupName, Name, DepartmentID
     FROM AdventureWorks2008R2.HumanResources.Department
     ORDER BY GroupName, Name') AS a;
GO

4. Please make sure that query should execute in first attempt else you need to execute all the steps from step one. Execute below query. For more than one server please repeat this for all the servers.
SQL

exec bts_ConfigureBizTalkLogShipping @nvcDescription = 'LogShippingSolution_ENV',
@nvcMgmtDatabaseName = 'BizTalkMgmtDB',
@nvcMgmtServerName = 'BizTalk Database Server',
@SourceServerName = null, -- null indicates that this destination server restores all databases
@fLinkServers = 1 -- 1 automatically links the server to the management database

5. Verify below jobs are created as in SQL Server Agent jobs. We need to monitor these jobs and these jobs should not fail.
1. BTS Log Shipping Get Backup History
2. BTS Server Log Shipping Restore Databases
3. BTS Log Shipping Restore To Mark
6. Open C:\Program Files (x86)\Microsoft BizTalk Server 2010\Bins32\Schema\Restore on the BizTalk server.
7. Edit SchemaUpdateInfo.xml  replace the name of the Source System and Destination System.
8. Add two more entries for BAM.

<Database Name="BAM Alerts Application DB" oldDBName="BAMAlertsApplication" 
    oldDBServer="SourceServer" newDBName=" BAMAlertsApplication" newDBServer="DestinationServer"/> 
<Database Name="BAM Alerts Instance DB" oldDBName="BAMAlertsNSMain" 
   oldDBServer="SourceServer" newDBName="BAMAlertsNSMain" newDBServer="DestinationServer"/>

9. Since we have more than two databases add other two messageboxes and set isMaster=0
10. SAVE the file and exist.

Appendix 3 – Restoring Database

1. If there is only one server in the destination system, make sure that all of the log backup sets (except for the most recent set) have been restored. For more information, see Viewing the History of Restored Backups. If all the log backup sets have not been restored, and the restore job is not currently running, run the restore job (manually if necessary). If there are outstanding backup sets that can be restored, the job will process them until they are all restored.
2. If there are multiple servers in the destination system, all servers must be restored to the same backup set. You must view the restore history on each server and make sure that the most recent log backup set restored is the same on all servers. If it is not, you must manually run the restore job on each server that needs the most recent log backup set restored.
3. The adm_BackupHistory table is the central history point for the log shipping process for the source system. All backup work performed is recorded to this table.
1. Connect to SQL Server management studio Disable three Log shipping related jobs i.e. “BTS Log Shipping – Get Backup History”, “BTS Server Log Shipping Restore Databases”
4. Start the job and “BTS Log Shipping - Restore To Mark” at step 1. If the job is successful, SQL Server Agent jobs and BizTalk Server databases are restored to the destination system.
5. Navigate to “drive:\Program Files\Microsoft BizTalk Server 2010\Schema\Restore”. Execute cscript UpdateDatabase.vbs SampleUpdateInfo.xml on the BizTalk Server.
6. Run the script mention in Step 5 on each BizTalk Server.