Script to throttle mass vMotions or svMotions

I wrote the shell of this as new datastores were added to environments and we rearranged VMs, and it got a little better each time until I finally made it a full function when we were going to migrate to new storage arrays in a few separate environments.

It’s purpose is to make sure not too many of my storage vMotions are running at once and impacting other VMs.  It sits in the middle of your script or one-liner pipeline.  It looks at the active tasks and counts how many storage or host vMotions are running.  If more than the throttle limit are running the script sleeps for a bit and checks again.  If less than the limit are running then it exits, passing control to the next action.

You  can configure counting running (s)vMotions at the vCenter or cluster level, the number of (s)vMotions allowed to be running at once, and the sleep interval.

I found the cluster option handy because we have multiple clusters in each vCenter connected to separate storage so the IO load in one cluster doesn’t impact other clusters.  If a host is put into maintenance mode and 20 vMotions are queued in one cluster the storage vMotions in another cluster can keep going.

It’s a bit long to post the code inline so I’m only using the github link.

https://github.com/monahancj/PowerCLI/blob/master/mVMwarePowerCLI/cluster/Wait-mTaskvMotions.ps1

Basic Cisco UCS fault report

This will connect to a list of UCS environments in a text file, build the report for each one, then send a single email with the report for each environment. The report contains:

– Unacknowledged faults
– Hardware locator LEDs that are on (it’s common to forget to turn them off when the work is done)
– If you are short port licenses it will also do a report showing the relevant counts and for convenience output the serial numbers of the FIs.

You’ll have to make the usual changes to directories, email addresses, etc.

You can also grab the script from http://http://poshcode.org/3923 .

<#
.Synopsis
    Connects to multiple UCS environments listed in a text file and generates a basic fault report for each one, then sends one email with the results.  In the report are unacknowledged faults, hardware locator LEDs that are on, and if you are short on port licenses it will show that.  Variables and paths will have to be changed to match your environment.
#>

param($CredentialFile=$null)

if ($CredentialFile -eq $null) { Write-Host "Missing the path to the credential file." -BackgroundColor DarkYellow -ForegroundColor DarkRed; break }


#-----------------------------
# Prep
#-----------------------------
function Get-Now { (get-date -uformat %Y%m%d) + "_" + (get-date -uformat %H%M%S) }
if ((Get-Module -Name CiscoUcsPS) -eq $null) { Import-Module -Name C:\Ops\Modules\CiscoUCSPowerTool\CiscoUcsPS.psd1 }

#Static parameters
Set-Variable -Name ScriptDir   -Value "C:\Scripts" -Scope Local
Set-Variable -Name ReportPath  -Value "C:\Scripts\Reports\UCS_Fault_Reports" -Scope Local
Set-Variable -Name ReportFile  -Value "$($ReportPath)\UCS_Fault_Report_$(Get-Now).html" -Scope Local
Write-Verbose "`nStatic parameters`n-----------------`nScriptDir = $($ScriptDir)`nReportFile = $($ReportFile)`n"
$mailTo = "alertsdistgroup@yourcompany.com"
$mailFrom = "UCS_Fault_Reports@yourcompany.com"
$mailSMTP = "smtp.yourcompany.com"

If ( !(Test-Path $ReportPath) ) { mkdir $ReportPath }
. $ScriptDir\Hals_PSCredentials.ps1
# Setup credentials as described in https://batchlife.wordpress.com/2012/11/14/powershell-script-to-backup-a-cisco-ucs-fabric-interconnect-using-the-cisco-powertool-cmdlets/

$UcsCred = Import-PsCredential -Path $CredentialFile


#-----------------------------
# Start doing the real work.
#-----------------------------

Disconnect-Ucs 2>$null # Connect-Ucs will fail if you're already connected to a UCS

"<pre>" | Out-File -FilePath $ReportFile -Width 400
Get-Date | Out-File -FilePath $ReportFile -Append -Width 400

Get-Content C:\Scripts\Repo\StaticInfo\UCSs.txt | ForEach-Object { 
    Write-Output "=========================================================================================================="
    Connect-Ucs -Name $_ -Credential $UcsCred | select name,version,username,virtualipv4address | ft -a
    if ($DefaultUcs -eq $null) {
        $mailBody = @()
        $mailBody += "$(Get-Now) - Error connecting to UCS`n`n"
        $mailBody += $_
        Send-MailMessage  -From $mailFrom -To $mailTo -Subject "Backup-mUcs Error-- Failed connecting to $($Ucs)" -Body ($mailBody | Out-String) -SmtpServer $mailSMTP
    }
    else {
        Get-UcsFault | Where-Object { ($_.Severity -ne 'cleared') -and ($_.Ack -ne 'yes') } | sort lasttransition -Descending | select lasttransition,severity,status,type,dn,descr | ft -a
        Get-UcsLocatorLed | ? { $_.operstate -eq 'on' } | sort dn | select dn,adminstate,color,operstate,id | ft -a
        If (Get-UcsFault | Where-Object { ($_.Severity -ne 'cleared') -and ($_.Dn -match 'F0676') } ) { 
            "`n------ Port Licenses Needed ----------------------------------------------------------------------------"
            Get-UcsLicense | Select-Object Ucs,Scope,Feature,Sku,AbsQuant,DefQuant,UsedQuant,@{n="RemQuant";e={$_.AbsQuant-$_.UsedQuant}},Status,PeerStatus,OperState,GracePeriodUsed | ft -a
            Get-UcsLicenseServerHostId | select ucs,rn,hostid | ft -a
        }
        Disconnect-Ucs 
    }
} | Out-File -FilePath $ReportFile -Append -Width 400

"</pre>" | Out-File -FilePath $ReportFile -Append -Width 400

$body = Get-Content $ReportFile

Send-MailMessage  -From $mailFrom -To $mailTo -Subject "UCS Fault Report $(Get-Date)" -Body ($body | Out-String) -BodyAsHTML -SmtpServer $mailSMTP

PowerShell script to backup a Cisco UCS fabric interconnect using the Cisco PowerTool cmdlets

We’ve got several UCS environments and thought it’d be a good idea to back them up. 🙂 The script is designed to run as a scheduled task. The deployment instructions are in the script. Thanks to Dan and Nick for testing and correcting the instructions.  Hopefully I didn’t make the instructions more confusing when I sanitized the script for server names and the like.

You’ll have to make the usual changes to directories, email addresses, etc.

You can also grab the script from http://poshcode.org/3767 .

<#
.Synopsis
Does a backup of a single UCS target.

.Description
Uses the UCS PowerTool cmdlet 'Backup-Ucs' and does all four types of UCS backup.
Filenames include the UCS name, date/time stamp, and backup type.
The backup directory is set by a variable in the script. If it doesn't exist the script will create. Since the backup files are XML I recommend enabling compression on the directory.

.Parameter Ucs
The UCS environment to backup. This can be either an IP address or the DNS name.

.Parameter CredentialFile
The location of the credential file to log on to the UCSM with. Defaults to the current directory.

.Example
Backup-mUcs.ps1 -Ucs ucspoc -CredentialPath C:\Temp\credentials.enc.xml

.Notes
NAME: Backup-mUcs.ps1
AUTHOR: Chris Monahan
LASTEDIT: 8/7/2012
KEYWORDS: UCS, PowerTool

.Link
https://batchlife.wordpress.com/2012/11/14/powershell-script-to-backup/

Requires -Powershell Version 2.0
Requires -Cisco PowerTool Version 0.99
Requires -Hal Rottenberg's PsCredential functions: http://poshcode.org/501 & http://halr9000.com/article/531

#>

<# Instructions for one time setup are at the bottom of the script #>

param($Ucs=$null, $CredentialFile=$null)

if ($Ucs -eq $null) { Write-Host "Enter the name/address of the UCS you're backing up" -BackgroundColor DarkYellow -ForegroundColor DarkRed; break }
if ($CredentialFile -eq $null) { Write-Host "Enter path to the credential file." -BackgroundColor DarkYellow -ForegroundColor DarkRed; break }

#-----------------------------
# Prep
#-----------------------------
#Static parameters
Set-Variable -Name ScriptDir -Value "\\server\share" -Scope Local
Set-Variable -Name BackupPath -Value "C:\Backups\UCSM_Configuration_Backups" -Scope Local
Set-Variable -Name BackupTypes -Value ('config-system','config-logical','config-all','full-state')
Write-Verbose "`nStatic parameters`n-----------------`nScriptDir = $($ScriptDir)`nBackupPath = $($BackupPath)`nBackupTypes = $($BackupTypes)`n"
$errmailTo = "name@address"
$errmailFrom = "name@address"
$errmailSMTP = "emailserver"

If ( !(Test-Path $BackupPath) ) { mkdir $BackupPath }
. $ScriptDir\Hals_PSCredentials.ps1

$UcsCred = Import-PsCredential -Path $CredentialFile

#-----------------------------
# Start doing the real work.
#-----------------------------

# Connect to the UCS
Connect-Ucs -Name $Ucs -Credential $UcsCred -ErrorVariable errConnectingUcs| select Ucs,UserName,Version
if ($errConnectingUcs -ne $null) {
$errmailBody = @()
$errmailBody += "$(Get-Now) - Error connecting to UCS`n`n"
$errmailBody += $errConnectingUcs
Send-MailMessage -From $errmailFrom -To $errmailTo -Subject "Backup-mUcs Error-- Failed connecting to $($Ucs)" -Body ($errmailBody | Out-String) -SmtpServer $errmailSMTP
break
}

# Get one of each type of UCS backup.
ForEach ($type in $backuptypes) {
Backup-Ucs -Type $type -PreservePooledValues -PathPattern ($BackupPath + '\${ucs}_${yyyy}${MM}${dd}_${HH}${mm}_' + $type + '.xml') -ErrorVariable errBackupUcs
if ($errBackupUcs -ne $null) {
$errmailBody = @()
$errmailBody += "$(Get-Now) - Error running backup of type $($type)`n`n"
$errmailBody += $error[0]
Send-MailMessage -From $errmailFrom -To $errmailTo -Subject "Backup-mUcs Error-- On $($Ucs) backup of type $($type) failed" -Body ($errmailBody | Out-String) -SmtpServer $errmailSMTP
}
}

# Don't leave a stale session on the UCS
Disconnect-Ucs

<# One time setup

1- Copy script to SCRIPTHOST server for the datacenter you're working in.
$> copy \\server\share\Hals_PSCredentials.ps1 C:\Scripts

2- Log on locally to the SCRIPTHOST with the service account the script will run under in Task Scheduler.

3- If necessary install the Cisco PowerTool module and configure it to load automatically in the PowerShell profile.
a- The PowerTool module can be copied from \\server\share\Modules to the same directory on the local scripthost.
b- The PowerShell profile may not exist yet. If it doesn't create file in the path specified by the "$profile" variable.
$> notepad $profile
1- Add the line "Import-Module C:\Ops\Modules\CiscoUCSPowerTool\CiscoUcsPS.psd1"
2- Save and exit the file

4- Create the credential file containing the UCS login credentials to use in the script. The credential file can only opened by the Windows user account that created it. That's why you have to log on with the Windows service account used to run the task in Task Scheduler. The Windows credentials encrypt the file, and inside the file are the UCS login username and password. You can store the credential files in any directory you want. The directory in the example is for demonstration.
$> mkdir C:\Scripts\CredentialFiles
$> . \\scripthost\share\Hals_PSCredentials.ps1
$> $creds = Get-Credential # Username/password to login to the UCS with.
$> Export-PSCredential -Credential $creds -Path C:\Scripts\CredentialFiles\somename.enc.xml

5- Copy \\server\share\Scripts\Run_Backup-mUcs.ps1 to the vmscripthost you're setting up.
a- This is the wrapper script that will be put into Task Scheduler. It will call Backup-mUcs.ps1 for each UCS environment to be backed up.
b- Edit the script so that it has the correct UCS DNS name(s) or IP address(es) for the datacenter you're configuring this for.

6- Test the script interactively first by running Backup-mUcs.ps1 and then by running Run_Backup-mUcs.ps1.

7- Add Run_Backup-mUcs.ps1 into Task Scheduler and test.
a- For now copy the Task Scheduler settings for the job from what's in the \\scripthost Task Scheduler. May put the full instructions in later.

8- Verify the scripthost server is getting backed up somehow. Preferably as a normal OS level full backup by the backup group. The size of the backups for our UCS's so far for four UCS environments is 12MB a day.

#>

If a VM has a C: drive with unallocated space guests OS customizations will fail

I didn’t find any information on this with an Internet search so I’m throwing it up here in case it helps someone else.

$> set-vm -vm somevm -OSCustomizationSpec w2k3_std_ent_x86_sp2_r2_tmpl

Confirmation

Proceed to configure the following parameters of the virtual machine with name 'somevm'?

[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "Y"):

set-vm : 10/10/2012 4:32:07 PM    Set-VM        Customization of the guest operating system 'winNetStandardGuest' is not supported in thisconfiguration. Microsoft Vista (TM) and Linux guests with Logical Volume Manager are supported only for recent ESX host and VMware Toolsversions. Refer to vCenter documentation for supported configurations.

At line:1 char:1

+ set-vm -vm qax1ftp201 -OSCustomizationSpec w2k3_std_ent_x86_sp2_r2_tmpl

+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    + CategoryInfo          : NotSpecified: (:) [Set-VM], UncustomizableGuest

    + FullyQualifiedErrorId : Client20_ClientSideTaskImpl_ThreadProc_UnhandledException,VMware.VimAutomation.ViCore.Cmdlets.Commands.SetVM

I was getting this error after cloning a VM.  The OS customizations didn’t run automatically with the cloning and I couldn’t set it manually.  At first I thought the customizations had a problem, but this part tipped me off: Microsoft Vista (TM) and Linux guests with Logical Volume Manager are supported only for recent ESX host and VMware Tools versions.

I checked the C: partition on the VM and only 13 GB of it was partitioned on the 20 GB physical disk.  After expanding the C: with gparted (it’s win2003) the guest customization applied successfully.  I don’t know if this would happen if the D: drive or something else had unallocated space.

UCS port license report.

This is simple, but handy.  It’s just some formatting around the PowerTool cmdlet Get-UcsLicense.  Then I either export it to a CSV or leave it on the screen, and I might wrap it in a loop to report on all our UCS FEs.

Below are two snippets.  First, connecting to one UCS and exporting to a CSV file.  And then connecting to several UCSs, filtering on UCSs that are short licenses, and printing to the screen.

Get-UcsLicense | Select-Object Ucs,Scope,Feature,Sku,AbsQuant,DefQuant,UsedQuant,@{n="RemQuant";e={$_.AbsQuant-$_.UsedQuant}},Status,PeerStatus,OperState,GracePeriodUsed | Export-Csv-Path "UCSLicenseReport_$($DefaultUcs | Select-Object -Expand Ucs)_$(Get-Now).csv" -NoTypeInformation


gc UCSs.txt | % {
   Connect-Ucs-Name$_-Credential$ucscred | select ucs,VirtualIpv4Address,username,version | ft-a
   Get-UcsLicense | Select-Object Ucs,Scope,Feature,Sku,AbsQuant,DefQuant,UsedQuant,@{n="RemQuant";e={$_.AbsQuant-$_.UsedQuant}},Status,PeerStatus,OperState,GracePeriodUsed | ? { $_.RemQuant -lt0 } | ft-a
   Disconnect-Ucs
}

Hope this helps

Chris

How I avoid working with the wrong VM, host, server.

At all costs I avoid typing or having to remember names.  🙂

Most of the work on VMs we do is managed by our ticketing system.  I’ve got a directory “Tickets” and for the ones where I’ll need to work with files there’s a sub-directory “Tickets\ticketnumber”.  Say there’s 15 VMs that need a hard drive size increase.  Those 15 VMs will go in a file “servers_ticketnum_comment.txt” with each VM name on one line.  Then all the operations I do will look like “get-content servers_ticketnum_comment.txt | % { do-something $_ }”

That’s the general prinicple.  Put the names of whatever you’re working on in a text file, CSV file, or variable, and keep reusing it.  You could use a Get-VM filter to select the VMs to work with, “get-vm *servertype*”, but if a new VM gets added with a similar name you could start making changes to the wrong VM(s).

It’s also handy if you need to get back to working on the ticket in a week you don’t have to set anything up, just use the file.  If 2 months later there’s a question about which VMs were modified you can check the file.  (It should be logged in your transcript, too.)  If it’s short term and not ticket work I might just use a variable.  Using CSV is for when, using the above example, each hard drive is getting increased by a different size.  The CSV columns would be “VM,SizeKB”.