How to schedule a vCheck (PowerShell) for vmware

I just created a .bat file with the following

C:\WINDOWS\system32\windowspowershell\v1.0\powershell.exe -PSConsoleFile “%ProgramFiles%\VMware\Infrastructure\vSphere PowerCLI\vim.psc1” -File “C:\vCheck\vCheck.ps1”

Watch those stupid quotation marks, WP tends to mangle them

Veeam error, consolidate-helper-0

As I mentioned the other day we had a little hiccup with veeam. https://michaelellerbeck.com/2013/02/28/fail-error-the-object-has-already-been-deleted-or-has-not-been-completely-created-veeam-6-5-0-10/

I don’t know if this related or not, but we ended up with some stuck snapshots.

The easiest way to find this is to use the Awesome RVTools (I should probably figure out a way to script this alert though)

Now I need to figure out how to remove these snapshot, bother.

So, I looked into getting scripted alerts. This guy seems to be the best:

http://www.virtu-al.net/vcheck-pluginsheaders/vcheck/

For what ever reason I had a hard time getting all the requirements setup just right at first. But eventually installing powershell 2.0 in server 2003, and then I installed VMware-PowerCLI-5.1.0-1012425.exe I was finally able to kick everything off…. now it is running a tremendous arsenal against my vmware environment. I’m pretty sure I need to go fix like a bazillion things now…. hmmm

Fun with #vmware ESXi5.1 #IBM 3690

So wanted to find the recommended settings for ‘UEFI’ for the 3690 for vmware. Finally found it here http://www.redbooks.ibm.com/redbooks/pdfs/sg247909.pdf page 262

To Fix: System Management Software 1 SEL Fullness: Log almost full #VMWARE

To clear it http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2011531

So you bought an #EqualLogic and now you are setting it up with #VMware ESXi 5.1

So I wish there was one consolidated doc of settings to change for ESXi 5.1 and EqualLogic. Anyone out there have more settings please let me know!!

To start from here http://en.community.dell.com/techcenter/storage/f/4466/t/19459676.aspx

1.)  Delayed ACK DISABLED

2.)  Large Receive Offload DISABLED

3.)  Make sure they are using either VMware Round Robin (with IOs per path changed to 3), or preferably MEM 1.1.0.

Additionally

4.) Set Login_timeout to 15 or 30 seconds

I contacted EqualLogic support and they replied:

Solution Title HOWTO: Disabling TCP Delayed ACK may improve read performance with ESX 3.x/4.x/5.x software iSCSI initiator.Solution Details ESX allows you to disable the delayed ACK on your ESX host through a configuration option and may improve the read performance of storage attached to ESX software through the iSCSI initiator. PLEASE NOTE: this change will require a reboot of the ESX server to take effect.

Disabling Delayed Ack in ESX 4.0, 4.1, and 5.x

1. Log in to the vSphere Client and select the host.

2. Navigate to the Configuration tab.

3. Select Storage Adapters.

4. Select the iSCSI vmhba to be modified.

5. Click Properties.

6. Modify the delayed Ack setting using the option that best matches your site’s needs.

Choose one of the below options, I, II or II, then move on to step 7 after making the changes:

Option I:

Modify the delayed Ack setting on a discovery address (recommended).

A. On a discovery address, select the Dynamic Discovery tab.

B. Select the Server Address tab.

C. Click Settings.

D. Click Advanced.

7. In the Advanced Settings dialog box, scroll down to the delayed Ack setting.

8. Uncheck Inherit From parent. (Does not apply for Global modification of delayed Ack)

9. Uncheck DelayedAck.

delayedack

10. Reboot the ESX host.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1002598

NOTE:  From http://communities.vmware.com/message/2078917#2078917

One thing about Delayed ACK, is that you have to verify that the change took place.  If you just change it, it appears that only new LUNs will inherit the value. (disable)   I find that, while in maint mode, removing the discovery address and any discovered targets (in static discovery tab), then disabling Delayed ACK and re-add in Discovery Address and rescan resolves this.

At the ESX console run:   #vmkiscsid –dump-db | grep Delayed     All the entries should end with =’0′ for disabled. ( note that is a – – two dashes before the dump-db silly WP font)

I ran it and experienced the same, only some of the connections had the DelayedAck set correctly. So definitely remove the discovery address and static targets to fix.

iscsi log dump

To disable LRO  from http://communities.vmware.com/thread/419234

HOWTO: Disable Large Receive Offload (LRO) in ESX v4/v5
Solution Details
Within VMware, the following command will query the current LRO value.# esxcfg-advcfg -g /Net/TcpipDefLROEnabledTo set the LRO value to zero (disabled):# esxcfg-advcfg -s 0 /Net/TcpipDefLROEnabledNOTE: a server reboot is required.

Info on changing LRO in the Guest network.

http://docwiki.cisco.com/wiki/Disable_LRO

To disable LRO, follow this procedure:

  1. Log into the ESXi host or its vCenter with vSphere Client.
  2. Select the host > Configuration Software:Advanced Settings.
  3. Select Net and scroll down slightly more than half way.
  4. Set the following parameters from 1 to 0:
    • Net.VmxnetSwLROSL
    • Net.Vmxnet3SwLRO
    • Net.Vmxnet3HwLRO
    • Net.Vmxnet2SwLRO
    • Net.Vmxnet2HwLRO
  5. Reboot the ESXi host to activate these changes.

Your guest VMs should now have normal TCP networking performance.

Set Login Time Out

From: http://pubs.vmware.com/vsphere-51/index.jsp?topic=%2Fcom.vmware.vsphere.storage.doc%2FGUID-2E5AE903-0240-4323-95A8-6BB3719E42DE.html

This vmware 5.1 document points to this KB article http://kb.vmware.com/kb/2007829

To change the login timeout from the vSphere client:
  1. Go to Storage Adapters > iSCSI Software Adapter > Properties.
  2. Select Advanced and scroll down to LoginTimeout.
  3. Change the value from 5 seconds to a larger value, such as 15 or 30 seconds.

logintimeout

 

 

 

 

Addendum

On second review the Official Documentation for the EqualLogic MEM (Rev 1.2, which covers vSphere 5.1)
http://www.equallogic.com/WorkArea/DownloadAsset.aspx?id=11000

Advises:

Deployment Considerations: iSCSI Login Timeout on vSphere 5.1 and5.0
The default value of 5 seconds for iSCSI logins on vSphere 5.x is too short in some
circumstances. For example: In a large configuration where the number of iSCSI
sessions to the array is close to the limit of 1024 per pool. If a severe network
disruption were to occur, such as the loss of a network switch, a large number of iSCSI
sessions will need to be reestablished. With such a large number of logins occurring,
some logins will not be completely processed within the 5 second default timeout
period.
Dell therefore recommends applying patch ESXi500-201112001 and increasing the ESXi
5.0 iSCSI Login Timeout to 60 seconds to provide the maximum amount of time for
such large numbers of logins to occur.
If the patch is installed prior to installing the EqualLogic MEM, the MEM installer will
automatically set the iSCSI Login Timeout to the Dell recommended value of 60
seconds.

 

Automatically setup is nice in theory but doesn’t appear to work with ESX5.1 ??? I had to set mine to 60 manually

I was able to find these three recommendations under the support portal under here https://support.equallogic.com/support/solutions.aspx?id=1444

They are

HOWTO: Disable Large Receive Offload (LRO) in ESX v4/v5

HOWTO: Change Login timeout value ESXi v5.0 // Requires ESX patch ESXI500-201112001

HOWTO: Disabling TCP Delayed ACK may improve read performance with ESX 3.x/4.x/5.x software iSCSI initiator

A gerneral system error occurred: Timed waiting for vpxa to start #vmware

A great discription of the error here http://serverfault.com/questions/450093/error-while-adding-new-host-to-vm-center

And the ding dong resolution is don’t try to add a ESXi 5.1 to a 5.0 vcenter

#EqualLogic #MEM #vmware install continued

So I’m following along with vmpete’s http://vmpete.com/2012/09/20/multipathing-in-vsphere-with-the-dell-equallogic-multipathing-extension-module-mem/  mem install…

You really do have to put the server in maintenance mode or it won’t let you run setup.pl 🙂

Like Pete suggests I used a full command string to setup. I looked up the definition of -nohwiscsi that means

Ignore any supported iSCSI offload network cards, and use
only the VMware software iSCSI initiator. If the
configuration script detects iSCSI offload network cards, it
will use this capability by default unless this flag is set.

My string looks like setup.pl -configure -server=10.56.1.1 -vswitch=vSwitch4 –mtu=9000 -nics=vmnic4,vmnic5,vmnic10,vmnic11 -ips=10.10.1.75,10.10.1.76,10.10.1.77,10.10.1.78 -netmask=255.255.0.0 -vmkernel=iscsi -nohwiscsi –enableswiscsi

Ok, that setup.pl is awesome! You won’t realize how cool it is until you had to create iscsi nics the old fashion way! Thanks #Equallogic

mem nics

So I wanted to show the team how this script ran, in order to do it again you need to unbind the physical network cards from the iSCSI VMKernel

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1030674

so starting with vmk1 (vmk0 usually is the management interface)

ESXi 5.0: # esxcli iscsi networkportal remove -n vmk1 -A vmhba33

Then I needed to install the Multipathing Extension Module…

Wow, piece of cake!