#Ravello #VEEAM and the chaos monkeys

Starting off a quick spin through Ravello. Ok, I got all registered up… now what

hmm I bet there is some way to import a VM image… although really an virtualized ESXi server would be more interesting.

Hitting the google, found some instructions here … https://www.ravellosystems.com/blog/create-esxi-5-5-iso-image/

Ok, seems pretty straight forward, just drop an empty ESXi container and then attach a CD-ROM and install…. although wouldn’t it be nice if Ravello already had the iso there already to go for me…. hmmm could be a security issue though. Although I’m already trusting them with all the keys of the castle so… why do I have to drop an empty ESXi, how about I drop an already made ESXi 5.5 box… Oh well, I’ll stop my complaining 🙂

Pretty easy, go to the library and the upload an .ISO (I would have liked a drag and drop of the file into the window to upload, but it was easy enough to use.)

So, now I choose that disk image and attach it to the empty ESX box.

Choose Disk Image

This window could be improved, it’s hard to tell what you are able to click on. Radio buttons would be appropriate.

P.S. its hard to delete a box from the canvas. I expected to be able to right click it and then option delete. Instead, its hidden under the more button, bah.


Ok, published. Looks like I should press the play button 🙂 err, go to the console 🙂

Ok, firing up the console… that is just plain cool!!!


Ok, so now I need to do some tweaks to allow the ESXi box to be added to the same cluster. So it has you make some changes to enable SSH on the ravello side and then turn on the SSH on the ESXi box…. and then tells you to make changes. It fails to mention you should SSH into the box now 🙂 but anyways after futzing I was able to ssh in, and make the changes.

OK, so now I have two ESX severs up and running. Now for some fun.

See next post 🙂 https://michaelellerbeck.com/2015/08/11/ravello-veeam-and-the-chaos-monkeys-part-2/

Where in I install graylog2 as a linux newb

Or just watch the video 🙂 https://everythingshouldbevirtual.com/ubuntu-graylog2-auto-install-script-video

First download Ubuntu


Use the 64-bit PC (AMD64) server install image I’m installing inside vmware so I choose Ubuntu 64bit as the vm type. The AMD64 might throw you off, it just means 64bit.

Mount and boot the .iso. Install using defaults. Done.

Next, try to get vmware tools installed.

I followed these instructions -> from here http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1022525

And it looks like they actually worked!


Ubuntu Server with only a command line interface

  1. Go to Virtual Machine > Install VMware Tools (or VM > Install VMware Tools).Note: If you are running the light version of Fusion, or a version of Workstation without VMware Tools, or VMware Player, you are prompted to download the Tools before they can be installed. Click Download Now to begin the download.
  2. In the Ubuntu guest, run these commands:
    1. Create a directory to mount the CD-ROM by running the command:sudo mkdir /mnt/cdromWhen prompted for a password, enter your Ubuntu admin user password.Note: For security reasons, the typed password is not displayed. You do not need to enter your password again for the next five minutes.
    2. Mount the CD-ROM by running the command:sudo mount /dev/cdrom /mnt/cdrom or sudo mount /dev/sr0 /mnt/cdrom
    3. The file name of the VMware Tools bundle varies depending on your version of the VMware product. Run this command to find the exact name:ls /mnt/cdrom
    4. Extract the contents of the VMware Tools bundle by running the command:tar xzvf /mnt/cdrom/VMwareTools-x.x.x-xxxx.tar.gz -C /tmp/Notex.x.x-xxxx is the version discovered in the previous step.
    5. Change directories into the VMware Tools distribution by running the command:cd /tmp/vmware-tools-distrib/
    6. Install VMware Tools by running the command:sudo ./vmware-install.pl -dNote: The -d switch assumes that you want to accept the defaults. If you do not use -d, press Return to accept each default or supply your own answers.
  3. Run this command to reboot the virtual machine after the installation completes:sudo reboot

Assign a static ip using instructions here ->https://help.ubuntu.com/10.04/serverguide/network-configuration.html

Static IP Address Assignment

To configure your system to use a static IP address assignment, add the static method to the inet address family statement for the appropriate interface in the file /etc/network/interfaces. The example below assumes you are configuring your first Ethernet interface identified as eth0. Change the addressnetmask, and gateway values to meet the requirements of your network.

auto eth0
iface eth0 inet static

By adding an interface configuration as shown above, you can manually enable the interface through the ifup command.

sudo ifup eth0

--- OK now to install. taken from here -> http://everythingshouldbevirtual.com/ubuntu-12-04-graylog2-installation

The salient point of the install is

Installation steps
sudo apt-get -y install git
cd ~
git clone https://github.com/mrlesmithjr/graylog2/
chmod +x ./graylog2/install_graylog2_20_ubuntu.sh

Now enter the following to start running the script.
cd ~
sudo ./graylog2/install_graylog2_20_ubuntu.sh
That will take a while, I don't know how long since I went home and it was done when I got back.

--tada it worked

---now setup the redirect Rsyslog is listening on UDP/514 and forwarding to Graylog2 which is listening on UDP/10514

Open your browser of choice and connect to http://ip.or.nameofgraylog2server:9000

Login with username admin and password is password123


Click on system


Click on nodes


Select action and then manage inputs


Select Syslog UDP from dropdown


Give it a name of syslog redirect and port 10514 and then click launch and close. (Rsyslog is listening on UDP/514 and forwarding to Graylog2 which is listening on UDP/10514)


You should now see your new input created and accepting traffic.


--- Ok, now to get our ESX traffic in. Taken from -> https://everythingshouldbevirtual.com/collecting-vsphere-syslog-data-with-graylog2

Now assuming that you have Graylog2 up and running let’s configure our ESXi hosts to send their syslogs to our new Graylog2 server.

Open your vSphere Client and select your host from vCenter or connect directly to your host. Select configuration and then advanced settings under the software section. Scroll down and expand Syslog and select global. Now underSyslog.global.loghost fill in udp://ipaddressofgraylog2:514 then click ok.



Now you have to allow syslog data to be sent from your host. In order to do this you must configure the firewall on the host to allow this. So on the configuration page select security profile under the Software section. Scroll down to syslog and enable the checkbox and click ok.


Once that is done you should start to see syslog data showing up in Graylog2. The only other thing you might want to do is make sure that each of your hosts are showing up as unique hosts within Graylog2. Other than that you are good to go. You can also configure other devices in your environment to send their syslog data back to your Graylog2 server.

OK, now I see stuff flowing in but I need to make some sort of dashboard! I'll figure that out next!



vCenter next prob

Why is vcenter 5.1 so much harder to install?




How to schedule a vCheck (PowerShell) for vmware

I just created a .bat file with the following

C:\WINDOWS\system32\windowspowershell\v1.0\powershell.exe -PSConsoleFile “%ProgramFiles%\VMware\Infrastructure\vSphere PowerCLI\vim.psc1” -File “C:\vCheck\vCheck.ps1”

Watch those stupid quotation marks, WP tends to mangle them

To Fix: System Management Software 1 SEL Fullness: Log almost full #VMWARE

To clear it http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2011531

So you bought an #EqualLogic and now you are setting it up with #VMware ESXi 5.1

So I wish there was one consolidated doc of settings to change for ESXi 5.1 and EqualLogic. Anyone out there have more settings please let me know!!

To start from here http://en.community.dell.com/techcenter/storage/f/4466/t/19459676.aspx

1.)  Delayed ACK DISABLED

2.)  Large Receive Offload DISABLED

3.)  Make sure they are using either VMware Round Robin (with IOs per path changed to 3), or preferably MEM 1.1.0.


4.) Set Login_timeout to 15 or 30 seconds

I contacted EqualLogic support and they replied:

Solution Title HOWTO: Disabling TCP Delayed ACK may improve read performance with ESX 3.x/4.x/5.x software iSCSI initiator.Solution Details ESX allows you to disable the delayed ACK on your ESX host through a configuration option and may improve the read performance of storage attached to ESX software through the iSCSI initiator. PLEASE NOTE: this change will require a reboot of the ESX server to take effect.

Disabling Delayed Ack in ESX 4.0, 4.1, and 5.x

1. Log in to the vSphere Client and select the host.

2. Navigate to the Configuration tab.

3. Select Storage Adapters.

4. Select the iSCSI vmhba to be modified.

5. Click Properties.

6. Modify the delayed Ack setting using the option that best matches your site’s needs.

Choose one of the below options, I, II or II, then move on to step 7 after making the changes:

Option I:

Modify the delayed Ack setting on a discovery address (recommended).

A. On a discovery address, select the Dynamic Discovery tab.

B. Select the Server Address tab.

C. Click Settings.

D. Click Advanced.

7. In the Advanced Settings dialog box, scroll down to the delayed Ack setting.

8. Uncheck Inherit From parent. (Does not apply for Global modification of delayed Ack)

9. Uncheck DelayedAck.


10. Reboot the ESX host.


NOTE:  From http://communities.vmware.com/message/2078917#2078917

One thing about Delayed ACK, is that you have to verify that the change took place.  If you just change it, it appears that only new LUNs will inherit the value. (disable)   I find that, while in maint mode, removing the discovery address and any discovered targets (in static discovery tab), then disabling Delayed ACK and re-add in Discovery Address and rescan resolves this.

At the ESX console run:   #vmkiscsid –dump-db | grep Delayed     All the entries should end with =’0′ for disabled. ( note that is a – – two dashes before the dump-db silly WP font)

I ran it and experienced the same, only some of the connections had the DelayedAck set correctly. So definitely remove the discovery address and static targets to fix.

iscsi log dump

To disable LRO  from http://communities.vmware.com/thread/419234

HOWTO: Disable Large Receive Offload (LRO) in ESX v4/v5
Solution Details
Within VMware, the following command will query the current LRO value.# esxcfg-advcfg -g /Net/TcpipDefLROEnabledTo set the LRO value to zero (disabled):# esxcfg-advcfg -s 0 /Net/TcpipDefLROEnabledNOTE: a server reboot is required.

Info on changing LRO in the Guest network.


To disable LRO, follow this procedure:

  1. Log into the ESXi host or its vCenter with vSphere Client.
  2. Select the host > Configuration Software:Advanced Settings.
  3. Select Net and scroll down slightly more than half way.
  4. Set the following parameters from 1 to 0:
    • Net.VmxnetSwLROSL
    • Net.Vmxnet3SwLRO
    • Net.Vmxnet3HwLRO
    • Net.Vmxnet2SwLRO
    • Net.Vmxnet2HwLRO
  5. Reboot the ESXi host to activate these changes.

Your guest VMs should now have normal TCP networking performance.

Set Login Time Out

From: http://pubs.vmware.com/vsphere-51/index.jsp?topic=%2Fcom.vmware.vsphere.storage.doc%2FGUID-2E5AE903-0240-4323-95A8-6BB3719E42DE.html

This vmware 5.1 document points to this KB article http://kb.vmware.com/kb/2007829

To change the login timeout from the vSphere client:
  1. Go to Storage Adapters > iSCSI Software Adapter > Properties.
  2. Select Advanced and scroll down to LoginTimeout.
  3. Change the value from 5 seconds to a larger value, such as 15 or 30 seconds.







On second review the Official Documentation for the EqualLogic MEM (Rev 1.2, which covers vSphere 5.1)


Deployment Considerations: iSCSI Login Timeout on vSphere 5.1 and5.0
The default value of 5 seconds for iSCSI logins on vSphere 5.x is too short in some
circumstances. For example: In a large configuration where the number of iSCSI
sessions to the array is close to the limit of 1024 per pool. If a severe network
disruption were to occur, such as the loss of a network switch, a large number of iSCSI
sessions will need to be reestablished. With such a large number of logins occurring,
some logins will not be completely processed within the 5 second default timeout
Dell therefore recommends applying patch ESXi500-201112001 and increasing the ESXi
5.0 iSCSI Login Timeout to 60 seconds to provide the maximum amount of time for
such large numbers of logins to occur.
If the patch is installed prior to installing the EqualLogic MEM, the MEM installer will
automatically set the iSCSI Login Timeout to the Dell recommended value of 60


Automatically setup is nice in theory but doesn’t appear to work with ESX5.1 ??? I had to set mine to 60 manually

I was able to find these three recommendations under the support portal under here https://support.equallogic.com/support/solutions.aspx?id=1444

They are

HOWTO: Disable Large Receive Offload (LRO) in ESX v4/v5

HOWTO: Change Login timeout value ESXi v5.0 // Requires ESX patch ESXI500-201112001

HOWTO: Disabling TCP Delayed ACK may improve read performance with ESX 3.x/4.x/5.x software iSCSI initiator

A gerneral system error occurred: Timed waiting for vpxa to start #vmware

A great discription of the error here http://serverfault.com/questions/450093/error-while-adding-new-host-to-vm-center

And the ding dong resolution is don’t try to add a ESXi 5.1 to a 5.0 vcenter