Reference Links

http://theithollow.com/

http://www.virtuallypeculiar.com/2015/12/unable-to-delete-orphanedstale-vmdk-file.html  - All Vmware Logs related

http://virtualpatel.blogspot.in/2013/06/vmware-network-ports-diagrams-vsphere.html - Command line poster

http://www.virtuallyghetto.com/2013/09/new-vcenter-server-simulator-20.html - for automation tips in vmware environment

https://www.vmadmin.co.uk - Good Vmware materials understandable

Understanding VMware Slot Sizes

VMware slot sizes are an important topic if you’re concerned with how many ESXi hosts are required to run your environment.

What is a Slot?

To begin this post, we need to understand what a slot is.  A slot is the minimum amount of CPU and memory resources required for a single VM in an ESXi cluster.  Slot size is an important concept because it affects admission control.

A VMware ESXi cluster needs a way to determine how many resources need to be available in the event of a host failure.  This slot calculation gives the cluster a way to reserve the right amount of resources.

How are Slots Sized?

The slot has two parts, the CPU component and the memory component.  Each of them has its own calculation.    If there are no virtual machine resource reservations in the cluster, then the slot size (for ESXi 5 at least) is 32 Mhz for CPU and 0 MBs + overhead for memory. (I’ve used 80 MBs as my memory overhead in the examples)

On to an incredibly simplistic diagram…

In the example below we have 2 ESXi hosts that have the same amount of resources available for virtual machines.  There are different sized VMs, but none of them have a reservation.  Doing a quick calculation we can determine that 384 slots are available on each host.

CPU Component:   4 X 3.0 GHz / 2000 MHz = 384 slots

Memory Component:   49 GBs / 4024 MBs = 627 slots

We take the lower value between the CPU slot size and the memory slot size to determine the number of virtual machines that can be started up under admission control.  So therefore we could safely start 384 machines on these ESXi hosts, have one fail, and have the other host start all of them.

(I should mention that it’s unlikely that you could get 384 vms on one of these hosts.  That would be a great consolidation ratio.)

Problem Scenario

What if you have a single large VM with a reservation, but the rest of the virtual machines are relatively small.

Let’s look at the same environment, but this time let’s make the larger VM have a reservation on it.

CPU Component:   4 X 3.0 GHz / 2000 MHz = 6 slots

Memory Component:   49 GBs / 4024 MBs = 12 slots

Admission control is going to tell us that only 6 slots are available on host B, so it will only allow 6 VMs on host A to be powered on.  Since I’m using a simplistic diagram with only two hosts, we know that these VMs will still fit on the host but since we use the largest slot size to determine how much we can fail over admission control will stop us from powering on VMs.

What are our options?

Option 1 – Don’t use reservations unless their is a good reason to do so.

Option 2 – We can manually configure the slot size on the cluster.

Navigate to the cluster settings and go to the HA Section, Click Edit and you’ll have the option of modifying the slot size.  Note that if you do this, some of your VMs will require multiple slots to run.  For instance the large VM we used in our example might take more than 1 slot depending on what size you make it.  The button below the slot size configuration may help you determine how many VMs will be affected by this change.

View Your Slot Size

If you’re curious about what the slot size is on your system, look at your cluster summary.  There will be an item listed for slot size.

Summary

If you’re in a situation where you think you need to add extra ESXi hosts to your cluster because you can’t power on virtual machines without exceeding your admission control rules, take a look at your slot sizes first.  It may save you some money on a host you don’t really need.

What is Private Cloud?

What is Private Cloud?

Private cloud is a type of cloud computing that delivers similar advantages to public cloud, including scalability and self-service, but through a proprietary architecture. Unlike public clouds, which deliver services to multiple organizations, a private cloud is dedicated to a single organization.

As a result, private cloud is best for businesses with dynamic or unpredictable computing 

needs that require direct control over their environments.

Public and private cloud deployment models differ. Public clouds, such as those from Amazon Web Services or Google Compute Engine, share a computing infrastructure across different users, business units or businesses. However, these shared computing environments aren't suitable for all businesses, such as those with mission-critical workloads, security concerns, uptime requirements or management demands. Instead, these businesses can provision a portion of their existing data center as an on-premises -- or private -- cloud.

A private cloud provides the same basic benefits of public cloud. These include self-service and scalability; multi-tenancy; the ability to provision machines; changing computing resources on-demand; and creating multiple machines for complex computing jobs, such as big data. Chargeback tools track computing usage, and business units pay only for the resources they use.

In addition, private cloud offers hosted services to a limited number of people behind a firewall, so it minimizes the security concerns some organizations have around cloud. Private cloud also gives companies direct control over their data.

But private clouds have some disadvantages. For example, on-premises IT -- rather than a third-party cloud provider -- is responsible for managing the private cloud. As a result, private cloud deployments carry the same staffing, management, maintenance and capital expenses as traditional data center ownership. Additional private cloud expenses include virtualization, cloud software and cloud management tools.

However, to reduce an organization's on-premises IT footprint, cloud providers, such asRackspace and VMware, can deploy private cloud infrastructures.

A business can also use a mix of a private and public cloud services with hybrid cloud deployment. This allows users to scale computing requirements beyond the private cloud and into the public cloud -- a capability called cloud bursting.

Difference Between Vpshere Version

	vSphere 4.0	vSphere 4.1	vSphere 5.0	vSphere 5.1	vSphere 5.5	vSphere 6.0
Hypervisor	ESX & ESXi	ESX & ESXi	ESXi	ESXi	ESXi	ESXi
CPUs per host	64	160	160	160	320	480
RAM per host	1 TB	1 TB	2  TB	2  TB	4 TB	12 TB
vCPU per VM	8	8	32	64	64	128
vMEM per VM	255 GB	255 GB	1 TB	1 TB	1 TB	4 TB
VMDK Size	2TB	2TB	2TB	2TB	62 TB	62 TB
Cluster Size	32	32	32	32	32	64
FT Max vCPU	1 vCPU	1 vCPU	1 vCPU	1 vCPU	1 vCPU	4 vCPU
VM Hardware Version	7	7	8	9	10	11
VMFS Version	3.33	3.46	5.54	5.58	5.60	5.61
Features	– Distributed Switch – Update Manager – Fault tolerance – Host Profiles – Storage vMotion – Linked Mode – Enhanced vMotion – Compatibility – Distributed Power Management – Hot Add VM CPU/Memory – VMDirectPath – VMXNET3	– Boot from SAN – Scripted Installation – Storage I/O Control – DRS Host Affinity – Memory Compression – Array Integration – Network I/O Control	– Auto Deploy – Storage DRS – vSphere Storage Appliance – Profile driven storage – vSphere Web Client – vCenter Server Appliance – ESXi Firewall	– Single Sign-On – Enhanced vMotion – Sparse Virtual Disks – Single-root I/O Virtualization – vSphere Replication – vSphere Data Protection – vShield Endpoint	– vSphere Flash Read Cache – Virtual SAN – Reliable Memory Technology – vSphere Big Data Extensions – vCenter Single Sign-On 2.0 – vSphere App HA – 40GBps NIC Support	– Virtual Volumes – vMotion across vSwitches – Cross vCenter vMotion – Native Linked Mode – Platform Services Controller – Certificate Authority (VMCA) – Content Library
Preferred Management	vSphere Client	vSphere Client	vSphere Client	vSphere Web Client	vSphere Web Client	vSphere Web Client
Authentication Management	–	–	–	Single Sign-On	Single Sign-On 5.5	Platform Services Controller
Backup Strategy	VMware Data Recovery (VDR)			vSphere Data Protection (VDP)
vMotion Support	– Boundary: Datacenter – Shared Storage required – max. 5ms RTT	– Boundary: Datacenter – Shared Storage required – max. 5ms RTT	– Boundary: Datacenter – Shared Storage required – max. 10ms RTT	– Boundary: Datacenter – max. 10ms RTT	– Boundary: Datacenter – max. 10ms RTT	– vMotion across vCenters – vMotion across vSwitches – max. 100ms RTT
Virtual SAN	–	–	–	–	VSAN 5.5	VSAN 6.0
HA Agent	AAM Automatic Availability Manager		FDM Fault Domain Manager
VAAI Primitives	–	– Atomic Test & Set – Cloning Blocks – Zeroing File Blocks	– Atomic Test & Set – Cloning Blocks – Zeroing File Blocks – UNMAP – Quota Exceeded Behavior – TP LUN Reporting – NFS Full File Clone – NFS Space Reservation – NFS Extended Stat – NFS Space Reclaim – NFS Fast File Clone
vCenter Type	Windows	Windows	Windows Linux (vCSA)	Windows Linux (vCSA)	Windows Linux (vCSA)	Windows Linux (vCSA)
vCSA Scale (vPostgres)	–	–	5 Hosts 50 VMs	5 Hosts 50 VMs	100 Hosts 3000 VMs	1000 Hosts 10000 VMs

What is LAHF and SAHF Instructions and NX/XD bit in CPU

NX/XD bit is a CPU feature calledNever eXecute, hence the NX name. What the NX bit does is enable the ability to mark certain areas of memory as non-executable with a flag. When this happens the processor will then refuse to execute any code that resides in those areas of memory. Any attempt to execute code from a page that is marked as no execute will result in a memory access violation. This feature adds a layer of security to a computer by providing a protected area against malicious code such as viruses and buffer overflow attacks.

AMD first added the NX bit feature to their AMD64 processor line starting with the Opteron processor starting in 2003. So you may be wondering about the XD part, well that is simply Intel’s name for the same feature which they refer to as eXecute Disable. Intel introduced support for the XD bit shortly after AMD with their Pentium 4 Prescott processor in 2004. Both the NX bit and the XD bit have the exact same functionality just different names so you will often see it as referred to as NX/XD. This feature has been standard on most processors for years now so almost every server built since 2006 should have it. Support for NX/XD is typically enabled or disabled in the server BIOS and is typically found under Processor options and labeled as something like “Execute Disable Bit”, “NX Technology” or “XD Support”.

Many virtualization admins know what NX/XD is but LAHF & SAHF CPU instructions are a processor function that you have probably never heard of. LAHF stands for Load AH from Flags and SAHF stands for Store AH into Flags. LAHF & SAHF are used to load and store instructions for certain status flags. Instructions are basic commands composed of one or more symbols that that are passed to a CPU as input. These instructions related to LAHF & SAHF are used for virtualization and floating-point condition handling. You really don’t need to understand how they work as they are related to the core CPU architecture but if you want to understand them better you can read more about them here.

Support for LAHF and SAHF instructions appeared shortly after NX/XD was introduced. AMD introduced support for the instructions with their Athlon 64, Opteron and Turion 64 revision D processors in March 2005 and Intel introduced support for the instructions with the Pentium 4 G1 stepping in December 2005. So again most most servers built after 2006 should have CPUs that support LAHF/SAHF. Similar to NX/XD which can be enabled or disabled in the server BIOS, support for LAHF/SAHF is typically tied into the Virtualization Technology (VT) option in a server BIOS which is often referred to Intel VT or AMD-V which is their respective support for virtualization CPU technology. The option to enable this on a HP Proliant BIOS is shown below:

So how do you know if your server’s CPUs support NX/XD & LAHF/SAHF? As I said before if you’ve purchased a server in the last 5 or so years, it most likely will support it. If it doesn’t support it the ESXi installer will warn you when you install it as shown below:

Interesting enough though it will still let you install it despite not having the required CPU features. Prior versions of vSphere used to give you an error saying your CPU doesn’t support Long Mode and wouldn’t let you install it. If you do get the error above the first thing to check in that case is if you have those options enabled in the BIOS, if you don’t see those options in the BIOS then your CPU may not support them. You can check your specific CPU’s specifications on Intel’s orAMD‘s websites. You can also check VMware’s Hardware Compatibility List but be aware that there are many processor types/server models not on the HCL that will still work despite not being on the list, they just are not officially supported.

Another way to know if your CPU’s support the required features is to use VMware’s CPU Identification Utility which is a small little ISO that you can boot your host from and it will check the CPU hardware to see if it will support vSphere. I’ve mounted it using the iLO management on server and have also mounted it to a VM’s CD-ROM and booted from it and ran it. Since the CPU hardware is not emulated it can see what type of physical CPU the host is using and what features it supports. The output of the CPU ID tool is shown below, this server fully support all the required CPU features for vSphere:

So there you have it, now you know more about NX/XD & LAHF/SAHF than you probably wanted to know but at least you have an understanding of what they are when you read about the CPU requirements in the vSphere documentation. You probably won’t find any modern servers that don’t support it but often times our data centers become server graveyards and contain a lot of older hardware that keeps getting re-used until they finally die which may not support it. So knowing what to look for when it comes to CPU features is good to know.

Interview Tips

The list of questions below is intentionally broken up into 5 groups:

1.     System Requirements for ESXi 5.5

2.     ESXi 5.5 Installation

3.     System Requirements for vCenter

4.     vCenter Installation

5.     Managing vSphere Resources

These 5 groups will test for technical proficiency that an experienced VMware administrator will [or should have].

The list is also what I use when interviewing someone at an administrator level who will manage the day-to-day support, build, and provisioning of:

·         vCenter

·         ESXi hosts

·         virtual servers

·         virtual networks

·         virtual storage

Note: This list of questions may also be used for screening vSphere Engineers or Architects.

System Requirements for ESXi

1.     Your company has a bunch of unused server hardware and you have been asked to build a vSphere environment. But before you start you want to make sure it is supported by the ESXi version you plan to install. Where would you find a list of hardware that is supported?

Answer: You would check the VMware Compatibility Guidehttp://www.vmware.com/resources/compatibility/

2.                 Your hardware is on the compatibility list and now you need to know what the minimum CPU and memory requirement of an ESXi host when building a 64bit OS that will take full advantage of ESXi features ?

Answer: 2 CPU Cores and 8 GB memory (bare minimum)

3.                 Before you start installing software what must be enabled in the server BIOS in order to run ESXi?

Answer: The NX/XD bit needs to be enabled.

More Details about NX/XD and LAHF and SAHF CPU instruction here

4.                 You also have an old pile of network adapters to choose from. Where would you find a list of network adapters (NIC) that are supported by the version of ESXi you are installing?

Answer: You would check the VMware Compatibility Guide (same link as above)

5.                 One of the servers you found wasn’t on the list and now you have a problem that is causing ESXi to purple screen so you call VMware for support. What would happen if they found out your hardware was not supported?

Answer: When hardware is not supported there are known issues that will happen and VMware cannot help to resolve problems due to unsupported hardware.

Unpacking the ESXi hardware interview questions.

Why are these questions about server hardware important?

Since 2006 I have had the awful job of having to clean up behind admins that would build vCenter and ESX/ESXi on any hardware they found. I’m guilty of it myself!

This is not only a bad practice but it also leads to unplanned maintenance and costly outages that impact users and customers.

ESXi will run on just about any hardware but that doesn’t mean it should.

An experienced vSphere admin will know better than to waste time (and a costly license) trying to run a 64bit OS on a server that doesn’t support it.

ESXi Installation Qs

6.                 You’ve done this a hundred times. What are 3 methods that can be used when doing an Interactive ESXi installation?

Answer: Boot from a CD/DVD, boot from a USB, and PXE boot from a deployment server on the network.

7.                 After so many ESXi builds you have figured out a faster way to do the install using a script. What’s the main difference between an interactive ESXi installation and a scripted installation?

Answer: The interactive install will require you to follow along and fill in the configuration information whereas, in a scripted install, the configuration information will be queried from an unattended text file (boot.cfg). Note: ESXi uses a Kickstart installer similar to Linux.

8.                 You can install ESXi in your sleep. What is the command to access the installer window on ESXi 5.5?

Answer: Shift+O will get you to the runweasel command prompt.

9.                 Darn, one of your servers is not loading. What’s the first thing to double-check when having a problem installing ESXi?

Answer: Make sure the hardware clock is set to UTC and the NX/XD bit is enabled in the BIOS.

10.            You’ve typed this stuff in a million times. Name 5 of the 10 bits of information required when installing ESXi?

Answer: Keyboard Layout, VLAN ID, IP Address, Subnet Mask, Gateway, Pri DNS, Sec DNS, Host Name, Installation Location, Root Password.

Now let’s unpack VMware interview questions about ESXi Installations.

ESXi is actually easy to install and setup; however, problems do arise when there are different admins or teams building out environments that share the same network, storage, and server resources. It is very important to have a standard build process so all hosts are built the same way every time.

Finding a good vSphere admin who can follow your standard, someone who can answer these basic questions, is a good start.

System Requirements for vCenter

11.            You decided not to run the vCenter appliance. What’s the minimum hardware requirements for a vCenter 5.x server?

Answer: It depends on whether other vSphere services such as single sign-on, inventory service, and the database will be running on the same hardware with vCenter.

12.            You’re going to upgrade vCenter and the old vCenter is running on Windows Server 2003. What Windows Operating System is required for vCenter 5.x?

Answer: Windows 2008 SP2

13.            You’re creating a list of ports you need the network admin to open on the firewall. What is port 902 used for on vCenter?

Answer: Port 902 is the default port vCenter uses to send data to hosts managed by vCenter.

14.            The old vCenter was running MS SQL Express but you are building 10 ESXi hosts. What’s the difference between using MS SQL Server Express and MS SQL Server for vCenter?

Answer: MS SQL Server Express is for vSphere installations with up to 5 ESXi hosts and 50 virtual servers.

15.            Your new boss doesn’t like MS SQL and asked you to use Oracle. Does vCenter support Oracle 11c or 12c?

Answer: Yes, but the JDBC driver must be installed manually.

Let’s review vCenter Interview Questions.

vCenter is at the heart of vSphere and when it’s not properly setup there will be problems!

These problems may be as small as a frustrated admins who can’t access and support virtual servers from a central management console. Or as large as a full-blown outage and corruption of the vCenter database.

For example, if a vSphere clouds grows too fast the additional load on hardware and system resources can cause performance problems that in worst cases can require a complete rebuild of vCenter on bigger hardware. Fortunately, each ESXi host can be managed separately if central management via vCenter is lost.

An experience vSphere admin who can answer these questions will likely know this work-around.

Managing vCenter Resources

Before we dive into the final 10 questions I want to stress how important finding a team player that will work closely with storage and network admins really is.

Tip! Pay close attention for any sign that the person you are interviewing does not think he/she owns your vSphere Cloud!

Storage is everything!

16.            You’ve learned over the years there’s more to vSphere than server hardware. What other 2 resources are just as important as servers and need to be properly planned out when designing and building a vSphere Cloud?

Answer: Storage and network resources are crucial for all vSphere Clouds.

17.            VMware vSphere give you options. What are the 4 typical ways storage can be added to a vSphere?

Answer: Storage can be added via iSCSI, FC, NFS and local disk (including DAS).

18.            When setting up a new datastore how many VMFS file systems should be created per LUN?

Answer: The best practice is to only create 1 VMFS file system per LUN.

19.            What is the best plan for a storage failure that impacts multiple datastores?

Answer: Always have a backup of the VMs on a separate storage environment that can be used to restore the lost virtual servers.

20.            Your host is a beast, dual socket with 8 core CPUs and 192 GB of memory. How many virtual servers can be added to a 1TB datastore?

Answer: It depends on the size of the VMs and the performance of the storage. Higher performing storage can be filled to capacity but space should be left for data growth and snapshots. A datastore should never be allowed to fill up 100%.

Unpacking these VMware Storage Qs.

Managing storage in a vSphere is an important job that can get a junior vSphere admins in big trouble!

Regardless of the protocol (iSCSI, NFS, FC) it requires a certain skill set which only comes with time and experience. Every new ESXi hosts which is added to vSphere requires someone to know how to balance the workloads across various storage tiers: local disk, SATA, SAS and SSD (depending on the needs of the environments and applications).

Far too many new VMware admins have learned the hard way that “Not all storage is the same” and very quickly a junior admin can cause an outage on a business critical database server simply by vMotioning a VM to a datastore that looks empty.

Networking is everything, too!

21.            This is a tough VMware interview question created just for you. Name 4 things that happen on the VMkernel networking layer?

Answer: vMotion, IP storage (iSCSI/NFS), Fault Tolerance and Virtual SAN.

22.            You have a VCP so this should be an easy interview question. What are 2 ways a vSphere admin can separate traffic from distinct environments (ex. Production and test) on the same hosts.

Answer: Either by creating separate vSwitches using dedicated NICs or if NICs are not available by creating separate port groups using different VLAN IDs on the same vSwitch.

23.            True or false. A Distributed Virtual Switch is very much like a physical switch that detects which VMs are logically connects to each port and uses that information to forward network traffic. Hint: It is not used for monitoring and administration across a datacenter.

Answer: False. A Distributed Virtual Switch acts as a single switch across all hosts in a datacenter to provide centralized provisioning, administration, and monitoring of virtual networks.

24.            Another True or false. NIC teams are “normally” put in active/active mode to allow fail-over in the event of a hardware failure.

Answer: False again. NIC teams are normally put in active/standby mode to allow fail-over in the event of a hardware failure. You can use active/active but this would not be standard and would require port channeling at the physical switch.

25.            This is a trick question so think about it! How many physical NICs are needed in an ESXi hosts for hosting 25 virtual servers on iSCSI storage split between 2 diverse environments (web/app).

Answer: The answer is purely subjective. It depends on how much separation is needed for performance and the level of redundancy built into the design for hardware failure. At minimum, maybe 2 (1 for data and 1 for VMkernel) but more should be used.

And finally, let’s review the Network Questions.

Since 2007, I estimate 80% of all the VMware problems I have dealt with were network or storage related. That’s not to say it was the network or storage admins fault. No, in most cases it was the VMware admins fault for not communicating and properly calculating the requirements before he/she asked for IP addresses or storage LUNs.

Like I said in the beginning. These questions are not for practicing to take a VCP test. They are designed to gauge the level of experience someone who is applying for your VMware admin or engineer job has before you give them the keys to start managing your company’s jewels.