Small Changes

It’s amazing what a small change can do to improve your attitude and outlook.  This past weekend, my wife and I celebrated our 20th wedding anniversary.  In today’s culture, being married to the same person for that length of time is quite an accomplishment.  As I reflected on the past 20 years, I started to ponder what made it work for us.  My conclusion?  Small Changes.

To be sure, we are not the same people today that we were 20 years ago when we met.  We were both in the Army, stationed at Ft. Carson in Colorado, me a never married boy of 21 and her a recently divorced mother of two.  Our courtship was quick, taking just under 10 months to get from first date to wedding vows.   Six days after we were married, she left for Korea and I didn’t see her for almost six months.

In our first 18 months together, we only spent the middle six together when our Korean tours overlapped.  This was back before the internet, Skype, Yahoo! chat, or any other digital forms of communications were available, limiting us to weekly phone calls that were so long distance that $600 monthly phone bills were the norm, rather than the exception.

We’ve lived all over the country; Colorado, Kentucky, New York, Michigan, Hawaii, and Texas.  We have been together through sickness and health, richer and poorer, and through good times and bad.

Over the years we have learned to adjust to each other, to try new things together, and to be comfortable with each other.  As life delivers it’s next mission, we make our small adjustments and keep moving on.  Sometimes it’s good, sometime’s it’s bad, but we always work together to make whatever adjustments are necessary.

Every time we make the small changes, it seems to reinvigorate what we have together.  It’s never dull, it’s never boring.  For the last 20 years I have been lucky enough to be married to my best friend.  I wouldn’t have it any other way.

Is Outsourcing Good or Bad?

I’ve started reading through the textbook for my Multinational Corporate Environment Book and the first Point/Counterpoint is on the subject of whether offshoring is good strategy.

The “yes” side of the argument starts off with the point that if it succeeds in reducing costs, it’s good.  They make the statement that it isn’t much different than other means of employment changes, such as technology improvements or recessions.  Much of the remainder of the argument in favor of offshoring revolves around this central theme.  They wrap up by concluding that these are less skilled jobs and that their departure leads to an high-value jobs here at home.

On the “no” side, they start by stating that cost of goods never seem to go down, even when the cost of production does.  They make the argument that the displaced workers find it hard to recover from the even, and even if their is higher value jobs, they likely won’t go to the people being displaced by outsourcing.  Probably the only point that I felt was beneficial in either argument was the following on the no side:

Outsourcing merely diverts companies’ attention from taking steps to find innovative means of more efficient production

Personally, I think that both sides of these arguments are off base.  I am a firm believer in the The Three Rules:

      • Better Before Cheaper
      • Revenue Before Costs
      • There are no other rules

If your outsourcing strategy complements the Three Rules, then great.  For much of what I have seen over the course of my career, these types of moves have always been centered around cost savings, not anything that would add value to the product or service we are offering.  We look for cheap labor, but don’t realize what we end up missing; a sense of ownership.

One of the things that I love about working with the teams here in Austin are the sense of ownership that everybody feels for the products that we ship and support.  When something doesn’t work the way it is supposed to, everybody is on it until it is solved.  There is no kicking it over the wall because the developer is in Ireland or the QA resource is in India.  People are here and hold each other accountable.  They can walk up and discuss problems immediately.  There is no time zone difference to deal with.

Personally, I’m not against offshoring.  I think that it has a place, as long as its sole reason is not cost savings.  That only gets you cheap.

Reflection

A lot of the reading that I do these days is around leadership.  As with most things in my life, once I start down a path I can’t help but devour everything that I can find on the topic.  I’m not looking for a leadership style, that was developed courteous of good old Uncle Sam.  What I am looking for is primarily tips and tricks that leaders do to make their jobs easier and their leadership more effective.  While I was going through my morning RSS Feeds I came across an article on CIO.com by Stephen Balzac titled The Leader Who Didn’t Play Well With Others.  Talk about an eye opener.  For much of my career, I have been that engineer.

Regardless of any official title, I have always been a thought leader in every organization that I have been in.  The relentless drive for efficiency and excellence have only intensified as formal responsibility has been given to me.  While I feel that I have been generally open with my teams during a decision making process and allowed for their input, I never really thought much about the sense of ownership or the lack of commitment that stems from not allowing them to drive outcomes.

It was about halfway through the article that I started to see myself.  I never really thought about it in those specific terms, but as I think back (especially early in my career) I recall a few such demoralizing incidents.  The good thing is that it is easily fixed.  I have recognized for some time that I need to reign in some of my perfectionist tenancies, which is generally leads to the “My Idea Is Better” realm.  I also need to remember how I happy and focused I feel when I have a sense of ownership over what I am doing.  This is something that I need to instill in my team.

It just goes to show you that no matter what you think you know, you always have more to learn.

Embarking on a New Adventure

For the last few years things have been pretty busy, both personally and professionally. I’ve been the Scoutmaster of my son’s Boy Scout troop, the Finance officer for our local American Legion Post, taken over as the leader of a great team of System Administrators, and learned a lot of new technologies. Throw in some interesting (and for my co-workers, entertaining) family and personal adventures and I haven’t had much time to do anything.

My time as Scoutmaster is winding down, and with only one last big trip (Summer Camp) to go before I hand the reigns over to my successor (Thanks Ray!), it’s time for me to find something to do with the extra “hour a week” I’ll gain. So I’ve decided to go back to school.

I earned my Bachelor’s degree in 2007. When I started my degree in 2001 I already had 10 years in IT, so I decided to earn a degree in a field where I would actually learn something. I chose business management. Much better than a computer science degree where I would learn about three year old technology as if it were the cutting edge (although at the time I was still in Detroit and the many of the auto companies were still using OS2 Warp4 so it may still have seemed like cutting edge.)

Now I have decided to go back and get my MBA with a focus initially in Information Technology, but eventually I think I will also take the courses on Leadership. Since my military days I have been fascinated by the the difference in how the military approaches leadership versus the how the civilian world handles it. I expect that the next 18 months will be very challenging and very rewarding. I look forward to it.

Personal Security

A coworker asked me a question yesterday about what I use to protect my personal data.  He knew that I was using a password vault of some sort, but was curious as to what I did for files, etc.  What followed was a pretty lengthy discussion of the tools I used and why.  Most of the why on specific tools was due to their ability to be used across all the platforms that I use.  On a daily basis I use Windows 7, Mac, Linux, Android, and until recently IOS.

Password Save – Keepass
I have been using Keepass as my password safe for probably the last 5 years or so.  It provides a number of key features that I really wanted, including auto type and password generation.  It also does a pretty good job with some of the websites that make you enter a username first and then goes to a second page to enter your password.  The password generation tool allows me to make long, very complex passwords that I don’t have to remember.  As a result, outside of my keepass password I only have to remember about 10 passwords.  Compare that to the 200+ entries I currently have in Keepass and you can see the benefit.
I keep a copy of the database in SpiderOak Hive (we will discuss SpiderOak further down) so that it syncs securely across all of my devices.  That way, if I am every using a computer that is not mine and I need to enter a password, I can pull it up on my phone and type it in manually.  I also keep a copy of it on my Portable Apps USB drive I keep on my keychain.

Computer Backup – SpiderOak
When I started using SpiderOak a few years ago, there were only a handful of online backup providers.  Carbonite was the big dog, but there was also Mozy and Backblaze.  There were a number of things that drew me to SpiderOak.  First of course was that it worked on all the platforms, and right behind it was that you paid for space rather than computers.  With most providers you paid a fee for a computer and could backup as much as you want, but when you are only backing up a couple of important gigs, per PC I think SpiderOak provides a better value.  I pay one fee annually for 100G and can backup as many computers or devices as I want to it.  The thing that really sealed the deal was SpiderOak’s Zero Knowledge privacy (https://spideroak.com/zero-knowledge/).  It essentially means that they have no way of decrypting the contents of your files once they reach their servers.

Cloud Drive Encryption – BoxCryptor Classic
Dropbox, Google Drive, and Microsoft’s OneDrive provide a lot of free storage (I think I have around 100G between them), but none of them provide encryption for your files, other than in-transit.  For those services I use BoxCryptor Classic.    It allows me to encrypt locally before they are synced to the cloud drive.  While BoxCryptor only has a Windows and Mac client, you can you EncFS on a Linux system to decrypt and mount the drive.  The free version allows you to encrypt the files of one provider, but it doesn’t encrypt the file names.  I use the paid version that lets me use it with as many providers as I want as well as encrypting file names.

Local Disk/Device Encryption – TrueCrypt
There is still local drives and devices, and for that I use TrueCrypt.  TrueCrypt allows me to encrypt my system drives, portable USB drives, and provides portable containers for other types of files.  Right now, I only have one encrypted container on my laptop, and it contains all of my financial information (Quicken, Turbo Tax, Budget, etc) so that if my laptop is ever lost they don’t get my bank information as well.

Saltstack and Bonding – Part 2

(Part 1 can be found here)

After I wrote my custom grains and was able to get the information that I needed (IP, netmask, gateway) to configure my bonding and bridging, my first attempt to get SaltStack to configure it was with the salt.states.network module:

bond0:
  network.managed:
    - type: bond
    - enabled: True
    - proto: dhcp
    - mode: 802.3ad
    - miimon: 100
    - require:
    - network: eth1
    - network: eth2
eth1:
  network.managed:
    - type: slave
    - proto: none
    - master: bond0

eth2:
  network.managed:
    - type: slave
    - proto: none
    - master: bond0

br0:
  network.managed:
    - enabled: True
    - type: bridge
    - ipaddr: {{ grains['bondnet_ip'] }}
    - netmask: {{ grains['bondnet_mask'] }}
    - gateway: {{ grains['bondnet_gw'] }}

This only partially worked for me.  It would configure the interfaces and change the necessary files, but it would leave the system in a broken state with the IP running on both br0 and eth0.  Once I logged in to the host and restarted networking, everthing was fine, but I don’t what to do that for every system that I build.

Next I tried to do it using file.managed and Jinja templates to make the changes and restart networking:

/etc/sysconfig/network:
  file.managed:
    - source: salt://kvm/network
    - user: root
    - group: root
    - mode: 644
    - template: jinja  
    - defaults:
      hostname: {{ grains['fqdn'] }}

/etc/sysconfig/network-scripts/ifcfg-eth0:
  file.managed:
    - source: salt://kvm/ifcfg-eth0
    - user: root
    - group: root
    - mode: 644

/etc/sysconfig/network-scripts/ifcfg-eth1:
  file.managed:
    - source: salt://kvm/ifcfg-eth1
    - user: root
    - group: root
    - mode: 644

/etc/sysconfig/network-scripts/ifcfg-bond0:
  file.managed:
    - source: salt://kvm/ifcfg-bond0
    - user: root
    - group: root
    - mode: 644

/etc/sysconfig/network-scripts/ifcfg-br0:
  file.managed:
    - source: salt://kvm/ifcfg-br0
    - user: root
    - group: root
    - mode: 644
    - template: jinja
    - defaults:
      ip: {{ grains['bondnet_ip'] }}
      netmask: {{ grains['bondnet_mask'] }}
      gateway: {{ grains['bondnet_gw'] }}

/etc/init.d/network restart>/tmp/out:
  cmd.wait:
    - watch:
    - file: '/etc/sysconfig/network-scripts/ifcfg-br0'

This didn’t work very well for me initially. It would restart the network after it changed ifcfg-br0, but it would change it before the rest of the network files, so it would break as well.  To solve that I added “- order” to each of the lines to make sure that the processed in the order that I wanted them to and my problem was solved:

/etc/sysconfig/network:
  file.managed:
    - source: salt://kvm/network
    - user: root
    - group: root
    - mode: 644
    - template: jinja
    - order: 1
    - defaults:
      hostname: {{ grains['fqdn'] }}

/etc/sysconfig/network-scripts/ifcfg-eth0:
  file.managed:
    - source: salt://kvm/ifcfg-eth0
    - user: root
    - group: root
    - mode: 644
    - order: 2

/etc/sysconfig/network-scripts/ifcfg-eth1:
  file.managed:
    - source: salt://kvm/ifcfg-eth1
    - user: root
    - group: root
    - mode: 644
    - order: 2

/etc/sysconfig/network-scripts/ifcfg-bond0:
  file.managed:
    - source: salt://kvm/ifcfg-bond0
    - user: root
    - group: root
    - mode: 644
    - order: 3

/etc/sysconfig/network-scripts/ifcfg-br0:
  file.managed:
    - source: salt://kvm/ifcfg-br0
    - user: root
    - group: root
    - mode: 644
    - template: jinja
    - order: 4
    - defaults:
      ip: {{ grains['bondnet_ip'] }}
      netmask: {{ grains['bondnet_mask'] }}
      gateway: {{ grains['bondnet_gw'] }}

/etc/init.d/network restart>/tmp/out:
  cmd.wait:
    - watch:
      - file: '/etc/sysconfig/network-scripts/ifcfg-br0'

Once I added the “- order”, everything worked exactly the way that I wanted it.  My KVM server was build and ready to be added to CloudStack with the proper network configuration.

Saltstack and Bonding – Part 1

For the last week or so I have been working on creating a SaLt State (SLS) file to build my KVM servers. Most of the SLS file was straight forward, adding a custom repo for CloudStack, installing the necessary packages for KVM and CloudStack.  The most difficult aspect of building it revolved around bonding and bridging the interfaces.

First, a little bit about my setup.  I use a Kickstart server to build all of my hosts.  It builds every host to the same minimalist configuration, with the only additional thing that it does post installation is install SaltStack.  I then use SaltStack to do all of the post installation; packages specific to a server type, configuration files that are specific to an environment, etc.

My SLS file is for all KVM hosts, so I wanted it to take the IP address, netmask, and gateway  already assigned to the host when it was kickstarted and then reconfigure the server using the same information.  The first challenge that I faced was getting that information.

My first attempt was to set a variable:

- defaults:
  ip: {{ salt['network.interfaces']()['eth0']['inet'][0]['address'] }}

This worked for the initial build, but then all future state.highstate runs would fail  because there is no IP address assigned to the eth0 interface because it was now bridged.  Being new at SaltStack, I tried to find a way to do some sort of if statement to check if eth0 had an address assigned, but I could not find a way.

In any event, there wasn’t a good way for me to get the subnet mask or the default gateway so I decided to write my own grains to collect the information.  I’m still learning python so there is probably a much better way to do this, but was able to come up with this:

import socket
import fcntl
import struct
import sys
import os

def bondnet_ip():
    '''
    Return the path
    '''
    # Provides:
    # path
    return {'bondnet_ip': socket.gethostbyname(socket.gethostname())}

def bondnet_mask():
    '''
    Return the netmask
    '''
    # Provides:
    # mask
    try :
        iface = "br0"
        mask = socket.inet_ntoa(fcntl.ioctl(socket.socket(socket.AF_INET, socket.SOCK_DGRAM), 35099, struct.pack('256s', iface))[20:24])
    except:
        pass

    try :
        iface = "eth0"
        mask = socket.inet_ntoa(fcntl.ioctl(socket.socket(socket.AF_INET, socket.SOCK_DGRAM), 35099, struct.pack('256s', iface))[20:24])
    except:
        pass

    return {'bondnet_mask': mask}

# Get the default Gateway
    def bondnet_gw():
    '''
    Return the Gateway
    '''
    """Read the default gateway directly from /proc."""
    with open("/proc/net/route") as fh:
    for line in fh:
        fields = line.strip().split()
        if fields[1] != '00000000' or not int(fields[3], 16) & 2:
            continue

        return {'bondnet_gw': socket.inet_ntoa(struct.pack("<L", int(fields[2], 16)))}

Now I can pull them as grains in and set the variable that way:

- defaults:
  ip: {{ grains['bondnet_ip'] }}
  netmask: {{ grains['bondnet_mask'] }}
  gateway: {{ grains['bondnet_gw'] }}

This has gotten pretty long, so part 2 will focus on how I used the new grains that I created to build my bonded and bridged interface.

Update 11/1:  Part 2 can be found here

Four Levels of Maturity

There are a lot of written words on maturity models, but none of them have ever seemed to fit my vision for where an infrastructure should be. In my mind, there are 4 primary levels of maturity:

    • Ad-Hoc
    • Standardization
    • Automation
    • Orchestration

Ad-Hoc.  At the Ad-Hoc level, everything is a manual process.  Every systems build is unique.

Standardization. When we reach standardization, everything is still a manual process, but it is a documented, repeatable process.

Automation. From standardization we move into automation.  At this point, the documented processes have been used to build tools to automate system builds and managment.  Tools such as Kickstart, Jumpstart, CFEngine, Chef, and Salt are used to deploy and build systems to exact specifications every time.

Orchestration.  The final phase is tying all of the automation processes together.  Orchestration ties all of the various automation tools together into one process from beginning to end.

I currently fall between standardization and automation.  While my server builds and deployments are mostly automated, we still have some manual processes that need to be completed, mostly revolving around IP Management and server managment (physical and virtual).  My goal is complete orchestration.

Looking to Grow

As a father, Scoutmaster, and leader, one of the key messages that I have for people is to never stop learning. I’ve never been a big fan of the saying “you can’t teach an old dog new tricks”, and I become even less so as I have gotten older. For me personally, if you stop learning you might as well stop living. There is no real joy in being stuck in the same old.

Lately, my looking to grow opportunity is centered around automation. I have been a big CFEngine user over the last few years, but I have started to become a little frustrated with the complexity of the syntax as well as some of the changes that have come in more recent versions. Recently I have started to look into SaltStack, which so far seems extremely promising. I have also started looking into using Python to do my programming, especially around APIs.

While I still have a long way to go before I am any sort of expert, I plan on sharing my experiences here on these pages. Since my use cases are a little more complex, or at the very least rare, I have not been able to find a lot of information on how to solve them through the web so I plan to share my challenges and solutions here. Hopefully I can help somebody solve a similar problem down the road.

Let the journey begin. Each day is an opportunity to learn something new.

Rants and Ramblings of Austin IT Professional