F5 LTM iRule Testing

I've been working with F5 LTMs for a few years now, and over time the configuration has accumulated some significant technical debt. Well, the time has come to clean it up, and unfortunately the configuration needs so much rework that it will be simpler just to rewrite all of the logic. That's where a testing tool comes in.

To rewrite all or a significant portion of the iRules, I needed a way to test the existing iRule logic and capture a snapshot of which logic works and which doesn't so that when the logic is revamped, it can be tested again to verify that the same functionality exists. After searching online for a while I came to the conclusion that the tool I needed didn't exist, so I decided to write one. It's called irule-tester, and it's written in Bash and leverages Curl to make the web requests.

I know some of you are going to cringe about it being written in Bash.  Hey, it works.  I was even able to put in a command-line argument that allows you to switch out the testing output format.  The results can be color-coded, plain text, or dumped in TAP (Test Anything Protocol) format so that the testing can be easily dropped into Jenkins for automated testing!

Write all iRule logic in a way that it can be tested.

When I started writing irule-tester, it quickly became obvious that some of my existing iRule logic would need to be changed to support reliable testing. This presented a chicken and egg scenario because some of the iRule logic needed to be rewritten to support the testing, but I needed the testing to validate that the changes were successful and didn't break anything. Because of this, I now believe that all iRule logic should be written in a way that allows external testing and validation of each iRule operation. To get through this, I created a function in irule-tester that just logs when a given piece of iRule logic can't currently be tested. This allows us to at least get the majority of the iRule logic tested and marks the part we can't as areas to improve in the future.

Environment agnostic LTM iRule methodology.

Another issue I've stumbled on over my time with the LTM is the complexity of performing periodic environment refreshes. From time to time, it becomes necessary for us to refresh our non-production environments from production, or to create a project environment based off of production and this is always painful. More specifically, it's difficult because each iRule contains environment-specific object names (classes, websites, pools, etc).

To solve this problem, I came up with a way to standardize the configuration across all of the environments in a way that allows true iRule code promotion across environments. The method involves creating an environment-specific 'variables' iRule which just sets variables for each object that is referenced in an iRule. This allows you to have a single iRule that sets variables that are different in each environment, and then have supporting iRules that carry out the traffic management logic and leverage those variables in place of the environment-specific objects.

You can see an example of what I'm talking about by looking at rule STD01 of my iRules standards on Github.

Why a blog?

I wanted to document the ups and downs of implementing Chef and how we are creating culture change in my IT organization.  I'm hopeful that by recording our experiences (good and bad) it will help others with their own journeys, and through reader comments, help us with ours.

Who am I?

Hey, I'm Jesse.  I've been in the IT industry since 1998 and I've worn hats of many shapes, colors, and sizes.  I've lead a migration from Banyan Vines to a Windows server infrastructure while enlisted in the Marine Corps, worked in an incident response center for the Navy, and managed and supported large classified networks at several bases for the Marine Corps.  At my current gig, I spent a few years as an IT security professional dealing with SOX audits and PCI compliance, and then finally landed in my current team, managing load-balancers, Linux hosts, and application containers.  The most recent shift in my job description has me focused on rolling out Chef, adopting Test-Driven Development principles, and being an evangelist for the DevOps movement.

If you want to learn more about me, you can check out my profile on LinkedIn.

So what am I up to?

My organization has been interested in Chef for a little over a year now, and the task of making our interests an implemented reality has been handed to me.  We've also started actively pursuing an adoption of a DevOps philosophy for about 6 months.  And after years of following waterfall development practices, our IT organization recently decided to invest in Scrum to reduce risk, improve product quality, and accelerate our product enhancement throughput.  We have a lot of change and a long path ahead of us, but we're on our way!

On my team, we're finally starting to make some ground on our roll-out of open source Chef, and I'm finding that when I wake up each morning I can't wait to get to the office.  In fact, I'm actually starting to look forward to my hour-long commute to the office so I can use that time to sort out my ideas for reaching the day's goals.  I'm definitely a fan of Chef so far, and I'm certain it's going to be an indispensable tool in our box going forward.

We've also started to build a testing framework around testing our load-balancer configuration and have started investing resources in continuous integration.  Both of these efforts excite me just as much as Chef because I think they will begin paving the way for a completely new way to manage our infrastructure.  There will definitely be future posts about these two topics in the near future.

What to expect in future posts

• A collection of links to some of the most helpful information I've come across as it applies to Chef, Continuous Integration/Deployment, and DevOps in general
• A complete explanation of our cookbook architecture and development process
• A first look at our F5 iRule testing framework
• How we are encouraging independent thought and critical thinking to solve problems