When I first came to the company I work for now, I was fairly new to the world of QA. I had come from 5 years as an IT Administrator and a couple years before that as a Technical Support Manager. The concept of testing software was brand new as well as something else I had never ventured into; Scripting.
One of my first tasks as a QA Engineer was to write a system wide test that would take our system back to front and run a battery of tests against it. This included 3 database boxes, several middleware machines, an .NET web app, several supporting test machines, a network appliance, and a client machine to simulate traffic through the network appliance. Once the test was written, it was my job to run it. This became my primary role during a development cycle. Take all the components, install the latest builds and make sure they play nice together. The first couple times was fine. Then one by one, other tasks were put on my plate. Eventually, the test I designed that took several days to run manually was becoming a pain in my backside due to time constrains. I approached my boss and suggested we automate it. Once approved, I took to the task of finding out how the heck I was going to get 30+ machines to work in concert with one another while at the same time test our product.
One of my coworkers, who some of you know as qa_warrior, is in charge of automated testing and the build system. At the time it was a myriad of python, vbscript, batch, perl, C#, and who knows what else. I thought to myself, my goodness, do I have to learn all this just to automate something?!? There has to be a better way. At my previous job, I heard of this revolutionary new scripting language that was suppose to blow the socks off of everyone. Monad. Worth a look I thought.
One book (Powershell In Action), tons of blogs, and a couple of Channel 9 interviews with Jeffrey Snover later, I was foaming at the mouth ready to bring this new found gem to my workplace. I approached qa_warrior with Powershell with reserved excitement since I was fairly new there. His response was “Just what I need, another scripting language to learn”. I handed him the Powershell In Action book and left it at that.
Days later, he comes back to me. “This stuff is REALLY cool! You can do this, and this, and that, and it ties into pretty much everything we have already done!” He was sold. Soon after that he went on vacation. Upon his return, he brought back the start of our new automation framework. It first started out as a simple xml file with data we may need during a test, a few libraries, some Cmdlets and some scripts that had useful functions in them to tie everything together.
Over the past year and a half, we have been expanding, refining and evangelizing what we lovingly call “FPTestLab”. What started with qa_warrior and myself, is now being worked on and used by the entire QA department. It is also being spread to other areas of the company like technical support, data services, development, and IT. Powershell is the mantra for many in our organization. (Of course there are those of us that live in the Linux world that swear by Perl. Looking at you John )
Today FPTestLab is tens of thousands (probably approaching hundreds of thousands soon) of lines of code that builds our product, deploys a test environment (again 30+ machines) through VMWare and PowerCLI, tests our product (we have developed our own testset/testcase/step/assert framework), reports on the results of tests and moves those results into SharePoint on a nightly basis. We have over 500 test cases written in pure Powershell that plug into FPTestLab. This may not sound like much but bringing disparate systems together like Quality Center, VMWare, SharePoint, and our own systems on a complex network was only possible due to Powershell and it’s ability to consume just about anything we throw at it.
With the release of Powershell Version 2, we have begun the process of moving our complex testing framework over to use the new remoting, background jobs, self documentation, events, proxy functions and modules. We first tried to make this move with CTP1. Yeah, I know. We were told not to by the Powershell team but the promise of remoting was just too good to leave alone. Unfortunately, it was not mature enough to sustain the workload we were throwing at it and had to go with nSoftware’s Powershell Server. Good product for simple V1 remoting.
Over the past two weeks I have taken my testset that was written in V1 and moved the code over to V2. The first milestone was “Does it run without code change in V2?”. Yes. Only 1 issue with ambiguous methods which is a known issue in the release notes. Next milestone was convert the remoting from nSoftware to Powershell V2 remoting. Today I checked in my last code changes and got a 100% successful pass with 100% Powershell remoting. 0 issues. This was a good day.
To more directly answer Jeffrey Snover and his request for what we have tested and on what platforms. Here is my best attempt at an answer.
- Windows Server 2003 SP2
- SQL Server 2000/2005
- Windows XP SP3
- VMWare vSphere 4.0
- SharePoint 2007
- Quality Center 9.x
- Cruise Control
- Advanced Functions
- Inline documentation
- Proxy functions
- Remoting – I can’t begin to explain how much we use this. 24×7 traffic between boxes. Connections up for up to 16 hours sometimes.
- Global functions, PSObjects, NoteProperties, and ScriptBlocks/Nested ScriptBlocks HEAVILY.
- Cross domain authentication
- HEAVY xml use
- Tons of web requests through system.web.httprequest .NET objects
- REST based web service calls
- HEAVY PowerCLI use
- Tons of in-house .NET libraries wrapped with Powershell
- Decent WMI use
- Use of sqlcmd for running .sql files and lots of calls through ADO.NET for straight queries
- Some use of direct win32 calls through Powershell and PInvoke.
- Calls to the SharePoint web services API
- HEAVY use of global vs. script vs. local scope variables
- Hashtables everywhere
- MD5 hashing then web requests of URLs (This one drove one of our engineers crazy)
I’m sure I am forgetting stuff but it’s definitely something we are proud of. Version 2 is opening up tons of new possibilities for us and so far has been rock solid with everything listed above on the XP/V2 combo.
Great work and thank you to the Powershell Team. You have made our jobs tremendously more efficient and I would gladly put my badge on the table again for Powershell given the chance.