Really how you will sell that unclear automation? You need to prepare the best proposal, solution, surveys, fact sheets, comparisons and of course happy case studies. Once your presentation is polished out – you are ready to present it to your customer, stakeholders or, in general, decision-makers.

In fact, people like and want real cases, they want to hear that you or whoever else did the same for some of “Fortune 100″ companies. Moreover they want to know how, why and for how much they had got that. Your potential Clients want to know risks, issues, lessons learnt and benefits of your proposal from other guys who had got the experience. They would like to ask them but usually cannot. And my target as seller is to forestall them, I’m trying to show them case studies.

Case studies could be personalized or anonymized. The last one appears as pattern (proper or anti) and still serves as message reflecting a real-life situations and experience. Someone may say it’s tricky to show anonymous reviews/surveys. I’d rather agree in most cases. But what if your market is so limited and closed for sharing information? Answer – nothing but tricky hinting. What if your study covers a huge group of customers/users? In this case you can extract some representative groups with the same obtained result/capabilities/behavior/model or generally segment a target audience groups. I apologize for my passion, but statistical analysis is what I like as well as test automation.

The fortunate case studies from me are cheerfully given but please consider them as patterns of proper or happy software automated testing.

Case #1 Various environments support, compatibility with other software, components…
Test automation framework is designed as platform-independent. So that test engineers easily run test sets against various environments and combinations of installed software in order to test compatibility and fault tolerance under different real-life usage scenarios. For Web systems it could be running on various browsers and versions, with different combinations of installed plug-ins and add-ons on different set of OSs

Case #2 Multiple Localization testing
Automated tests are not tied to GUI visible captions, it means recognition is implemented by checking invariant attributes (ID, location binding in hierarchy, appearance index…). However localization validation is not nightmare. For that, localization database to be designed which feeds data for checking localization attributes in GUI. Basically you can run a verification of GUI either at runtime or as batch-state-checkpoint. The last one is better as it brings a separate level of testing called GUI localization testing. Other words, whole automation scope is split out on functional testing (the same code without parameterization will test product behavior for all locales) and GUI testing (will just verify appearance using dynamic parameterization from localization DB).

Case #3 Unstable GUI but stable API
Some companies made and forced a decision of testing by pulling API. It means end-User UI is not touched upon testing. The drawback is that the approach will not cover real user interface but test coverage and stability will be incomparable to functional testing through GUI. Yet other advantages are less expensive overall automation, quicker execution and integration with codebase of AUT.
However project decided to have GUI testing anyway. For that, test automation guys came up with GUI testing as additional verification layer which was inlined to API tests. It’s really simple and clever – call or change something in application by API, then verify result again with API and additionally and independently in GUI.

Case #4 Business needs to support software As-Is-It-Now for a few years but fixing bugs will be applied by patches, CFs and so forth
This case is a usual for COTS and enterprise systems. Support requires testing and main effort is regression. Business should make sure that fix does not affect existing functionality. This is very risky to ship critical fix without a round of regression testing. Here is automation can come as cheap or almost free solution (someone just have to run and review results)

Case #5 System does not evaluate significantly between releases. New release is just old one + some fixes + additional features
It’s again nice point to leave regression for test automation. The rest effort is spending on new features and bugs. From this point automation becomes very effective as returning investements on automation is continious process. It’s like planting – initially you just spend after a while you gather in the crops but a little effort on support still have place.

Case #6 Often commits, often builds
Making test automation as part of continuous integration in aggressive project environment for providing ASAP and often feedback about build quality. Some tests could be run as sanity checking against each code pre and/or post commit. The rest suite is run upon each new build arrival

Case #7 Mission critical project shipping
In this case stakeholders may agree to have automated tests as part of delivery package. For instance a new build is not accepted if test automation does not yet ready as planned for a build/release and if whole test suite does not result a Green (passed) result. This is really challengeable. Just imagine test automation is a inseparable part from a product. I like this approach especially for self-tests and embedded automated tests. Take a look at the example

Case #8 Our system is a legacy back-end data processing and some unstable (under continuous development) to changes UI. Almost no money on test automation
under this case goes well a claim “We just need to support that unknown legacy system with small money”. Well, if small effort, consider automation of system logs parsing, checking datatbase consistency, reliability and data integrity by running queries periodically, automatic memory dumps processing (learn in depth), build in a memory leaks checkers, continous health monitoring of environment

Case #9 Test automation is looks like chaotic but it’s smart distributed one indeed
Imagine you have a cloud of various environment and you have a smart dispatcher which decides which tests, where and how to run based on its acknowledges about the cloud. This is amazing abstraction that presents accelerated test running relying on wide spread test stands and acknowledged by stands load/usage. System completely controls execution, engineer just to check results out in real-time. IMO it’s state of art!

Case #10 Integration with 3-rd party systems
Test automation daemon may sit as listener somewhere in integration point. Automated test should understand native communication language, so listening traffic and verifying result is just a deal. Let say we can create Web services testing (SOAP/WSDL) which utilizes the same data schema as it incorporated in web service itself. Test can proxy, interpret, stub or listen data.

Case #11 Security is a priority number one on Agile (shipping often) project
It’s not a secret security testing can be perfectly modeled and automated. The core of security tests will work against most of web projects (XSS, SQL injections, session hijacking, tampering, sniffing and so forth). If a project runs short iterations and releasing frequency is high, security tests could be run over and over again against pages and contained. For instance test engine walks through all web pages and then feeds basic dangerous scripts (JS, SQL, VBS, sh) to all fields, then triggering form submission. You can even simulate DOS attacks and brute force password hacking. Although there are already many tools which will do it for you, thus our job is just to automate execution and reporting.

Is it enough? It does not but I need to stop this story in a fear the story will never finish. Actually any project can produce different cases and practices since inviting a best fit to project solution demands creativity and out of the ordinary thinking. Patterns themselves don’t require using them as is, they are rather reusable good practices which could be ported and adopted if it make sense in your project. Using combination of patterns may bear new patterns or a single pattern of level higher than reused ones.

It would be very nice if we will share specific cases, patterns and lessons learnt of fortunate test automation. I just doubt why bloggers running test automation exchange their successes and achievements too rarely for the industry; instead we can found risks, problems, obstacles and why-dont-run-low-ROI-automation mostly in the internet.

• Easy code maintenance: to call an object located on a page, just write something like this:

var conext = Common.GetAppTop();

var res = recognizeControl ( w.Panel(0).Panel(0).Panel(0).TextNode(0).Link(0), ["ObjectType","innerHTML"], ["Link","Log in"], context);

• Robust change management: if object direct path was changed on a page, test log will report an error but the control will be found indirectly either within a particular html element (e.g. cell, row) or within whole web page.
• Automatic checking of correspondence between properties values of direct object path and specified object properties for indirect search. In this way, we achieve discipline of test development process and may reveal hidden mismatches. For instance, this code can reveal that “Log In” link has changed innerText property to “Log Innn” but test will continue work with this control because direct object path is correct.

In conclusion, the proposed approach of object recognition provides checking relevant properties and expected object location simultaneously. The code snippet of the proposed pattern is presented below.

The provided code should be saved as .vbs, e.g. HostsHealthMon.vbs

To call the script, run in CL this (or make cmd file to wrap this call):
cscript.exe HostsHealthMonitor.vbs>>\\YouRStorage\HostsHealtMon_Log.txt

Option Explicit

Dim Hosts, RootUser, RootPwd, forceAlarmCPU, forceAlarmHD, forceAlarmRAM, forceAlarmConnection
Dim ptr, partitionRedLimit, CPUUsageRedLimit, RAMFreeRedLimit, IterationDelay, resetCount
Dim lastCacheLogPath
Dim httpServerUrl

'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
' Description:    The Sub is responsible for initialization and reseting (on beginning of each cycle) settings
' Populate this set of variables With your data as descibed in comments for each block to configure run
'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
Sub InitVars ()
  
   'Machines for checking details: host name or IP, User Id and password
    Hosts = Array("host1", "10.90.23.255", "host02", "host03", "Server01")
    RootUser = Array("Admin", "User", "Admin", "User01", "TestUser")
    RootPwd = Array("123west", "123west", "123west", "123west", "123west")
   
    'Flags reseting
    forceAlarmConnection = Array (True, True, True, True, True)
    forceAlarmCPU = Array (True, True, True, True, True)
    forceAlarmHD = Array (True, True, True, True, True)
    forceAlarmRAM = Array (True, True, True, True, True)
   
    'Threasholds
    partitionRedLimit = "5"
    CPUUsageRedLimit = "95"
    RAMFreeRedLimit  = "50"
   
    'Settings for defining frequency of the reporting
    IterationDelay = 100
    resetCount = 5
   
    'Only latest results will be stored in this file (no history). E.g. a host had a problem with CPU load in previous full iteration and now it soes not, then the log file will contain nothing about that
    lastCacheLogPath = "\\hostStorage\HostsHealtMon_Cache.txt"
    Call Delete_File ( lastCacheLogPath )

End Sub

'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
' Description:    removes specified file
' Parameter       sFile - name of folder to be removed
'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
Public Function Delete_File (ByVal sFile)
On Error Resume Next
    Const c_Proc = "Delete_File"
    Const c_DeleteReadOnly = True
   
    Dim iResult, oFSO

    ' Define default error code
    iResult = True

    Set oFSO = CreateObject("Scripting.FileSystemObject")

    ' Check if specified folder exists
    If Not oFSO.FileExists(sFile) Then
        iResult = True

    ' The file found
    Else
       
        ' Remove specified folder
        Call oFSO.DeleteFile (sFile, c_DeleteReadOnly)

        ' Check operation result
        If Err.Number <> 0 Then
            Wscript.Echo "Can't remove the file '" &amp; sFile
            iResult = False
        Else
            iResult = True
        End If

    End If ' The folder found
    Set oFSO = Nothing
    Delete_File = iResult

End Function

'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
' Description:    Writes in file
' Parameters:    objFile - file object
'                strText - text to write
' Result:        True / False
'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
Public Function FileWrite(ByRef sFile, ByVal sText)
    Dim iResult
    Dim objFSO, objFile
    iResult = True
    Const ForAppending = 8
   
    Set objFSO = CreateObject("Scripting.FileSystemObject")
       
        Err.Clear
        Set objFile = objFSO.OpenTextFile (sFile, ForAppending, True)

        If Err.Number <> 0 Then
            Wscript.Echo "Error appending to file " &amp; sFile
            iResult = False
            Err.Clear
            Else
                objFile.WriteLine sText
                objFile.Close
           
        End If
               
    Set objFSO = Nothing
    Set objFile = Nothing
   
    FileWrite = iResult
End Function


Sub Main()

    InitVars ()
   
    Do While True
        On Error Resume Next
        ptr=0
        resetCount = resetCount - 1
        For Each Host In Hosts
            Set objSWbemLocator = CreateObject("WbemScripting.SWbemLocator")
           
            Dim alarmStr, tmpStr
            alarmStr = ""
            tmpStr = ""
           
            Err.Clear
            Set objWMIService = objSWbemLocator.ConnectServer(Host, "root\CIMV2", RootUser (ptr), RootPwd (ptr),  "MS_409")
           
            If Err.Number <> 0 Then
                'Wscript.Echo "Warning establishing RPC connection with " &amp; Host &amp; ": "&amp; Err.description
                tmpStr = Date &amp; " " &amp; Time &amp; " " &amp; "Error: Unable to establish RPC connection with " &amp; Host &amp; ": "&amp; Err.description
                If CBool(forceAlarmConnection(ptr)) = True Then
                    Wscript.Echo tmpStr
                    alarmStr = alarmStr &amp; " <br />" &amp; tmpStr
                    forceAlarmConnection(ptr) = False
                End If   
            End If
           
            'Wscript.Echo Date &amp; " " &amp; Time &amp; " " &amp; Host
           
            'Show free space on logical disk
            Set colItems = objWMIService.ExecQuery( _
                "SELECT * FROM Win32_LogicalDisk",,48)
            For Each objItem In colItems
                'Wscript.Echo "FreeSpace: " &amp; objItem.FreeSpace
                If objItem.FreeSpace <>"" Then
                    If CInt(Left(objItem.FreeSpace, Len(objItem.FreeSpace)- 9)) < CInt(partitionRedLimit) Then
                        tmpStr = Date &amp; " " &amp; Time &amp; " " &amp; "ALARM event - Free space on a partition at : " &amp; Host &amp; " = "  &amp; Left(objItem.FreeSpace, Len(objItem.FreeSpace)- 9) &amp; " GBs"
                        If CBool(forceAlarmHD(ptr)) = True Then
                            Wscript.Echo tmpStr
                            alarmStr = alarmStr &amp; " <br />" &amp; tmpStr
                            forceAlarmHD(ptr) = False
                        End If
                    End If
                   
                'Else
                '    forceAlarmHD(ptr) = True       
                End If
            Next
           
            'Show CPU load percentage
            Set colItems = objWMIService.ExecQuery( _
                "SELECT * FROM Win32_Processor",,48)
            For Each objItem In colItems
                'Wscript.Echo "CPU load on: " &amp; Host &amp; " = "  &amp; objItem.LoadPercentage
                If CInt(objItem.LoadPercentage) > CInt(CPUUsageRedLimit) Then
                    WScript.Sleep(3000)
                    If CInt(objItem.LoadPercentage) > CInt(CPUUsageRedLimit) Then 'Double check after 5 sec
                   
                        tmpStr = Date &amp; " " &amp; Time &amp; " " &amp; "ALARM event - CPU usage on: " &amp; Host &amp; " = "  &amp; objItem.LoadPercentage &amp; " %"
                       
                        If CBool(forceAlarmCPU(ptr)) = True Then
                            Wscript.Echo tmpStr
                            alarmStr = alarmStr &amp; " <br />" &amp; tmpStr
                            forceAlarmCPU(ptr) = False
                        End If
                       
                    End If
                   
                'Else
                '    forceAlarmCPU(ptr) = True       
                End If
            Next
           
            'Show Memory MB free space
            Set colItems = objWMIService.ExecQuery( _
                "SELECT * FROM Win32_PerfFormattedData_PerfOS_Memory",,48)
            For Each objItem In colItems

                If CInt(objItem.AvailableMBytes) < CInt(RAMFreeRedLimit) Then
                    WScript.Sleep(1000)
                    If CInt(objItem.AvailableMBytes) < CInt(RAMFreeRedLimit) Then 'Double check after 5 sec
                   
                        tmpStr = Date &amp; " " &amp; Time &amp; " " &amp; "ALARM event - RAM free space: " &amp; Host &amp; " = "  &amp; objItem.AvailableMBytes &amp; " MB"
                       
                        If CBool(forceAlarmRAM(ptr)) = True Then
                            Wscript.Echo tmpStr
                            alarmStr = alarmStr &amp; " <br />" &amp; tmpStr
                            forceAlarmRAM(ptr) = False
                        End If                       
                       
                    End If   
                   
                End If
            Next
       
        'Wscript.Echo alarmStr
        If alarmStr<>"" And alarmStr<>" <br />" Then
            Call FileWrite(lastCacheLogPath, alarmStr)
        End If
       
        Set objSWbemLocator = Nothing
        ptr = ptr + 1
       
        Next
       
        WScript.Sleep(IterationDelay)
       
        If resetCount <= 0 Then
            Call InitVars()
        End If
       
    Loop
   
End Sub

Call Main

1. Prototype or simple model
2. Design as skeleton
3. Design refactoring
4. Some stuff is implemented
5. Refactoring should be done over new stuff
6. Code and design are ready
7. But code and design fit badly (inefficient memory management, resource leaks, security flows, unstable run, weak code support, improper interfaces and technologies, code smells…)
8. Run new refactoring. Dangerous and nightmare as hard!

* This imagination from standpoint of refactoring only. Change requests, new features and bugs and emergency wishes can appear between any of listed steps! Some projects recursively call Step #1 along any Step #X. Better case if you enjoy a few iterations 4-5-4, 6-7-6 or 7-3-7

The tool for refactoring is refactoring
The standard simple approach is to collect all code issues in defect tracker, assign to a proper player, come up with refactoring method, define expected results and benefits, estimate cost of fix and plan it. Once a refactoring item is ready – it should be tested inside and outside (Integration), unit tests (if exists) should give green (or rework them to meet with code changes). The last step is integration to production, QA testing and verification that “Refactoring is took place and results in expected out”

Usually refactoring is a step after code review or informal design/code inspection. The motivation to run refactoring should be as inside as outside, i.e. inside – when lead developer or architect initiate code review process and outside – when QA or customers or management have complains on product quality caused by bad design or coding.

Why refactoring is painful? Simply, more works done behind – more risks and potential affecting points on each refactoring item. And higher cost of refactoring. The same as for defects discovering and fixing – rate is continiously rising to production. Do you remeber this curve?

Consequently, frequent code review – less painful refactoring and less cost totally on refactoring. The general motivation of refactoring should be improve quality attributes of software, don’t be driven by this task as something fancy or trendy.

Now, it’s time for technical matter – some refactoring methods most used in industry. The good list of refactorings is listed here (http://www.refactoring.com/catalog/). The list is clearly complicated and easy to understand for those who are familiar with refactoring and coding practices, almost all links show up briefly meaning and how to apply with code excerpts and or UML diagrams. Most of the items can be fit and applied to automated testing either your code in scripting languages or real object oriented languages.

Basically we can break down refactoring approached to 2 groups: running top-down and running down-top. The first presumes review and improvements from highest levels (architecture and design) lowering to lines of code. The second method – is vice versa. Refactoring is not only finding and fixing issues in code pieces, it’s also about designing proper patterns up to level of general architecture. Some time ago I made a post on patterns applied for automated testing 20 Essential design patterns for automated testing, however we can use it to revise the current design.

http://en.wikipedia.org/wiki/Theory_of_errors

http://en.wikipedia.org/wiki/Errors_and_residuals_in_statistics

Though, the notion normal (Gaussian) distribution appears potentially helpful for s/w project planning and analysis, but it’s strongly recommended to play with this tool carefully and what more important – with statistically sufficient and significant data set. Yes there is a constraint – the project should be big enough. However, does it really important to forecast new defects and persistent ones on a project having 5000 SLOC? Don’t think so.

Obviously this theory may help to:
- estimate volume of non-covered/discovered defects. By this – rebuild test plan to achieve proper test coverage.
- predict volume and distribution of newly revealed product defects. By this – come up and/or numerically adjust with project sign off date. Somehow metrics could shift project release or may play start point of more resources negotiation.
- model efficient test automation (functional and unit) coverage. By this – achieve high ROI on automation.

All you need is to:
- select a sampling variable. In our case likely it is number of valid reported defects, e.g. weekly/daily. It depends periods you are going to operate on.
- calculate mean over this variable
- calculate variance of this sampling variable
- Then build and graph Probability density function (PDF)

This graph as well as mean and variance calculations are easy task using Excel formulas and graphs. The resulting graph should be build together with original sample in time series (histogram). The view of the curve itself shows “normality» of this sample. Overlapping of two graphs shows divergence of normality with real state. But don’t hurry to make judgments on this curve it’s just tip for you to feel confidence of project control. E.g. the curve with open tail (end in time series) may signal that testing should be prolonged as there are undiscovered defects as expected statistically.

The next advanced application is prediction. To make it you need to build either ideal normal distribution or use existing one. Then the restore function will give you remaining sample sub-set (show future). So that, you may say how many defects will be found for example week by week. Or what functionality needs to be tested with more effort

Finally to calculate precision over your calculations you have to come up with confidence intervals of your observations.

1. Test automation is just spending money without return
2. Automated testing is always profitable. Test automation fits to any kind of project
3. Unit testing is better and finds more defects then functional automated testing
4. Record-playback is automated testing
5. Any person with IT background can be auto tester
6. It’s easy to combine both manual and automate dtesting in a single role person
7. Test automation ROI is impossible to predict and calculate
8. Test automation estimation is not real activity
9. Test automation completely differs from s/w development
10. Bad developer can be good enough for automation
11. Developers can effectively develop functional auto tests
12. Auto testing is not about QA
13. Auto testing should not be a part of overall QA and QC
14. Manual testing coverage is preferable on any project/functionality and any project phase
15. Nobody should rely on automated testing
16. It’s impossible to create robust and efficient auto testing at all
17. Everything can be automated
18. Is auto testing a sort of machine intelligence? Though, I don’t trust that
19. Test automation should be capable to predict any system and environmental outcomes. Whatever issue occur – auto test suite should remove a problem and/or think out a new workflow path, recovery scenario, invent workaround from a scratch
20. Test automation is about writing a few code lines which result in testing major functionality. It’s easy!

I wish to hear more myths and strange points you ever met on test automation. Let’s share yours here commenting to this post.

Try it now!

]]> http://blog.bugpub.com/2012/05/10/private-bug-tracker-easiest-to-embedded-and-engaging-sites-vistors/feed/ 0 http://blog.bugpub.com/2012/05/09/voice-of-community-what-are-we-up-to-on-linkedin/?utm_source=rss&utm_medium=rss&utm_campaign=voice-of-community-what-are-we-up-to-on-linkedin http://blog.bugpub.com/2012/05/09/voice-of-community-what-are-we-up-to-on-linkedin/#comments Wed, 09 May 2012 12:15:18 +0000 bugpub http://blog.bugpub.com/?p=471 Check out some of our votes and polls which we run for fun!

The last two guys – security and safety – i believe are proper candidates to be automated. Why? Because of security flows can be introduced while any development and maintenance phase. You might polish out your s/w from security flows but new change request can bring a defect in design which may subsequently make your s/w unsecured (e.g. improper handling cookies or turning off fields validation common mechanism)

Such tests like SQL injections, XSS, session hijacking, cookies substitutions, overflows (as negative tests) and unvalidated I/O, access and permissions control can be easily simulated by test automation and can be run repetitively.
Btw, denial of services can be modeled with load, stress and volume testing.

The resource owasp.org provides tons of info about security and hacking on Web. You can find there lots of helpful data.
They have a comprehensive list of potential vulnerabilities in s/w:
http://www.owasp.org/index.php/Category:Vulnerability
and a category of attacks:
http://www.owasp.org/index.php/Category:Attack
I strongly appreciate this society for their job!

Handy add-ons to Firefox can simplify your process of developing and running hacks. The simple list of essential tools:
Tamper Data
SQL InjectMe
XSSMe
View Cookies
CookieSwap
HackBar