Formal Test Approach
A formal test consists of running the CCS or TSC according to a documented test procedure, and recording the result in a test report, which we keep. To be released (appear on this web site), the software must pass these formal tests. The tests are largely automated, and are very thorough for monitoring, control and automation features. User interfaces are tested only insofar as they are used or explicitly tested.
The test documentation is "inspired by" the ESA ECSS-E-ST-40 standard for software engineering processes and covers 80% to 90% of the CCS requirements for most missions.
Test and Release Frequency
We aim to formally test both CCS and TSC at least once per 3 months, or as required by specific projects and contracts.
CCS and TSC are automated test systems and so naturally they are used for their own self-tests. We run the bulk of our tests automatically every night as part of the build process for all platforms. These do not require human interaction. If one of these tests fails, we find out about it the next day. A formal test is not started if there has been any problem with the automated test.
Nevertheless the formal test includes both the automated and user interaction; our formally run tests are executed and recorded by a human operator.
CCS, TSC and uNIS are available on 64-bit platforms. In addition, TSC is available in 32-bit as well for compatibility with certain data-source plugins and Tcl packages.
We formally test using Windows 10 and RHEL7. Day-to-day development is done on Windows 7.
Different projects are successfully using a variety of different versions (including Windows Server 2008, 2012, Windows 8, Windows 8.1).
Supported Linux distributions are RHEL7 and OpenSUSE Leap.
Windows XP is no longer supported.
Daily BuildThe daily build is, as the name suggests, an installer that has been built overnight from our configuration control most recent check-ins. We use it during our testing to confirm informally that a new feature is working or a bug is fixed. Our developers perform tests on their changes in a debug environment before checking in, so these builds are usually (but not guaranteed) stable. We use them to confirm that our changes still work in the release packaging that would be installed by a customer. They can be used if you need to check quickly whether we have successfully implemented a change or bug fix for you. The daily builds are automatically tested as far as possible without a user interface.
Note that we certainly do not recommend using such a release for flight operations or a critical test.