Test Driven Development (TDD) as a concept is widely used in most areas of software development. Software developers using TDD start with writing test cases, based on what the software or piece of code is expected to do, then write the code, so that the test passes and they keep repeating these steps on a regular basis. This concept enables developers to find issues at the earliest possible phase when developing new features, so they will be in the know of any issues before merging the changes into the existing code base. A large number of studies prove that writing tests is crucial when developing software and will result in a decrease in production issues and of course less headache with fixing bugs.
In an ideal world, this should be no different in the BI space. BI developers not only visualize data but they also make a lot of data transformations, implementing various business logic into the visualization layer, blending data from multiple sources, etc. This simply results in a very high risk of potentially showing data incorrectly.
Test Driven Development steps in general purpose programming languages
In Business Intelligence, however, due to various reasons like the heavy reliance on third-party software solutions such as Tableau, Power BI etc., the different skillset of BI developers and the nature of BI dashboards, using Test Driven Development would be very complicated. Even the most simple test cases are very difficult to set up with traditional web frameworks.
Let’s say you want to check if a particular filter exists on a Tableau dashboard. Checking it will be difficult once you realized that Tableau renders the page content dynamically full of HTML <canvas> elements where CSS/XPath selectors are not working, hence you can’t interact with these objects on the page. Testing will quickly get even more complicated once you start implementing business logic into calculated fields/dimensions and want to validate if these figures are calculated correctly. But leaving these objects untested sooner or later will cause pain and a increases the risk of production issues on a business critical dashboard.
Prevailing test frameworks, like Selenium or xUnit are designed for web applications or general purpose programming languages and not compatible with these BI solutions, so BI Developers don’t have many testing tools to rely upon.
With a variety of flexible test cases, with Kinesis CI you can implement Test Driven Development for Tableau. Kinesis CI is using the Tableau JS/REST APIs and further internal communication methods to interact with the vizzes where web testing frameworks are not efficient. Moreover, it comes with built-in data comparison tasks that are mandatory to validate the figures shown on the dashboard.
In the below blog post, I’m presenting the concept of TDD and show you a detailed example on how to develop a Tableau dashboard, using the TDD concept. Every Tableau professional who’s developing dashboards for decision makers should heavily rely on automated testing for their dashboards, to make sure they work as expected, and existing functionality is maintained after rolling out changes.
TDD for Tableau with Kinesis CI
Using Kinesis CI, the Test-driven Development cycle with Tableau looks as follows:
Test Driven Development steps in Tableau with Kinesis CI
This steps are similar to the ones in the original TDD lifecycle but there are a few small differences:
You will need to use two tools at the different stages. Tableau Desktop/Server as usual and Kinesis CI as an additional tool to design and run test cases for Tableau
Make the Viz is equivalent to the standard Write Code step
Pimp to Viz is equivalent to the standard Refactor step
Real life TDD example for a Tableau Dashboard
If you work in an Agile environment, you most probably will get the requirement as a user story, something like the below:
As the Chief Financial Officer I want to have a Tableau dashboard that shows the daily sales in a chart with Year on Year figures in a selected period and country So That I will have a quick insight into how the business is progressing on a daily basis
Let’s say you or a business analyst discussed this in more details and they made a draft how they can imagine the dashboard in action
First draft of the dashboard
You were thinking of testing and also defined four basic acceptance criteria:
Should have one sheet with the name ‘Daily Sales’
Should have a country filter with three values: ‘United States of America’, ‘United Kingdom’, ‘Australia’
Should have two date parameters for ‘Start Date’ and ‘End Date’
Two figures always should be on the chart: ‘Total Sales’ and ‘Total Sales Previous Year’. Both of them should always be greater than zero as a day with negative sales is not feasible
There are quite a few options to make mistakes, when the Tableau dashboard is developed, for instance, the filter is missing, no YoY figures, wrong color coding, or malfunctioning calculated fields. To prevent these mistakes from occurring, you should write test cases that verify, if the acceptance criteria are met.
Step 1: Write Failing Test
Traditionally once you get the requirements, you start developing the dashboard, however, with Test Driven Development you need to define the tests first. In Kinesis CI, in Functional testing, every test case is made up by a series of tasks, that are run one after another by the automated test runner to simulate the actions of a user as a test case. In this example, we will create functional tests with a few steps to cover every acceptance criteria: Login to Tableau, Open Viz, Assert Worksheets Exist, Assert Parameter Equals, Assert Filter Equals and Assert Data Rules.
Login to Tableauand Open Viz as preparatory tasks, to simulate a user navigating to the relevant Tableau dashboard. The Assert Parameter Equals and Assert Filter Equals tasks will verify the existence of Start/End date parameters and the country filter with the required values: United States of America, United Kingdom and Australia. Finally, the Assert Data Rules task verifies if ‘Total Sales’ and ‘Total Sales Previous Year’ are both larger than zero.
Series of selected tasks will test the functional acceptance criteria
At this stage, you will save the test and push the Kinesis CI project into your git repository.
Step 2: Run and Fail Test
Running the test first time validates that the test itself is working correctly. It should fail because the dashboard doesn’t exist, as we haven’t started to build this dashboard.
First failing test run
Step 3: Write Code and Pass Test (Make the Viz)
OK, now it is time to open your Tableau Desktop and develop the dashboard. Based on the requirements you will end up something like this:
First version of the viz focusing only to the functional requirements
In this step we should focus on the functional requirements and we need to make sure it meets the acceptance criteria. We should not spend too much time at this stage on cosmetics and fancy layouts. We can do that a little bit later. Ideally, once the viz is developed, all of the above test cases should pass, verifying that the required user criteria are met.
Step 4: Ensure all old test cases Pass
In this step we need to run every test case previously done, making sure the new features are not breaking the existing ones. At the moment we only have one test case, hence it is not adding too much extra, but this step will be a crucial one going forward.
Passing test cases on a functionally working dashboard
Note that the Assert Data Rules task allows you to create advanced data validation rules by excel like formulas that reference Tableau dimensions and measures. To validate the sales figures from this year and last year we added two formulas. Both value should be greater than zero but null values are allowed. Every formula returns TRUE or FALSE values and if it false then the test will fail too. The underlying data exported from the Tableau workbook at test run time and validated for every row.
OR([SUM(In Scope - During - Amount)] > 0, ISBLANK([SUM(In Scope - During - Amount)])) -- Formula to validate sales amount this year
OR([SUM(In Scope - LY - Amount)] > 0, ISBLANK([SUM(In Scope - LY - Amount)])) -- Formula to validate sales amount in the previous year
Tests are green so we can proceed to the last step.
Step 5: Refactor (Pimp the Viz)
At this stage you can ‘pimp your viz’ and improve the layout for a visually pleasing Tableau dashboard, that is in-line with any formatting requirements of your organization. Every test should be green at this stage too and if needed you can add an extra Assert Image Equals task in Kinesis CI test case to validate the layout of the dashboard.
Functionally tested and nice looking daily sales Tableau dashboard (Viz inspired by Eva Murray’s Tri My Data blog)
Every test should be green at this stage too. Next time when you need to modify something on the dashboard you will REPEAT the entire flow so you will go back to point 1) and will make new test cases first focusing on the new feature requested by the business. i.e. adding new filters, worksheets, filter actions, calculated measures etc.
Of course, with the above TDD Functional test example, we’ve not covered all the testing concepts that non-BI developers have been using for a long time and for a very good reason. The good news is that most of these are now possible with Tableau too so you will not only be convinced that your dashboard is looking great but you can feel confident that the underlying business logic is always working correctly. In the following posts, I’ll be going through further examples of test-driven concepts employed in BI, such as:
Keeping Tableau dashboards and their test cases together under version control in Git
Running Tableau tests automatically after each commit (Bamboo, Jenkins, Teamcity integrations)