Continuous Testing

I was reading the "Software you can keep in your head" book the other day - not sure if that's the exact title rn sry. It reminded that we should start up a deployment pipeline for bankbog.dk as early as possible. I have been procrastinating over "choosing" the "right" architecture for too long I guess. Now it's time to create a deployment pipeline and we will start the testing part of it.

A short rant about testing

Bankbog is a solution that makes use of a database and the database "stuff" is an important part of the testing harness. Some people ask: "Why not just "fake" the database for tests? It's an implementation detail and not important to the application logic." Or "That's not tenable in the long run when you get too big of a system." There is more than one reason:

1. It is important - we want to test those migration scripts, the sql statements, everything, in an automated way that does not require us to boot up the application and click around / send http requests to validate the security was implemented correctly on endpoints etc.
2. Not only is the database important, the user-facing API is extremely important to test as well, so testing bankbog.dk goes through the API of course. And as little as possible is mocked/faked - everything is as real as production.
3. In fact, pretty much everything - not only application core - is important to be included in tests.

In fact - I find it most productive to test from the public API endpoints since they rarely change, but whatever is behind there can and does (and should) often change - especially in the beginning of a new project - and testing classes more directly makes the tests dependent on too many implmentation details and will only increase the refactoring effort needlessly.

It can be argued that it is hard to hit every scenario very transparently in this way and that is true especially when the system becomes bigger, but I truly believe these kinds of tests gives the most bang for the buck in the beginning - especially because all the implementation "details" will change so often and quickly that we don't want to be slowed down by a thousand papercuts, i.e. fixing tests, but the API and expected return seldom changes.

Alright, rant's over

Bankbog.dk is under source control, like everything else, and I use GitHub to host it. Lucky that they have GitHub Actions that enable us to specify a pipeline for continuous integration.

I have had mixed luck with doing dotnet test on a whole solution. Sometimes it works, sometimes it does not. That's a little unfortunate since that would be the best command to use, to automatically include new test project in continuous integration. I am not sure what it is with GitHub Actions that goes wrong - maybe it's due to two/more processes running side by side and that's "flaky" (only speculation of course, and based on the "sometimes it works, sometimes it doesn't" observation).

Anyways, there's always a workaround of course and it is simple enough to just run test projects in each their own "step".

By the way: don't put secrets in appsettings files. The host, port, user, and password to connect to the database used during continuous integration is considered a secret - at least by me. In this case, we can use the "Secrects" feature in GitHub and a task made by Microsoft for doing variable substitution.

The appsettings.test.continuous_integration.json file has configuration that looks like this: (yes just empty strings, and yes of course a lot of other unimportant stuff is omitted.)

{
  "MySqlDbConfig": {
    "Server": "",
    "Port": "",
    "User": "",
    "Pass": ""
  }
}

And with that in mind, here's a real example of how such a file could look:

name: Build and Test

on:
  push:
    branches: [ "main" ]
  pull_request:
    branches: [ "main" ]

jobs:
  build-and-test:

    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v3

    - name: setup dotnet 8.0.x
      uses: actions/setup-dotnet@v2
      with:
        dotnet-version: 8.0.x

    - name: restore
      run: dotnet restore

    - name: build
      run: dotnet build --no-restore

    - name: appsettings variable substitution
      uses: microsoft/variable-substitution@v1
      with:
        files: '**/appsettings.test.continuous_integration.json'
      env:
        MySqlDbConfig.Server: ${{ secrets.INTEGRATION_TEST_DB_HOST }}
        MySqlDbConfig.Port: ${{ secrets.INTEGRATION_TEST_DB_PORT }}
        MySqlDbConfig.User: ${{ secrets.INTEGRATION_TEST_DB_USER }}
        MySqlDbConfig.Pass: ${{ secrets.INTEGRATION_TEST_DB_PASS }}

    - name: test dmb.database.test
      run: ASPNETCORE_ENVIRONMENT=continuous_integration dotnet test **/dmb.database.test --no-build --nologo

    - name: test dmb.users.test
      run: ASPNETCORE_ENVIRONMENT=continuous_integration dotnet test **/dmb.users.test --no-build --nologo

And with that, we are on our way to continuous delivery by way of continuous integration.