Testing

Unit Tests

Unit tests perform function-by-function tests to ensure their individual functionality. This test suite runs without depending on the server-side and thus it is executed in Travis CI for every push.

How to run

$ python -m pytest -m 'not integration' tests

Integration Tests

Integration tests combine multiple invocations of high-level interfaces to make underlying API requests to a running gateway server to test the full functionality of the client as well as the manager.

They are marked as “integration” using the @pytest.mark.integration decorator to each test case.

Warning

The integration tests actually make changes to the target gateway server and agents. If some tests fail, those changes may remain in an inconsistent state and requires a manual recovery such as resetting the database and populating fixtures again, though the test suite tries to clean up them properly.

So, DO NOT RUN it against your production server.

Prerequisite

Please refer the README of the manager and agent repositories to set up them. To avoid an indefinite waiting time for pulling Docker images:

  • (manager) python -m ai.backend.manager.cli etcd rescan-images

  • (agent) docker pull

    • lablup/python:3.6-ubuntu18.04

    • lablup/lua:5.3-alpine3.8

The manager must also have at least the following active suerp-admin account in the default domain and the default group.

  • Example super-admin account:

    • User ID: admin@lablup.com

    • Password wJalrXUt

    • Access key: AKIAIOSFODNN7EXAMPLE

    • Secret key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

One or more testing-XXXX domain, one or more testing-XXXX groups, and one ore more dummy users are created and used during the tests and destroyed after running tests. XXXX will be filled with random identifiers.

Tip

The halfstack configuration and the example-keypairs.json fixture is compatible with this integration test suite.

How to run

Execute the gateway and at least one agent in their respective virtualenvs and hosts:

$ python -m ai.backend.client.gateway.server
$ python -m ai.backend.client.agent.server
$ python -m ai.backend.client.agent.watcher

Then run the tests:

$ export BACKEND_ENDPOINT=...
$ python -m pytest -m 'integration' tests