Instructor Guide

This guide covers how to create test cases, choose assertions, serialize to YAML, write custom test harnesses, and integrate with nbgrader’s assignment pipeline.

Defining Test Cases

A TestCase stores one set of inputs and the corresponding expected outputs:

from pytest_nbgrader.cases import TestCase

case = TestCase(
    inputs=((2, 3), {}),
    expected=((5,), {}),
)

Both inputs and expected are (positional_args, keyword_args) tuples. The exact format depends on the submission type.

Function Inputs

For function submissions, inputs holds the arguments passed to the function:

# Tests: add(2, 3) == 5
TestCase(inputs=((2, 3), {}), expected=((5,), {}))

# Tests: greet(name="Alice") == "Hello, Alice"
TestCase(inputs=((), {"name": "Alice"}), expected=(("Hello, Alice",), {}))

The expected positional tuple wraps the return value — a function returning 5 expects (5,).

Code String Inputs

For code string submissions (variable assignments), inputs are keyword arguments that become the execution scope, and expected outputs are the variables that should exist after execution:

# Student code: "result = x + y"
# Test: with x=10, y=20, result should be 30
TestCase(
    inputs=((), {"x": 10, "y": 20}),
    expected=((), {"x": 10, "y": 20, "result": 30}),
)

Note

The expected dict must include the input variables too, because they remain in scope after execution.

Class Inputs

For class submissions, inputs is a list of (args, kwargs) pairs — one per instantiation:

# Instantiate Point(1, 2) and Point(3, 4)
TestCase(
    inputs=[((1, 2), {}), ((3, 4), {})],
    expected=((), {}),
)

The resulting objects are compared using attribute assertions like equal_attributes or close_attributes.

Testing Exceptions

To test that code raises an exception, set raises=True:

TestCase(
    inputs=((1, 0), {}),
    expected=((), {}),
    raises=True,
)

Use the raises assertion to verify the exception type (see Assertions Reference).

Timing Constraints

To enforce execution time bounds:

TestCase(
    inputs=((large_input,), {}),
    expected=((result,), {}),
    timing=(None, 2.0),  # must finish within 2 seconds
)

Use with the time_bounds assertion.

Choosing Assertions

Assertions verify the relationship between expected and actual outputs. Each assertion function takes (case, outputs, *args, **kwargs) and returns (ExitCode.OK, "") on success or (ExitCode.TESTS_FAILED, error_info) on failure.

For Functions

• equal_value() — exact equality of return values

• almost_equal() — approximate equality for floats (numpy-based, with atol/rtol)

• raises() — expected exception was raised (use with TestCase(raises=True))

• time_bounds() — execution time within TestCase.timing bounds

For Code Strings

• equal_value() — exact equality of named variables (pass variable names as *args)

• equal_scope() — all expected variables are defined

• equal_types() — variable types match

• almost_equal() — approximate equality of named variables

For Classes

• equal_attributes() — attribute values match between expected and actual instances

• close_attributes() — attribute values approximately match (numpy-based)

• has_method() — object has required methods or attributes

For File Output

• file_contents() — written files match expected contents byte-for-byte

See Assertions Reference for complete signatures and examples.

Adding Prerequisites

Prerequisites validate the submission before test cases run. They are optional — in many courses, assertions alone are sufficient.

Signature Validation

Verify that a student’s function has the correct parameter names:

import inspect
from pytest_nbgrader.prerequisites import has_signature

ref_sig = inspect.signature(lambda a, b: None)

subtask = TestSubtask(
    cases=[...],
    assertions={...},
    prerequisites={"signature": (has_signature, ((ref_sig,), {}))},
)

has_signature compares parameter names, and optionally types, defaults, and return annotations via the *strict_comparisons and **comparisons arguments.

Module Output Checking

For path/module submissions, verify stdout/stderr output during import:

from pytest_nbgrader.prerequisites import writes

prerequisites={"output": (writes, ((), {"out": "Hello, World!\n"}))}

File Write Checking

Verify which files a module creates, deletes, or modifies during import:

from pathlib import Path
from pytest_nbgrader.prerequisites import writes_file

prerequisites={"files": (writes_file, ((), {"created": {Path("output.txt")}}))}

Packaging with TestSubtask

A TestSubtask bundles test cases with assertions and prerequisites:

from pytest_nbgrader.assertions import almost_equal, equal_value
from pytest_nbgrader.cases import TestCase, TestSubtask

subtask = TestSubtask(
    cases=[
        TestCase(inputs=((2, 3), {}), expected=((5,), {})),
        TestCase(inputs=((0, 0), {}), expected=((0,), {})),
    ],
    assertions={equal_value: ((), {})},
)

The Assertions Dict

The assertions dict maps assertion function objects to their extra arguments:

assertions = {
    equal_value: (("a", "b"), {}),              # check variables a, b for equality
    almost_equal: (("x",), {"atol": 1e-6}),     # check x with tolerance
}

The format is {function: (extra_positional_args, extra_keyword_args)}. The case and outputs arguments are passed automatically by the harness.

When pytest runs, each case is tested against each assertion, creating a Cartesian product of len(cases) × len(assertions) test nodes.

Generating Test Cases Programmatically

For robust grading, generate many test cases from parameter spaces:

import itertools
from pytest_nbgrader.assertions import equal_value, equal_types
from pytest_nbgrader.cases import TestCase, TestSubtask

left = (1, 2.0, True, "xyz")
right = (3, 4.0, False, "abc")

cases = [
    TestCase(
        inputs=((), {"a": x, "b": y}),
        expected=((), {"a": y, "b": x}),
    )
    for x, y in itertools.product(left, right)
]

assertions = {
    equal_value: (("a", "b"), {}),
    equal_types: (("a", "b"), {}),
}

subtask = TestSubtask(cases=cases, assertions=assertions)

This creates 16 test cases × 2 assertions = 32 test nodes.

Serializing to YAML

Test cases are serialized to YAML for distribution to students.

Single Subtask

from pathlib import Path
from pytest_nbgrader.dumper import dump_subtask

dump_subtask(subtask, to=Path("tests/Addition/basic.yml"))

Full Exercise

For multi-task exercises, organize subtasks into a nested dict and use dump_exercise:

from pytest_nbgrader.dumper import dump_exercise

exercise = {
    "Addition": {
        "basic": subtask_basic,
        "edge_cases": subtask_edge,
    },
    "Multiplication": {
        "correctness": subtask_mul,
    },
}

dump_exercise(exercise)

This creates the directory tree:

tests/
  Addition/
    basic.yml
    edge_cases.yml
  Multiplication/
    correctness.yml

What the YAML Looks Like

A serialized YAML file contains the full TestSubtask — assertions, cases, and metadata — using Python-specific YAML tags:

!!python/object:pytest_nbgrader.cases.TestSubtask
assertions:
  ? !!python/name:pytest_nbgrader.assertions.equal_value ''
  : !!python/tuple
  - !!python/tuple
    - a
    - b
  - {}
cases:
- !!python/object:pytest_nbgrader.cases.TestCase
  expected: !!python/tuple
  - !!python/tuple []
  - a: 2
    b: 1
  inputs: !!python/tuple
  - !!python/tuple []
  - a: 1
    b: 2
  raises: false
  timing: !!python/tuple
  - null
  - null

The !!python/object and !!python/name tags let yaml.unsafe_load() reconstruct the original Python objects at load time. You don’t need to read or edit these files — they are generated by the dumper and consumed by the plugin automatically.

Warning

YAML files are loaded using yaml.unsafe_load(), which can execute arbitrary Python code during deserialization. Only distribute YAML files through trusted channels (e.g., your institution’s LMS or nbgrader’s exchange directory). Students should never load YAML files from untrusted sources.

Writing Custom Test Harnesses

For exercises that need custom logic beyond the standard assertion pipeline — such as testing class methods, generators, or complex object interactions — write a Python test file alongside the YAML data.

# tests/MyTask/tests.py
import pytest
from pytest_nbgrader.assertions import close_attributes
from pytest_nbgrader.cases import format_result

class TestCircle:
    """Custom test harness for Circle class."""

    def test_instantiation(self, submission, cases, verbosity):
        args, kwargs = cases.inputs[0]
        instance = submission(*args, **kwargs)
        result, message = close_attributes(
            cases, instance, "radius", "area", "circumference",
            atol=1e-6, rtol=1e-6,
        )
        if result is not pytest.ExitCode.OK:
            pytest.fail(format_result(cases.inputs[0], result, message))

In the student notebook, run both the YAML tests and the custom harness:

pytest.main([
    "-qq", "-x",
    "--cases", "tests/MyTask/data.yml",
    "tests/MyTask/tests.py::TestCircle",
])

Using runner.main()

The runner module provides a convenience wrapper around pytest.main() that handles path construction, temporary symlinks to the harness and conftest, and the standard flags:

from pytest_nbgrader.runner import main

main(task="Addition", subtask="basic")

This is equivalent to:

pytest.main([
    "-p", "no:pytest-nbgrader",
    "--cases=tests/Addition/basic.yml",
    "harness.py::TestClass",
])

runner.main() creates temporary symlinks to the built-in harness.py and conftest.py in the current directory, runs pytest, then cleans up. It accepts additional *args that are forwarded to pytest.main().

Note

The example course uses raw pytest.main() calls for more control over flags and paths. Use runner.main() when the default harness and conftest are sufficient and you want a simpler API.

Integrating with nbgrader

pytest-nbgrader works alongside nbgrader’s assignment pipeline. This section explains how to set up a course so that test cases flow correctly from instructor to student.

How the Pipeline Works

nbgrader’s generate_assignment command copies notebooks from source/ to release/, applying preprocessors that clear solutions, lock cells, and strip hidden tests. Critically, non-notebook files are copied as-is — this includes the tests/ directory with your YAML files.

The flow:

source/{assignment}/              nbgrader generate_assignment
  _data_generation.ipynb  ────────────── (excluded by config)
  exercise.ipynb          ──────────────► release/{assignment}/exercise.ipynb
  tests/                  ──────────────► release/{assignment}/tests/
    Task/subtask.yml                        Task/subtask.yml

Students receive the release version. After submission, nbgrader collects notebooks into submitted/, then autograde runs them in autograded/. The tests/ directory must be present at each stage for pytest to find the YAML files.

Directory Layout

A typical course directory:

HomeworkAssignments/
  nbgrader_config.py
  source/
    02_arithmetics/
      _data_generation.ipynb     # instructor-only: generates YAML
      exchange_variables.ipynb   # notebook with solution + test cells
      tests/
        ExchangeVariables/
          test_for_equal_value.yml
          test_for_equal_types.yml
  release/                       # generated by nbgrader
    02_arithmetics/
      exchange_variables.ipynb   # solutions cleared, cells locked
      tests/                     # copied as-is from source
        ExchangeVariables/
          test_for_equal_value.yml

The source/ directory is instructor-only. It contains:

• Data generation notebooks (prefixed with _) that create the YAML test files

• Assignment notebooks with model solutions in solution cells and pre-built test cells

• tests/ directories with the serialized YAML output

Configuring nbgrader

Use nbgrader_config.py to prevent instructor-only files from leaking to students:

c = get_config()

c.CourseDirectory.ignore = [
    '*data_generation*.ipynb',  # data generation notebooks
    '_*',                       # any file/dir starting with _
    '.ipynb_checkpoints',
    '*.pyc',
    '__pycache__',
    '.pytest_cache',
]

# Only process notebooks not starting with _
c.CourseDirectory.notebook_id = '[!_]*'

The ignore list ensures _data_generation.ipynb and _model_solutions/ directories are excluded from generate_assignment. The notebook_id pattern provides an additional safeguard — only non-underscored notebooks are treated as assignment notebooks.

Writing Test Cells in Notebooks

In the source notebook, write the test cells alongside the solution. These cells will be locked (read-only) in the released version so students cannot modify them.

A typical pattern uses two cells. The first submits the student’s solution and sets up shared arguments:

import pathlib
import pytest
from pytest_nbgrader import loader

loader.Submission.submit(_i)
cases = pathlib.Path('tests') / 'TaskName'
args = ['-qq', '-x',
        '-W', 'ignore::_pytest.warning_types.PytestAssertRewriteWarning',
        '--cases']

The second cell runs the actual tests:

assert pytest.main([*args, cases / 'equal_value.yml']) is pytest.ExitCode.OK
assert pytest.main([*args, cases / 'equal_types.yml']) is pytest.ExitCode.OK

The assert ... is pytest.ExitCode.OK pattern makes the cell raise an AssertionError if any test fails, which nbgrader’s autograder interprets as a failed grade cell.

In nbgrader, mark these cells with the appropriate metadata:

• Set locked: true so students cannot edit the test cells

• Set grade: true and assign points for cells that should be autograded

• Set solution: true on the cell where students write their answer

Tip

Use _i (IPython’s magic variable for the previous cell’s source code) to capture code string submissions. This avoids asking students to wrap their code in quotes — they just write normal Python in the solution cell, and the next cell captures it automatically.

Data Generation Workflow

The data generation notebook is the instructor’s workspace for creating test cases. A typical workflow:

1. Import assertion functions, TestCase, TestSubtask, and the dumper

2. Define model solution functions that compute expected outputs

3. Generate test cases from parameter combinations (e.g., with itertools.product)

4. Package into TestSubtask objects with appropriate assertions

5. Serialize with dump_exercise() — output goes to tests/ alongside the notebook

import itertools
from pytest_nbgrader.assertions import equal_value, equal_types
from pytest_nbgrader.cases import TestCase, TestSubtask
from pytest_nbgrader import dumper

left = (1, 2.0, True, 'xyz')
right = (3, 4.0, False, 'abc')

cases = [
    TestCase(
        inputs=((), {"a": x, "b": y}),
        expected=((), {"a": y, "b": x}),
    )
    for x, y in itertools.product(left, right)
]

exercise = {
    "ExchangeVariables": {
        "equal_value": TestSubtask(
            cases=cases,
            assertions={equal_value: (("a", "b"), {})},
        ),
        "equal_types": TestSubtask(
            cases=cases,
            assertions={equal_types: (("a", "b"), {})},
        ),
    },
}

dumper.dump_exercise(exercise)

Run this notebook once to populate the tests/ directory. Then run nbgrader generate_assignment to produce the student-facing release — the YAML files are copied automatically.