pytest_nbgrader package

Pytest plugin for using with nbgrader and generating test cases.

Submodules

pytest_nbgrader.assertions module

Module for testing actual and expected outputs of a TestCase.

Export functions for asserting relations between student outputs and expected outputs: - close_attributes – - equal_attributes – - has_import – - has_method – - calls – - equal_contents – Do output containers have contents exactly as expected? - almost_equal – Are outputs almost equal to expected? - equal_value – Are outputs exactly equal to expected? - equal_type – Are output objects of expected types? - equal_scope – Is the output scope as expected (i.e. same variables present)?

pytest_nbgrader.assertions.almost_equal(case: TestCase, outputs: tuple, *args: str, atol: float = 1e-07, rtol: float = 1e-07, **kwargs: object) → ExitCode | tuple[ExitCode, Any, Any][source]

Test for closeness between actual and expected TestCase outputs.

Parameters:

  • case (TestCase) – Test case with expected outputs.

  • outputs (tuple) – Actual outputs from student submission.

  • *args (str) – Variable names for which near equality is tested.

  • atol (float, optional) – Absolute tolerance, by default 1e-7.

  • rtol (float, optional) – Relative tolerance, by default 1e-7.

  • **kwargs (dict) – Unused keyword arguments.

Returns:

pytest.ExitCode.OK if all values are equal up to tolerance, otherwise pytest.ExitCode.TESTS_FAILED.

Return type:

Enum

pytest_nbgrader.assertions.calls(case: TestCase, outputs: tuple, caller: str, **callees: list[tuple[tuple, dict]]) → ExitCode | tuple[ExitCode, Any, Any][source]

Test if function ‘caller’ of imported module calls callees in the prescribed manner.

Parameters:

  • case (TestCase) – Test case with expected outputs.

  • outputs (tuple) – Actual outputs (positional, named, elapsed) from student submission. The module or class under test is taken from outputs[0][0].

  • caller (str) – Name of the function to call on the object.

  • **callees (list of tuple) – Mapping of callee names to lists of (args, kwargs) expected call tuples.

Returns:

pytest.ExitCode.OK if all calls match, otherwise a failure tuple.

Return type:

Enum

pytest_nbgrader.assertions.close_attributes(case: TestCase, outputs: object, *args: str, **kwargs: float) → ExitCode | tuple[ExitCode, Any, Any][source]

Assert close values for given attributes between expected and outputs.

Parameters:

  • case (TestCase) – Test case with expected outputs.

  • outputs (object) – Actual output object whose attributes are compared.

  • *args (str) – Attribute names to compare.

  • **kwargs (dict) – Tolerances forwarded to np.testing.assert_allclose.

Returns:

pytest.ExitCode.OK on success, or a failure tuple.

Return type:

Enum

pytest_nbgrader.assertions.equal_attributes(case: TestCase, outputs: tuple, *args: str, **kwargs: object) → ExitCode | tuple[ExitCode, Any, Any][source]

Test if all attributes of expected and actual return object are equal.

Parameters:

  • case (TestCase) – Test case with expected outputs.

  • outputs (tuple) – Actual outputs (positional, named, elapsed) from student submission. The return object is taken from outputs[0][0]; the expected object from case.expected[0][0].

  • *args (str) – Attribute names to compare.

  • **kwargs (dict) – Unused keyword arguments.

Returns:

pytest.ExitCode.OK if equal, otherwise pytest.ExitCode.TESTS_FAILED.

Return type:

Enum

pytest_nbgrader.assertions.equal_contents(case: TestCase, outputs: tuple, *args: str, **kwargs: object) → ExitCode | tuple[ExitCode, Any, Any][source]

Test if containers have equal contents between actual and expected outputs.

Parameters:

  • case (TestCase) – Test case with expected outputs.

  • outputs (tuple) – Actual outputs from student submission.

  • *args (str) – Variable names which hold containers to be tested.

  • **kwargs (dict) – Unused keyword arguments.

Returns:

pytest.ExitCode.OK if all pairs of containers have equal contents, otherwise pytest.ExitCode.TESTS_FAILED.

Return type:

Enum

pytest_nbgrader.assertions.equal_scope(case: TestCase, outputs: tuple, *args: object, **kwargs: object) → ExitCode | tuple[ExitCode, Any, Any][source]

Test for equal scope between expected and actual TestCase outputs.

Parameters:

  • case (TestCase) – Test case with expected outputs.

  • outputs (tuple) – Actual outputs from student submission.

  • *args (tuple) – Unused positional arguments.

  • **kwargs (dict) – Unused keyword arguments.

Returns:

pytest.ExitCode.OK if expected and actual have equal variable names, otherwise pytest.ExitCode.TESTS_FAILED.

Return type:

Enum

pytest_nbgrader.assertions.equal_types(case: TestCase, outputs: tuple, *args: str, **kwargs: object) → ExitCode | tuple[ExitCode, Any, Any][source]

Test for equal types between expected and actual TestCase outputs.

Parameters:

  • case (TestCase) – Test case with expected outputs.

  • outputs (tuple) – Actual outputs from student submission.

  • *args (str) – Variable names to check.

  • **kwargs (dict) – Unused keyword arguments.

Returns:

pytest.ExitCode.OK if all values have equal types, otherwise pytest.ExitCode.TESTS_FAILED.

Return type:

Enum

pytest_nbgrader.assertions.equal_value(case: TestCase, outputs: tuple, *args: str, **kwargs: object) → ExitCode | tuple[ExitCode, Any, Any][source]

Test for exact equality between expected and actual TestCase outputs.

Parameters:

  • case (TestCase) – Test case with expected outputs.

  • outputs (tuple) – Actual outputs from student submission.

  • *args (str) – Variable names for which exact equality is tested.

  • **kwargs (dict) – Unused keyword arguments.

Returns:

pytest.ExitCode.OK if all values are exactly equal, otherwise pytest.ExitCode.TESTS_FAILED.

Return type:

Enum

pytest_nbgrader.assertions.file_contents(case: TestCase, outputs: tuple, *args: object, **kwargs: object) → ExitCode | tuple[ExitCode, Any, Any][source]

Test if files in TestCase.expected[1] have the prescribed contents.

Parameters:

  • case (TestCase) – Test case with expected[1] mapping filenames to expected contents.

  • outputs (tuple) – Actual outputs from student submission (unused by this assertion).

  • *args (tuple) – Unused positional arguments.

  • **kwargs (dict) – Unused keyword arguments.

Returns:

pytest.ExitCode.OK if contents are bitwise identical, otherwise pytest.ExitCode.TESTS_FAILED.

Return type:

Enum

pytest_nbgrader.assertions.has_import(case: TestCase, outputs: tuple, *args: Path, **kwargs: Path | None) → ExitCode | tuple[ExitCode, Any, Any][source]

Test if an object has a module-level import.

Parameters:

  • case (TestCase) – Test case with expected outputs.

  • outputs (tuple) – Actual outputs (positional, named, elapsed) from student submission. The return object is taken from outputs[0][0].

  • *args (pathlib.Path) – Expected import paths to check (positional).

  • **kwargs (pathlib.Path or None) – Mapping of object names to expected import paths (None = locally defined).

Returns:

pytest.ExitCode.OK if all imports match, otherwise pytest.ExitCode.TESTS_FAILED.

Return type:

Enum

pytest_nbgrader.assertions.has_method(case: TestCase, outputs: tuple, *args: str, **kwargs: type) → ExitCode | tuple[ExitCode, Any, Any][source]

Test if return object has given methods/attributes.

Parameters:

  • case (TestCase) – Test case with expected outputs.

  • outputs (tuple) – Actual outputs (positional, named, elapsed) from student submission. The return object is taken from outputs[0][0].

  • *args (str) – Attribute names that must exist on the return object.

  • **kwargs (dict) – Mapping of attribute names to expected type hints.

Returns:

pytest.ExitCode.OK if all methods exist with correct types, otherwise pytest.ExitCode.TESTS_FAILED.

Return type:

Enum

pytest_nbgrader.assertions.raises(case: TestCase, outputs: tuple | Exception, *args: type[Exception], **kwargs: object) → ExitCode | tuple[ExitCode, Any, Any][source]

Test if case raised an exception as prescribed.

Parameters:

  • case (TestCase) – Test case with raises flag indicating expected exception.

  • outputs (tuple or Exception) – Actual outputs or raised exception from student submission.

  • *args (type) – Exception types that are expected to be raised.

  • **kwargs (dict) – Unused keyword arguments.

Returns:

pytest.ExitCode.OK if the expected exception was raised, otherwise pytest.ExitCode.TESTS_FAILED.

Return type:

Enum

pytest_nbgrader.assertions.time_bounds(case: TestCase, outputs: tuple, *args: object, **kwargs: object) → ExitCode | tuple[ExitCode, Any, Any][source]

Test for execution time within bounds, if provided.

Parameters:

  • case (TestCase) – Test case with timing tuple of (lower, upper) bounds.

  • outputs (tuple) – Actual outputs; outputs[2] is the execution time.

  • *args (tuple) – Unused positional arguments.

  • **kwargs (dict) – Unused keyword arguments.

Returns:

pytest.ExitCode.OK if execution time is within bounds, otherwise pytest.ExitCode.TESTS_FAILED.

Return type:

Enum

pytest_nbgrader.cases module

Classes for test cases using code objects or functions.

Export classes for TestCases, listing *args and **kwargs for inputs and (both actual and expected) outputs from a student submission: - TestCase: Data class for storing inputs and expected outputs of tests. - Test: Class for executing submissions on TestCases.

class pytest_nbgrader.cases.TestCase(inputs: tuple[tuple, dict]=<factory>, expected: tuple[tuple, dict]=<factory>, raises: bool = False, timing: tuple[float | None, float | None]=(None, None))[source]

Bases: object

Inputs and expected outputs of a single test case.

expected: tuple[tuple, dict]
inputs: tuple[tuple, dict]
raises: bool = False
timing: tuple[float | None, float | None] = (None, None)
class pytest_nbgrader.cases.TestSubtask(cases: list[~pytest_nbgrader.cases.TestCase], assertions: dict, prerequisites: dict = <factory>)[source]

Bases: object

Test cases, prerequisites, and assertions for a single subtask.

assertions: dict
cases: list[TestCase]
prerequisites: dict
class pytest_nbgrader.cases.Timer[source]

Bases: object

Context manager for measuring execution time.

property elapsed: float

Return elapsed time in seconds.

Returns:

Elapsed time since entering the context.

Return type:

float

end: float | None
start: float
pytest_nbgrader.cases.execute(submission: object, case: TestCase) → tuple[tuple, dict, float][source]
pytest_nbgrader.cases.execute(submission: LambdaType, case: TestCase) → tuple[tuple, dict, float]
pytest_nbgrader.cases.execute(submission: CodeType, case: TestCase) → tuple[tuple, dict, float]
pytest_nbgrader.cases.execute(submission: ModuleSpec, case: TestCase) → tuple[tuple, dict, float]
pytest_nbgrader.cases.execute(submission: type, case: TestCase) → tuple[tuple, dict, float]

Execute a submission on a test case.

Parameters:

  • submission (object) – The student submission to execute.

  • case (TestCase) – Test case with inputs and expected outputs.

Returns:

A (positional_outputs, named_outputs, elapsed_time) triple.

Return type:

tuple[tuple, dict, float]

pytest_nbgrader.cases.format_result(inputs: tuple[tuple, dict], result: ExitCode, message: str | None = None, exception: str | None = None) → str[source]

Format case result for logging nicely.

Parameters:

  • inputs (tuple) – A (args, kwargs) pair of test case inputs.

  • result (Enum) – The pytest exit code for this case.

  • message (str or None, optional) – Failure message, by default None.

  • exception (str or None, optional) – Formatted traceback, by default None.

Returns:

Formatted result string.

Return type:

str

pytest_nbgrader.conftest module

Pytest configuration module.

pytest internals:

pytest_addoption – add custom options to test a single subtask at a time pytest_generate_tests – generate tests programmatically from tests.pickle

pytest fixtures:

verbosity – provide verbosity level for outputs submission – provide student submission

PYTEST_DONT_REWRITE

pytest_nbgrader.conftest.pytest_addoption(parser: Parser) → None[source]

Add custom options for test case location and auto-generation.

Parameters:

parser (_pytest.config.argparsing.Parser) – The pytest argument parser.

pytest_nbgrader.conftest.pytest_generate_tests(metafunc: Metafunc) → None[source]

Programmatically generate tests from deserialized test cases.

Parameters:

metafunc (pytest.Metafunc) – The metafunc object for parametrizing test functions.

pytest_nbgrader.conftest.pytest_sessionfinish(session: Session) → None[source]

Unlink the previously generated tests file at session finish.

Parameters:

session (pytest.Session) – The pytest session object.

pytest_nbgrader.conftest.pytest_sessionstart(session: Session) → None[source]

Collect manual and automatic test cases.

Parameters:

session (pytest.Session) – The pytest session object.

pytest_nbgrader.conftest.submission() → object[source]

Inject submission object into pytest as fixture.

Returns:

The stored student submission.

Return type:

object

pytest_nbgrader.conftest.verbosity(request: FixtureRequest) → int[source]

Inject verbosity from global config.

Parameters:

request (pytest.FixtureRequest) – The pytest fixture request object.

Returns:

The verbosity level.

Return type:

int

pytest_nbgrader.dumper module

Serialize test cases to YAML files.

pytest_nbgrader.dumper.dump_exercise(exercise: dict[str, dict[str, TestSubtask]], to: Path = PosixPath('tests')) → None[source]

Dump all subtasks of an exercise to a directory tree.

Parameters:

  • exercise (dict[str, dict[str, TestSubtask]]) – Nested mapping of task names to subtask dictionaries.

  • to (pathlib.Path, optional) – Target directory, by default Path("tests").

pytest_nbgrader.dumper.dump_subtask(subtask: TestSubtask, to: Path = PosixPath('tests.yml'), append: bool = False) → None[source]

Dump a single subtask to a YAML file.

Parameters:

  • subtask (TestSubtask) – The subtask to serialize.

  • to (pathlib.Path, optional) – Target file path, by default Path("tests.yml").

  • append (bool, optional) – Whether to append to an existing file, by default False.

pytest_nbgrader.dumper.dump_task(subtasks: dict[str, TestSubtask], to: Path) → None[source]

Dump all subtasks of a single task to a directory.

Parameters:

  • subtasks (dict[str, TestSubtask]) – Mapping of subtask names to TestSubtask objects.

  • to (pathlib.Path) – Target directory for the YAML files.

pytest_nbgrader.harness module

Check submissions with pytest and fixtures.

class pytest_nbgrader.harness.TestClass[source]

Bases: object

Generic pytest class.

test_assertion(test_execution: tuple, assertions: tuple, verbosity: int) → None[source]

Run assertions against results of test execution.

Parameters:

  • test_execution (tuple) – A (case, outputs) pair from the test_execution fixture.

  • assertions (tuple) – A (function, (args, kwargs)) pair for the assertion check.

  • verbosity (int) – Verbosity level for output formatting.

test_execution(submission: object, cases: TestCase, verbosity: int) → tuple[TestCase, tuple[tuple, dict, float] | Exception][source]

Run student submission on test cases.

Parameters:

  • submission (object) – The student submission to execute.

  • cases (TestCase) – Test case with inputs and expected outputs.

  • verbosity (int) – Verbosity level for output formatting.

Returns:

A (case, result) pair.

Return type:

tuple

test_prerequisites(submission: object, prerequisites: tuple) → None[source]

Run prerequisites tests against student submission.

Parameters:

  • submission (object) – The student submission under test.

  • prerequisites (tuple) – A (function, (args, kwargs)) pair for the prerequisite check.

pytest_nbgrader.loader module

Interface module between ipynb and pytest.

Exports:

Submission – interface class to hold one submission

class pytest_nbgrader.loader.Submission[source]

Bases: object

Store submission object from notebooks.

submission: object = None

The stored submission object.

classmethod submit(submission: object) → None[source]
classmethod submit(cell: str) → CodeType
classmethod submit(function: LambdaType) → LambdaType
classmethod submit(clss: type) → type
classmethod submit(module: Path) → ModuleSpec

Store a generic submission.

Parameters:

submission (object) – The submission object to store.

pytest_nbgrader.prerequisites module

Module for testing prerequisites a student’s submission needs to fulfill.

Export functions for asserting properties of student code: - has_signature – is a function compatible with the supplied signature?

Might in the future test for static attributes or methods of classes

pytest_nbgrader.prerequisites.has_signature(function: ~collections.abc.Callable[[...], ~typing.Any], ref_sig: ~inspect.Signature, *strict_comparisons: str, compare_names: ~collections.abc.Callable[[list[str], list[str]], bool] = <slot wrapper '__eq__' of 'list' objects>, **comparisons: ~collections.abc.Callable[[~typing.Any, ~typing.Any], bool]) → ExitCode[source]

Test if function is compatible with passed signature.

Parameters:

  • function (callable) – The function whose signature is to be tested.

  • ref_sig (inspect.Signature) – Reference signature to compare against.

  • *strict_comparisons (str) – Parameter attributes to compare using strict equality.

  • compare_names (callable, optional) – Function to compare parameter name lists, by default list.__eq__.

  • **comparisons (callable) – Mapping of parameter attributes to comparison functions.

Returns:

pytest.ExitCode.OK if signature matches, otherwise pytest.ExitCode.TESTS_FAILED.

Return type:

Enum

pytest_nbgrader.prerequisites.writes(spec: ModuleSpec, *args: object, name: str | None = None, out: str | None = None, err: str | None = None, **kwargs: object) → ExitCode[source]

Test stdout and stderr writes of module execution as name.

Parameters:

  • spec (importlib.machinery.ModuleSpec) – Module specification to be executed.

  • *args (tuple) – Unused positional arguments.

  • name (str or None, optional) – __name__ of module at execution time, by default None.

  • out (str or None, optional) – Expected stdout output, skipped if None.

  • err (str or None, optional) – Expected stderr output, skipped if None.

  • **kwargs (dict) – Unused keyword arguments.

Returns:

pytest.ExitCode.OK if stdout/stderr match expectations, otherwise pytest.ExitCode.TESTS_FAILED.

Return type:

Enum

pytest_nbgrader.prerequisites.writes_file(spec: ModuleSpec, *args: object, name: str | None = None, created: set[Path] | None = None, deleted: set[Path] | None = None, modified: set[Path] | None = None) → ExitCode | tuple[ExitCode, Any, Any][source]

Test file writes of module execution as name.

Parameters:

  • spec (importlib.machinery.ModuleSpec) – Module specification to be executed.

  • *args (tuple) – Unused positional arguments.

  • name (str or None, optional) – __name__ of module at execution time, by default None.

  • created (set or None, optional) – Expected set of created file paths, by default None.

  • deleted (set or None, optional) – Expected set of deleted file paths, by default None.

  • modified (set or None, optional) – Expected set of modified file paths, by default None.

Returns:

pytest.ExitCode.OK if file operations match expectations, otherwise pytest.ExitCode.TESTS_FAILED.

Return type:

Enum

pytest_nbgrader.runner module

Notebook-side entry point for running pytest with test cases.

Bases: object

Context manager for a temporary symlink to a module file.

Parameters:

  • module (module) – The Python module to symlink.

  • destination (str or None, optional) – Destination filename, by default uses the module’s filename.

custom: bool
module: ModuleType
path: Path

Bases: object

Context manager for multiple temporary symlinks.

Parameters:

  • *args (module) – Modules to create symlinks for.

  • **kwargs (module) – Mapping of destination names to modules.

pytest_nbgrader.runner.main(*args: str, task: str | None = None, subtask: str | None = None, case_dir: str = 'tests', auto: bool = True, **kwargs: object) → ExitCode | int[source]

Wrap around pytest to inject test cases and set up config.

Parameters:

  • *args (str) – Additional arguments passed to pytest.main.

  • task (str or None, optional) – Task name subdirectory, by default None.

  • subtask (str or None, optional) – Subtask name for the YAML file, by default None.

  • case_dir (str, optional) – Directory containing test case files, by default "tests".

  • auto (bool, optional) – Whether to auto-generate test class, by default True.

  • **kwargs (dict) – Additional keyword arguments passed to pytest.main.

Returns:

The pytest exit code.

Return type:

int