Skip to content

Commit 59bbfda

Browse files
MicK7MicK7
authored
Add github CI to pytest_parallel
* Add github workflow * use modern decorator * add nice badges * move fixtures
1 parent d7b3608 commit 59bbfda

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

41 files changed

+1314
-1000
lines changed

‎.github/workflows/test.yml‎

Lines changed: 91 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,91 @@
1+
name: CI
2+
3+
on:
4+
push:
5+
branches: [dev*, master]
6+
pull_request:
7+
branches:
8+
- master
9+
10+
# A workflow run is made up of one or more jobs that
11+
# can run sequentially or in parallel
12+
jobs:
13+
pylint:
14+
runs-on: ubuntu-latest
15+
steps:
16+
- uses: actions/checkout@v3
17+
- uses: mpi4py/setup-mpi@v1
18+
- name: Set up Python
19+
uses: actions/setup-python@v4
20+
with:
21+
python-version: "3.10"
22+
- name: Install dependencies
23+
run: |
24+
python -m pip install --upgrade pip
25+
pip install pylint==2.14.5 pytest
26+
python setup.py egg_info
27+
pip install -r *.egg-info/requires.txt
28+
- name: Analysing the code with pylint
29+
run: |
30+
pylint --unsafe-load-any-extension=y --disable=fixme $(git ls-files '*.py') || true
31+
32+
build:
33+
needs: [pylint]
34+
runs-on: ${{ matrix.os }}
35+
strategy:
36+
fail-fast: false
37+
matrix:
38+
py-version:
39+
- "3.8"
40+
- "3.9"
41+
- "3.10"
42+
- "3.11"
43+
py-arch:
44+
- x64
45+
mpi:
46+
- mpich
47+
- openmpi
48+
- intelmpi
49+
- msmpi
50+
os:
51+
- ubuntu-latest
52+
- macos-latest
53+
- windows-2022
54+
exclude:
55+
- os: macos-latest
56+
mpi: intelmpi
57+
- os: macos-latest
58+
mpi: msmpi
59+
- os: windows-2022
60+
mpi: mpich
61+
- os: windows-2022
62+
mpi: openmpi
63+
- os: windows-2022
64+
mpi: intelmpi
65+
- os: ubuntu-latest
66+
mpi: msmpi
67+
- os: ubuntu-latest
68+
py-version: 3.8
69+
mpi: mpich
70+
- os: ubuntu-latest
71+
py-version: 3.9
72+
mpi: mpich
73+
name: ${{ matrix.mpi }} - ${{matrix.py-version}} - ${{matrix.os}}
74+
steps:
75+
- name: Checkout
76+
uses: actions/checkout@v3
77+
- name: Use Python
78+
uses: actions/setup-python@v4
79+
with:
80+
python-version: ${{ matrix.py-version }}
81+
- name: Setup MPI
82+
uses: mpi4py/setup-mpi@v1
83+
with:
84+
mpi: ${{ matrix.mpi }}
85+
- run: pip install .
86+
# MPI parallel testing
87+
- run: python -m pytest ./test/test_pytest_parallel.py
88+
# Unit testing
89+
- run: python -m pytest ./pytest_parallel/test
90+
91+

‎.travis.yml‎

Lines changed: 0 additions & 31 deletions
This file was deleted.

‎README.md‎

Lines changed: 17 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,16 @@
11
pytest_parallel
22
===============
33

4-
## Introduction ##
4+
**pytest_parallel** extends [PyTest](http://pytest.org) to support parallel testing using mpi4py.
5+
6+
[![Python 3](https://img.shields.io/static/v1?label=Python&logo=Python&color=3776AB&message=3)](https://www.python.org/)
7+
![CI workflow](https://github.com/onera/pytest_parallel/actions/workflows/test.yml/badge.svg)
8+
[![Linux OK](https://img.shields.io/static/v1?label=Linux&logo=Linux&color=yellow&message=%E2%9C%93)](https://en.wikipedia.org/wiki/Linux)
9+
[![macOS OK](https://img.shields.io/static/v1?label=macOS&logo=Apple&color=silver&message=%E2%9C%93)](https://en.wikipedia.org/wiki/macOS)
10+
[![Windows OK](https://img.shields.io/static/v1?label=Windows&logo=Windows&color=white&message=%E2%9C%93)](https://en.wikipedia.org/wiki/Windows)
11+
![License-MPL](https://img.shields.io/badge/license-MPL%202.0-blue.svg)
512

6-
**pytest_parallel** is a PyTest plugin that allows you to implement MPI parallel tests using PyTest and mpi4py.
13+
## Introduction ##
714

815
```Python
916
import pytest_parallel
@@ -15,25 +22,26 @@ def test_fail_one_rank(comm):
1522
if comm.Get_rank()==1: assert False
1623
```
1724

18-
Here we declare a test that should run on two processes. When the test suite is run, the test will execute on two MPI processes. The `comm` fixture is an mpi4py communicator that is private to the test.
25+
Here a test that should run on two processes is declared. When the test suite is run, the test will execute on two MPI processes. The `comm` fixture is an mpi4py communicator that is private to the test.
1926

2027
The test can be run on two processes, e.g. with:
2128

2229
```Bash
2330
mpirun -np 2 pytest --color=yes test_pytest_parallel.py
2431
```
2532

26-
And this will result in the following output:
33+
And the following output will be produced:
2734

2835
![example failing test](doc/images/test_fail.png)
2936

30-
If the test is launch with too few MPI processes then it is skipped. For instance, launching it with:
37+
If there is not enough MPI processes to run the test, it will be skipped.
38+
For instance, the following launching command:
3139

3240
```Bash
3341
mpirun -np 1 pytest --color=yes test_pytest_parallel.py
3442
```
3543

36-
would result in:
44+
would lead to:
3745

3846
![example skipped test](doc/images/test_skip.png)
3947

@@ -45,7 +53,7 @@ The `comm` fixture that you get when decorating your test with `pytest_parallel.
4553

4654
The `pytest_parallel.mark.parallel(n_procs)` decorator takes one argument, `n_procs`.
4755

48-
`n_procs` is generally an integer that specify the size of the communicator that will be given to the test through the `comm` fixture.
56+
`n_procs` is generally an integer that specifies the size of the communicator that will be given to the test through the `comm` fixture.
4957

5058
`n_procs` can also be a list of integers to parametrize the test. For instance, the following test
5159

@@ -59,7 +67,7 @@ will run two times: once with `comm` being a communicator of size 2, once with `
5967

6068
## Schedulers ##
6169

62-
**pytest_parallel** comes with 3 schedulers. To understand how they work, let's take the following example:
70+
**pytest_parallel** comes with three kind of schedulers. To understand how they work, let's take the following example:
6371

6472
```Python
6573
import pytest_parallel
@@ -111,7 +119,7 @@ We see that processes 0,1 and 2 wait at the first step for process 3 to finish.
111119

112120
### Dynamic scheduler ###
113121

114-
The **dynamic** scheduler spawns a new MPI process that act as the master scheduler and sends work to the original processes. The scheduler tries to schedule tests requiring the most processes first. The scheduler tries to send work to idle process until all the processes are busy executing one test, or when not enough processes are ready accept a test. It then waits for a signal that workers have finished their test to schedule further work.
122+
The **dynamic** scheduler spawns a new MPI process which acts as the master scheduler and sends work to the original processes. The scheduler tries to schedule tests requiring the most processes first. The scheduler tries to send work to idle process until all the processes are busy executing one test, or when not enough processes are ready to accept a test. It then waits for a signal that workers have finished their test to schedule further work.
115123

116124
Example:
117125
![static scheduler sequence diagram - bad case](doc/images/dyn_anim.png)

‎pyproject.toml‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
[build-system]
2+
requires = [
3+
"setuptools",
4+
]
5+
build-backend = "setuptools.build_meta"
6+
7+
[tool.setuptools.packages.find]
8+
where = ["."]
9+
exclude = ["pytest_mpi_check"]
10+
namespaces = false
11+
12+
[project]
13+
name = "pytest_parallel"
14+
description = "Plugin to manage test in distributed way with MPI Standard"
15+
readme = "README.md"
16+
authors = [
17+
{name = "Bruno Maugars", email = "bruno.maugars@onera.fr"},
18+
{name = "Berenger Berthoul", email = "berenger.berthoul@onera.fr"},
19+
]
20+
maintainers = [
21+
{name = "Bruno Maugars", email = "bruno.maugars@onera.fr"},
22+
]
23+
license = {text = "Mozilla Public License 2.0"}
24+
keywords = [
25+
"pytest",
26+
"report",
27+
]
28+
classifiers = [
29+
"Development Status :: 4 - Beta",
30+
"Intended Audience :: Developers",
31+
"Intended Audience :: Information Technology",
32+
"Intended Audience :: Science/Research",
33+
"License :: OSI Approved :: Mozilla Public License 2.0 (MPL 2.0)",
34+
"Operating System :: Unix",
35+
"Operating System :: POSIX :: Linux",
36+
"Operating System :: MacOS :: MacOS X",
37+
"Operating System :: OS Independent",
38+
"Programming Language :: Python",
39+
"Programming Language :: Python :: 3",
40+
"Programming Language :: Python :: 3.8",
41+
"Programming Language :: Python :: 3.9",
42+
"Programming Language :: Python :: 3.10",
43+
"Programming Language :: Python :: 3.11",
44+
"Programming Language :: Python :: 3 :: Only",
45+
"Programming Language :: Python :: Implementation :: CPython",
46+
"Topic :: Software Development :: Testing",
47+
"Topic :: Software Development :: Libraries :: Python Modules",
48+
]
49+
requires-python = ">=3.8"
50+
dependencies = [
51+
"pytest>=6.2.5",
52+
"mpi4py",
53+
"numpy",
54+
]
55+
version = "1.0.0"
56+
57+
[project.urls]
58+
Homepage = "https://github.com/onera/pytest_parallel"
59+
Source = "https://github.com/onera/pytest_parallel"
60+
Documentation = "https://github.com/onera/pytest_parallel"
61+
"Release notes" = "https://github.com/onera/pytest_parallel/releases"
62+
63+
[project.entry-points.pytest11]
64+
parallel = "pytest_parallel.plugin"
65+
parallel_fixtures = "pytest_parallel.fixtures"

‎pytest_mpi_check/__init__.py‎

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -4,11 +4,12 @@
44
"""
55
__version__ = "0.1"
66

7+
78
def assert_mpi(comm, rank, cond, *args):
8-
if comm.rank == rank:
9-
if isinstance(cond, bool):
10-
assert cond
9+
if comm.rank == rank:
10+
if isinstance(cond, bool):
11+
assert cond
12+
else:
13+
assert cond(*args)
1114
else:
12-
assert cond(*args)
13-
else:
14-
pass
15+
pass

‎pytest_mpi_check/_decorator.py‎

Lines changed: 10 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,13 @@
11
import pytest
22

3+
34
def mark_mpi_test(n_proc_list):
4-
if isinstance(n_proc_list, int):
5-
n_proc_list = [n_proc_list] # One integer `i` is considered equivalent to `[i]`
6-
7-
def parallel_impl(tested_fun):
8-
return pytest.mark.parametrize('sub_comm', n_proc_list, indirect=['sub_comm']) (
9-
(tested_fun)
10-
)
11-
return parallel_impl
5+
if isinstance(n_proc_list, int):
6+
n_proc_list = [n_proc_list] # One integer `i` is considered equivalent to `[i]`
7+
8+
def parallel_impl(tested_fun):
9+
return pytest.mark.parametrize("sub_comm", n_proc_list, indirect=["sub_comm"])(
10+
(tested_fun)
11+
)
12+
13+
return parallel_impl

‎pytest_mpi_check/plugin.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1 +1,2 @@
11
from pytest_parallel.plugin import *
2+
from pytest_parallel.fixtures import *

0 commit comments

Comments
 (0)