Skip to content

Latest commit

 

History

History
271 lines (189 loc) · 20.9 KB

tox.md

File metadata and controls

271 lines (189 loc) · 20.9 KB

tox

  • tox 只能用在 setup.py based project
  • tox 支援的 test tool 有 pytest、nose 與 unittest
  • tox 背後用 virtualenv 來安排不同的 test environment

參考資料：

  • tox 3.0.0rc4 : Python Package Index
    • 開宗明義 vision: standardize testing in Python
    • 基於 virtualenv，專注在 testing；利用 virtualenv 檢查在不同的 test environment (主要是不同的 Python version) 都能安裝成功、通過測試 (支援不同的 test tool)；過程中會自動建立 virtualenv，也難怪會有 "virtualenv managment" 的說法。
    • 做為 CI server 的 frontend??
  • Welcome to the tox automation project — tox 3.0.0rc3.dev10 documentation #ril
    • 自動/標準化 Python 的 testing，支援 "setup.py based project" (透過 setup.py 安裝到不同的 virtual environment)，而 test tool 則支援 pytest、nose、unittest 等。
    • 支援 CPython 2.7/3.4+、Jython 及 pypy。
    • Current features ...
  • Run/Debug Configuration: Tox - Help | PyCharm PyCharm 支援 tox 設定 #ril

跟 setup.py、requirements.txt 的關係 ??

  • tox --develop (install package in the venv using 'setup.py develop') 這種用法的存在，感覺 tox 像是 setup.py 的 frontend。
  • the tox automation project https://tox.readthedocs.io/en/latest/ 提到 "installs your setup.py based project into each virtual environment"，所以執行 tox 時若 setup.py 不存在，會丟出 tox.MissingFile: MissingFile: path/to/project/setup.py 的錯誤。
  • Tox tricks and patterns | ionel's codelog (2015-04-14) 若專案沒有 setup.py 但又想用 tox，在 tox.ini 增加 [tox] skipsdist = true 就可以。

有哪些知名的專案在用 tox?

跟 Docker 的關係 ??

跟 pyenv 的關係 ??

Hello, World!

def test_hello_world(cli):
    cli.src('tox.ini', r"""
    [tox]
    envlist = py27,py36

    [testenv]
    deps = pytest
    commands = pytest
    """)

    # setup.py in the same directory
    cli.src('setup.py', r"""
    from setuptools import setup

    setup(
        name='hello-world',
        packages=['hello'],
    )
    """)

    # production code
    cli.src('hello/__init__.py')
    cli.src('hello/hello.py', r"""
    def say_hello(who='World'):
        return 'Hello, %s!' % who
    """)

    # test code; at least one test
    cli.src('tests/test_hello.py', r"""
    from hello import hello

    def test_hello():
        assert hello.say_hello() == 'Hello, World!'
        assert hello.say_hello('pytest') == 'Hello, pytest!'
    """)

    r = cli.run('tox')
    assert 'py27: commands succeeded' in r.out
    assert 'py36: commands succeeded' in r.out

    # directories .tox/ and *.egg-info/ got created
    assert cli.exists('.tox/')
    assert cli.exists('hello_world.egg-info/')

參考資料：

  • Basic example - Welcome to the tox automation project — tox 3.0.0rc3.dev10 documentation #ril

    • 透過 pip install tox 安裝，寫 tox.ini (也可以用 tox-quickstart 產生)，之後就能用 tox 指令來做事。

    • tox.ini 放在 setup.py 同一目錄下，裡面宣告要進行測試的多個 (virtualenv) test environment，以及會用到什麼套件，例如：

      [tox]
envlist = py27,py36

[testenv]
deps = pytest
commands = pytest

    • 執行 tox 就會打包 sdist package，分別安裝到 Python 2.7 與 3.6 的 virtualenv 進行測試 (第一次跑 tox 還要安裝 dependencies 比較花時間)；有打包進 package 才測得到!!

    • 注意不同版本的 Python 要自己先安裝好，否則會發生錯誤，這不是 tox 可以幫忙做掉的。

  • Basic usage — tox 3.0.0rc3.dev10 documentation #ril

Configuration ??

使用 pytest ??

tox (CLI) ??

tox-quickstart (CLI) ??

測試已經打包好的 package ??

Private Package Index ??

搭配 setup.py test 使用

tox 一定要搭配 setup.py 使用，但 setuptools 測試的慣例是 setup.py test，如何讓 setup.py test 用 tox 執行測試?

tox-setuptools 可以讓你輕鬆完成這件事，不過 2016/10 後，整合 tox 與 setup.py test 就不被鼓勵，因為 setup.py test 預期在同一個 invocation interpreter，而不會像 tox 另外建立 virtualenvs，這會衍生一些問題，所以若真要整合的話，官方建議整合最終的 test tool (例如 pytest)。

不過按照 tox 的說法，Python 漸漸不以 setup.py 做為 tool entry point，整合 setup.py test 好像也沒什麼必要？

As the python eco-system rather moves away from using setup.py as a tool entry point it’s maybe best to not go for any setup.py test integration.

-- Integration with “setup.py test” command - Basic usage]

參考資料：

CI Integration ??

Virtualenv Test Environment, Default Environment, Matrix ??

參考資料：

tox.ini 要怎麼寫?

  • 可以用 tox-quickstart 產生，再做修改。
[tox]
envlist = py26, py27

[testenv]
commands = pytest
deps =
    pytest

參考資料：

產生 JUnit XML report?

  • Using Tox with the Jenkins Integration Server — tox documentation http://tox.readthedocs.io/en/latest/example/jenkins.html 加上 --junitxml=junit-{envname}.xml 就會產生 JUnit XML reports；這裡 --junitxml 是 pytest 的用法，但 tox 提供 {envname} 這個變數。
  • 如何合併多個 environment 的結果?

合併多個測試環境的 Coverage ??

  • python - How can I combine coverage results with tox? - Stack Overflow
    • Tim Martin: 假設有安裝 pytest-coverage，可以在 [tool:pytest].addopts 加上 --cov-append (其實就是為 pytest 加上 --cov-append 參數)，這會合併多次 test run 的結果，有助於把拆分 integration/unit tests。但後來提問的 Martin Thoma 說這並沒有解決問題 XD

執行速度很慢 ?? {: #slow }

  • tox startup is very slow due to setup.py and .tox virtualenvs · Issue #293 · tox-dev/tox

    • spookylukey: It could be considered a performance bug in setuptools, but fixing that is looking quite hard. I'm wondering if fixing it via tox would be better - after all, tox is triggering the really poor performance of setup.py by PUTTING THOUSANDS OF FILES IN THE WORKING DIRECTORY, and then suffering from it.

      • Solution 1: specify a toxworkdir outside the package directory - http://tox.readthedocs.org/en/latest/config.html#tox-global-settings

        The problem with this is that it is difficult to find a place which would be appropriate when you consider multiple people working on a project, possibly on different platforms. Also, it needs to be applied in every project that notices the problem.

        但為什麼把 toxworkdir 從 {toxinidir}/.tox 改到其他地方就會變快?

      • Solution 2: have a different default value for toxworkdir, something like ~/.cache/tox/virtualenvs/{project}, where {project} gets substituted by something appropriate, like a mangled version of the working directory, perhaps combined with a platform name or something. This might also address issue #44 https://bitbucket.org/hpk42/tox/issues/44/cant-share-a-tox-directory-between-os-x

    • spookylukey: pypa/setuptools#450 was closed without a patch being merged, but pypa/setuptools#764 was merged, which hopefully addresses this.

    • spookylukey: With my test case, instead of tox having a startup overhead of: 10 minutes (cold start) or 40 seconds (warm start) ...it now has a startup overhead of: 40 seconds (cold start) or 1-2 seconds (warm start). Yay! This makes a world of difference.

      To see these benefits, you will need setuptools 28.5 or later in the virtualenv that has tox in it, and I think also in the virtualenvs that tox builds - so you may need to use tox --recreate to force the new version of setuptools to be installed. You also need to avoid use of global_include in your MANIFEST file, which will force all files to be scanned - and is probably not what you wanted anyway, because it will include files from hidden directories like .git, .hg, .tox etc.

      The tests aren't exact for a bunch of reasons, but should give a rough idea. The speedup I'm seeing is DUE TO TOX HAVING A LARGE NUMBER OF VIRTUALENVS, so in other cases it will be less extreme. The point to note is that you are no longer punished for having a large number of virtualenvs, which were being unnecessarily scanned by setuptools before (due to them being stored within .tox in the project directory).

      什麼是 code/warm start ?? 不過把 .tox 搬出 project directory 真的變很快!! 跑 tox --notest 準備多個 virtualenvs 時就感受得到。

安裝設置 {: #setup }

  • 在 virtualenv 下用 pip install tox 安裝，就會有 tox 與 tox-quickstart 指令可用。

參考資料：

參考資料 {: #reference }

社群：

手冊：