The Python Toolbox

Thu 01 February 2018 by Moshe Zadka

I have written before about Python tooling. However, as all software, things have changed -- and I wanted to write a new post, with my current understanding of best practices.

Testing

As of now, pytest has achieved official victory. Unless there are overwhelming reasons to use something else, strongly consider using it as your test runner. The only good reason is an existing project, with an existing test runner -- and even there, checking if pytest will just run your tests as is is worthwhile.

Even when using Twisted, unless you are implementing a new reactor, tests should be using in-memory reactors -- so the usefulness of trial is limited.

Static checking

In the static checking arena, flake8 and pylint are both still going strong. There are less and less reasons not to use both, especially if starting a project from scratch.

The flake8 plugin ecosystem is rich, and it is useful to look around and see if useful things are there. For example, the ebb-lint plugin is extremely strict about coding conventions -- hard to introduce to existing code bases, but wonderful when starting out a new one.

In the meantime, pylint has decent static code flow analysis which can often prevent bugs. Note, however, that Python static code analysis is hard, and pylint is not perfect. For example, it will often strugle with attrs-based classes.

Last but not least, mypy has made a showing in the field. It supports both Python 2 an 3, and allows annotating functions with types, in order to find mismatches before running the code. This is especially useful at API "boundaries", where the unit tests tend to cross less.

Test metarunners

The tox testing system is still the golden standard in Python. It can test complicated dependency matrixes, and allows configuring test commands flexibly. Its documentation is somewhat lacking, sadly -- it is often the case new tricks are only apparently after reading someone else's tox file.

Output

Building wheels, especially if the project has no native-code extensions, is nowadays considered standard. The best place to build wheels is in tox, when configuring a test that will build a wheel, install it, and then test against the installed wheel.

The best and most reliable way to upload wheels, and source distributions, to PyPI is to use twine. It used to be a good idea to test against the test PyPI server, but nowadays it is best to set up a devpi server for local testing.

When building applications, pex is a great output format. It allows a one-file install.

Future

The future is bright -- pip 10 is slated to come out, supporting the pyproject.toml format -- and hopefully the next post in the series will explain how to make packages using flit, with no setup.py.


Jupyter for SRE

Sat 30 December 2017 by Moshe Zadka

Jupyter is a tool that came out of the data science community. In science, being able to replicate experiments is of the utmost importance -- so a tool where you can "show your work" is helpful. However, being able to show your work -- have colleagues validate what you have done, repeat …

read more

Write Python like an expert

Sun 17 December 2017 by Moshe Zadka

Ten tricks to level up your Python.

Trick 0 -- KISS

Experts know about the weird dark corners of Python -- but do not use them in production code. The first tip is remembering that while Python has some interesting corners, they are best avoided in production code.

Make your code as …

read more

Interesting text encodings (and the people who love them)

Wed 13 December 2017 by Moshe Zadka

(Thanks to Tom Prince and Nelson Elhage for suggestions for improvement.)

Nowadays, almost all text will be encoded in UTF-8 -- for good reasons, it is a well thought out encoding. Some of it will be in Latin 1, AKA ISO-8859-1, which is popular in the western world. Less of it …

read more

Exploration Driven Development

Sun 10 December 2017 by Moshe Zadka

"It's ok to mess up your own room."

Sometime there is a problem where the design is obvious -- at least to you. Maybe it's simple. Maybe you've solved one like that many times. In those cases, just go ahead -- use Test-Driven-Development, lint your code as you're writing, and push a …

read more

Abstraction Cascade

Tue 14 November 2017 by Moshe Zadka

(This is an adaptation of part of the talk Kurt Rose and I gave at PyBay 2017)

An abstraction cascade is a common anti-pattern in legacy system. It is useful to understand how to recognize it, how it tends to come about, how to fix it -- and most importantly, what …

read more

Gather

Mon 13 November 2017 by Moshe Zadka

Gather is a plugin framework -- and it now has its own blog.

Use it! If you like it, tell us about it, and if there is a problem, tell us about that.

read more

Brute Forcing AES

Wed 27 September 2017 by Moshe Zadka

Thanks to Paul Kehrer for reviewing! Any mistakes or oversights that are left are my responsibility.

AES's maximum key size is 256 bits (there are also 128 and 192 bit versions available). Is that enough? Well, if there is a cryptographic flaw in AES (i.e., a way to recover …

read more

Announcing NColony 17.9.0

Tue 19 September 2017 by Moshe Zadka

I have released NColony 17.9.0, available in a PyPI near you.

New this version:

  • CalVer
  • Python 3 support!
  • You can ask to, explicitly, inherit environment variables from the monitoring process.
  • Website

Thanks to Mark Williams for reviewing many pull requests.

read more

SSH to EC2

Wed 30 August 2017 by Moshe Zadka

(Thanks to Donald Stufft for reviewing this post, and to Glyph Lefkowitz for inspiring much of it.)

(JP Calderone wrote a Twisted version of this approach.)

It is often the case that after creating an EC2 instance in AWS, the next step is SSHing. This might be because the machine …

read more