Write Python like an expert

Sun 17 December 2017 by Moshe Zadka

Ten tricks to level up your Python.

Trick 0 -- KISS

Experts know about the weird dark corners of Python -- but do not use them in production code. The first tip is remembering that while Python has some interesting corners, they are best avoided in production code.

Make your code as straightforward as possible.

Trick 1 -- The power of lists

The humble list, or even humbler [], pack a lot of punch -- for those who know how to use it.

It serves, of course, as a useful array type. It is also a good stack, using append and pop(), with the correct (amortized) performance characteristic. The .sort() method is sophisticated enough it is one of the few cases where Python actually broke new theoretical grounds on a sorting algorithm -- timsort was originally invented for it.

Trick 2 -- The power of dicts

The humble dict, or even humbler {}, also pack a lot of punch.

While many use string keys, it is important to remember any immutable type is possible as keys, including tuples and frozensets. This helps writing caches, memoizers or even a passable sparse array.

The keyword argument constructor also gives it a lot of power for making simple and readable APIs.

Trick 3 -- Iterators and generators

The iterator protocol is one of the most powerful aspects of Python. Experts understand it deeply, and know how to use it to make code shorter, more readable, more composable and more debuggable.

One of the easiest ways to accomplish it is to write functions that accept an iterator and return an iterator: and remembering that generators are really good syntactic sugar for writing functions which return iterators.

If a code base has a lot of functions that return iterators, the iterator algebra functions in itertools become immediately higher value.

Trick 4 -- Collections

The collections module has a lot of wonderful functionality.

For code that needs defaults, defaultdict.

For code that needs counting, Counter.

For FIFOs, deque.

Trick 5 -- attrs

One thing that is not wonderful about the collections module is the namedtuple class.

In almost every way imaginable, the attrs package is better. Also, for things that wouldn't be namedtuples otherwise, attrs is still better.

Trick 6 -- First class functions and types

Return functions. Store them in lists, or dictionaries. Keep classes in a double-ended queue. These are not a "Python does what". These are ways to avoid boilerplate or needless indirections.

Trick 7 -- Unit tests and lint

Experts hate having to waste time. Writing unit tests makes sure they have to fix any given bug only once. Correctly configuring a linter makes sure they do not have to comment on every pull request with a list of nitpicks.

Trick 8 -- Immutability

Immutable data structures, such as those available from the Pyrsistent library, are useful for avoiding a lot of bugs. "Global mutable state is the root of all evil" -- and if you cannot get rid of things being global (modules, function defaults and other things) it is often possible to make them mutable.

Immutable data structures are much easier to reason about, and much harder to make bugs that are hard to find and trigger.

Trick 9 -- Not reinventing the wheel

If something is available as a wheel, don't reinvent it. PyPI has ~125K packages, at times of writing. It is almost certain that it has something that takes care of some of the task you are currently working on.

How to know what's worthwhile?

Follow Planet Python, check Awesome python and, if it is within reach, try to go to Python meetups or conferences. (If it's not, of even if it is, PyVideo has the videos -- but talking to other Python programmers is extremely useful.)

Interesting text encodings (and the people who love them)

Wed 13 December 2017 by Moshe Zadka

(Thanks to Tom Prince and Nelson Elhage for suggestions for improvement.)

Nowadays, almost all text will be encoded in UTF-8 -- for good reasons, it is a well thought out encoding. Some of it will be in Latin 1, AKA ISO-8859-1, which is popular in the western world. Less of it …

read more

Exploration Driven Development

Sun 10 December 2017 by Moshe Zadka

"It's ok to mess up your own room."

Sometime there is a problem where the design is obvious -- at least to you. Maybe it's simple. Maybe you've solved one like that many times. In those cases, just go ahead -- use Test-Driven-Development, lint your code as you're writing, and push a …

read more

Abstraction Cascade

Tue 14 November 2017 by Moshe Zadka

(This is an adaptation of part of the talk Kurt Rose and I gave at PyBay 2017)

An abstraction cascade is a common anti-pattern in legacy system. It is useful to understand how to recognize it, how it tends to come about, how to fix it -- and most importantly, what …

read more


Mon 13 November 2017 by Moshe Zadka

Gather is a plugin framework -- and it now has its own blog.

Use it! If you like it, tell us about it, and if there is a problem, tell us about that.

read more

Brute Forcing AES

Wed 27 September 2017 by Moshe Zadka

Thanks to Paul Kehrer for reviewing! Any mistakes or oversights that are left are my responsibility.

AES's maximum key size is 256 bits (there are also 128 and 192 bit versions available). Is that enough? Well, if there is a cryptographic flaw in AES (i.e., a way to recover …

read more

Announcing NColony 17.9.0

Tue 19 September 2017 by Moshe Zadka

I have released NColony 17.9.0, available in a PyPI near you.

New this version:

  • CalVer
  • Python 3 support!
  • You can ask to, explicitly, inherit environment variables from the monitoring process.
  • Website

Thanks to Mark Williams for reviewing many pull requests.

read more

SSH to EC2

Wed 30 August 2017 by Moshe Zadka

(Thanks to Donald Stufft for reviewing this post, and to Glyph Lefkowitz for inspiring much of it.)

(JP Calderone wrote a Twisted version of this approach.)

It is often the case that after creating an EC2 instance in AWS, the next step is SSHing. This might be because the machine …

read more

Python as a DSL

Mon 07 August 2017 by Moshe Zadka and Mark Williams

This is a joint post by Mark Williams and Moshe Zadka. You are probably reading it on one of our blogs -- if so, feel free to look at the other blog. We decided it would be fun to write a post together and see how it turns out. We definitely …

read more

Image Editing with Jupyter

Tue 25 July 2017 by Moshe Zadka

With the news about MS Paint going away from the default MS install, it might be timely to look at other ways to edit images. The most common edit I need to do is to crop images -- and this is what we will use as an example.

My favorite image …

read more