Checking in JSON

Tue 08 January 2019 by Moshe Zadka

JSON is a useful format. It might not be ideal for hand-editing, but it does have the benefit that it can be hand-edited, and it is easy enough to manipulate programmatically.

For this reason, it is likely that at some point or another, checking in a JSON file into your repository will seem like a good idea. Perhaps it is even beyond your control: some existing technology uses JSON as a configuration file, and the easiest thing is to go with it.

It is useful to still keep the benefit of programmatic manipulation. For example, if the JSON file encodes a list of numbers, and we want to add 1 to every even number, we can do:

with open("myfile.json") as fp:
    content = json.load(fp)
content = [x + (2 % i) for i, x in enumerate(content)]
with open("myfile.json", "w") as fp:
    json.dumps(fp, content)

However, this does cause a problem: presumably, before, the list was formatted in a visually-pleasing way. Having dumped it, now the diff is unreadable -- and hard to audit visually.

One solution is to enforce consistent formatting.

For example, using pytest, we can write the following test:

def test_formatting():
    with open("myfile.json") as fp:
        raw = fp.read()
    content = json.loads(raw)
    redumped = json.dumps(content, indent=4) + "\n"
    assert raw == redumped

Assuming we gate merges to the main branches on passing tests, it is impossible to check in something that breaks the formatting. Automated programs merely need to remember to give the right options to json.dumps. However, what happens when humans make mistakes?

It turns out that Python already has a command-line tool to reformat:

$ python -m json.tool myfile.json > myfile.json.formatted
$ mv myfile.json.formatted myfile.json

A nice test failure will remind the programmer of this trick, so that it is easy to do and check in.


Office Hours

Sat 08 December 2018 by Moshe Zadka

If you want to speak to me, 1-on-1, about anything, I want to be able to help. I am a busy person. I have commitments. But I will make the time to talk to you.

Why?

  • I want to help.
  • I think I'll enjoy it. I like talking to people …
read more

Common Mistakes about Generational Garbage Collection

Wed 28 November 2018 by Moshe Zadka

(Thanks to Nelson Elhage and Saivickna Raveendran for their feedback on earlier drafts. All mistakes that remain are mine.)

When talking about garbage collection, the notion of "generational collection" comes up. The usual motivation given for generational garbage collection is that "most objects die young". Therefore, we put the objects …

read more

The Conference That Was Almost Called "Pythaluma"

Wed 07 November 2018 by Moshe Zadka

As my friend Thursday said in her excellent talk (sadly, not up as of this time) naming things is important. Avoiding in-jokes is, in general, a good idea.

It is with mixed feelings, therefore, that my pun-loving heart reacted to Chris's disclosure that the most common suggestion was to call …

read more

Why No Dry Run?

Sat 06 October 2018 by Moshe Zadka

(Thanks to Brian for his feedback. All mistakes and omissions that remain are mine.)

Some commands have a --dry-run option, which simulates running the command but without taking effect. Sometimes the option exists for speed reasons: just pretending to do something is faster than doing it. However, more often this …

read more

Managing Dependencies

Sun 02 September 2018 by Moshe Zadka

(Thanks to Mark Rice for his helpful suggestions. Any mistakes or omissions that remain are my responsibility.)

Some Python projects are designed to be libraries, consumed by other projects. These are most of the things people consider "Python projects": for example, Twisted, Flask, and most other open source tools. However …

read more

Tests Should Fail

Thu 02 August 2018 by Moshe Zadka

(Thanks to Avy Faingezicht and Donald Stufft for giving me encouragement and feedback. All mistakes that remain are mine.)

"eyes have they, but they see not" -- Psalms, 135:16

Eyes are expensive to maintain. They require protection from the elements, constant lubrication, behavioral adaptations to protect them and more. However …

read more

Thank you, Guido

Thu 02 August 2018 by Moshe Zadka

When I was in my early 20s, I was OK at programming, but I definitely didn't like it. Then, one evening, I read the Python tutorial. That evening changed my mind. I woke up the next morning, like Neo in the matrix, and knew Python.

I was doing statistics at …

read more

Composition-oriented programming

Sun 01 July 2018 by Moshe Zadka

A common way to expose an API in Python is as inheritance. Though many projects do that, there is a better way.

But first, let's see. How popular is inheritance-as-an-API, anyway?

Let's go to the Twisted website. Right at the center of the screen, at prime real-estate, we see:

What's …

read more

Avoiding Private Methods

Fri 01 June 2018 by Moshe Zadka

Assume MyClass._dangerous(self) is a private method. We could have implemented the same functionality without a private method as follows:

  • Define a class InnerClass with the same __init__ as MyClass
  • Define InnerClass.dangerous(self) with the same logic of MyClass._dangerous
  • Make MyClass into a wrapper class over InnerClass …
read more