Jupyter for SRE

Sat 30 December 2017 by Moshe Zadka

Jupyter is a tool that came out of the data science community. In science, being able to replicate experiments is of the utmost importance -- so a tool where you can "show your work" is helpful. However, being able to show your work -- have colleagues validate what you have done, repeat it if needs be, and learn new techniques -- is also useful in the world of Site Reliability Engineering and DevOps.

The Jupyter console allows us to experiment (carefullly!) with APIs, running one function at a time, and validating the results. It allows building the needed automation from simple atoms, all the while learning how to use them.

The Jupyter Python kernel is popular in the data science community because so many data science libraries are available for Python. Luckily, the same is true in the SRE/DevOps community.

import github3
import os

The GitHub API has several client libraries in Python. My personal favorite is github3 -- I find its interface allowing remarkably idiomatic Python.

fname=os.path.expanduser("~/.git-token")
with open(fname) as fin:
    token = fin.read().strip()
gh = github3.login(token=token)

I have prepared a GitHub authentication token in a file. This allows the NoteBook to be published widely, without leaking access credentials. Never put usernames and passwords in Jupyter notebooks!

repositories = list(gh.organization('twisted').iter_repos())
repositories[:3], len(repositories)
([<Repository [twisted/txmongo]>,
  <Repository [twisted/twisted]>,
  <Repository [twisted/txaws]>],
 26)

This is a list of the repositories in the Twisted GitHub organization. It is nice to be able to validate we got a reasonable value by checking the first three. In previous versions of the notebook, my usage of github3 had an error -- and this was a list of all repositories I had access to, including private ones! Being able to inspect values interactively meant I could fix the bug easily.

repositories[1].branch('trunk').commit.sha
'51d23b6cb004e60ed699b3b100b4c476f5aebc4e'

As an example of why this might be useful, we are checking the commit hash of the trunk branch. This can be used in validating which version we are running somewhere, or checking if there have been new commits.

The GitHub API is big, and this is not meant to be an exhaustive guide to it. However, this approach is powerful -- it can be used, for example, to automatically create pull requests for a list of repositories. This comes in handy when needing to change the build structure, for example.

from fabric import api as fabpi

The Fabric library, (here used in its fabric3 fork) is an all-purpose ad-hoc library for operations.

Again, a full tutorial is beyond the scope of this article.

fabpi.local("uname -a", capture=True)

However, the advantages of running Fabric from Jupyter notebook are big. Because Fabric is specifically designed for ad-hoc operations, there is often no way to know exactly what someone did. Even with a fabfile, the logging is often lacking.

Running the functions from a notebook means an official log of what was done -- that can easily be attached to the relevant ticket, or to the post-mortem. (Either the post-mortem that the operations were meant to fix, or the ones they inevitably caused).

import docker
client = docker.DockerClient(base_url='unix://var/run/docker.sock')

The Docker client is also available as a Python library. Once again, the possibilities are endless.

[im for im in client.images.list() if (im.tags or [''])[0].startswith('debian')]
[<Image: 'debian:latest'>,
 <Image: 'debian:stable-slim', 'moshez/debian:2017-10-26T10-58-56-455022'>]
import boto3

The boto3 library is an API for the Amazon Web Services -- which includes everything from load balancers, through orchestrating containers, to sending e-mail.

The Jupyter console is a great adjunct to the AWS web console -- while results can often be inspected in the web console, any actions done from the notebook can be repeated, tweaked, and automated.

ec2 = boto3.client('ec2', region_name='us-west-2')

Summary

For a team of SRE/DevOps engineers who are already using Python, the Jupyter notebook allows a great way to communicate about actions taken. It logs the inputs and the outputs, while allowing editing and clarifying.

Note that it is not a useful auditing tool, and should not be used as such. It is meant as a communications tool, attaching notebooks to tickets or e-mails in order to clarify, in a way that can be fed back into a computer for testing and tweaking.


Write Python like an expert

Sun 17 December 2017 by Moshe Zadka

Ten tricks to level up your Python.

Trick 0 -- KISS

Experts know about the weird dark corners of Python -- but do not use them in production code. The first tip is remembering that while Python has some interesting corners, they are best avoided in production code.

Make your code as …

read more

Interesting text encodings (and the people who love them)

Wed 13 December 2017 by Moshe Zadka

(Thanks to Tom Prince and Nelson Elhage for suggestions for improvement.)

Nowadays, almost all text will be encoded in UTF-8 -- for good reasons, it is a well thought out encoding. Some of it will be in Latin 1, AKA ISO-8859-1, which is popular in the western world. Less of it …

read more

Exploration Driven Development

Sun 10 December 2017 by Moshe Zadka

"It's ok to mess up your own room."

Sometime there is a problem where the design is obvious -- at least to you. Maybe it's simple. Maybe you've solved one like that many times. In those cases, just go ahead -- use Test-Driven-Development, lint your code as you're writing, and push a …

read more

Abstraction Cascade

Tue 14 November 2017 by Moshe Zadka

(This is an adaptation of part of the talk Kurt Rose and I gave at PyBay 2017)

An abstraction cascade is a common anti-pattern in legacy system. It is useful to understand how to recognize it, how it tends to come about, how to fix it -- and most importantly, what …

read more

Gather

Mon 13 November 2017 by Moshe Zadka

Gather is a plugin framework -- and it now has its own blog.

Use it! If you like it, tell us about it, and if there is a problem, tell us about that.

read more

Brute Forcing AES

Wed 27 September 2017 by Moshe Zadka

Thanks to Paul Kehrer for reviewing! Any mistakes or oversights that are left are my responsibility.

AES's maximum key size is 256 bits (there are also 128 and 192 bit versions available). Is that enough? Well, if there is a cryptographic flaw in AES (i.e., a way to recover …

read more

Announcing NColony 17.9.0

Tue 19 September 2017 by Moshe Zadka

I have released NColony 17.9.0, available in a PyPI near you.

New this version:

  • CalVer
  • Python 3 support!
  • You can ask to, explicitly, inherit environment variables from the monitoring process.
  • Website

Thanks to Mark Williams for reviewing many pull requests.

read more

SSH to EC2

Wed 30 August 2017 by Moshe Zadka

(Thanks to Donald Stufft for reviewing this post, and to Glyph Lefkowitz for inspiring much of it.)

(JP Calderone wrote a Twisted version of this approach.)

It is often the case that after creating an EC2 instance in AWS, the next step is SSHing. This might be because the machine …

read more

Python as a DSL

Mon 07 August 2017 by Moshe Zadka and Mark Williams

This is a joint post by Mark Williams and Moshe Zadka. You are probably reading it on one of our blogs -- if so, feel free to look at the other blog. We decided it would be fun to write a post together and see how it turns out. We definitely …

read more