Python as a DSL

Introduction

A Domain Specific Language is a natural solution to many problems. However, creating a new language from whole cloth is both surprisingly hard and, more importantly, surprisingly hard to get right.

One needs to come up with a syntax that is easy to learn, easy to get right, hard to get wrong, and has the ability to give meaningful errors when incorrect input is given. One needs to carefully document the language, supplying at least a comprehensive reference, a tutorial, and a best practices guide all with examples.

On top of this, one needs to write a toolchain for the language that is as high quality as the one users are used to from other languages.

All of this raises a tempting question: can we use an existing language? In this manner, many languages have been used, or abused, as domain specific languages -- Lisp variants (such as Scheme) were among the first to be drafted, but were quickly followed by languages like TCL, Lua, and Ruby.

Python, being popular in quite a few niches, has also been used as a choice for things related to those niches -- the configuration format for Jupyter, the website structure specification in Pyramid the build directives for SCons, and the target specification for Pants.

In this post, we will show examples of Python as a Domain Specific Language (or DSL) and explain how to do it well -- and how to avoid doing it badly.

As programmers we use a variety of languages to solve problems. Usually these are "general purpose" languages, or languages whose design allows them to solve many kinds of problems equally well. Python certainly fits this description. People use it to solve problems in astronomy and biology, to answer questions about data sets large and small, and to build games, websites, and DNS servers.

Python programmers know how much value there is in generality. But sometimes that generality makes solving a problem tedious or otherwise difficult. Sometimes, a problem or class of problems requires so much set up, or has so many twists and turns, that its obvious solution in a general purpose language becomes complicated and hard to understand.

Domain specific languages are languages that are tailored to solve specific problems. They contain special constructions, syntax, or other affordances that organize patterns common to the problems they solve.

Emacs Lisp, or Elisp, is a Domain Specific Language focused on text editing. Emacs users can teach Emacs to do novel things by extending the editor with Elisp.

Here's an example of an Elisp function that swaps ' with " and vice-versa when the cursor is inside a Python string:

(defun python-swap-quotes ()
  "Swap single and double quotes."
  (interactive)
  (save-excursion
    (let ((state (syntax-ppss)))
      (when (eq 'string (syntax-ppss-context state))
        (let* ((left (nth 8 state))
               (right (1- (scan-sexps left 1)))
               (newquote (if (= ?' (char-after left))
                             ?\" ?')))
          (dolist (loc (list left right))
            (goto-char loc)
            (delete-char 1)
            (insert-char newquote 1)))))))

This is clearly Lisp code, and parts of it, such as defining a function with defun or variables with let, is not specific to text editing or even Emacs.

(interactive), however, is a special extension to Elisp that makes the function that encloses it something a user can assign to a keyboard short cut or select from a menu inside Emacs. Similarly, (save-excursion ...) ensures that file the user is editing and the location of the cursor is restored fter the code inside is run. This allows the function to jump around within a file or even multiple files without disturbing a user's place.

Lots of Elisp code makes use of special extensions, but Python programmers don't complain about their absence, because they're of no use outside Emacs. That specialization makes Elisp a DSL.

The language of Dockerfiles is also a domain specific language. Here's a simple hello world Dockerfile:

FROM scratch
COPY hello /
ENTRYPOINT ["/hello"]

The word that begins each line instructs Docker to perform some action on the arguments that follow, such as copying the file hello from the current directory into the image's root directory. Some of these commands have meaning specifically to Docker, such as the FROM command to underlay the image being built with a base image.

Note that unlike Elisp, Dockerfiles are not Turing complete, but both are DSLs. Domain specificity is distinct from mathematical concepts like decidability. It's a term we use to describe how specialized a language is to its problem domain, not a theoretical Computer Science term.

Code written in a domain specific language should be clearer and easier to understand because the language focuses on the domain, while the programmer focuses on the specific problem.

The Elisp code won't win any awards for elegance or robustness, but it benefits from the brevity of (interactive) and (save-excursion ..). Most of the function consists of the querying and computation necessary to find and rewrite Python string literals. Similarly, the Dockerfile doesn't waste the reader's attention on irrelevant details, like how a base image is found and associated with the current image. These DSLs keep their programs focused on their problem domains, making them easier to understand and extend.

Naive Usage of Python as a DSL

Programmers describe things that hide complexity behind a dubiously simple facade as magic. For some reason, when the idea of using Python as a DSL first comes up, many projects choose the strategy we will call "magical execution context". It is more common in projects written in C/C++ which embed Python, but happens quite a bit in pure-Python projects.

The prototypical code that creates a magical execution context might look something like:

namespace = dict(DomainFunction1=my_domain_function1,
                 DomainFunction2=my_domain_function2)
with open('Domainspecificfile') as fp:
    source = fp.read()
exec(source, globals=namespace)
do_something_with(namespace['special_name'])

Real-life DSLs usually have more names in their magical execution contexts (often ranging in the tens or sometimes hundreds), and DSL runtimes often have more complicated logic around finding the files they parse. However, this platonic example is useful to keep in mind when reading through the concrete examples.

While various other projects were automatable with Python, SCons might be the oldest surviving project where Python is used as a configuration language. It also happens to be implemented in Python -- but aside from making the choice of Python as a DSL easier to implement, it has no bearing on our discussion today.

An SCons file might look like this:

src_files = Split("""main.c
                     file1.c
                     file2.c""")
Program('program', src_files)

Code can also be imported from other files:

SConscript(['drivers/SConscript',
            'parser/SConscript',
            'utilities/SConscript'])

Note that it is not possible, via this method, to reuse any logic other than build settings across the files -- a function defined in one of them is not available elsewhere else.

At 12 years old, Django is another venerable Python project, and like the similarly venerable Ruby on Rails, it's no stranger to magic. Once upon a time, Django's database interaction APIs were magical enough that they constituted a kind of domain-specific language with a magical execution context.

Like modern Django, you would define your models by subclassing a special class, but unlike modern Django, they were more than just plain old Python classes.

A Django application in a module named best_sellers.py might have had a model that looked like this:

from django.core import meta

class Book(meta.Model):
      name = meta.CharField(maxlength=70)
      author = meta.CharField(maxlength=70)
      sold = meta.IntegerField()
      release_date = meta.DateTimeField(default=meta.LazyDate())

      def get_best_selling_authors(self):
          cursor = db.cursor()
          cursor.execute("""
          SELECT author FROM books WHERE release_date > '%s'
          GROUP BY author ORDER BY sold DESC
          """ % (db.quote(datetime.datetime.now() - datetime.timedelta(weeks=1)),))
          return [row[0] for row in cursor.fetchall()]

      def __repr__(self):
          return self.full_name

A user would then use it by like so:

from django.models.best_sellers import books
print books.get_best_selling_authors()

Django transplated the Book model into its own magic models module and renamed it books. Note the subtle transformation in the midst of more obvious magic: the Book model was lowercased and automatically pluralized.

Two magic globals were injected into the model's instance methods: db, the current database connection, and datetime, the Python standard library module. That's why our example module doesn't have to import them.

The intent was to reduce boilerplate by exploiting Python's dynamicism. The result, however, diverged from Python's expected behaviors and also invented new, idiosyncratic boilerplate; in particular, the injection of special globals prevented methods from accessing variables defined in their source modules, so methods had to directly import any module they used, forcing programmers to repeat themselves.

Django's developers came to see these features as "warts" and removed them before the 0.95 release. It's safe to say that the "magic-removal" process succeeded in improving Django's usability.

Python has well-documented built-ins. People who read Python code are, usually, familiar with those. Any symbol which is not a built-in or a reserved word is imported.

Any DSL will have its own, extra built-ins. Ideally, those are well documented -- but even when they are, this is a source of documentation separate from the host language. This code can never be used by something outside the DSL. A good example for such potential usage is for unit testing the code. Once a DSL catches on, it often inspires creation of vast amounts of code. The example of Elisp is particularly telling.

Another problem with such code is that it's often not obvious what the code itself is allowed to import or call. Is it safe to do long-running operations? If logging to a file, is logging properly set-up? Will the code double log messages, or does it cache the first time it is used? As an example, there are a number of questions about how to share code between SCons on StackOverflow, with explanations about the trade-offs between using an SConscript file or using Python modules and import.

Last, but not least, other Python code often implicitly assumes that functions and classes are defined by modules. This means that either it is ill-advised to write such in the DSL -- perhaps defining classes might lead to a memory leak because the contents are used in exec multiple times -- or, worse, that random functionality will break. For example, do tracebacks work correctly? Does pickle?

A New Hope

As seen from the examples of SCons and old, magical Django, naively using Python as a DSL is problematic. It gives up a lot of the benefits of using a pre-existing language, and results in something that is in the Python uncanny valley -- just close enough to Python that the distinctions result in a sense of horror, not cuteness.

One way to avoid the uncanny valley is to step further away and avoid confusion -- implement a little language using PyParsing that is nothing like Python. But valleys have two sides. We can solve the problem by just using pure, unadulterated Python. It turns out that removing an import statement at the top of the file does not reduce much overhead when specializing to a domain.

We explore, by example, good ways to use Python as DSL. We start by showing how even a well written module, taking advantage of some of the power of Python, can create a de-facto DSL. Taking it to the next level, frameworks (which call user code) can also be used to build DSLs in Python. Most powerfully, especially combined with libraries and frameworks, Python plugin systems can be used to avoid even the need for a user-controlled entry point, allowing DSLs which can be invoked from an arbitrary program.

Python is a flexible language and subtle use of its features can result in a flexible DSL that still looks like Python.

We explore four examples of such DSLs -- NumPy, Stan, Django ORM, and Pyramid.

x[4,...,5,:]

class ItemGetterer(object):
    def __getitem__(self, idx):
        return idx

x = ItemGetterer()
print(x[4,...,5,:])

from nevow import flat, tags, stan

video = stan.Tag('video')

aDocument = tags.html[
                tags.head[
                    tags.title["Title"]
                ],
                tags.body[
                    tags.h1["Heading" ],
                    tags.p(class_="life")["A paragraph about life."],
                    video["Your video here!"],
                ]
            ]
with open('output.html', 'w') as fp:

from nevow import flat, tags
bullets = [tags.li["bullet {}".format(i)] for i in range(10)]
aDocument = tags.html[
                tags.body[
                    tags.ul[bullets]
                ]
            ]
with open('output.html', 'w') as fp:
    fp.write(flat.flatten(aDocument))

from django.db import models


class Question(models.Model):
    question_text = models.CharField(max_length=200)
    pub_date = models.DateTimeField('date published')


class Choice(models.Model):
    question = models.ForeignKey(Question, on_delete=models.CASCADE)
    choice_text = models.CharField(max_length=200)
    votes = models.IntegerField(default=0)

--
-- Create model Choice
--
CREATE TABLE "polls_choice" (
    "id" serial NOT NULL PRIMARY KEY,
    "choice_text" varchar(200) NOT NULL,
    "votes" integer NOT NULL
);
--
-- Create model Question
--
CREATE TABLE "polls_question" (
    "id" serial NOT NULL PRIMARY KEY,
    "question_text" varchar(200) NOT NULL,
    "pub_date" timestamp with time zone NOT NULL
);

# Removed imports

## The function definition can go anywhere
@view_config(route_name='home')
def my_home(context, request):
    return 'OK'

## This goes in whatever file we pass to our WSGI host
config = Configurator()
config.add_route('home', '/')
config.scan('.')
app = config.make_wsgi_app()

Plugins

When code lives in real Python modules, and uses real Python APIs, it is sometimes useful for it to be executed automatically based on context. After all, one thing that DSL systems like SCons give us is automatically executing the SConscript when we run scons at the command line.

One tool that can be used for this is a plugin system. While a comprehensive review of plugin systems is beyond our scope here, we will give a few examples of using such systems for specific domains.

One of the oldest plugin systems is twisted.plugin. While it can be used as a generic plugin system, the main usage of it -- and a good case study of using it as a plugin system for DSLs -- is to extend the twist command line. These are also known as tap plugins, for historical reasons.

Here is a minimal example of a Twisted tap plugin:

# Removed imports
@implementer(IServiceMaker, IPlugin)
class SimpleServiceMaker(object):
    tapname = "simple-dsl"
    description = "The Simplest DSLest Plugin"

    class options(usage.Options):
        optParameters = [["port", "p", 1235, "Port number."]]

    def makeService(self, options):
        return internet.TCPServer(int(options["port"]),
                                  Factory.forProtocol(Echo))

serviceMaker = SimpleServiceMaker()

In order to be a valid plugin, this file must be placed under twisted.plugins. The usage.Options class defines a DSL, of sorts, for describing command-line options. We used only a small part of it here, but it is both powerful and flexible.

Note that this is completely valid Python code -- in fact, it will be imported as a module. This allows us to import it as well, and to test it using unit tests.

In fact, because this is regular Python code, usually serviceMakers are created using a helper class -- twisted.application.service.ServiceMaker. The definition above, while correct, is not idiomatic.

The gather library does not have a DSL. It does, however, function well as an agnostic plugin discovery mechanism. Because of that, it can be built into other systems -- that do provide a Pythonic DSL -- to serve as the autodiscovery mechanism.

# In  a central module:
ITS_A_DSLISH_FUNCTION = gather.Collector()

## Define the DSL as
## -- functions that get two parameters
##    -- conf holds some general configuration
##    -- send_result is used to register the result
def run_the_function_named(name, conf, send_result):
    res = ITS_A_DSLISH_FUNCTION.collect()
    return res[name](conf, result)

# In a module registering a DSL function
@ITS_A_DSLISH_FUNCTION.register(name='my_dslish_name')
def some_func(conf, send_result):
    with conf.temp_stuff() as some_thing:
         send_result(some_thing.get_value())

Conclusion

Python is a good language to use for DSLs. So good, in fact, that attrs, a DSL for defining classes, has achieved enormous popularity. Operator overloading, decorators, the with operator and generators, among other things, combine to allow novel usage of the syntax in specific problem domains. The existence of a big body of documentation of the language and its best practices, along with a thriving community of practicioners, is also an asset.

In order to take advantage of all of those, it is important to use Python as Python -- avoid magical execution contexts and novel input search algorithms in favor of the powerful code organization model Python already has -- modules.

Most people who want to use Python as a DSL are also Python programmers. Consider allowing your program's users to use the same tools that have made you successful.

As Glyph said in a related discussion, "do you want to confuse, surprise, and annoy people who may be familiar with Python from elsewhere?" Assuming the answer is "no", consider using real modules as your DSL mechanism.