Posted on by Michael Catanzaro

On Python Shebangs

So, how do you write a shebang for a Python program? Let’s first set aside the python2/python3 issue and focus on whether to use env. Which of the following is correct?

#!/usr/bin/env python
#!/usr/bin/python

The first option seems to work in all environments, but it is banned in popular distros like Fedora (and I believe also Debian, but I can’t find a reference for this). Using env in shebangs is dangerous because it can result in system packages using non-system versions of python. python is used in so many places throughout modern systems, it’s not hard to see how using #!/usr/bin/env in an important package could badly bork users’ operating systems if they install a custom version of python in /usr/local. Don’t do this.

The second option is broken too, because it doesn’t work in BSD environments. E.g. in FreeBSD, python is installed in /usr/local/bin. So FreeBSD contributors have been upstreaming patches to convert #!/usr/bin/python shebangs to #!/usr/bin/env python. Meanwhile, Fedora has begun automatically rewriting #!/usr/bin/env python to #!/usr/bin/python, but with a warning that this is temporary and that use of #!/usr/bin/env python will eventually become a fatal error causing package builds to fail.

So obviously there’s no way to write a shebang that will work for both major Linux distros and major BSDs. #!/usr/bin/env python seems to work today, but it’s subtly very dangerous. Lovely. I don’t even know what to recommend to upstream projects.

Next problem: python2 versus python3. By now, we should all be well-aware of PEP 394. PEP 394 says you should never write a shebang like this:

#!/usr/bin/env python
#!/usr/bin/python

unless your python script is compatible with both python2 and python3, because you don’t know what version you’re getting. Your python script is almost certainly not compatible with both python2 and python3 (and if you think it is, it’s probably somehow broken, because I doubt you regularly test it with both). Instead, you should write the shebang like this:

#!/usr/bin/env python2
#!/usr/bin/python2
#!/usr/bin/env python3
#!/usr/bin/python3

This works as long as you only care about Linux and BSDs. It doesn’t work on macOS, which provides /usr/bin/python and /usr/bin/python2.7, but still no /usr/bin/python2 symlink, even though it’s now been six years since PEP 394. It’s hard to understate how frustrating this is.

So let’s say you are WebKit, and need to write a python script that will be truly cross-platform. How do you do it? WebKit’s scripts are only needed (a) during the build process or (b) by developers, so we get a pass on the first problem: using /usr/bin/env should be OK, because the scripts should never be installed as part of the OS. Using #!/usr/bin/env python — which is actually what we currently do — is unacceptable, because our scripts are python2 and that’s broken on Arch, and some of our developers use that. Using #!/usr/bin/env python2 would be dead on arrival, because that doesn’t work on macOS. Seems like the option that works for everyone is #!/usr/bin/env python2.7. Then we just have to hope that the Python community sticks to its promise to never release a python2.8 (which seems likely).

…yay?

14 Replies to “On Python Shebangs”

    1. I do believe this is the winning solution — none of the other proposals in the comments so far fit our requirements — but I can’t tell for sure because I’m afraid my eyeballs have begun to bleed….

  2. Have your build system find the python executable and replace it in your scripts. (sucks for projects not using a decent build system but thats another problem)

  3. If those scripts are used during the build processus, then have your build system detect the version of Python to use, then run the scripts with `${PYTHON} script.py` instead.

    (Or have the build system generate the shebang, effectively “building” the script)

    1. That’s the rub, most of the scripts in question are not used during the build. We need them to work from the command line without the developer having to think about which language they’re written in (half the scripts are perl). And they should be usable from the source directory, so no generating shebangs at build time, either.

      1. If those were tools written in C, you’d have to build them before you use them.

        There’s really no reason not to do the same for Python scripts and build them as well. (Where “build” means generating the right shebang)

        This really is the right way. It’s how we, Python developers, actually develop Python apps. (see anarcat’s comment about the setup.py entry point) We never write shebangs for our Python apps, we let the build system write it for us.

        However… Given your scripts are only meant for developers and shouldn’t end up installed in thevdistro, then the Fedora objection to /use/bin/env just doesn’t apply and using `/use/bin/env python2.7` as you suggested seems perfectly fine for your use-case. I think you’re searching for a problem where there isn’t one with this option.

        1. So this is why we don’t write scripts in C. :P You shouldn’t have to build WebKit in order to be able to run build-webkit. (OK, that one is admittedly a perl script.)

          /usr/bin/env python2.7 is probably what we’re going to do. Only problem with this is that it’s horrible and breaks if python2.8 is ever released. But I think we have to assume that will never happen….

          1. Python 2.8 won’t happen.

            I don’t think usong `/usr/bin/env python2.7` is horrible. But if you think so, then the correct solution has been suggested a few times by different commenters already.

            As for scripts written in C… Isn’t that how the Autotools detect features, by building and running lots of tiny C tools during `configure`?

            There really is no difference here: if you want a tool to run on any system without making any assumption about this system, you need a build step to detect how the system works. If you don’t want a build step, you must make assumptions.

  4. It seems to me that

    #!/usr/bin/env python3

    ought to work pretty much anywhere where you have a Python 3. Unless you want to support systems (Mac OS?) which only have Python 2? But what happens in 2020 when Python 2 support ends?

    Personally, I would make my scripts compatible with Python 2 _and_ Python 3 and use

    #!/usr/bin/env python

    but perhaps my many years of Python experience make the task seem simpler than it is.

    Would you like me to take a look at your scripts and see how hard it would be to make them support both Python versions?

    1. I suspect it would be pretty hard, because we have a large number of scripts. See WebKit/Tools/Scripts and also WebKit/Tools/Scripts/webkitpy.

  5. the proper way to do this is to have a `setup.py` file that has an `entry_point` field. then installing (with pip) will do the right thing for you.

    1. actually, i take that back: `entry_point` is great when you have a larger program, but totally breaks down when all you have is a single script. say i have a program called `undertime.py`, it lives (along many others) under (say) `scripts`. It’s (deliberately) not part of a package. I have two options to install this:

      1. setup(scripts=[‘undertime.py’]): that works, but won’t fix the shebang and installs the script in /usr/local/bin/undertime.py (with an ugly suffix)

      2. entry_points={‘console_scripts’: [‘undertime=undertime:main’]},: that “works” in that the shebang is correct, but the program doesn’t actually execute correctly, because `undertime` is not a “package”. if you try to install that using packages=[‘undertime’], you get:

      error: package directory ‘undertime’ does not exist

      So no, there is no clean way of packaging this out of the box, without reverting to some ugly hacks like moving your script into a subdirectory and calling it `undertime/__init__.py` or something.

      And by the way, how do you run the packaging scripts again? Ah yes, setup.py which is, you guessed it, a python script itself.

      We can’t escape that one folks. :)

      setuptools and python packaging is a minefield, to be honest… to show it as a solution to any problem is generally a mistake.

  6. See the python packaging guide, where it mentions console_scripts (and entry points) is the correct way to install scripts. https://packaging.python.org/tutorials/distributing-packages/#console-scripts

    It’s a simple mapping of the script name to a function to call in one of your packages. Note, that the Python Packaging Guide has best practices for doing things with python and packaging.

    “`
    entry_points={
    ‘console_scripts’: [
    ‘some_script=’
    ‘some_package.some_module:some_function’,
    ],
    },
    “`

    1. We’re not going to change WebKit to use setuptools. We just have a Scripts directory with some scripts. Some are python. They’re not installed.

Comments are closed.