22

✏️ Editing: python_eggs.rst

=====================================
The Internal Structure of Python Eggs
=====================================

STOP! This is not the first document you should read!

----------------------
Eggs and their Formats
----------------------

A "Python egg" is a logical structure embodying the release of a
specific version of a Python project, comprising its code, resources,
and metadata. There are multiple formats that can be used to physically
encode a Python egg, and others can be developed. However, a key
principle of Python eggs is that they should be discoverable and
importable. That is, it should be possible for a Python application to
easily and efficiently find out what eggs are present on a system, and
to ensure that the desired eggs' contents are importable.

There are two basic formats currently implemented for Python eggs:

1. ``.egg`` format: a directory or zipfile *containing* the project's
   code and resources, along with an ``EGG-INFO`` subdirectory that
   contains the project's metadata

2. ``.egg-info`` format: a file or directory placed *adjacent* to the
   project's code and resources, that directly contains the project's
   metadata.

Both formats can include arbitrary Python code and resources, including
static data files, package and non-package directories, Python
modules, C extension modules, and so on.  But each format is optimized
for different purposes.

The ``.egg`` format is well-suited to distribution and the easy
uninstallation or upgrades of code, since the project is essentially
self-contained within a single directory or file, unmingled with any
other projects' code or resources.  It also makes it possible to have
multiple versions of a project simultaneously installed, such that
individual programs can select the versions they wish to use.

The ``.egg-info`` format, on the other hand, was created to support
backward-compatibility, performance, and ease of installation for system
packaging tools that expect to install all projects' code and resources
to a single directory (e.g. ``site-packages``).  Placing the metadata
in that same directory simplifies the installation process, since it
isn't necessary to create ``.pth`` files or otherwise modify
``sys.path`` to include each installed egg.

Its disadvantage, however, is that it provides no support for clean
uninstallation or upgrades, and of course only a single version of a
project can be installed to a given directory. Thus, support from a
package management tool is required. (This is why setuptools' "install"
command refers to this type of egg installation as "single-version,
externally managed".)  Also, they lack sufficient data to allow them to
be copied from their installation source.  easy_install can "ship" an
application by copying ``.egg`` files or directories to a target
location, but it cannot do this for ``.egg-info`` installs, because
there is no way to tell what code and resources belong to a particular
egg -- there may be several eggs "scrambled" together in a single
installation location, and the ``.egg-info`` format does not currently
include a way to list the files that were installed.  (This may change
in a future version.)

Code and Resources
==================

The layout of the code and resources is dictated by Python's normal
import layout, relative to the egg's "base location".

For the ``.egg`` format, the base location is the ``.egg`` itself. That
is, adding the ``.egg`` filename or directory name to ``sys.path``
makes its contents importable.

For the ``.egg-info`` format, however, the base location is the
directory that *contains* the ``.egg-info``, and thus it is the
directory that must be added to ``sys.path`` to make the egg importable.
(Note that this means that the "normal" installation of a package to a
``sys.path`` directory is sufficient to make it an "egg" if it has an
``.egg-info`` file or directory installed alongside of it.)

Project Metadata
=================

If eggs contained only code and resources, there would of course be
no difference between them and any other directory or zip file on
``sys.path``.  Thus, metadata must also be included, using a metadata
file or directory.

For the ``.egg`` format, the metadata is placed in an ``EGG-INFO``
subdirectory, directly within the ``.egg`` file or directory.  For the
``.egg-info`` format, metadata is stored directly within the
``.egg-info`` directory itself.

The minimum project metadata that all eggs must have is a standard
Python ``PKG-INFO`` file, named ``PKG-INFO`` and placed within the
metadata directory appropriate to the format.  Because it's possible for
this to be the only metadata file included, ``.egg-info`` format eggs
are not required to be a directory; they can just be a ``.egg-info``
file that directly contains the ``PKG-INFO`` metadata.  This eliminates
the need to create a directory just to store one file.  This option is
*not* available for ``.egg`` formats, since setuptools always includes
other metadata.  (In fact, setuptools itself never generates
``.egg-info`` files, either; the support for using files was added so
that the requirement could easily be satisfied by other tools, such
as distutils).

In addition to the ``PKG-INFO`` file, an egg's metadata directory may
also include files and directories representing various forms of
optional standard metadata (see the section on `Standard Metadata`_,
below) or user-defined metadata required by the project.  For example,
some projects may define a metadata format to describe their application
plugins, and metadata in this format would then be included by plugin
creators in their projects' metadata directories.

Filename-Embedded Metadata
==========================

To allow introspection of installed projects and runtime resolution of
inter-project dependencies, a certain amount of information is embedded
in egg filenames.  At a minimum, this includes the project name, and
ideally will also include the project version number.  Optionally, it
can also include the target Python version and required runtime
platform if platform-specific C code is included.  The syntax of an
egg filename is as follows::

name ["-" version ["-py" pyver ["-" required_platform]]] "." ext

The "name" and "version" should be escaped using the ``to_filename()``
function provided by ``pkg_resources``, after first processing them with
``safe_name()`` and ``safe_version()`` respectively.  These latter two
functions can also be used to later "unescape" these parts of the
filename.  (For a detailed description of these transformations, please
see the "Parsing Utilities" section of the ``pkg_resources`` manual.)

The "pyver" string is the Python major version, as found in the first
3 characters of ``sys.version``.  "required_platform" is essentially
a distutils ``get_platform()`` string, but with enhancements to properly
distinguish Mac OS versions.  (See the ``get_build_platform()``
documentation in the "Platform Utilities" section of the
``pkg_resources`` manual for more details.)

Finally, the "ext" is either ``.egg`` or ``.egg-info``, as appropriate
for the egg's format.

Normally, an egg's filename should include at least the project name and
version, as this allows the runtime system to find desired project
versions without having to read the egg's PKG-INFO to determine its
version number.

Setuptools, however, only includes the version number in the filename
when an ``.egg`` file is built using the ``bdist_egg`` command, or when
an ``.egg-info`` directory is being installed by the
``install_egg_info`` command. When generating metadata for use with the
original source tree, it only includes the project name, so that the
directory will not have to be renamed each time the project's version
changes.

This is especially important when version numbers change frequently, and
the source metadata directory is kept under version control with the
rest of the project.  (As would be the case when the project's source
includes project-defined metadata that is not generated from by
setuptools from data in the setup script.)

Egg Links
=========

In addition to the ``.egg`` and ``.egg-info`` formats, there is a third
egg-related extension that you may encounter on occasion: ``.egg-link``
files.

These files are not eggs, strictly speaking. They simply provide a way
to reference an egg that is not physically installed in the desired
location. They exist primarily as a cross-platform alternative to
symbolic links, to support "installing" code that is being developed in
a different location than the desired installation location. For
example, if a user is developing an application plugin in their home
directory, but the plugin needs to be "installed" in an application
plugin directory, running "setup.py develop -md /path/to/app/plugins"
will install an ``.egg-link`` file in ``/path/to/app/plugins``, that
tells the egg runtime system where to find the actual egg (the user's
project source directory and its ``.egg-info`` subdirectory).

``.egg-link`` files are named following the format for ``.egg`` and
``.egg-info`` names, but only the project name is included; no version,
Python version, or platform information is included.  When the runtime
searches for available eggs, ``.egg-link`` files are opened and the
actual egg file/directory name is read from them.

Each ``.egg-link`` file should contain a single file or directory name,
with no newlines.  This filename should be the base location of one or
more eggs.  That is, the name must either end in ``.egg``, or else it
should be the parent directory of one or more ``.egg-info`` format eggs.

As of setuptools 0.6c6, the path may be specified as a platform-independent
(i.e. ``/``-separated) relative path from the directory containing the
``.egg-link`` file, and a second line may appear in the file, specifying a
platform-independent relative path from the egg's base directory to its
setup script directory.  This allows installation tools such as EasyInstall
to find the project's setup directory and build eggs or perform other setup
commands on it.

-----------------
Standard Metadata
-----------------

In addition to the minimum required ``PKG-INFO`` metadata, projects can
include a variety of standard metadata files or directories, as
described below.  Except as otherwise noted, these files and directories
are automatically generated by setuptools, based on information supplied
in the setup script or through analysis of the project's code and
resources.

Most of these files and directories are generated via "egg-info
writers" during execution of the setuptools ``egg_info`` command, and
are listed in the ``egg_info.writers`` entry point group defined by
setuptools' own ``setup.py`` file.

Project authors can register their own metadata writers as entry points
in this group (as described in the setuptools manual under "Adding new
EGG-INFO Files") to cause setuptools to generate project-specific
metadata files or directories during execution of the ``egg_info``
command.  It is up to project authors to document these new metadata
formats, if they create any.

``.txt`` File Formats
=====================

Files described in this section that have ``.txt`` extensions have a
simple lexical format consisting of a sequence of text lines, each line
terminated by a linefeed character (regardless of platform).  Leading
and trailing whitespace on each line is ignored, as are blank lines and
lines whose first nonblank character is a ``#`` (comment symbol).  (This
is the parsing format defined by the ``yield_lines()`` function of
the ``pkg_resources`` module.)

All ``.txt`` files defined by this section follow this format, but some
are also "sectioned" files, meaning that their contents are divided into
sections, using square-bracketed section headers akin to Windows
``.ini`` format.  Note that this does *not* imply that the lines within
the sections follow an ``.ini`` format, however.  Please see an
individual metadata file's documentation for a description of what the
lines and section names mean in that particular file.

Sectioned files can be parsed using the ``split_sections()`` function;
see the "Parsing Utilities" section of the ``pkg_resources`` manual for
for details.

Dependency Metadata
===================

``requires.txt``
----------------

This is a "sectioned" text file.  Each section is a sequence of
"requirements", as parsed by the ``parse_requirements()`` function;
please see the ``pkg_resources`` manual for the complete requirement
parsing syntax.

The first, unnamed section (i.e., before the first section header) in
this file is the project's core requirements, which must be installed
for the project to function.  (Specified using the ``install_requires``
keyword to ``setup()``).

The remaining (named) sections describe the project's "extra"
requirements, as specified using the ``extras_require`` keyword to
``setup()``.  The section name is the name of the optional feature, and
the section body lists that feature's dependencies.

Note that it is not normally necessary to inspect this file directly;
``pkg_resources.Distribution`` objects have a ``requires()`` method
that can be used to obtain ``Requirement`` objects describing the
project's core and optional dependencies.

``setup_requires.txt``
----------------------

Much like ``requires.txt`` except represents the requirements
specified by the ``setup_requires`` parameter to the Distribution.

``dependency_links.txt``
------------------------

A list of dependency URLs, one per line, as specified using the
``dependency_links`` keyword to ``setup()``.  These may be direct
download URLs, or the URLs of web pages containing direct download
links. Please see the setuptools manual for more information on
specifying this option.

``depends.txt`` -- Obsolete, do not create!
-------------------------------------------

This file follows an identical format to ``requires.txt``, but is
obsolete and should not be used.  The earliest versions of setuptools
required users to manually create and maintain this file, so the runtime
still supports reading it, if it exists.  The new filename was created
so that it could be automatically generated from ``setup()`` information
without overwriting an existing hand-created ``depends.txt``, if one
was already present in the project's source ``.egg-info`` directory.

``namespace_packages.txt`` -- Namespace Package Metadata
========================================================

A list of namespace package names, one per line, as supplied to the
``namespace_packages`` keyword to ``setup()``.  Please see the manuals
for setuptools and ``pkg_resources`` for more information about
namespace packages.

``entry_points.txt`` -- "Entry Point"/Plugin Metadata
=====================================================

This is a "sectioned" text file, whose contents encode the
``entry_points`` keyword supplied to ``setup()``.  All sections are
named, as the section names specify the entry point groups in which the
corresponding section's entry points are registered.

Each section is a sequence of "entry point" lines, each parseable using
the ``EntryPoint.parse`` classmethod; please see the ``pkg_resources``
manual for the complete entry point parsing syntax.

Note that it is not necessary to parse this file directly; the
``pkg_resources`` module provides a variety of APIs to locate and load
entry points automatically.  Please see the setuptools and
``pkg_resources`` manuals for details on the nature and uses of entry
points.

The ``scripts`` Subdirectory
============================

This directory is currently only created for ``.egg`` files built by
the setuptools ``bdist_egg`` command.  It will contain copies of all
of the project's "traditional" scripts (i.e., those specified using the
``scripts`` keyword to ``setup()``).  This is so that they can be
reconstituted when an ``.egg`` file is installed.

The scripts are placed here using the distutils' standard
``install_scripts`` command, so any ``#!`` lines reflect the Python
installation where the egg was built.  But instead of copying the
scripts to the local script installation directory, EasyInstall writes
short wrapper scripts that invoke the original scripts from inside the
egg, after ensuring that sys.path includes the egg and any eggs it
depends on.  For more about `script wrappers`_, see the section below on
`Installation and Path Management Issues`_.

Zip Support Metadata
====================

``native_libs.txt``
-------------------

A list of C extensions and other dynamic link libraries contained in
the egg, one per line.  Paths are ``/``-separated and relative to the
egg's base location.

This file is generated as part of ``bdist_egg`` processing, and as such
only appears in ``.egg`` files (and ``.egg`` directories created by
unpacking them).  It is used to ensure that all libraries are extracted
from a zipped egg at the same time, in case there is any direct linkage
between them.  Please see the `Zip File Issues`_ section below for more
information on library and resource extraction from ``.egg`` files.

``eager_resources.txt``
-----------------------

A list of resource files and/or directories, one per line, as specified
via the ``eager_resources`` keyword to ``setup()``.  Paths are
``/``-separated and relative to the egg's base location.

Resource files or directories listed here will be extracted
simultaneously, if any of the named resources are extracted, or if any
native libraries listed in ``native_libs.txt`` are extracted.  Please
see the setuptools manual for details on what this feature is used for
and how it works, as well as the `Zip File Issues`_ section below.

``zip-safe`` and ``not-zip-safe``
---------------------------------

These are zero-length files, and either one or the other should exist.
If ``zip-safe`` exists, it means that the project will work properly
when installed as an ``.egg`` zipfile, and conversely the existence of
``not-zip-safe`` means the project should not be installed as an
``.egg`` file.  The ``zip_safe`` option to setuptools' ``setup()``
determines which file will be written. If the option isn't provided,
setuptools attempts to make its own assessment of whether the package
can work, based on code and content analysis.

If neither file is present at installation time, EasyInstall defaults
to assuming that the project should be unzipped.  (Command-line options
to EasyInstall, however, take precedence even over an existing
``zip-safe`` or ``not-zip-safe`` file.)

Note that these flag files appear only in ``.egg`` files generated by
``bdist_egg``, and in ``.egg`` directories created by unpacking such an
``.egg`` file.

``top_level.txt`` -- Conflict Management Metadata
=================================================

This file is a list of the top-level module or package names provided
by the project, one Python identifier per line.

Subpackages are not included; a project containing both a ``foo.bar``
and a ``foo.baz`` would include only one line, ``foo``, in its
``top_level.txt``.

This data is used by ``pkg_resources`` at runtime to issue a warning if
an egg is added to ``sys.path`` when its contained packages may have
already been imported.

(It was also once used to detect conflicts with non-egg packages at
installation time, but in more recent versions, setuptools installs eggs
in such a way that they always override non-egg packages, thus
preventing a problem from arising.)

``SOURCES.txt`` -- Source Files Manifest
========================================

This file is roughly equivalent to the distutils' ``MANIFEST`` file.
The differences are as follows:

* The filenames always use ``/`` as a path separator, which must be
  converted back to a platform-specific path whenever they are read.

* The file is automatically generated by setuptools whenever the
  ``egg_info`` or ``sdist`` commands are run, and it is *not*
  user-editable.

Although this metadata is included with distributed eggs, it is not
actually used at runtime for any purpose.  Its function is to ensure
that setuptools-built *source* distributions can correctly discover
what files are part of the project's source, even if the list had been
generated using revision control metadata on the original author's
system.

In other words, ``SOURCES.txt`` has little or no runtime value for being
included in distributed eggs, and it is possible that future versions of
the ``bdist_egg`` and ``install_egg_info`` commands will strip it before
installation or distribution.  Therefore, do not rely on its being
available outside of an original source directory or source
distribution.

------------------------------
Other Technical Considerations
------------------------------

Zip File Issues
===============

Although zip files resemble directories, they are not fully
substitutable for them.  Most platforms do not support loading dynamic
link libraries contained in zipfiles, so it is not possible to directly
import C extensions from ``.egg`` zipfiles.  Similarly, there are many
existing libraries -- whether in Python or C -- that require actual
operating system filenames, and do not work with arbitrary "file-like"
objects or in-memory strings, and thus cannot operate directly on the
contents of zip files.

To address these issues, the ``pkg_resources`` module provides a
"resource API" to support obtaining either the contents of a resource,
or a true operating system filename for the resource.  If the egg
containing the resource is a directory, the resource's real filename
is simply returned.  However, if the egg is a zipfile, then the
resource is first extracted to a cache directory, and the filename
within the cache is returned.

The cache directory is determined by the ``pkg_resources`` API; please
see the ``set_cache_path()`` and ``get_default_cache()`` documentation
for details.

The Extraction Process
----------------------

Resources are extracted to a cache subdirectory whose name is based
on the enclosing ``.egg`` filename and the path to the resource.  If
there is already a file of the correct name, size, and timestamp, its
filename is returned to the requester.  Otherwise, the desired file is
extracted first to a temporary name generated using
``mkstemp(".$extract",target_dir)``, and then its timestamp is set to
match the one in the zip file, before renaming it to its final name.
(Some collision detection and resolution code is used to handle the
fact that Windows doesn't overwrite files when renaming.)

If a resource directory is requested, all of its contents are
recursively extracted in this fashion, to ensure that the directory
name can be used as if it were valid all along.

If the resource requested for extraction is listed in the
``native_libs.txt`` or ``eager_resources.txt`` metadata files, then
*all* resources listed in *either* file will be extracted before the
requested resource's filename is returned, thus ensuring that all
C extensions and data used by them will be simultaneously available.

Extension Import Wrappers
-------------------------

Since Python's built-in zip import feature does not support loading
C extension modules from zipfiles, the setuptools ``bdist_egg`` command
generates special import wrappers to make it work.

The wrappers are ``.py`` files (along with corresponding ``.pyc``
and/or ``.pyo`` files) that have the same module name as the
corresponding C extension.  These wrappers are located in the same
package directory (or top-level directory) within the zipfile, so that
say, ``foomodule.so`` will get a corresponding ``foo.py``, while
``bar/baz.pyd`` will get a corresponding ``bar/baz.py``.

These wrapper files contain a short stanza of Python code that asks
``pkg_resources`` for the filename of the corresponding C extension,
then reloads the module using the obtained filename.  This will cause
``pkg_resources`` to first ensure that all of the egg's C extensions
(and any accompanying "eager resources") are extracted to the cache
before attempting to link to the C library.

Note, by the way, that ``.egg`` directories will also contain these
wrapper files.  However, Python's default import priority is such that
C extensions take precedence over same-named Python modules, so the
import wrappers are ignored unless the egg is a zipfile.

Installation and Path Management Issues
=======================================

Python's initial setup of ``sys.path`` is very dependent on the Python
version and installation platform, as well as how Python was started
(i.e., script vs. ``-c`` vs. ``-m`` vs. interactive interpreter).
In fact, Python also provides only two relatively robust ways to affect
``sys.path`` outside of direct manipulation in code: the ``PYTHONPATH``
environment variable, and ``.pth`` files.

However, with no cross-platform way to safely and persistently change
environment variables, this leaves ``.pth`` files as EasyInstall's only
real option for persistent configuration of ``sys.path``.

But ``.pth`` files are rather strictly limited in what they are allowed
to do normally.  They add directories only to the *end* of ``sys.path``,
after any locally-installed ``site-packages`` directory, and they are
only processed *in* the ``site-packages`` directory to start with.

This is a double whammy for users who lack write access to that
directory, because they can't create a ``.pth`` file that Python will
read, and even if a sympathetic system administrator adds one for them
that calls ``site.addsitedir()`` to allow some other directory to
contain ``.pth`` files, they won't be able to install newer versions of
anything that's installed in the systemwide ``site-packages``, because
their paths will still be added *after* ``site-packages``.

So EasyInstall applies two workarounds to solve these problems.

The first is that EasyInstall leverages ``.pth`` files' "import" feature
to manipulate ``sys.path`` and ensure that anything EasyInstall adds
to a ``.pth`` file will always appear before both the standard library
and the local ``site-packages`` directories.  Thus, it is always
possible for a user who can write a Python-read ``.pth`` file to ensure
that their packages come first in their own environment.

Second, when installing to a ``PYTHONPATH`` directory (as opposed to
a "site" directory like ``site-packages``) EasyInstall will also install
a special version of the ``site`` module.  Because it's in a
``PYTHONPATH`` directory, this module will get control before the
standard library version of ``site`` does.  It will record the state of
``sys.path`` before invoking the "real" ``site`` module, and then
afterwards it processes any ``.pth`` files found in ``PYTHONPATH``
directories, including all the fixups needed to ensure that eggs always
appear before the standard library in sys.path, but are in a relative
order to one another that is defined by their ``PYTHONPATH`` and
``.pth``-prescribed sequence.

The net result of these changes is that ``sys.path`` order will be
as follows at runtime:

1. The ``sys.argv[0]`` directory, or an empty string if no script
   is being executed.

2. All eggs installed by EasyInstall in any ``.pth`` file in each
   ``PYTHONPATH`` directory, in order first by ``PYTHONPATH`` order,
   then normal ``.pth`` processing order (which is to say alphabetical
   by ``.pth`` filename, then by the order of listing within each
   ``.pth`` file).

3. All eggs installed by EasyInstall in any ``.pth`` file in each "site"
   directory (such as ``site-packages``), following the same ordering
   rules as for the ones on ``PYTHONPATH``.

4. The ``PYTHONPATH`` directories themselves, in their original order

5. Any paths from ``.pth`` files found on ``PYTHONPATH`` that were *not*
   eggs installed by EasyInstall, again following the same relative
   ordering rules.

6. The standard library and "site" directories, along with the contents
   of any ``.pth`` files found in the "site" directories.

Notice that sections 1, 4, and 6 comprise the "normal" Python setup for
``sys.path``.  Sections 2 and 3 are inserted to support eggs, and
section 5 emulates what the "normal" semantics of ``.pth`` files on
``PYTHONPATH`` would be if Python natively supported them.

For further discussion of the tradeoffs that went into this design, as
well as notes on the actual magic inserted into ``.pth`` files to make
them do these things, please see also the following messages to the
distutils-SIG mailing list:

* http://mail.python.org/pipermail/distutils-sig/2006-February/006026.html
* http://mail.python.org/pipermail/distutils-sig/2006-March/006123.html

Script Wrappers
---------------

EasyInstall never directly installs a project's original scripts to
a script installation directory.  Instead, it writes short wrapper
scripts that first ensure that the project's dependencies are active
on sys.path, before invoking the original script.  These wrappers
have a #! line that points to the version of Python that was used to
install them, and their second line is always a comment that indicates
the type of script wrapper, the project version required for the script
to run, and information identifying the script to be invoked.

The format of this marker line is::

"# EASY-INSTALL-" script_type ": " tuple_of_strings "\n"

The ``script_type`` is one of ``SCRIPT``, ``DEV-SCRIPT``, or
``ENTRY-SCRIPT``.  The ``tuple_of_strings`` is a comma-separated
sequence of Python string constants.  For ``SCRIPT`` and ``DEV-SCRIPT``
wrappers, there are two strings: the project version requirement, and
the script name (as a filename within the ``scripts`` metadata
directory).  For ``ENTRY-SCRIPT`` wrappers, there are three:
the project version requirement, the entry point group name, and the
entry point name.  (See the "Automatic Script Creation" section in the
setuptools manual for more information about entry point scripts.)

In each case, the project version requirement string will be a string
parseable with the ``pkg_resources`` modules' ``Requirement.parse()``
classmethod.  The only difference between a ``SCRIPT`` wrapper and a
``DEV-SCRIPT`` is that a ``DEV-SCRIPT`` actually executes the original
source script in the project's source tree, and is created when the
"setup.py develop" command is run.  A ``SCRIPT`` wrapper, on the other
hand, uses the "installed" script written to the ``EGG-INFO/scripts``
subdirectory of the corresponding ``.egg`` zipfile or directory.
(``.egg-info`` eggs do not have script wrappers associated with them,
except in the "setup.py develop" case.)

The purpose of including the marker line in generated script wrappers is
to facilitate introspection of installed scripts, and their relationship
to installed eggs.  For example, an uninstallation tool could use this
data to identify what scripts can safely be removed, and/or identify
what scripts would stop working if a particular egg is uninstalled.

MR SKB