Project Standards

For a project under Farama to be listed as mature, it must comply with the following standards. We work to maintain these standards for our existing projects, and update new projects to meet them. These standards may evolve over time, and we'll publicly announce any major changes to them.

General Standards

Standards we strongly encourage any reinforcement learning environment to comply with
Release Details
  • Be released on PyPI, so that users can easily install libraries via `pip` without having to clone Git repositories or run shell scripts.
  • Include a full package description for the PyPI page (generally by repurposing the readme file), so that users can quickly see what a package is when searching on PyPI.
  • Deploy packages to PyPI via GitHub Actions that run when a Release is published on GitHub. This ensures that releases are fully automated to remove human error, and allows anyone to easily take over a project if needed.
Standardization
  • Support the Gymnasium and/or PettingZoo API at the newest version (supporting other APIs in addition to these are fine), so that users can drop in and try out different environments as easily as possible, and code can be more modular and interchangeable.
  • Support all current versions of Python (currently 3.8-3.12). Different users may need to use different versions of Python, often as specified by outside forces like company policy or their package manager, and we want to comprehensively support this.
Code Standardization
  • Have Python code comply with the black, flake8, and isort linting/formatting tools. This keeps the code formatting consistent across different projects.
  • Add type hinting where reasonable, at least in the most publicly facing portions of a repository. Adding title hinting to functions can help new users get accustomed to code easier and can sometimes help prevent or catch obscure bugs.
  • Include docstrings for all substantial functions so that new users can easily understand what they do. Use the pydocstyle and Google styles so that they have a standard formatting.
Development Workflow
  • Have continuous integration testing for API compliance, determinism, and title checking with Pyright. All tests must pass on both Linux and macOS. This way, we can ensure that all libraries remain well-tested and robust.
  • Include git pre-commit hooks. Including linting checks in pre-commit hooks ensures that all committed code follows a consistent style, and allows new contributors to adapt their code automatically.
Quality of Life
  • Include a thorough documentation website in the same format as the other mature projects if the library is large enough to warrant one. We want both experienced researchers and newcomers alike to be able to quickly see what a library is and reference functionality as needed without having to refer to source.
  • Run on both Linux and macOS. The vast majority of the software in this field is run on Linux servers, and a huge number of researchers and students develop and test on macOS.
Binaries
  • Be deterministically compiled if we're releasing it as a binary or if it compiles on installation, to reduce the likelihood of reproducibility issues.
  • Ship any binaries for both x86_64 and Arm architectures and both Linux and macOS operating systems.
  • All supported operating systems should have wheels provided. (Note: we're temporarily handling Arm-based macOS wheels by having them compile on installation due to an issue with Github Actions support.)
Good Science
  • Be fully deterministic when given a specific random seed, so that any result can be exactly reproduced (ignoring potential issues due to floating point arithmetic errors).
  • Include explicit versioning in environment and wrapper names in a format like 'Pong-v5' to make it clear to researchers when results are from differently configured environments.
  • Support serialization for all environments via Pickle or EzPickle. This allows environments to be saved for experiment reproduction and for environment vectorization.
User Experience
  • Include polished rendering for all environments. Rendering should be inviting and appealing, especially for newcomers to reinforcement learning.
  • Include descriptive warning and error messages. When something goes wrong, it should be as easy as possible to identify the problem. Python f-Strings and assert statements are encouraged to catch common errors.
Other
  • Use an MIT or similarly styled open source license (Apache 2.0 is fine). Listing an open source license is needed to properly distribute free and open source software. We prefer MIT styled licenses as they let users use the software in any way they'd like, and this style has become ubiquitous across projects in this field.
  • Pull documentation website information from environments directly instead of manually specifying it. This prevents human errors in transcribing information or forgetting to update it when it changes in the future.

Farama Standards

Specific standards we enforce for projects inside Farama that generally don't apply to other projects
Farama Branding
  • Have a logo in the same style as the other projects. A nice logo is a small feature that makes projects more memorable and makes the whole space a little more inviting to interested new users. Since we list all the logos together, we try to make sure they match.
  • Have a documentation website at [project].farama.org. Having everything under a single domain name makes things simple and consistent for users (and makes things a bit easier for us to manage).
  • Include the standard Farama code of conduct. We believe that all members of our community should be respected. Additionally, listing a code of conduct under our projects is a requirement for some open source donations.
Design Philosophy
  • Do it right the first time, and eliminate technical debt. We strive to maintain flexible APIs which can be easily used for many years to come. Breaking changes may be required when bringing in outside projects, in order to align code with our standards and ensure it is maintainable in the future.
  • APIs should be user-friendly and prevent incorrect usage. New users should be able to easily use APIs correctly without needing to dig through source code.
  • Use deprecation warnings whenever possible. This gives users time to adapt their code and mitigates the severity of breaking changes. Releasing breaking changes without warning is strongly discouraged. When breaking changes are made, they should include detailed error messages to inform the user how to fix the problem.
Quality
  • Include only environments which are scientifically important or commonly used. We value quality over quantity for environments. Environments which are not commonly used and have limited scientific value should be removed, or included only in tutorials. Toy environments should be specified as such, and included only if they are widely used.
  • Projects should be highly polished and look professional. For example, projects should have a comprehensive README and documentation site, with straightforward wording that makes sense to new users. Avoid excessive technical jargon in favor of intuitive explanations, examples and images.
  • Rendering should be high-quality and consistent between environments. We usually use Pygame for all 2D Python graphics, and create pixel art when native rendering does not exist.
Organization
  • Limit projects to a single purpose. Ideally, standard APIs, utilities, environments and datasets should be split into separate repositories. This makes maintenance more manageable and results in a simpler user experience.
  • External dependencies should be limited, and include only well-maintained projects. If an external dependency stops being maintained, benchmarks can no longer be maintained long term. For example, PyBullet is no longer maintained, and MuJoCo-py bindings should be updated to DeepMind's MuJoCo bindings.
  • Use semantic versioning. We limit minor releases and patches to `0.0.x` version bumps, with `0.x.0` for standard releases. We reserve `x.0.0` versions for initial maturity releases (1.0.0) and for major releases with significant API changes and/or major additional functionality.
Development Practices
  • Limit commits and PR's to a single purpose. This makes the reviewing process significantly easier and prevents errors from slipping into the codebase.
  • Avoid committing to master. Instead, develop locally in feature branches and use PR's for review. This mitigates merge conflicts and allows multiple developers to work simultaneously without issues. Exceptions can be made for small bugfixes, but it is best to use separate branches or patches.
  • Communicate clearly with the community about upcoming changes. Future development plans should be communicated through release notes, social media announcements, and public roadmaps. In the case of breaking changes, specific instructions must be provided to inform users how to update their code.
Security
  • Verify and sign commits with GPG or SSH keys. This ensures commits are from trusted sources, and prevents potential attacks through spoofed identity.
  • List Jordan Terry and Mark Towers as owners on PyPI, to make sure that we don't lose control of a package in the case that a primary maintainer becomes unavailable due to unexpected circumstances.
  • Have maintainers with GitHub or PyPI permissions use TOTP based 2FA or better (and secure passwords, emails addresses, etc.), to reduce the likelihood of supply chain security attacks involving any of our projects.
Other
  • Use Farama Notifications. This allows us to add warnings after releases about issues that users of old versions need to be aware of.
  • Include Google analytics in all documentation websites, so we can see how popular they are and what users are doing on the websites.
  • Include a "Sponsor this project" button on the GitHub repository. To do this, add a FUNDING.yml file in /.github/ directory, with the following line: `github: Farama-Foundation`.