Project Standards

For a project under Farama to be listed as mature, it must comply with the following standards. We work to maintain these standards for our existing projects, and update new projects to meet them. These standards may evolve over time, and we'll publicly announce any major changes to them.
Good Science
  • Be fully deterministic when given a specific random seed, so that any result can be exactly reproduced (ignoring potential issues due to floating point arithmetic errors).
  • Include explicit versioning in environment and wrapper names in a format like "Pong-v5" to make it clear to researchers when results are from differently configured environments.
Release Details
  • Be released on PyPI, so that users can easily install libraries via `pip` without having to clone Git repositories.
  • Include a full package description for the PyPI page (generally by repurposing the readme file), so that users can quickly see what a package is when searching on PyPI.
  • Deploy packages to PyPI via GitHub Actions that run when a Release is published on GitHub. This ensures that releases are fully automated to remove human error, and allows anyone to easily take over a project if needed.
Binaries
  • Be deterministically compiled if we're releasing it as a binary or if it compiles on installation, to reduce the likelihood of reproducibility issues.
  • Ship any binaries for both x86_64 and Arm architectures and both Linux and macOS operating systems.
  • All supported operating systems should have wheels provided. (Note: we're temporarily handling Arm-based macOS wheels by having them compile on installation due to an issue with Github Actions support.)
Quality of Life
  • Include a thorough documentation website in the same format as the other mature projects if the library is large enough to warrant one. We want both experienced researchers and newcomers alike to be able to quickly see what a library is and reference functionality as needed without having to refer to source.
  • Run on both Linux and macOS. The vast majority of the software in this field is run on Linux servers, and a huge number of researchers and students develop and test on macOS.
Code Standardization
  • Have Python code comply with the black, flake8, and isort linting/formatting tools. This keeps the code formatting consistent across different projects.
  • Add type hinting where reasonable, at least in the most publicly facing portions of a repository. Adding type hinting to functions can help new users get accustomed to code easier and can sometimes help prevent or catch obscure bugs.
  • Include docstrings for all substantial functions so that new users can easily understand what they do. Use the pydocstyle and Google styles so that they have a standard formatting.
Development Workflow
  • Have continuous integration testing for API compliance, determinism, and type checking with Pyright. All tests must pass on both Linux and macOS. This way, we can ensure that all libraries remain well-tested and robust.
  • Include a git pre-commit with black, flake8 and isort. Including these checks in pre-commit reduces headaches in getting PRs merged by making it clear to contributors early on when their code will need changes to comply with the style guide.
Farama Branding
  • Have a logo in the same style as the other projects. A nice logo is a small feature that makes projects more memorable and makes the whole space a little more inviting to interested new students. Since we list all the logos together, we try to make sure they match.
  • Have the documentation website be at .farama.org. Having everything under a single domain name makes things simple and consistent for users (and makes things a bit easier for us to manage).
  • Include the standard Farama code of conduct. We believe that all members of our community should be respected. Additionally, listing a code of conduct under our projects is a requirement for some open source donations.
Standardization
  • Support the Gymnasium and/or PettingZoo API at the newest version (supporting other APIs in addition to these are fine), so that users can drop in and try out different environments as easily as possible, and code can be more modular and interchangeable.
  • Support all current versions of Python (currently 3.7-3.10). Different users may need to use different versions of Python, often as specified by outside forces like company policy or their package manager, and we want to comprehensively support this.
Security
  • List Jordan Terry and Mark Towers as owners on PyPI, to make sure that we don't lose control of a package in the case that a primary maintainer becomes unavailable due to unexpected circumstances.
  • Have maintainers with GitHub or PyPI permissions use TOTP based 2FA or better (and secure passwords, emails addresses, etc.), to reduce the likelihood of supply chain security attacks involving any of our projects.
Other
  • Include Google analytics in all documentation websites, so we can see how popular they are and what users are doing on the websites.
  • Use an MIT or similarly styled open source license (Apache 2.0 is fine). Listing an open source license is needed to properly distribute free and open source software. We prefer MIT styled licenses as they let users use the software in any way they'd like, and this style has become ubiquitous across projects in this field.