A simpler and OS independant setup of the workstation — My journey with the Amaranth HDL
Where I tell how I could simplify the requirements to make life easier for other people wanting to use my projects.
[This is a part of my series about Amaranth HDL. Especially, this is a followup to this article.]
TL;DR:
git clone http://git-something/some-group/some-project.git
cd some-project
python -m pdm sync
python -m pdm run whatever-script-provided-to-do-useful-stuff
Shortly after proposing to add my series of article to the list of tutorials, whitequark suggested that I used Yowasp (“Unofficial WebAssembly-based packages for Yosys, nextpnr, and more”) instead of compiling and installing Yosys, Symbiyosys and nextpnr : simply add those yowasp variants into the dependencies of the python project, and that’s all. The only tools needed to be build and installed by yourself would be the tool to upload a bitstream into a FPGA.
Beside this, I also succeeded in using Amaranth HDL to write a small demo that display something on an HDMI screen, and I got some feedbacks about how difficult it was to use with a windows workstation, because I wrote bash scripts, and got a proposal to use PDM, a modern Python package and dependency manager that allow to defines “scripts” to be called in a more platform independant way.
Thus I worked on those feedbacks to make life easier, and I ended up with the following instructions. By chance, I have access to a Mac OS workstation and a Windows 10 workstation for the time being (with the right to install something), so I could verify what to do for those platforms too.
Python and pip are the only requirement to build the bitstream file
Yes, that’s all that you have to install in an OS-dependant way.
On linux based OSes, the package manager usually is the way to go (if it is not already installed by default).
On Windows, just download the installer from the official python site.
On MacOS, I went with the homebrew package manager and then installed python and pip without any trouble.
Once you have python, you will use PDM as the build tool for your project, and it can be installed with a simple python -m pip instal pdm.
Define your custom actions using python and PDM only
I usually write some small shell scripts as macros to perform a series of calls to manage my python projects : calling the formatter, perform integration test and computing coverage, etc…
The goal is to not rely on shell scripts anymore, and to call the converted script (say ‘xxx’) using python -m pdm run xxx .
Porting simple action (like simply calling blake ) will be as simple as adding this in the pyproject.toml file :
[tool.pdm.scripts]
#...
reformat = "python3 -m black ."Then, to reformat my sources, I invoke python -m pdm run reformat .
More complex now, to clean all the artifacts created for building the bitstream or performing integrated tests, I had a script like this :
#!/bin/bash
# clean
echo "Cleaning..."
rm -Rf tmp.test_* 2>&1 >/dev/null
rm dist/* 2>&1 >/dev/null
rm -Rf .pytest_cache htmlcov .coverage 2>&1 >/dev/nullThe cleaning integrate a lot of steps, and one usually doesn’t care whether the files to delete actually exists to begin with, what matters is that there is no artifact remaining after this action.
Here I converted this shell script (‘reclean’) into a python script (‘reclean.py’), and I also enriched the output using the rich library :
from glob import glob
import os
import shutil
from rich import print
for pattern in ["tmp.test_*", "build-tests/*", "dist/*", "test_*"]:
print(f" [yellow]Cleaning[/yellow] {pattern}...")
for path in glob(pattern, recursive=0):
print(f" :wastebasket: [bright_black]{path}[/bright_black]")
if os.path.isdir(path):
shutil.rmtree(path)
else:
os.remove(path)
for path in [".coverage", "htmlcov", ".pytest_cache"]:
if os.path.exists(path):
print(f" [yellow]Removing[/yellow] {path}...")
if os.path.isdir(path):
shutil.rmtree(path)
else:
os.remove(path)
print(f":white_check_mark: [bold green]DONE Cleaning.[/bold green]")Then, in my pyproject.toml, I added this :
[tool.pdm.scripts]
# ...
_clean_tons_of_files = "python3 reclean.py"
clean = { composite = [ "_clean_tons_of_files" ] }You will notice that the actual call to ‘reclean.py’ is not named clean , but is given another name ( _clean_tons_of_files ). The clean script is in fact a composition that for now is calling the other script that actually call reclean.py , but it could evolve later to call other subscripts doing other kind of actions that are deemed parts of what it means to clean.
Anyway, I now clean my project folder using python -m pdm run clean .
Those two ways of defining scripts are put together for something more involved : performing the tests and get coverage reports. The original script was doing this :
#!/bin/bash
# prepare workspace
./reclean
./reformat
# rebuild, install and test
echo "Rebuild, Reinstall and Run tests..."
python3 -m build && python3 -m pip install --force-reinstall dist/*.whl && \
python3 -m coverage run --source=amaranth_stuff --branch -m pytest && \
python3 -m coverage report -m && \
python3 -m coverage htmlI already took care of the cleaning task and reformat task, now I can update my pyproject.toml file like this :
[tool.pdm.scripts]
#...
_coverage_run = "python3 -m coverage run --source=ecp5 --branch -m pytest"
_coverage_report = "python3 -m coverage report -m"
_coverage_html = "python3 -m coverage html"
_ci_only = { composite = ["_coverage_run","_coverage_report","_coverage_html"] }
ci = { composite = ["clean", "reformat", "_ci_only"] }And now I launch all the test using python -m pdm run ci .
Using the Yowasp version of yosys, nextpnr,…
The first thing is to add the dependencies with pdm. I defined several groups of developper dependencies, and thus did the following actions :
python -m pdm add -dG ci yowasp-yosys
python -m pdm add -dG ecp5 yowasp-nextpnr-ecp5Obviously, the dependency for nextpnr depends on your target FPGA. Anyway, pdm modify the pyproject.toml file like this :
[tool.pdm.dev-dependencies]
#...
ci = [
#...
"yowasp-yosys>=0.34.0.0.post591",
]
ecp5 = [
"yowasp-nextpnr-ecp5>=0.6.0.0.post445",
]Now, to tell Amaranth to use those version of the tools instead of trying to use native tools that would have to be compiled and installed separately, one defines some environment variables into a separate file, let’s call it .env.toolchain :
AMARANTH_USE_YOSYS=system
YOSYS=yowasp-yosys
SBY=yowasp-sby
SMTBMC=yowasp-yosys-smtbmc
# for ECP5 family
NEXTPNR_ECP5=yowasp-nextpnr-ecp5
ECPPACK=yowasp-ecppackThen, let’s reference this file as a shared option of the scripts in the pyproject.toml file :
[tool.pdm.scripts]
_.env_file = ".env.toolchain"Do not forget to commit the pdm.lock file
Doing that, anyone cloning your project and having python, pip and pdm will just have to perform the following :
git clone http://git-something/some-group/some-project.git
cd some-project
python -m pdm sync
python -m pdm run whatever-script-you-provide-to-do-useful-stuffSee the full pyproject.toml for a project that build a bitstream to be uploaded to a ECP5 FPGA. See also another one for a library.
Side effect : simpler workflow
Now that dependencies are managed using pdm, the workflow is simplified using the sync action of pdm :
- name: Install dependencies
run: |
#...
python -m pip install --upgrade pip pdm
python -m pdm syncLaunching the tests and other verification is also simplified :
- name: Run project CI
run: |
python -m pdm run ciSee the full workflow.
A programming tool is still required to program the FPGA…
As I said at the beginning, you still have to install the appropriate tool to program your FPGA. It may be as simple as to use a package manager or as involved as installing tons of build tools –that in itself can be a kind of Pandora box on some systems, yes I am looking at you Windows–, downloading the source code, compile and install.