Developer Workstation Setup

Running das with Visual Studio Code Remote Containers, on Mac (official/recommended method)

Intro

This guide assumes you have already been onboarded to the engineering team, have a GitHub account and membership within the PADAS organization, you have been granted developer-level repository access, and your local development tools (git, VS Code, etc) are installed and ready to go. The bulk of this guide follows our recommended and supported approach, developing inside of remote containers, using Visual Studio Code on a Mac. If you’d like more background on remote containers, you can review the documentation here.

Pre-Requisites

• Visual Studio Code (VSCode)

• Python 3.10 - python 3.10 is what our ER server runs on. You may opt to use a newer version locally yourself.

• Docker Desktop

• Git and/or GitHub CLI

• make (xcode-select --install or brew install make) for OSX users.

• gcloud CLI

• GEOS (brew install geos) in case of running native

• pre-commit (brew install pre-commit)

Steps

1. Create a local directory on your machine to put ER source code. Then clone our server and web client repositories:

   1. https://github.com/PADAS/das

   2. https://github.com/PADAS/das-web-react

2. Configure Docker Desktop with what RAM, CPU and disk space you have to spare. I gave it 8 cpus, 8 gb ram and 50 gb disk space. (optional, you can use the command line option if your OS has it)

3. Set up your local environment’s runtime settings by following the instructions in the Server Settings section.

4. Set up your pre-commit hooks by following the instructions in the Pre-commit hooks section.

5. Run make start from the root of das to spin up docker containers for Postgres and Redis.

   • This step has to be run on your local machine, not inside the remote container environment

   • If these containers get torn down for any reason, you’ll have to run this step again to continue working with das

   • Troubleshooting connecting to the postgres db from ER server project: We have seen depending on the postgres image used, the database server is not configured to allow password method connections. Review the pg_hba.conf by hand in that case. We want to ensure the postgres server is configured to allow password authentication method.

     • Add this line or similar:

       host all all 127.0.0.1/32 password
       

     • Then execute this in psql to reload the pg_hba.conf file:

       SELECT pg_reload_conf();
       

6. Set up VSCode extensions

   1. Install Python extensions to your liking

   2. Install the Remote Containers extension pack

7. Restart VSCode and open the das project

8. VS Code should now prompt you to open the project in a remote container (see toast messages at the bottom right of your screen). Click the link in the notification to do so.

   • You can also manually trigger this action by using key command Cmd+Shift+P, then selecting “Remote-Containers: Open Folder in Container” from the dropdown menu.

9. Building the container will take a while as it installs Ubuntu and its other dependencies. Once complete, your development environment is nearly set up and you are provided with a bash shell inside the remote container, from which to issue development and management commands.

10. Now you are ready to migrate the db.

    • Use the VSCode launch task Django migrate db

    • VSCode launch tasks can be found near the top left of the “run and debug” section (where it says “Django:Run”).

11. After the migrations complete, we want to install the necessary data fixtures for a new EarthRanger instance (event types, maps, permission sets and a default admin). You only need to do this once after installing a new database.

    1. First use the VSCode launch task Django initial data

    2. Optional: set up a second tenant in your local. This is an advanced configuration.

       • Use the VSCode launch task New tenant initial data

       • A prompt will appear when you select this task. Type in the hostname from your .env file (which should be localhost unless you’re doing something fancy) and hit Return.

12. Set up PyTest so you can test your code as you develop.

    1. Use the VSCode launch task PyTest --createdb

    2. Once complete, run your tests by using the VSCode launch task PyTest

13. Run the application

    • You made it! Now it’s time to run das. Choose one of these options from the launch settings in VSCode in the “run and debug” panel:

      • Django:Run - stand up the API server using runserver

      • Django:Gunicorn - similar to Django:Run, but uses the Gunicorn server instead

      • Django:GunicornRT - Run the API stack and the realtime services. Only use this when you need to debug the realtime services alongside the API.

      • ER Workers - runs the celery worker that pulls jobs from all of our celery queues. In production we break this up into three workers, but for simplicity here, we only run one worker with several threads

      • Django:MQL - this is our pubsub service. Facilitates message passing between services.

      • Django dumpdata - This runs the dumpdata management command

    • To run the complete stack, launch these three tasks. You should rarely need to do this.

      1. Django:GunicornRT

      2. ER Workers

      3. Django:MQL

Additional Information

• If you manually set up your own database server, configure like this after logging in using psql:

  CREATE ROLE das WITH LOGIN SUPERUSER PASSWORD 'password';
  CREATE DATABASE das ENCODING utf8 owner das;
  \c das
  CREATE EXTENSION IF NOT EXISTS "postgis";
  

• If you plan on using VSCode-integrated source control tools, update your Git config inside the container:

  git config --global user.email "<email>@earthranger.com"
  git config --global user.name "<your name>"
  

---

Server Settings

System settings in das are defined by two mechanisms:

1. The Django settings system, which defines the baseline system settings

   • This includes a series of settings.py files and an .env file, which you must create when setting up your local environment.

2. The Tenant Management Service (TMS), which augments individual sites with their specific settings.

   • This is preloaded for local development using fixtures generated during the loaddata phase of local developer installation. You don’t have to run the TMS locally.

Override settings using “.env” file

You must create your .env file. Copy the .env.template file from das into das/das and rename it to .env.

Modify that .env as needed, such as changing the FQDN to wherever you run your server (typically localhost). Most of the defaults from .env.template should be correct for an out-of-the-box setup following this guide.

Ensure these values are in your .env file:

DB_PASSWORD=das
GEOS_LIBRARY_PATH= #one of these VALUES:
# /usr/lib/x86_64-linux-gnu/libgeos_c.so.1 IF YOU ARE ON AN X86_64 ARCHITECTURE CHIP
# /usr/lib/aarch64-linux-gnu/libgeos_c.so.1 IF YOU ARE ON AN ARM ARCHITECTURE CHIP

# hint, when running on an ARM system the GEOS path would be
# GEOS_LIBRARY_PATH='/usr/lib/aarch64-linux-gnu/libgeos_c.so.1'

FQDN=localhost # unless you've configured your host otherwise
# without TMS, we load tenant settings from local data.
# comment out TMS_API_CLIENT and TENANT_ID if you have TMS configured
TMS_API_CLIENT="core.tms.DjangoSettingsClient"
# develop.pamdas.org tenant id
TENANT_ID="28bf3d60-195f-41c7-ad75-d249d4d32430"

MEMORY_STORE_CLIENT="utils.persistent.RedisStorage"
MEMORY_STORE_HOST="host.docker.internal" #this connects to the local redis container instead of the TMS Memorystore redis instance used in production deployments
MEMORY_STORE_PORT=6379
MEMORY_STORE_DATABASE=4

GITGUARDIAN_API_KEY="<your key here>" # the API key from your GitGuardian account when you set up your pre-commit hooks
GITGUARDIAN_DOTENV_PATH=/workspaces/das/das/.env # set a shell environment variable for the pre-commit ggshield library to find

Our advanced local development setup has pgcat configured. If you decide to simplify things, add the following to your .env file, to instruct our Django settings to use the simple brew installed postgresql server.

DB_POOLER_NAME=das
DB_POOLER_HOST=host.docker.internal
DB_POOLER_PORT=5432

For environments outside OSX:

DB_HOST=postgis
MEMORY_STORE_HOST="redis"
REDIS_HOST="redis"

Extra info about how our Django settings work:

The primary Django settings exist in the das_server app (das/das/das_server), which is the project’s root Django app.

• das_server/settings.py - primary settings file. It contains default settings, and additionally loads environment variables to update some defaults.

• das_server/local_settings_docker.py - these inherit from settings.py. This django_settings_module is used in our kubernetes pod deployments. Kubernetes secrets and configmap is used to inject environment variables into the running pods.

• test_scripts/unittest_settings.py - used by our CI unit test runner in CircleCI, executing in a docker compose-built environment. This settings file inherits from local_settings_docker, loading environment variables and overriding a few settings in this file.

To avoid mixing up settings between development and testing environments, use the VSCode launch tasks available for running pieces of the stack in development and testing modes.

---

Set up pre-commit hooks

1. Install pre-commit libraries and establish the git hooks: pre-commit install

   • To manually run pre-commit: pre-commit run --all-files

   • To update pre-commit: pre-commit autoupdate

2. Install GitGuardian (GG) by following this documentation to detect secrets before they are pushed.

   • Once installed, authorize ggshield by running ggshield auth login

     • You will have to set up a GG account if you don’t have one already.

     • In addition to a pre-commit hook for GG, you can manually trigger a repo scan at any time by executing this command: ggshield secret scan repo /path/to/your/repo

     • Add the appropriate GG variable values to the .env file as described in the Server Settings section.

Run Redis and Database in containers

The project has a Makefile in the root directory with a single task:

• start: Start storage containers (redis and postgres)

Run with: make start at project root directory.

Daily development task examples

I finish my changes, want to run new/existing unit tests

• If it is the first time running unit tests through VSCode, you must first run the VSCode task: PyTest --createdb. This will create a new database called test_das to be used in future tests and avoid the creation of a db every time. (Remember to run this task again if you made changes with migrations.)

• Run PyTest - this launches a prompt to write your desired path of test. If you want to hard code the path for a while, just click on the gear next to the task and change the last line in args property and overwrite ${input:pytest_path_args} with your desired path (don’t commit the file launch.json).

I want to run a manage.py command

You have two options:

1. Run Django manage command task and write the command in the prompt

2. If your command has arguments or something special, edit Django manage command - click on the gear next to the name and overwrite the args property:

   "args": [
       "${input:django_command}",
       "${input:django_command_args}"
   ],
   

   with your desired command:

   "args": [
       "makemigrations", "activity", "--empty"
   ],
   

   Remember to not commit launch.json changes.

I want to use another text editor

PyCharm

Francisco helped set up PyCharm and can help you too.

Neovim

Arthur and Francisco are neovim users and have successfully set up the project to work with neovim using the devcontainer and devpod.

---

Mac Native Install (alternative to devcontainer)

Here we are running the das project on bare metal, rather than in a docker container. Prepare for some fiddling to get the project running. This is not recommended for a straightforward install. Prefer using the docker approach in VS Code.

1. Install Brew

Brew makes it easy to install the necessary libraries to run the ER stack locally. Download brew here.

2. Install Python 3.10 in a venv using uv

1. brew install uv

2. cd das (into the directory you cloned the das repo)

3. uv venv --python=3.10 .venv

4. source .venv/bin/activate

5. Notes: be sure there is no .python-version file in the root directory

3. Install GDAL, libproj and geos

brew install gdal
brew install proj
brew install geos

4. Install Project Dependencies

Use uv to install the requirements. Requirements are found in pyproject.toml, and are pinned in uv.lock.

uv sync --group dev

5. Set GEOS Path in .env

See the Server Settings section on your local .env file.

Other Mac Utilities

• oh my zsh - handy utility for managing your zsh