yggdrasil/server/README.md

# Sensor Readouts

Two Flask services with separate responsibilities and separate databases:

- **main app** (`app.py`) — all user-facing views: login, home, API docs and
  the `/sensors` dashboard. It owns the **users** database (`main_app.db`) and
  proxies data requests to the sensors API.
- **sensors app** (`sensors_app.py`) — API only. It owns the **sensors**
  database (`sensors.db`, tables `sensor` and `sensor_readout`) and has no
  knowledge of users. Its API is open to every user on the local machine.

```
browser ──> main app :8000 ──HTTP──> sensors API :8001
                │                          │
           main_app.db                sensors.db
```

## Tech Stack

- **Python 3.13+**
- **Flask 3** — web framework
- **SQLAlchemy** (via Flask-SQLAlchemy) — ORM
- **SQLite** — databases
- **Alembic** — database migrations
- **Chart.js** — frontend charting library

## Project Structure

```
server/
├── app.py                    # Main app: views + API proxy
├── sensors_app.py            # Sensors app: API + sensors database
├── sensors_client.py         # HTTP client used by the main app
├── auth.py                   # Flask-Login setup
├── config.py                 # Config (databases, secret key, sensors URL)
├── models.py                 # Sensor, SensorReadout (sensors DB)
├── user_models.py            # User (main_app.db)
├── seed.py                   # Sample sensor data
├── migrate_legacy_db.py      # Import old combined data.db into split DBs
├── run.sh                    # Development launcher (main/sensors + port)
├── run-prod.sh               # Production launcher (gunicorn)
├── gunicorn.conf.py          # Gunicorn config (bind from BIND/PORT)
├── alembic.ini               # Sensors DB migrations
├── alembic/                  # Sensors DB migration scripts
├── alembic_users.ini         # Users DB migrations
├── alembic_users/            # Users DB migration scripts
├── sensors.db                # Sensors database (sensors app)
├── main_app.db               # Users database (main app)
├── data.db                   # Legacy combined DB (not used by default)
├── requirements.txt
└── templates/
    ├── home.html
    ├── login.html
    ├── docs.html
    └── sensors.html
```

## Setup

```bash
# Create and activate virtual environment
python3 -m venv .venv
source .venv/bin/activate

# Install dependencies
pip install -r requirements.txt

# Sensors database
alembic upgrade head

# Users database
alembic -c alembic_users.ini upgrade head

# (Optional) Sample sensor data
python seed.py
```

The `sensors.db` and `main_app.db` files are created by the migrations and are
ignored by git (`*.db`).

### Migrating the old combined database

Previous versions stored users, sensors and readouts in `data.db`. The new
layout uses `sensors.db` and `main_app.db`. To copy the old rows (the legacy file
is only read, never modified):

```bash
alembic upgrade head
alembic -c alembic_users.ini upgrade head
python migrate_legacy_db.py
```

For custom paths use `--legacy`, `--sensors` and `--main-app`, for example:

```bash
python migrate_legacy_db.py --legacy old.db --sensors /data/sensors.db \
    --main-app /data/main_app.db
```

After verifying the new databases you can archive or remove `data.db`.

## Users

User accounts live only in `main_app.db` (main app). There is no public
registration; accounts are created by an administrator.

Main app pages (`/`, `/home`, `/api/docs`, `/sensors`) and dashboard read
endpoints (`GET /api/sensors/:id/types`, `/readouts`, `/readouts/hourly`)
require login. Device endpoints (`GET/POST /api/sensors`,
`POST /api/sensors/:id/readouts`) stay open so devices can push data without a
browser session.

Set a secret key for sessions (required in production):

```bash
export SECRET_KEY='long-random-value'
```

If `SECRET_KEY` is unset, the main app falls back to `.secret_key` in this
directory (created automatically by `run-prod.sh`).

### First admin

```bash
# Option 1: flags
flask --app app create-admin --username admin --password 'secret'

# Option 2: environment variables (e.g. on first deploy)
ADMIN_USERNAME=admin ADMIN_PASSWORD=secret flask --app app create-admin

# Option 3: interactive (asks for username/password)
flask --app app create-admin
```

Running `create-admin` for an existing username resets its password and grants
`is_admin=True`. Log in at `/login`; you land on `/home`.

## Running

The first argument selects the app (`main` or `sensors`), the second the port.
The main app (`:8000`) listens on all interfaces; the sensors API (`:8001`)
listens on `127.0.0.1` by default and is open to every local user without
authentication.

### Development

```bash
./run.sh                 # main app on 8000
./run.sh main 8000
./run.sh sensors 8001
```

### Production

```bash
pip install -r requirements.txt

./run-prod.sh main 8000
./run-prod.sh sensors 8001
```

Both scripts use `.venv` when present. `run-prod.sh` runs the matching
migrations before starting each service.

To expose the sensors API on the network:

```bash
HOST=0.0.0.0 ./run.sh sensors 8001
# or
BIND=0.0.0.0:8001 ./run-prod.sh sensors 8001
```

### Configuration

| Variable | Default | Purpose |
|---|---|---|
| `SENSORS_API_URL` | `http://localhost:8001` | Sensors API address used by the main app |
| `SENSORS_DATABASE_URL` | `sqlite:///<server>/sensors.db` | Sensors database URL |
| `SENSORS_DB_PATH` | `<server>/sensors.db` | SQLite path when URL is unset |
| `USERS_DATABASE_URL` | `sqlite:///<server>/main_app.db` | Users database URL |
| `USERS_DB_PATH` | `<server>/main_app.db` | SQLite path when URL is unset |
| `SECRET_KEY` | `.secret_key` file | Main app session signing key |
| `HOST` | main `0.0.0.0`, sensors `127.0.0.1` | Listen address |
| `BIND` | `<HOST>:<port>` | Gunicorn bind override |

`DATABASE_URL`/`DB_PATH` are accepted as legacy aliases for the sensors DB.

When the services run on different machines, point `SENSORS_API_URL` at the
address reachable from the main app process. For example, with the sensors API
on your laptop (port 8001) and the main app on a VPS:

```bash
# On the laptop: expose the local sensors API as VPS port 5432
ssh -R 5432:localhost:8001 vps

# On the VPS
SENSORS_API_URL=http://localhost:5432 ./run-prod.sh main 8000
```

### systemd services

```bash
sudo cp yggdrasil.service yggdrasil-sensors.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now yggdrasil yggdrasil-sensors
```

| Service | Port | Routes |
|---|---|---|
| main | 8000 | `/`, `/login`, `/home`, `/sensors`, `/api/docs`, `/api/*` (proxy) |
| sensors | 8001 | `/api/*` (sensors and readouts) |

### Tests

```bash
python -m unittest discover -s tests -v
```

## Databases

### sensors.db

Owned by the sensors app. Contains only sensor data.

**sensor**
| Column | Type | Description |
|---|---|---|
| `id` | Integer | Primary key |
| `name` | String(128) | Unique sensor name |
| `created_at` | DateTime | Creation timestamp (UTC) |

**sensor_readout**
| Column | Type | Description |
|---|---|---|
| `id` | Integer | Primary key |
| `sensor_id` | Integer | Foreign key to `sensor.id` |
| `type` | String(64) | Readout type (e.g. `temperature`, `pressure`, `humidity`) |
| `value` | Float | Measured value |
| `timestamp` | DateTime | Readout timestamp (UTC) |

### main_app.db (users)

Owned by the main app. Contains only accounts.

**user**
| Column | Type | Description |
|---|---|---|
| `id` | Integer | Primary key |
| `username` | String(80) | Unique username |
| `password_hash` | String(255) | Password hash |
| `is_admin` | Boolean | Administrator flag |
| `created_at` | DateTime | Creation timestamp (UTC) |

### Inspecting the databases on Linux

The databases are ordinary SQLite files. Use the `sqlite3` CLI (install it with
`sudo apt install sqlite3` on Debian/Ubuntu):

```bash
# List tables
sqlite3 sensors.db ".tables"
sqlite3 main_app.db ".tables"

# Show the schema (all tables or one table)
sqlite3 sensors.db ".schema"
sqlite3 sensors.db ".schema sensor_readout"

# Show columns of one table
sqlite3 sensors.db "PRAGMA table_info(sensor);"

# Inspect rows (readable column output)
sqlite3 -header -column sensors.db \
    "SELECT id, name, created_at FROM sensor LIMIT 10;"
sqlite3 -header -column sensors.db \
    "SELECT id, sensor_id, type, value, timestamp FROM sensor_readout
     ORDER BY timestamp DESC LIMIT 10;"

# Users (do not print password_hash unnecessarily)
sqlite3 -header -column main_app.db \
    "SELECT id, username, is_admin, created_at FROM user;"

# Counts and migration version
sqlite3 sensors.db "SELECT COUNT(*) FROM sensor_readout;"
sqlite3 sensors.db "SELECT * FROM alembic_version;"
sqlite3 main_app.db "SELECT * FROM alembic_version;"
```

Inside an interactive `sqlite3` session you can also use:

```sql
.tables                 -- list tables
.headers on             -- show column names
.mode column            -- align columns
.schema sensor          -- show CREATE TABLE
SELECT * FROM sensor;   -- run a query
.quit                   -- exit
```

For read-only inspection add `-readonly`:

```bash
sqlite3 -readonly -header -column sensors.db \
    "SELECT * FROM sensor_readout LIMIT 5;"
```

If `sqlite3` is not available, the same tables can be listed with Python:

```bash
.venv/bin/python - <<'PY'
import sqlite3
con = sqlite3.connect("sensors.db")
print([row[0] for row in con.execute(
    "SELECT name FROM sqlite_master WHERE type='table'"
)])
PY
```

### Migrations

```bash
# Sensors DB
alembic upgrade head
alembic revision --autogenerate -m "description"

# Users DB
alembic -c alembic_users.ini upgrade head
alembic -c alembic_users.ini revision --autogenerate -m "description"
```

## API

Public API (main app): `http://localhost:8000/api`
Sensors API (local machine): `http://localhost:8001/api`

The main app proxies every `/api/sensors*` request to the sensors API, so
clients and devices can keep using port 8000.

### Endpoints

| Method | Path | Auth | Description |
|---|---|---|---|
| `GET` | `/api/sensors` | public | List all sensors |
| `POST` | `/api/sensors` | public | Create a new sensor |
| `GET` | `/api/sensors/:id/types` | login | List readout types for a sensor |
| `GET` | `/api/sensors/:id/readouts` | login | Latest readouts (max 300, filterable by `?type=`) |
| `GET` | `/api/sensors/:id/readouts/hourly` | login | Hourly averaged readouts |
| `POST` | `/api/sensors/:id/readouts` | public | Add a new readout |

Calling the sensors API directly on `:8001` skips login; it is open to every
local user by design.

### Examples

```bash
# List sensors
curl http://localhost:8000/api/sensors

# Create a sensor
curl -X POST http://localhost:8000/api/sensors \
     -H "Content-Type: application/json" \
     -d '{"name": "bmp280"}'

# Get temperature readouts for sensor 1 (requires a logged-in session)
curl -b cookies.txt "http://localhost:8000/api/sensors/1/readouts?type=temperature"

# Get hourly averaged data (requires a logged-in session)
curl -b cookies.txt "http://localhost:8000/api/sensors/1/readouts/hourly?type=temperature"

# Add a readout
curl -X POST http://localhost:8000/api/sensors/1/readouts \
     -H "Content-Type: application/json" \
     -d '{"type": "temperature", "value": 22.5}'

# Direct local access with no login
curl http://localhost:8001/api/sensors/1/readouts
```

Full API documentation with request/response examples is available at `/api/docs`.

## Frontend

The dashboard in the main app at `/sensors` provides:

- **Sensor dropdown** — select a sensor from the list
- **Type dropdown** — select a readout type (populated based on the chosen sensor)
- **Line chart** — raw or hourly-averaged values for the selected range
- **Data table** — raw readouts with formatted timestamps (latest 300 entries)