yggdrasil/server/README.md

# Sensor Readouts

Two Flask services with separate responsibilities and separate databases:

- **main app** (`app.py`) — all user-facing views: login, home, notes,
  notifications, clipboard, calendar, API docs and the `/sensors` dashboard. It
  owns the **main app** database (`main_app.db`: users, notes, notifications and
  clipboard file metadata) and proxies data requests to the sensors API.
- **sensors app** (`sensors_app.py`) — API only. It owns the **sensors**
  database (`sensors.db`, tables `sensor` and `sensor_readout`) and has no
  knowledge of users. Its API is open to every user on the local machine.

```
browser ──> main app :8000 ──HTTP──> sensors API :8001
                │                          │
           main_app.db                sensors.db
```

## Tech Stack

- **Python 3.13+**
- **Flask 3** — web framework
- **SQLAlchemy** (via Flask-SQLAlchemy) — ORM
- **SQLite** — databases
- **Alembic** — database migrations
- **Chart.js** — frontend charting library

## Project Structure

```
server/
├── app.py                    # Main app: views, notes + API proxy
├── sensors_app.py            # Sensors app: API + sensors database
├── sensors_client.py         # HTTP client used by the main app
├── auth.py                   # Flask-Login setup
├── config.py                 # Config (databases, secret key, sensors URL)
├── models.py                 # Sensor, SensorReadout (sensors DB)
├── user_models.py            # User, Note, NoteItem, Notification, ClipboardFile
├── seed.py                   # Sample sensor data
├── migrate_legacy_db.py      # Import old combined data.db into split DBs
├── run.sh                    # Development launcher (main/sensors + port)
├── run-prod.sh               # Production launcher (gunicorn)
├── gunicorn.conf.py          # Gunicorn config (bind from BIND/PORT)
├── alembic.ini               # Sensors DB migrations
├── alembic/                  # Sensors DB migration scripts
├── alembic_users.ini         # Users DB migrations
├── alembic_users/            # Users DB migration scripts
├── sensors.db                # Sensors database (sensors app)
├── main_app.db               # Users database (main app)
├── clipboard/                # Uploaded clipboard files (runtime, not in git)
├── data.db                   # Legacy combined DB (not used by default)
├── requirements.txt
└── templates/
    ├── home.html
    ├── login.html
    ├── notes.html
    ├── notifications.html
    ├── clipboard.html
    ├── calendar.html
    ├── docs.html
    └── sensors.html
```

## Setup

```bash
# Create and activate virtual environment
python3 -m venv .venv
source .venv/bin/activate

# Install dependencies
pip install -r requirements.txt

# Sensors database
alembic upgrade head

# Users database
alembic -c alembic_users.ini upgrade head

# (Optional) Sample sensor data
python seed.py
```

The `sensors.db` and `main_app.db` files are created by the migrations and are
ignored by git (`*.db`).

### Migrating the old combined database

Previous versions stored users, sensors and readouts in `data.db`. The new
layout uses `sensors.db` and `main_app.db`. To copy the old rows (the legacy file
is only read, never modified):

```bash
alembic upgrade head
alembic -c alembic_users.ini upgrade head
python migrate_legacy_db.py
```

For custom paths use `--legacy`, `--sensors` and `--main-app`, for example:

```bash
python migrate_legacy_db.py --legacy old.db --sensors /data/sensors.db \
    --main-app /data/main_app.db
```

After verifying the new databases you can archive or remove `data.db`.

## Users

User accounts, notes and notifications live only in `main_app.db` (main app).
There is no public registration; accounts are created by an administrator.

Main app pages (`/`, `/home`, `/notes`, `/notifications`, `/api/docs`,
`/sensors`) and dashboard read endpoints (`GET /api/sensors/:id/types`,
`/readouts`, `/readouts/hourly`) require login. Device endpoints (`GET/POST /api/sensors`,
`POST /api/sensors/:id/readouts`) stay open so devices can push data without a
browser session.

Set a secret key for sessions (required in production):

```bash
export SECRET_KEY='long-random-value'
```

If `SECRET_KEY` is unset, the main app falls back to `.secret_key` in this
directory (created automatically by `run-prod.sh`).

### First admin

```bash
# Option 1: flags
flask --app app create-admin --username admin --password 'secret'

# Option 2: environment variables (e.g. on first deploy)
ADMIN_USERNAME=admin ADMIN_PASSWORD=secret flask --app app create-admin

# Option 3: interactive (asks for username/password)
flask --app app create-admin
```

Running `create-admin` for an existing username resets its password and grants
`is_admin=True`. Log in at `/login`; you land on `/home`.

When the main app runs as a systemd service, run the command as the same user
and from the same directory as the service (see `yggdrasil.service`):

```bash
cd /home/mateusz/prog/yggdrasil/server
sudo -u mateusz .venv/bin/flask --app app create-admin
```

To pass credentials without an interactive prompt:

```bash
sudo -u mateusz env ADMIN_USERNAME=admin ADMIN_PASSWORD='secret' \
    .venv/bin/flask --app app create-admin
```

If the unit sets `USERS_DB_PATH` or `USERS_DATABASE_URL`, add the same variable
to the command, for example:

```bash
sudo -u mateusz env USERS_DB_PATH=/data/main_app.db \
    .venv/bin/flask --app app create-admin
```

The service does not need to be restarted; the new account is used on the next
login.

## Clipboard

Logged-in users can upload, download and delete files at `/clipboard`. Files
are stored on the server in the clipboard directory (random file name) and only
metadata (original name, content type, size, owner) is kept in `main_app.db`.

Limits:

- one file: **100 MB** (`CLIPBOARD_MAX_FILE_BYTES`)
- total storage per user: **1 GB** (`CLIPBOARD_MAX_TOTAL_BYTES`)

The per-file limit is enforced while uploading; the per-user total quota is
checked before a file is stored.

Apply the users database migration once (the production launcher does this
automatically):

```bash
alembic -c alembic_users.ini upgrade head
```

## Calendar

The main app provides a `/calendar` page with two views for browsing sensor
readouts:

- **month** (`/calendar?view=month&year=YYYY&month=MM`) — calendar grid of the
  given month, each day lists readouts (sensor, time, type, value)
- **week** (`/calendar?view=week&date=YYYY-MM-DD`) — list of the seven days of
  the week containing the given date

Readouts are fetched from the sensors API through the
`GET /api/sensors/:id/readouts/range?from=...&to=...` endpoint.

## Notes

Logged-in users manage their own private short notes at `/notes`. Each note has
a title, a text body and an optional checklist. Unchecked items are shown first;
when an item is checked it is saved as done, moves to the bottom of the checklist
and is displayed with a line through it. Checklist item text can be edited at any
time.

Notes are stored in `main_app.db` in the `note` and `note_item` tables. Apply
the users database migration once (the production launcher does this
automatically):

```bash
alembic -c alembic_users.ini upgrade head
```

## Notifications

Each user has their own short text notifications in `main_app.db`. A
notification is either **new** or **read**. The main navigation shows the number
of new notifications next to the `Notifications` link. Opening a notification
from the `/notifications` page marks it as read and moves it below the new ones.
A notification can be deleted from the `/notifications` page with the `[x]`
button (endpoint: `POST /notifications/<id>/delete`).

Create notifications with the CLI (run it as the same user as the service):

```bash
cd /home/mateusz/prog/yggdrasil/server

# For one user
sudo -u mateusz .venv/bin/flask --app app create-notification \
    --username admin --message "Backup finished."

# For every user
sudo -u mateusz .venv/bin/flask --app app create-notification \
    --all --message "Planned maintenance tonight."
```

Apply the users database migration once (the production launcher does this
automatically):

```bash
alembic -c alembic_users.ini upgrade head
```

## Running

The first argument selects the app (`main` or `sensors`), the second the port.
The main app (`:8000`) listens on all interfaces; the sensors API (`:8001`)
listens on `127.0.0.1` by default and is open to every local user without
authentication.

Do not run the main app on the sensors port (`8001`) while `SENSORS_API_URL` is
unset: it defaults to `http://localhost:8001`, so the main app would proxy API
requests to itself. Both launchers refuse to start in that configuration.

### Development

```bash
./run.sh                 # main app on 8000
./run.sh main 8000
./run.sh sensors 8001
```

### Production

```bash
pip install -r requirements.txt

./run-prod.sh main 8000
./run-prod.sh sensors 8001
```

Both scripts use `.venv` when present. `run-prod.sh` runs the matching
migrations before starting each service.

To expose the sensors API on the network:

```bash
HOST=0.0.0.0 ./run.sh sensors 8001
# or
BIND=0.0.0.0:8001 ./run-prod.sh sensors 8001
```

### Configuration

| Variable | Default | Purpose |
|---|---|---|
| `SENSORS_API_URL` | `http://localhost:8001` | Sensors API address used by the main app |
| `SENSORS_DATABASE_URL` | `sqlite:///<server>/sensors.db` | Sensors database URL |
| `SENSORS_DB_PATH` | `<server>/sensors.db` | SQLite path when URL is unset |
| `USERS_DATABASE_URL` | `sqlite:///<server>/main_app.db` | Users database URL |
| `USERS_DB_PATH` | `<server>/main_app.db` | SQLite path when URL is unset |
| `SECRET_KEY` | `.secret_key` file | Main app session signing key |
| `CLIPBOARD_DIR` | `<server>/clipboard` | Directory where uploaded files are stored |
| `CLIPBOARD_MAX_FILE_BYTES` | `104857600` (100 MB) | Maximum size of a single uploaded file |
| `CLIPBOARD_MAX_TOTAL_BYTES` | `1073741824` (1 GB) | Maximum total size of files per user |
| `HOST` | main `0.0.0.0`, sensors `127.0.0.1` | Listen address |
| `BIND` | `<HOST>:<port>` | Gunicorn bind override |

`DATABASE_URL`/`DB_PATH` are accepted as legacy aliases for the sensors DB.

When the services run on different machines, point `SENSORS_API_URL` at the
address reachable from the main app process. For example, with the sensors API
on your laptop (port 8001) and the main app on a VPS:

```bash
# On the laptop: expose the local sensors API as VPS port 5432
ssh -R 5432:localhost:8001 vps

# On the VPS
SENSORS_API_URL=http://localhost:5432 ./run-prod.sh main 8000
```

### systemd services

Main app only:

```bash
sudo cp yggdrasil.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now yggdrasil
```

Main app and sensors API:

```bash
sudo cp yggdrasil.service yggdrasil-sensors.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now yggdrasil yggdrasil-sensors
```

Check whether the main service is active:

```bash
systemctl status yggdrasil
systemctl is-active yggdrasil      # prints: active / inactive / failed
systemctl is-enabled yggdrasil     # prints: enabled / disabled
```

Quick HTTP check:

```bash
curl -I http://localhost:8000/login
ss -ltnp | grep ':8000'
```

Logs:

```bash
journalctl -u yggdrasil -f
journalctl -u yggdrasil -n 50 --no-pager
journalctl -u yggdrasil-sensors -f   # if the sensors service is installed
```

The unit files currently use `User=mateusz` and
`WorkingDirectory=/home/mateusz/prog/yggdrasil/server`; adjust them if the
deployment path or user is different.

| Service | Port | Routes |
|---|---|---|
| main | 8000 | `/`, `/login`, `/home`, `/clipboard`, `/calendar`, `/notes`, `/notifications`, `/sensors`, `/api/docs`, `/api/*` (proxy) |
| sensors | 8001 | `/api/*` (sensors and readouts) |

### Tests

```bash
python -m unittest discover -s tests -v
```

## Databases

### sensors.db

Owned by the sensors app. Contains only sensor data.

**sensor**
| Column | Type | Description |
|---|---|---|
| `id` | Integer | Primary key |
| `name` | String(128) | Unique sensor name |
| `created_at` | DateTime | Creation timestamp (UTC) |

**sensor_readout**
| Column | Type | Description |
|---|---|---|
| `id` | Integer | Primary key |
| `sensor_id` | Integer | Foreign key to `sensor.id` |
| `type` | String(64) | Readout type (e.g. `temperature`, `pressure`, `humidity`) |
| `value` | Float | Measured value |
| `timestamp` | DateTime | Readout timestamp (UTC) |

### main_app.db (users, notes and notifications)

Owned by the main app. Contains accounts, notes and notifications.

**user**
| Column | Type | Description |
|---|---|---|
| `id` | Integer | Primary key |
| `username` | String(80) | Unique username |
| `password_hash` | String(255) | Password hash |
| `is_admin` | Boolean | Administrator flag |
| `created_at` | DateTime | Creation timestamp (UTC) |

**note**
| Column | Type | Description |
|---|---|---|
| `id` | Integer | Primary key |
| `user_id` | Integer | Foreign key to `user.id` (note owner) |
| `title` | String(200) | Note title (existing notes get `Untitled`) |
| `content` | Text | Note body |
| `created_at` | DateTime | Creation timestamp (UTC) |
| `updated_at` | DateTime | Last modification timestamp (UTC) |

**note_item**
| Column | Type | Description |
|---|---|---|
| `id` | Integer | Primary key |
| `note_id` | Integer | Foreign key to `note.id` |
| `text` | String(255) | Checklist item text |
| `is_done` | Boolean | Checked state (checked items render last, struck through) |
| `created_at` | DateTime | Creation timestamp (UTC) |

**notification**
| Column | Type | Description |
|---|---|---|
| `id` | Integer | Primary key |
| `user_id` | Integer | Foreign key to `user.id` (recipient) |
| `message` | String(500) | Short notification text |
| `is_read` | Boolean | `false` = new, `true` = read |
| `created_at` | DateTime | Creation timestamp (UTC) |
| `read_at` | DateTime | When the notification was read, or `NULL` |

### Inspecting the databases on Linux

The databases are ordinary SQLite files. Use the `sqlite3` CLI (install it with
`sudo apt install sqlite3` on Debian/Ubuntu):

```bash
# List tables
sqlite3 sensors.db ".tables"
sqlite3 main_app.db ".tables"

# Show the schema (all tables or one table)
sqlite3 sensors.db ".schema"
sqlite3 sensors.db ".schema sensor_readout"

# Show columns of one table
sqlite3 sensors.db "PRAGMA table_info(sensor);"

# Inspect rows (readable column output)
sqlite3 -header -column sensors.db \
    "SELECT id, name, created_at FROM sensor LIMIT 10;"
sqlite3 -header -column sensors.db \
    "SELECT id, sensor_id, type, value, timestamp FROM sensor_readout
     ORDER BY timestamp DESC LIMIT 10;"

# Users (do not print password_hash unnecessarily)
sqlite3 -header -column main_app.db \
    "SELECT id, username, is_admin, created_at FROM user;"

# Counts and migration version
sqlite3 sensors.db "SELECT COUNT(*) FROM sensor_readout;"
sqlite3 sensors.db "SELECT * FROM alembic_version;"
sqlite3 main_app.db "SELECT * FROM alembic_version;"
```

Inside an interactive `sqlite3` session you can also use:

```sql
.tables                 -- list tables
.headers on             -- show column names
.mode column            -- align columns
.schema sensor          -- show CREATE TABLE
SELECT * FROM sensor;   -- run a query
.quit                   -- exit
```

For read-only inspection add `-readonly`:

```bash
sqlite3 -readonly -header -column sensors.db \
    "SELECT * FROM sensor_readout LIMIT 5;"
```

If `sqlite3` is not available, the same tables can be listed with Python:

```bash
.venv/bin/python - <<'PY'
import sqlite3
con = sqlite3.connect("sensors.db")
print([row[0] for row in con.execute(
    "SELECT name FROM sqlite_master WHERE type='table'"
)])
PY
```

### Migrations

```bash
# Sensors DB
alembic upgrade head
alembic revision --autogenerate -m "description"

# Users DB
alembic -c alembic_users.ini upgrade head
alembic -c alembic_users.ini revision --autogenerate -m "description"
```

## API

Public API (main app): `http://localhost:8000/api`
Sensors API (local machine): `http://localhost:8001/api`

The main app proxies every `/api/sensors*` request to the sensors API, so
clients and devices can keep using port 8000.

### Endpoints

| Method | Path | Auth | Description |
|---|---|---|---|
| `GET` | `/api/sensors` | public | List all sensors |
| `POST` | `/api/sensors` | public | Create a new sensor |
| `GET` | `/api/sensors/:id/types` | login | List readout types for a sensor |
| `GET` | `/api/sensors/:id/readouts` | login | Latest readouts (max 300, filterable by `?type=`) |
| `GET` | `/api/sensors/:id/readouts/hourly` | login | Hourly averaged readouts |
| `POST` | `/api/sensors/:id/readouts` | public | Add a new readout |

Calling the sensors API directly on `:8001` skips login; it is open to every
local user by design.

### Examples

```bash
# List sensors
curl http://localhost:8000/api/sensors

# Create a sensor
curl -X POST http://localhost:8000/api/sensors \
     -H "Content-Type: application/json" \
     -d '{"name": "bmp280"}'

# Get temperature readouts for sensor 1 (requires a logged-in session)
curl -b cookies.txt "http://localhost:8000/api/sensors/1/readouts?type=temperature"

# Get hourly averaged data (requires a logged-in session)
curl -b cookies.txt "http://localhost:8000/api/sensors/1/readouts/hourly?type=temperature"

# Add a readout
curl -X POST http://localhost:8000/api/sensors/1/readouts \
     -H "Content-Type: application/json" \
     -d '{"type": "temperature", "value": 22.5}'

# Direct local access with no login
curl http://localhost:8001/api/sensors/1/readouts
```

Full API documentation with request/response examples is available at `/api/docs`.

## Frontend

The dashboard in the main app at `/sensors` provides:

- **Sensor dropdown** — select a sensor from the list
- **Type dropdown** — select a readout type (populated based on the chosen sensor)
- **Line chart** — raw or hourly-averaged values for the selected range
- **Data table** — raw readouts with formatted timestamps (latest 300 entries)