mateusz/yggdrasil
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

sensors is separate API with td

Browse Source
This commit is contained in:
mateusz
2026-11-01 12:07:04 +01:00
parent b7048b401d
commit 85179c8a80
24 changed files with 1229 additions and 321 deletions
Download Patch File Download Diff File Expand all files Collapse all files
server/README.md
+256 -69
View File
@@ -1,13 +1,26 @@
# Sensor Readouts
A Flask web application for managing and visualizing sensor data. It provides a REST API for creating sensors and storing readouts, along with a frontend dashboard that displays real-time charts and tabular data.
Two Flask services with separate responsibilities and separate databases:
- **main app** (`app.py`) — all user-facing views: login, home, API docs and
the `/sensors` dashboard. It owns the **users** database (`main_app.db`) and
proxies data requests to the sensors API.
- **sensors app** (`sensors_app.py`) — API only. It owns the **sensors**
database (`sensors.db`, tables `sensor` and `sensor_readout`) and has no
knowledge of users. Its API is open to every user on the local machine.
```
browser ──> main app :8000 ──HTTP──> sensors API :8001
│ │
main_app.db sensors.db
```
## Tech Stack
- **Python 3.13+**
- **Flask 3** — web framework
- **SQLAlchemy** (via Flask-SQLAlchemy) — ORM
- **SQLite** — database
- **SQLite** — databases
- **Alembic** — database migrations
- **Chart.js** — frontend charting library
@@ -15,18 +28,31 @@ A Flask web application for managing and visualizing sensor data. It provides a
```
server/
├── app.py # Flask application factory and API routes
├── models.py # SQLAlchemy models (Sensor, SensorReadout)
├── seed.py # Database seeder with sample data
├── requirements.txt # Python dependencies
├── data.db # SQLite database (auto-created)
├── alembic.ini # Alembic configuration
├── alembic/ # Migration scripts
│ ├── env.py
│ └── versions/
├── app.py # Main app: views + API proxy
├── sensors_app.py # Sensors app: API + sensors database
├── sensors_client.py # HTTP client used by the main app
├── auth.py # Flask-Login setup
├── config.py # Config (databases, secret key, sensors URL)
├── models.py # Sensor, SensorReadout (sensors DB)
├── user_models.py # User (main_app.db)
├── seed.py # Sample sensor data
├── migrate_legacy_db.py # Import old combined data.db into split DBs
├── run.sh # Development launcher (main/sensors + port)
├── run-prod.sh # Production launcher (gunicorn)
├── gunicorn.conf.py # Gunicorn config (bind from BIND/PORT)
├── alembic.ini # Sensors DB migrations
├── alembic/ # Sensors DB migration scripts
├── alembic_users.ini # Users DB migrations
├── alembic_users/ # Users DB migration scripts
├── sensors.db # Sensors database (sensors app)
├── main_app.db # Users database (main app)
├── data.db # Legacy combined DB (not used by default)
├── requirements.txt
└── templates/
├── sensors.html # Dashboard with chart and table
└── docs.html # API documentation page
├── home.html
├── login.html
├── docs.html
└── sensors.html
```
## Setup
@@ -39,20 +65,50 @@ source .venv/bin/activate
# Install dependencies
pip install -r requirements.txt
# Run database migrations
# Sensors database
alembic upgrade head
# (Optional) Seed the database with sample data
# Users database
alembic -c alembic_users.ini upgrade head
# (Optional) Sample sensor data
python seed.py
```
The `sensors.db` and `main_app.db` files are created by the migrations and are
ignored by git (`*.db`).
### Migrating the old combined database
Previous versions stored users, sensors and readouts in `data.db`. The new
layout uses `sensors.db` and `main_app.db`. To copy the old rows (the legacy file
is only read, never modified):
```bash
alembic upgrade head
alembic -c alembic_users.ini upgrade head
python migrate_legacy_db.py
```
For custom paths use `--legacy`, `--sensors` and `--main-app`, for example:
```bash
python migrate_legacy_db.py --legacy old.db --sensors /data/sensors.db \
--main-app /data/main_app.db
```
After verifying the new databases you can archive or remove `data.db`.
## Users
There is no public registration. All accounts are created by an administrator.
Pages (`/`, `/home`, `/sensors`, `/api/docs`) and dashboard data endpoints
(`GET /api/sensors/:id/types`, `/readouts`, `/readouts/hourly`) require login.
`GET /api/sensors` (list, needed by devices to resolve name to id) and
`POST /api/*` stay open so devices can push data without a browser session.
User accounts live only in `main_app.db` (main app). There is no public
registration; accounts are created by an administrator.
Main app pages (`/`, `/home`, `/api/docs`, `/sensors`) and dashboard read
endpoints (`GET /api/sensors/:id/types`, `/readouts`, `/readouts/hourly`)
require login. Device endpoints (`GET/POST /api/sensors`,
`POST /api/sensors/:id/readouts`) stay open so devices can push data without a
browser session.
Set a secret key for sessions (required in production):
@@ -60,15 +116,12 @@ Set a secret key for sessions (required in production):
export SECRET_KEY='long-random-value'
```
If `SECRET_KEY` is unset, the main app falls back to `.secret_key` in this
directory (created automatically by `run-prod.sh`).
### First admin
The database starts with an empty `user` table. Create the first admin
with the CLI command (flags, env vars, or interactive prompts):
```bash
# Run database migrations first
alembic upgrade head
# Option 1: flags
flask --app app create-admin --username admin --password 'secret'
@@ -79,47 +132,95 @@ ADMIN_USERNAME=admin ADMIN_PASSWORD=secret flask --app app create-admin
flask --app app create-admin
```
Running `create-admin` for an existing username resets its password and
grants `is_admin=True`. After that log in at `/login` — you land on `/home`.
Running `create-admin` for an existing username resets its password and grants
`is_admin=True`. Log in at `/login`; you land on `/home`.
## Running
The first argument selects the app (`main` or `sensors`), the second the port.
The main app (`:8000`) listens on all interfaces; the sensors API (`:8001`)
listens on `127.0.0.1` by default and is open to every local user without
authentication.
### Development
```bash
python app.py
./run.sh # main app on 8000
./run.sh main 8000
./run.sh sensors 8001
```
The server starts on `http://localhost:5000`.
### Production
```bash
# Install dependencies (includes gunicorn)
pip install -r requirements.txt
# Run with gunicorn
gunicorn -c gunicorn.conf.py "app:create_app()"
./run-prod.sh main 8000
./run-prod.sh sensors 8001
```
Gunicorn starts on `http://0.0.0.0:8000` with `cpu * 2 + 1` workers.
Both scripts use `.venv` when present. `run-prod.sh` runs the matching
migrations before starting each service.
#### systemd service
To expose the sensors API on the network:
```bash
sudo cp yggdrasil.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now yggdrasil
HOST=0.0.0.0 ./run.sh sensors 8001
# or
BIND=0.0.0.0:8001 ./run-prod.sh sensors 8001
```
| Route | Description |
|---|---|
| `/sensors` | Dashboard with interactive chart and data table |
| `/api/docs` | API documentation with curl examples |
### Configuration
## Database
| Variable | Default | Purpose |
|---|---|---|
| `SENSORS_API_URL` | `http://localhost:8001` | Sensors API address used by the main app |
| `SENSORS_DATABASE_URL` | `sqlite:///<server>/sensors.db` | Sensors database URL |
| `SENSORS_DB_PATH` | `<server>/sensors.db` | SQLite path when URL is unset |
| `USERS_DATABASE_URL` | `sqlite:///<server>/main_app.db` | Users database URL |
| `USERS_DB_PATH` | `<server>/main_app.db` | SQLite path when URL is unset |
| `SECRET_KEY` | `.secret_key` file | Main app session signing key |
| `HOST` | main `0.0.0.0`, sensors `127.0.0.1` | Listen address |
| `BIND` | `<HOST>:<port>` | Gunicorn bind override |
Two tables are used:
`DATABASE_URL`/`DB_PATH` are accepted as legacy aliases for the sensors DB.
When the services run on different machines, point `SENSORS_API_URL` at the
address reachable from the main app process. For example, with the sensors API
on your laptop (port 8001) and the main app on a VPS:
```bash
# On the laptop: expose the local sensors API as VPS port 5432
ssh -R 5432:localhost:8001 vps
# On the VPS
SENSORS_API_URL=http://localhost:5432 ./run-prod.sh main 8000
```
### systemd services
```bash
sudo cp yggdrasil.service yggdrasil-sensors.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now yggdrasil yggdrasil-sensors
```
| Service | Port | Routes |
|---|---|---|
| main | 8000 | `/`, `/login`, `/home`, `/sensors`, `/api/docs`, `/api/*` (proxy) |
| sensors | 8001 | `/api/*` (sensors and readouts) |
### Tests
```bash
python -m unittest discover -s tests -v
```
## Databases
### sensors.db
Owned by the sensors app. Contains only sensor data.
**sensor**
| Column | Type | Description |
@@ -137,64 +238,150 @@ Two tables are used:
| `value` | Float | Measured value |
| `timestamp` | DateTime | Readout timestamp (UTC) |
### main_app.db (users)
Owned by the main app. Contains only accounts.
**user**
| Column | Type | Description |
|---|---|---|
| `id` | Integer | Primary key |
| `username` | String(80) | Unique username |
| `password_hash` | String(255) | Password hash |
| `is_admin` | Boolean | Administrator flag |
| `created_at` | DateTime | Creation timestamp (UTC) |
### Inspecting the databases on Linux
The databases are ordinary SQLite files. Use the `sqlite3` CLI (install it with
`sudo apt install sqlite3` on Debian/Ubuntu):
```bash
# List tables
sqlite3 sensors.db ".tables"
sqlite3 main_app.db ".tables"
# Show the schema (all tables or one table)
sqlite3 sensors.db ".schema"
sqlite3 sensors.db ".schema sensor_readout"
# Show columns of one table
sqlite3 sensors.db "PRAGMA table_info(sensor);"
# Inspect rows (readable column output)
sqlite3 -header -column sensors.db \
"SELECT id, name, created_at FROM sensor LIMIT 10;"
sqlite3 -header -column sensors.db \
"SELECT id, sensor_id, type, value, timestamp FROM sensor_readout
ORDER BY timestamp DESC LIMIT 10;"
# Users (do not print password_hash unnecessarily)
sqlite3 -header -column main_app.db \
"SELECT id, username, is_admin, created_at FROM user;"
# Counts and migration version
sqlite3 sensors.db "SELECT COUNT(*) FROM sensor_readout;"
sqlite3 sensors.db "SELECT * FROM alembic_version;"
sqlite3 main_app.db "SELECT * FROM alembic_version;"
```
Inside an interactive `sqlite3` session you can also use:
```sql
.tables -- list tables
.headers on -- show column names
.mode column -- align columns
.schema sensor -- show CREATE TABLE
SELECT * FROM sensor; -- run a query
.quit -- exit
```
For read-only inspection add `-readonly`:
```bash
sqlite3 -readonly -header -column sensors.db \
"SELECT * FROM sensor_readout LIMIT 5;"
```
If `sqlite3` is not available, the same tables can be listed with Python:
```bash
.venv/bin/python - <<'PY'
import sqlite3
con = sqlite3.connect("sensors.db")
print([row[0] for row in con.execute(
"SELECT name FROM sqlite_master WHERE type='table'"
)])
PY
```
### Migrations
```bash
# Create a new migration after model changes
# Sensors DB
alembic upgrade head
alembic revision --autogenerate -m "description"
# Apply pending migrations
alembic upgrade head
# Roll back one step
alembic downgrade -1
# Users DB
alembic -c alembic_users.ini upgrade head
alembic -c alembic_users.ini revision --autogenerate -m "description"
```
## API
Base URL: `http://localhost:5000/api`
Public API (main app): `http://localhost:8000/api`
Sensors API (local machine): `http://localhost:8001/api`
The main app proxies every `/api/sensors*` request to the sensors API, so
clients and devices can keep using port 8000.
### Endpoints
| Method | Path | Description |
|---|---|---|
| `GET` | `/api/sensors` | List all sensors |
| `POST` | `/api/sensors` | Create a new sensor |
| `GET` | `/api/sensors/:id/types` | List readout types for a sensor |
| `GET` | `/api/sensors/:id/readouts` | Latest readouts (max 300, filterable by `?type=`) |
| `GET` | `/api/sensors/:id/readouts/hourly` | Hourly averaged readouts (filterable by `?type=`) |
| `POST` | `/api/sensors/:id/readouts` | Add a new readout |
| Method | Path | Auth | Description |
|---|---|---|---|
| `GET` | `/api/sensors` | public | List all sensors |
| `POST` | `/api/sensors` | public | Create a new sensor |
| `GET` | `/api/sensors/:id/types` | login | List readout types for a sensor |
| `GET` | `/api/sensors/:id/readouts` | login | Latest readouts (max 300, filterable by `?type=`) |
| `GET` | `/api/sensors/:id/readouts/hourly` | login | Hourly averaged readouts |
| `POST` | `/api/sensors/:id/readouts` | public | Add a new readout |
Calling the sensors API directly on `:8001` skips login; it is open to every
local user by design.
### Examples
```bash
# List sensors
curl http://localhost:5000/api/sensors
curl http://localhost:8000/api/sensors
# Create a sensor
curl -X POST http://localhost:5000/api/sensors \
curl -X POST http://localhost:8000/api/sensors \
-H "Content-Type: application/json" \
-d '{"name": "bmp280"}'
# Get temperature readouts for sensor 1
curl "http://localhost:5000/api/sensors/1/readouts?type=temperature"
# Get temperature readouts for sensor 1 (requires a logged-in session)
curl -b cookies.txt "http://localhost:8000/api/sensors/1/readouts?type=temperature"
# Get hourly averaged data
curl "http://localhost:5000/api/sensors/1/readouts/hourly?type=temperature"
# Get hourly averaged data (requires a logged-in session)
curl -b cookies.txt "http://localhost:8000/api/sensors/1/readouts/hourly?type=temperature"
# Add a readout
curl -X POST http://localhost:5000/api/sensors/1/readouts \
curl -X POST http://localhost:8000/api/sensors/1/readouts \
-H "Content-Type: application/json" \
-d '{"type": "temperature", "value": 22.5}'
# Direct local access with no login
curl http://localhost:8001/api/sensors/1/readouts
```
Full API documentation with request/response examples is available at `/api/docs`.
## Frontend
The dashboard at `/sensors` provides:
The dashboard in the main app at `/sensors` provides:
- **Sensor dropdown** — select a sensor from the list
- **Type dropdown** — select a readout type (populated based on the chosen sensor)
- **Line chart** — hourly averaged values rendered with Chart.js
- **Line chart** — raw or hourly-averaged values for the selected range
- **Data table** — raw readouts with formatted timestamps (latest 300 entries)