Backup and Restore

This page specifies the Plant Tracer backup, restore, and course migration tool, src/dbbackup.py.

This is an operational backup for selected Plant Tracer data, not a full AWS account snapshot. It is intended to run from a developer or administrator machine with AWS credentials. The target DynamoDB table prefix is specified on the command line and supersedes DYNAMODB_TABLE_PREFIX. If PLANTTRACER_S3_BUCKET is configured during restore, restored movie objects are written to that bucket; otherwise restore uses the bucket recorded for each movie object in the archive. A future web-admin download/upload workflow may be added later.

Goals

src/dbbackup.py supports:

  • full and selective backups;

  • full and selective restores;

  • restoring a complete course, user, or movie dependency set;

  • dry-run login-link generation for restored users, with sending only when --send is present;

  • course and movie migration without leaving inconsistent DynamoDB records or stale S3 object references.

The implementation preserves existing IDs on restore. In particular, course_id, user_id, and movie_id are restored exactly as recorded in the backup.

Command-Line Contract

The CLI keeps backup, restore, inspection, login-link sending, and migration as separate subcommands:

poetry run dbbackup backup --table-prefix PREFIX --output file.ptb [--all | --course-id C | --user-email E | --movie-id M] [--include-deleted]
poetry run dbbackup restore --table-prefix PREFIX file.ptb [--all | --course-id C | --user-email E | --movie-id M] [--commit] [--threads N] [--regenerate-zips]
poetry run dbbackup inspect file.ptb [--verbose]
poetry run dbbackup list-prefixes
poetry run dbbackup send-restore-links --table-prefix PREFIX file.ptb [--all | --course-id C | --user-email E] [--send]
poetry run dbbackup migrate-course --table-prefix PREFIX --from-course-id A --to-course-id B [--user-email E] [--commit]

--table-prefix is the authoritative table prefix for commands that touch DynamoDB. It overrides DYNAMODB_TABLE_PREFIX if that environment variable is also set.

restore and migrate-course default to preflight mode. They must inspect the target state and report planned writes or blockers, but they must not write anything unless --commit is present.

backup defaults to --all when no selector is supplied. Restore and send-restore-links require an explicit selector so operators choose the restore or email scope deliberately.

list-prefixes does not take --table-prefix. It uses the current DynamoDB connection settings, lists all tables, reports only prefixes for which all application tables exist, and prints a fixed-width table with the course, user, and movie counts plus a single UTC from/to range for each complete prefix. The range is the minimum and maximum timestamp found in user created fields, movie created_at/date_uploaded fields, optional movie status_updated_at fields, and optional course created/created_at fields when those attributes are present. Current course rows do not have a schema-defined creation timestamp, and frame/trackpoint rows store frame numbers rather than timestamps. If the current AWS SSO token is expired, list-prefixes asks whether it should run aws sso login and retry once.

Backup Scope

Backups include application records needed to reconstruct courses, users, movies, and movie annotations:

  • users

  • courses

  • course_users

  • movies

  • movie_frames

Backups do not include:

  • api_keys;

  • unique_emails;

  • logs;

  • frame ZIP artifacts;

  • original uploaded movie files, by default;

  • traced movie artifacts.

Deleted movies are excluded by default. backup includes deleted movies only when --include-deleted is present.

unique_emails is regenerated during restore from restored user records. api_keys are intentionally not restored; after restore, the operator should be able to send new login emails to every restored account, creating fresh API keys. Audit logs are operational history and are not part of the backup. A separate log-dump command should export logs to a syslog-like text file.

Movie Objects

Each backed-up movie includes the rotated and shrunk MP4 used for analysis. This file is distinct from the traced MP4. The backup member name should be based on the movie ID, for example movies/{movie_id}.mp4.

Original uploaded movies are not included by default. If an implementation later adds an explicit option to include originals, those files should be named separately, for example movies/{movie_id}-orig.mp4.

Frame ZIP files are not backed up because they can be large and can be regenerated from the rotated and shrunk MP4. Restore should provide an option to regenerate omitted ZIP files. The data model needs a separate needs_zipfile concept so a restored movie can distinguish “trackpoints are stale and retracing is needed” from “trackpoints are current but the generated ZIP artifact is missing.”

Backup Format

The backup file extension is .ptb. The file is a ZIP archive containing:

  • a manifest JSON file;

  • a human-readable README file;

  • one JSON Lines file per backed-up DynamoDB table;

  • one MP4 file per backed-up movie.

The archive layout should be:

manifest.json
README
tables/users.jsonl
tables/courses.jsonl
tables/course_users.jsonl
tables/movies.jsonl
tables/movie_frames.jsonl
movies/{movie_id}.mp4

Each DynamoDB record is written as one JSON object per line using DynamoDB AttributeValue JSON, matching the native DynamoDB API item shape. For example, a string attribute is stored as {"S": "value"} and a number attribute is stored as {"N": "10.0"}. This preserves DynamoDB value types, including decimal numbers, sets, binary values, booleans, nulls, lists, and maps.

The manifest records at least:

  • backup format version;

  • Plant Tracer application version;

  • creation timestamp;

  • operator identity, when available;

  • source hostname or environment name, when available;

  • source AWS account/region details, when available;

  • source DynamoDB table prefix;

  • source S3 bucket;

  • selection filters used for the backup;

  • backup options, including whether originals were included;

  • per-table record counts;

  • per-movie object names, sizes, and checksums.

The README must contain a full warning that the archive contains student data, course data, movie metadata, trackpoint annotations, and movie files. It must state that the archive is not password protected, should be stored and transmitted as sensitive data, and can be restored into a Plant Tracer deployment by an operator with suitable AWS credentials.

Selective Backup

Backup selection is separate from restore selection. Backup supports:

  • all courses, users, and movies, which is the default when no selector is supplied;

  • one course by course_id;

  • one user by email address;

  • one movie by movie_id.

A selective backup must include the dependency records needed to restore the selected data consistently.

backup prints progress to stderr while it examines DynamoDB tables, runs preflight checks, reports the S3 bucket or buckets used by selected movie objects, and creates the archive. During preflight it verifies that each selected movie MP4 object exists in S3. If a selected movie’s MP4 is missing, backup prints a warning, records the warning in the manifest, omits that movie row and its frame rows from the archive, and continues backing up the remaining data. Other S3 access failures still abort the backup.

If the output archive already exists, backup first reads its manifest. It will overwrite the archive only when the existing archive’s source DynamoDB table prefix matches the requested --table-prefix after normalizing the trailing dash. This prevents accidentally replacing the metadata tables from a different DynamoDB installation. When overwriting a same-prefix archive, backup reuses an existing archived movie MP4 if the manifest shows the same S3 bucket and key, copying that zip member into the new archive instead of downloading it again. If the current S3 object is missing but the same object is already present in the existing archive, backup records a warning and reuses the existing archived copy.

For a user backup, include the user, the user’s primary course, every course that contains one of the user’s movies, the corresponding enrollment rows, the user’s selected movie rows, their frame rows, and their movie MP4 files.

For a movie backup, include only the movie owner user record, the movie’s course, the needed enrollment row, the movie row, frame rows, and movie MP4. Do not include course administrator users solely because they administer the movie’s course.

Archive Inspection

inspect defaults to a concise summary: manifest metadata, per-table record counts, the S3 bucket or buckets a restore would write movie objects to, backed-up movie object count, and total backed-up movie bytes. It does not dump individual records by default.

inspect --verbose adds one-line summaries for each backed-up user, course, course enrollment, movie, and movie object. Frame records are not dumped individually. Instead, verbose inspection reports the total number of frame records and groups consecutive frame ranges that have the same trackpoint count and trackpoint labels, producing one line per trackpoint set.

Restore Behavior

Restore supports:

  • all backed-up data;

  • one course by course_id;

  • one user by email address;

  • one movie by movie_id.

Restore preserves IDs exactly. If no DynamoDB tables exist under the specified --table-prefix, restore treats the target prefix as a new installation. In preflight mode it reports that the tables will be created and exits without creating them. With --commit, restore creates the full table set in parallel before writing restored rows and movie objects. If any table creation fails, restore deletes tables it created and fails before writing S3 objects or DynamoDB rows. If only some required target tables exist, restore blocks because a partial prefix can hide existing data or schema drift.

Restore defaults to preflight mode. Without --commit, it validates the archive, validates target dependencies, reports blockers, planned changes, and the S3 bucket or buckets movie objects would be restored to, prints no restore was done because --commit was not provided, and exits without writing DynamoDB records or S3 objects. Stderr is enough for preflight failure reporting in the first implementation.

When PLANTTRACER_S3_BUCKET is not set, committed restore falls back to the bucket recorded for each movie object in the archive. Before writing anything, it prints a warning naming the archive bucket or buckets and requires the operator to type yes. To avoid that confirmation, set PLANTTRACER_S3_BUCKET explicitly.

Committed restore uploads movie MP4 objects through a thread pool and prints a status line after each movie object is uploaded. It also writes movie_frames in parallel by movie_id; each movie’s frame rows remain batched together, and restore prints a status line when all frame rows for that movie have been written. --threads controls the S3 upload and frame-write worker count and defaults to 6.

If any restored email address already exists in the target users table, restore blocks before writing data. A future enhancement should allow movies from the backup to be restored under the existing email address with a different user_id. That remapping is out of scope for the first implementation.

Restore does not restore API keys. The send-restore-links command provides a follow-up option to send fresh login emails to restored users. That login-link command defaults to a dry run; it sends email only when --send is present.

Backup records a manifest warning that the backup may not be transactionally consistent if the production app is active during backup. The implementation does not force downtime because current utilization is low.

Regenerating ZIP Artifacts

Restore should provide an option to regenerate ZIP files after movie records and MP4 objects are restored. Regeneration must use the same local/deployed processing path used elsewhere in the project: Lambda through HTTP/SQS or the local Lambda bridge. The restore path should not invent a separate video processing implementation.

When --regenerate-zips is present, restore should enqueue regeneration work, poll until completion, and print status to stderr once per second using \r to overprint the current line. It should emit a real newline every 30 seconds so logs remain readable. ZIP regeneration times out after 5 minutes.

The current implementation accepts --regenerate-zips but fails closed with an operator-facing error because the regeneration queue/polling integration has not been implemented yet.

Course and Movie Migration

Course migration is part of src/dbbackup.py but is logically separate from backup and restore.

Moving a user or course must update every field and index table needed to avoid database inconsistency, including:

  • user course lists and primary course fields where applicable;

  • course_users rows;

  • course admin lists when affected;

  • movie course_id values;

  • S3 object keys and all S3 URNs stored in movie records or frame records.

Plant Tracer S3 keys include course_id/movie_id. Therefore migrating a movie to a new course must move or copy its objects under the new course prefix and rewrite the stored URNs. Leaving movie objects under the old course prefix is not acceptable for the migration command. Course migration should completely rewrite affected records to the new course ID rather than preserving the old course ID as restore metadata.

Security and Privacy

.ptb files are not password protected in the first implementation. Operators must treat them as sensitive student-data archives. The archive README must make this explicit.

Because the backup includes movie files and metadata, it should be stored only where student data is allowed, transmitted only over approved channels, and deleted when it is no longer operationally needed.

Testing Requirements

Backup and restore tests belong under the existing make pytest target. Tests must use DynamoDB Local and MinIO through existing fixtures and Makefile environment, not mocks.

The first substantive integration test:

  • creates one course with one user and one movie;

  • backs up that single course;

  • verifies that restore without --commit performs preflight only;

  • restores with --commit and verifies the restored DynamoDB records and MP4 object match the source backup data;

  • restores or migrates the same data to a different course and verifies that course references and S3 object keys are consistent.

inspect is also covered so operators can view manifest data and counts before running restore.

Open Follow-Ups

The first implementation depends on or is intentionally deferring these related issues:

  • #1081: distinguish missing ZIP artifacts from stale trace data.

  • #1082: add a syslog-style audit log dump command.

  • #1083: allow restore when the target email address already exists.

Documentation

When this feature is implemented, update at least: