# Data Ownership

This page summarizes where Plant Tracer stores data, which resource owns each
record or object, and which identifier fields connect the stores. The detailed
storage references are [DynamoDB](DynamoDB.rst), [S3](S3.rst),
[Movie attribution and research metadata](MOVIE_METADATA.rst),
[Environment Variables](EnvironmentVariables.rst), and
[Plant Tracer Movie Processing](ProcessingPhases.rst).

## Ownership Boundary

Plant Tracer has two different persistence lifetimes:

- **S3 bucket:** The bucket named by `PLANTTRACER_S3_BUCKET` is pre-existing and
  stack-independent. It must outlive CloudFormation/SAM stacks because it is the
  durable archive for uploaded movies and derived movie artifacts.
- **Artifacts bucket:** `s3://planttracer-artifacts` is shared manual
  infrastructure for temporary operational artifacts. Objects under
  `artifacts/` expire after 7 days; objects outside that prefix are retained and
  should be limited to bucket documentation or control files such as
  `README.md`.
- **DynamoDB tables:** Tables are owned by the deployment stack or by the local
  DynamoDB test/dev environment. Every table name is prefixed by
  `DYNAMODB_TABLE_PREFIX`; local creation normalizes a non-empty prefix to end
  in `-`.

Deleting or replacing a stack may remove DynamoDB tables, but it must not delete
the S3 archive. Metadata that has to survive beyond the database, such as
research-use and attribution choices, is also written into S3 object metadata
and the MP4 file as described in [MOVIE_METADATA](MOVIE_METADATA.rst).

Local development uses the same ownership model with DynamoDB Local and MinIO:
`AWS_REGION=local`, `AWS_ENDPOINT_URL_DYNAMODB=http://localhost:8000`,
`AWS_ENDPOINT_URL_S3=http://localhost:9000`, and the default bucket
`planttracer-local`.

## Identifier Names

These are the stable identifier fields used across the application, API, and
storage records:

| Identifier | Shape | Source |
| --- | --- | --- |
| `user_id` | `u` + UUID, for example `u550e8400-e29b-41d4-a716-446655440000`; `u0` is the root user | Generated by `odb.new_user_id()`; validated by `odb.is_user_id()` |
| `api_key` | `a` + 32 hex characters, for example `a550e8400e29b41d4a716446655440000`; `a123456789012345678901234567890bc` is the demo key | Generated by `odb.new_api_key()`; validated by `odb.is_api_key()` |
| `course_id` | Human/admin supplied string, for example a course code; there is intentionally no `new_course_id()` | Stored on course, user, enrollment, movie, and log records |
| `course_key` | Registration passphrase string | Stored on `courses`; indexed for self-registration |
| `movie_id` | `m` + UUID, for example `m550e8400-e29b-41d4-a716-446655440000` | Generated by `odb.new_movie_id()`; validated by `odb.is_movie_id()` |
| `frame_number` | Zero-based integer for visible movie frames | Sort key in `movie_frames`; `-100` is reserved for the internal marker-map item |
| `marker_id` | `marker-` + first 16 hex characters of SHA-256(label), with `-N` collision suffix if needed | Stored in frame trackpoints and the per-movie marker map |
| `log_id` | String supplied by log writers | Partition key in `logs` |
| `email` | Email address string | User contact field and uniqueness key in `unique_emails` |

## DynamoDB Tables

The table list is defined for local development in `src/app/odbmaint.py` and for
deployment in `template.yaml`. Item shapes are validated where possible by
Pydantic models in `src/app/schema.py`; some operational attributes are added by
`src/app/odb.py` at runtime.

All names below omit the deployment prefix. With `DYNAMODB_TABLE_PREFIX=demo-`,
the `movies` table is named `demo-movies`.

| Table | Pydantic model | Key fields | Owned data |
| --- | --- | --- | --- |
| `users` | `User` | `user_id` | Registered user profile: `email`, `user_name`, `created`, `enabled`, `primary_course_id`, `primary_course_name`, enrolled `courses`, and `admin_for_courses` |
| `unique_emails` | `UniqueEmail` | `email` | Email uniqueness guard used transactionally with `users` |
| `api_keys` | `ApiKey` plus runtime fields | `api_key` | Login/API token records: `user_id`, `enabled`, `use_count`, `created`, `first_used_at`, and `last_used_at` |
| `courses` | `Course` | `course_id` | Course metadata: `course_name`, `course_key`, `admins_for_course`, and `max_enrollment` |
| `course_users` | `CourseUser` | `course_id`, `user_id` | Enrollment join rows connecting users to courses |
| `movies` | `Movie` | `movie_id` | Movie metadata and artifact pointers, including ownership, visibility, processing state, geometry, capture interval, trim bounds, research metadata, and S3 URNs |
| `movie_frames` | `MovieFrame` and internal marker-map item | `movie_id`, `frame_number` | Per-frame trackpoints, optional frame URNs, and the marker map stored at `frame_number=-100` |
| `logs` | `LogEntry` | `log_id` | Audit/query log records keyed by `log_id` and indexed by `ipaddr`, `user_id`, and `course_id`/`time_t` |

### Relationship Rules

- A course can have many users through `course_users`; enrollment is also
  denormalized on each user in `courses`.
- A user can have many API keys. Login times live on `api_keys`, not `users`.
- A user can upload many movies. Each movie has one owning `user_id` and one
  `course_id`.
- A course can contain many movies. Course membership determines visibility for
  many movie list/read paths; movie owner and course admin checks control edits.
- A movie can have many `movie_frames` rows. Visible frames use
  `frame_number >= 0`.
- A movie has one internal marker-map companion row in `movie_frames` at
  `frame_number=-100`. It stores `markers`, `marker_labels`, and
  `marker_aliases`; visible frame trackpoints may store `marker_id` and resolve
  the current label through that map.
- DynamoDB and S3 are connected by URN fields on the movie and frame records,
  not by bucket scans.

## Movies Table Fields

The `movies` table is the main cross-store record. It is centered on the
`schema.Movie` model, with additional operational fields that are written by
processing code and older migrations. Current rows store or use:

- identity and ownership: `movie_id`, `user_id`, `user_name`, `course_id`,
  optional `orig_movie`
- display metadata: `title`, `description`
- state and visibility: `status`, `published`, `deleted`, `needs_retracing`,
  `version`
- timestamps and sizes: `created_at`, `date_uploaded`, `total_frames`,
  `total_bytes`
- playback/analysis metadata: `fps`, `fpm`, `width`, `height`,
  `trackpoint_origin`, `rotation`, `trim_start_frame`, `trim_end_frame`
- S3 references: `movie_data_urn`, `movie_zipfile_urn`, `first_frame_urn`, and
  runtime `movie_traced_urn`
- processing helpers: `processing_state`, `zip_frame_processing`,
  `last_frame_tracked`
- research metadata: `research_use`, `credit_by_name`, `attribution_name`

Some old rows or in-flight code paths may also contain legacy/operational fields
such as `rotation_steps` and `status_updated_at`. Code that reads movie records
must tolerate missing optional fields during upload, processing, and migration.

## Movie Frame Fields

Visible `movie_frames` rows use:

- `movie_id`
- `frame_number`
- `trackpoints`
- optional `frame_urn` when a JPEG frame object has been written

Each trackpoint uses the `Trackpoint` schema: `x`, `y`, `label`, optional
`marker_id`, optional `color`, optional `undeletable`, optional `frame_number`,
optional `status`, and optional `err`.

The internal marker-map item uses the same table key pattern but reserves
`frame_number=-100` and stores `markers`, `marker_labels`, and `marker_aliases`
instead of visible frame trackpoints.

## S3 Objects

S3 object names are deterministic and are converted into URNs of the form
`s3://{PLANTTRACER_S3_BUCKET}/{object_name}`.

The live naming helpers are:

- movie object template: `{course_id}/{movie_id}{ext}`
- frame object template: `{course_id}/{movie_id}/{frame_number:06d}{ext}`

The current S3 artifacts are:

| Artifact | Typical key or URN field | Owner/lifetime |
| --- | --- | --- |
| Original uploaded movie | `movie_data_urn`, usually `s3://bucket/{course_id}/{movie_id}.mov` | Durable archive; browser uploads directly with a presigned POST |
| Per-frame JPEG, when persisted | `frame_urn`, usually `s3://bucket/{course_id}/{movie_id}/{frame_number:06d}.jpg` | Derived artifact; may be regenerated from the movie |
| ZIP of analysis frames | `movie_zipfile_urn`, derived from `movie_data_urn` with a `_zipfile` suffix before the extension | Derived artifact written by lambda-resize tracing |
| Traced movie | `movie_traced_urn`, derived from `movie_data_urn` with a `_traced` suffix before the extension | Derived artifact written by lambda-resize tracing |

During upload, `/api/new-movie` first creates the DynamoDB movie row with
`status="uploading"` and returns a presigned POST for the final S3 key. The
browser uploads directly to S3/MinIO. There is no live S3 bucket notification
pipeline and no live `uploads/` staging-prefix path.

## Operational Artifacts Bucket

`s3://planttracer-artifacts` is a project-wide bucket for temporary operational
artifacts such as CloudWatch Synthetics screenshots and other generated stack
diagnostics. It is not the movie archive and is not owned by the lambda-only app
stack.

The bucket has these intended settings:

- PlantAdmin AWS account `****0669`, region `us-east-1`
- public access blocked
- default SSE-S3 encryption
- versioning disabled
- lifecycle rule expiring current objects under `artifacts/` after 7 days
- lifecycle rule aborting incomplete multipart uploads under `artifacts/` after
  1 day

Application and monitoring stacks should write under tool- and stack-specific
prefixes such as `artifacts/synthetics/{stack-name}/`. The root `README.md`
documents the bucket policy and is retained because it is outside `artifacts/`.
Do not put uploaded movies, derived movie artifacts, DynamoDB backups, release
artifacts, or other durable data in this bucket.

## Metadata Durability

DynamoDB is the application authority for current movie state and queryable
metadata. S3 is the durable byte archive. Because S3 outlives DynamoDB, selected
metadata is duplicated:

- `research_use`, `credit_by_name`, and `attribution_name` are stored on the
  movie row, in signed S3 object metadata, and in MP4 metadata.
- `fpm` is stored on the movie row and included as S3 object metadata at upload;
  processed MP4 metadata receives a best-effort snapshot when traced.

Later edits may update DynamoDB without rewriting every archived S3 object, so
the current database row remains authoritative while embedded/object metadata is
the survival copy for archived bytes.

## Source Files

Use these files as the implementation sources when updating this page:

- `src/app/schema.py` - Pydantic record shapes
- `src/app/odb.py` - identifier generation, table access, cross-record logic,
  marker-map storage, and movie metadata updates
- `src/app/odbmaint.py` - local DynamoDB table definitions
- `src/app/s3_presigned.py` - S3 URN parsing, key generation, and presigned POST
  metadata fields
- `lambda-resize/src/resize_app/movie_glue.py` - derived ZIP/traced movie writes
- `template.yaml` - deployed DynamoDB tables and the stack-independent S3 bucket
  parameter