Data Ownership

This page summarizes where Plant Tracer stores data, which resource owns each record or object, and which identifier fields connect the stores. The detailed storage references are DynamoDB, S3, Movie attribution and research metadata, Environment Variables, and Plant Tracer Movie Processing.

Ownership Boundary

Plant Tracer has two different persistence lifetimes:

  • S3 bucket: The bucket named by PLANTTRACER_S3_BUCKET is pre-existing and stack-independent. It must outlive CloudFormation/SAM stacks because it is the durable archive for uploaded movies and derived movie artifacts.

  • Artifacts bucket: s3://planttracer-artifacts is shared manual infrastructure for temporary operational artifacts. Objects under artifacts/ expire after 7 days; objects outside that prefix are retained and should be limited to bucket documentation or control files such as README.md.

  • DynamoDB tables: Tables are owned by the deployment stack or by the local DynamoDB test/dev environment. Every table name is prefixed by DYNAMODB_TABLE_PREFIX; local creation normalizes a non-empty prefix to end in -.

Deleting or replacing a stack may remove DynamoDB tables, but it must not delete the S3 archive. Metadata that has to survive beyond the database, such as research-use and attribution choices, is also written into S3 object metadata and the MP4 file as described in MOVIE_METADATA.

Local development uses the same ownership model with DynamoDB Local and MinIO: AWS_REGION=local, AWS_ENDPOINT_URL_DYNAMODB=http://localhost:8000, AWS_ENDPOINT_URL_S3=http://localhost:9000, and the default bucket planttracer-local.

Identifier Names

These are the stable identifier fields used across the application, API, and storage records:

Identifier

Shape

Source

user_id

u + UUID, for example u550e8400-e29b-41d4-a716-446655440000; u0 is the root user

Generated by odb.new_user_id(); validated by odb.is_user_id()

api_key

a + 32 hex characters, for example a550e8400e29b41d4a716446655440000; a123456789012345678901234567890bc is the demo key

Generated by odb.new_api_key(); validated by odb.is_api_key()

course_id

Human/admin supplied string, for example a course code; there is intentionally no new_course_id()

Stored on course, user, enrollment, movie, and log records

course_key

Registration passphrase string

Stored on courses; indexed for self-registration

movie_id

m + UUID, for example m550e8400-e29b-41d4-a716-446655440000

Generated by odb.new_movie_id(); validated by odb.is_movie_id()

frame_number

Zero-based integer for visible movie frames

Sort key in movie_frames; -100 is reserved for the internal marker-map item

marker_id

marker- + first 16 hex characters of SHA-256(label), with -N collision suffix if needed

Stored in frame trackpoints and the per-movie marker map

log_id

String supplied by log writers

Partition key in logs

email

Email address string

User contact field and uniqueness key in unique_emails

DynamoDB Tables

The table list is defined for local development in src/app/odbmaint.py and for deployment in template.yaml. Item shapes are validated where possible by Pydantic models in src/app/schema.py; some operational attributes are added by src/app/odb.py at runtime.

All names below omit the deployment prefix. With DYNAMODB_TABLE_PREFIX=demo-, the movies table is named demo-movies.

Table

Pydantic model

Key fields

Owned data

users

User

user_id

Registered user profile: email, user_name, created, enabled, primary_course_id, primary_course_name, enrolled courses, and admin_for_courses

unique_emails

UniqueEmail

email

Email uniqueness guard used transactionally with users

api_keys

ApiKey plus runtime fields

api_key

Login/API token records: user_id, enabled, use_count, created, first_used_at, and last_used_at

courses

Course

course_id

Course metadata: course_name, course_key, admins_for_course, and max_enrollment

course_users

CourseUser

course_id, user_id

Enrollment join rows connecting users to courses

movies

Movie

movie_id

Movie metadata and artifact pointers, including ownership, visibility, processing state, geometry, capture interval, trim bounds, research metadata, and S3 URNs

movie_frames

MovieFrame and internal marker-map item

movie_id, frame_number

Per-frame trackpoints, optional frame URNs, and the marker map stored at frame_number=-100

logs

LogEntry

log_id

Audit/query log records keyed by log_id and indexed by ipaddr, user_id, and course_id/time_t

Relationship Rules

  • A course can have many users through course_users; enrollment is also denormalized on each user in courses.

  • A user can have many API keys. Login times live on api_keys, not users.

  • A user can upload many movies. Each movie has one owning user_id and one course_id.

  • A course can contain many movies. Course membership determines visibility for many movie list/read paths; movie owner and course admin checks control edits.

  • A movie can have many movie_frames rows. Visible frames use frame_number >= 0.

  • A movie has one internal marker-map companion row in movie_frames at frame_number=-100. It stores markers, marker_labels, and marker_aliases; visible frame trackpoints may store marker_id and resolve the current label through that map.

  • DynamoDB and S3 are connected by URN fields on the movie and frame records, not by bucket scans.

Movies Table Fields

The movies table is the main cross-store record. It is centered on the schema.Movie model, with additional operational fields that are written by processing code and older migrations. Current rows store or use:

  • identity and ownership: movie_id, user_id, user_name, course_id, optional orig_movie

  • display metadata: title, description

  • state and visibility: status, published, deleted, needs_retracing, version

  • timestamps and sizes: created_at, date_uploaded, total_frames, total_bytes

  • playback/analysis metadata: fps, fpm, width, height, trackpoint_origin, rotation, trim_start_frame, trim_end_frame

  • S3 references: movie_data_urn, movie_zipfile_urn, first_frame_urn, and runtime movie_traced_urn

  • processing helpers: processing_state, zip_frame_processing, last_frame_tracked

  • research metadata: research_use, credit_by_name, attribution_name

Some old rows or in-flight code paths may also contain legacy/operational fields such as rotation_steps and status_updated_at. Code that reads movie records must tolerate missing optional fields during upload, processing, and migration.

Movie Frame Fields

Visible movie_frames rows use:

  • movie_id

  • frame_number

  • trackpoints

  • optional frame_urn when a JPEG frame object has been written

Each trackpoint uses the Trackpoint schema: x, y, label, optional marker_id, optional color, optional undeletable, optional frame_number, optional status, and optional err.

The internal marker-map item uses the same table key pattern but reserves frame_number=-100 and stores markers, marker_labels, and marker_aliases instead of visible frame trackpoints.

S3 Objects

S3 object names are deterministic and are converted into URNs of the form s3://{PLANTTRACER_S3_BUCKET}/{object_name}.

The live naming helpers are:

  • movie object template: {course_id}/{movie_id}{ext}

  • frame object template: {course_id}/{movie_id}/{frame_number:06d}{ext}

The current S3 artifacts are:

Artifact

Typical key or URN field

Owner/lifetime

Original uploaded movie

movie_data_urn, usually s3://bucket/{course_id}/{movie_id}.mov

Durable archive; browser uploads directly with a presigned POST

Per-frame JPEG, when persisted

frame_urn, usually s3://bucket/{course_id}/{movie_id}/{frame_number:06d}.jpg

Derived artifact; may be regenerated from the movie

ZIP of analysis frames

movie_zipfile_urn, derived from movie_data_urn with a _zipfile suffix before the extension

Derived artifact written by lambda-resize tracing

Traced movie

movie_traced_urn, derived from movie_data_urn with a _traced suffix before the extension

Derived artifact written by lambda-resize tracing

During upload, /api/new-movie first creates the DynamoDB movie row with status="uploading" and returns a presigned POST for the final S3 key. The browser uploads directly to S3/MinIO. There is no live S3 bucket notification pipeline and no live uploads/ staging-prefix path.

Operational Artifacts Bucket

s3://planttracer-artifacts is a project-wide bucket for temporary operational artifacts such as CloudWatch Synthetics screenshots and other generated stack diagnostics. It is not the movie archive and is not owned by the lambda-only app stack.

The bucket has these intended settings:

  • PlantAdmin AWS account ****0669, region us-east-1

  • public access blocked

  • default SSE-S3 encryption

  • versioning disabled

  • lifecycle rule expiring current objects under artifacts/ after 7 days

  • lifecycle rule aborting incomplete multipart uploads under artifacts/ after 1 day

Application and monitoring stacks should write under tool- and stack-specific prefixes such as artifacts/synthetics/{stack-name}/. The root README.md documents the bucket policy and is retained because it is outside artifacts/. Do not put uploaded movies, derived movie artifacts, DynamoDB backups, release artifacts, or other durable data in this bucket.

Metadata Durability

DynamoDB is the application authority for current movie state and queryable metadata. S3 is the durable byte archive. Because S3 outlives DynamoDB, selected metadata is duplicated:

  • research_use, credit_by_name, and attribution_name are stored on the movie row, in signed S3 object metadata, and in MP4 metadata.

  • fpm is stored on the movie row and included as S3 object metadata at upload; processed MP4 metadata receives a best-effort snapshot when traced.

Later edits may update DynamoDB without rewriting every archived S3 object, so the current database row remains authoritative while embedded/object metadata is the survival copy for archived bytes.

Source Files

Use these files as the implementation sources when updating this page:

  • src/app/schema.py - Pydantic record shapes

  • src/app/odb.py - identifier generation, table access, cross-record logic, marker-map storage, and movie metadata updates

  • src/app/odbmaint.py - local DynamoDB table definitions

  • src/app/s3_presigned.py - S3 URN parsing, key generation, and presigned POST metadata fields

  • lambda-resize/src/resize_app/movie_glue.py - derived ZIP/traced movie writes

  • template.yaml - deployed DynamoDB tables and the stack-independent S3 bucket parameter