AWS Video Analysis Pipeline Architecture: A Practical Guide to Object Tracking and Long-Term Memory

From One-Shot Analysis to Longitudinal Intelligence: Why Long-Term Memory Matters

Among the many AI video applications out there, the vast majority of systems remain stuck at the "one-shot analysis" stage — upload a video, get back a batch of results, and that's it. But the truly valuable use cases often demand "longitudinal memory": the system must not only understand the current video but also remember the user's past behaviors, habits, and observation data, and provide more targeted feedback accordingly.

This concept is borrowed from longitudinal studies in medicine and social science — repeatedly collecting data on the same subject at different points in time to track trends over time. In AI systems, this means the model is no longer stateless — it needs to maintain a persistent knowledge base tied to a specific user or entity, accumulating context across multiple interactions. This is fundamentally different from the conversation memory mechanisms in mainstream LLMs: conversation memory is typically confined to the context window of a single session, whereas longitudinal memory requires spanning days, weeks, or even months, supported by structured storage and retrieval mechanisms.

Recently, a developer shared on Reddit the requirements for a system they were prototyping, sparking considerable discussion about architecture design. The core scenario is: users upload 2–3 minute short videos, and the system needs to let users identify and select themselves in the video, continuously track that person through occlusion and multi-person interference, feed relevant frames into a Vision Language Model (VLM) for analysis, store structured observation results, and retrieve historical data in future uploads to provide feedback combining old and new material.

Vision Language Models (VLMs) are a core branch of multimodal AI that fuse computer vision with natural language processing in a unified model architecture. Notable examples include OpenAI's GPT-4o, Google's Gemini, and Anthropic's Claude. These models use a visual encoder (such as ViT) to convert images into token sequences, which are then fed into a Transformer alongside text tokens for joint reasoning. The key capability of VLMs is that they can not only "see" image content but also describe, reason about, and answer questions regarding visual content in natural language. In video analysis scenarios, VLMs typically receive keyframes extracted from the video rather than the full video stream — because the computational cost and API call expenses of frame-by-frame processing would skyrocket.

This set of requirements bundles two typically independent technical domains together: video understanding and object tracking and long-term AI memory. The following is a pragmatic technical breakdown centered around the architecture proposed by the author.

Interpreting the Original Architecture: The Complete Data Flow from S3 to Analysis

The data pipeline proposed by the author is as follows:

S3 → SQS → GPU/Video Processing Worker → Object Tracking → VLM → PostgreSQL/pgvector → Memory Retrieval → Analysis

This is a typical asynchronous event-driven architecture — logically clean and well-suited for an MVP.

• S3: Landing storage for video files, naturally suited for large file uploads and lifecycle management.
• SQS: Acts as a decoupling layer, separating upload events from time-consuming GPU processing tasks and preventing the upload interface from being blocked by long-running jobs.
• GPU Worker: Handles compute-intensive tasks such as video decoding, frame extraction, and object tracking.
• VLM: Performs semantic understanding on tracked target frames, outputting structured observations.
• pgvector: Stores observation results as vectors for subsequent semantic retrieval.

SQS plays a critical buffering and decoupling role in this architecture. Amazon SQS (Simple Queue Service) is a fully managed message queue service provided by AWS. The "decoupling" means the upload service and GPU processing service don't call each other directly — they communicate indirectly through the message queue. This delivers three core benefits: First, the upload interface can return a response in milliseconds, so users don't have to wait several minutes for video processing to complete. Second, when upload volume suddenly spikes, messages queue up and GPU Workers consume them at their own processing capacity without crashing from overload. Third, if a message fails to process, SQS provides built-in retry mechanisms and Dead Letter Queues, ensuring tasks aren't silently lost. This pattern is a classic practice of Event-Driven Architecture.

The author deliberately emphasized "keeping the MVP simple rather than building a massive ML platform from the start" — and this instinct is correct. Premature platform-level investment often slows down validation of core hypotheses.

Compute Layer Selection: ECS/EC2, SageMaker, or Lambda

The technology choice for the video CV processing layer is the critical decision point of the entire architecture. Based on the task characteristics, here's how to approach it.

Why Lambda Is Not Suitable for Video Processing

Lambda has a 15-minute execution time limit, limited ephemeral disk space (up to 10GB), and no GPU support. For tasks that require decoding several minutes of video and running tracking models frame by frame, Lambda is fundamentally mismatched. It's better suited for lightweight glue logic such as triggering orchestration, generating pre-signed URLs, and writing metadata.

ECS on EC2 (GPU Instances) Is the Solid MVP Choice

For the MVP stage, the recommendation is ECS + EC2 GPU instances (such as the g4dn/g5 series) pulling tasks from SQS. The benefits of this approach are:

• Full control over the runtime environment (CUDA, model dependencies, tracking libraries)
• Auto-scaling based on queue depth, scaling to zero when idle to save costs
• A gentler learning curve compared to SageMaker, without excessive abstraction

When ECS (Elastic Container Service) is paired with EC2 GPU instances, Application Auto Scaling policies can enable automatic scaling based on SQS queue depth. The specific mechanism works as follows: CloudWatch monitors the number of visible messages in the SQS queue (the ApproximateNumberOfMessagesVisible metric). When the backlog exceeds a threshold, it triggers Scale Out, launching new GPU instances to join the ECS cluster. When the queue is empty and instances have been idle beyond a certain duration, it triggers Scale In, terminating excess instances. Scale-to-zero is critical for cost control — GPU instances (e.g., g5.xlarge at roughly $1/hour) continue to incur charges even when idle. For an MVP with unpredictable request volume, idle costs could account for the majority of total GPU spending. It's worth noting that GPU instance cold start times typically take 3–5 minutes (including instance boot and Docker image pull), which means the first request after scaling to zero will have noticeable latency. This requires a trade-off between cost and response time.

When SageMaker Makes Sense

SageMaker is better suited for stages where you've already moved into large-scale training/inference and need managed endpoints and model version management. Introducing it during the MVP phase often brings unnecessary complexity and abstraction overhead. The author's judgment of "not wanting to build a big platform right away" aligns perfectly with this.

Object Tracking: The Most Underestimated Stage in the Pipeline

The author is particularly concerned about "continuously tracking the same person through occlusion and multi-person scenarios" — and this is actually the most technically challenging part of the entire system, far trickier than calling a VLM.

Recommended Tech Stack

Mainstream Multi-Object Tracking (MOT) approaches to consider:

• YOLO + ByteTrack / BoT-SORT: A classic detection + tracking combination with a mature community and open-source availability.
• ReID (Re-Identification) models: This is the key to handling occlusion and multi-person interference. When a target briefly disappears and reappears, pure motion-based tracking IDs tend to break, whereas appearance-based ReID can re-associate the same identity.

ByteTrack and BoT-SORT are two mainstream algorithms in the current multi-object tracking field, both belonging to the tracking-by-detection paradigm — first using a detector (like YOLO) to detect all targets frame by frame, then using association algorithms to link cross-frame detections into continuous trajectories. ByteTrack's core innovation is that it doesn't discard low-confidence detections but instead fully leverages these "weak signals" through a two-stage association strategy, significantly improving tracking continuity in crowded scenes. BoT-SORT builds on this by incorporating Camera Motion Compensation (CMC) and appearance feature matching, achieving a more refined fusion of motion and appearance models. Both achieve leading performance on public benchmarks like MOT17 and have open-source implementations ready for direct integration into production pipelines.

Person Re-Identification (ReID) is a dedicated research area in computer vision aimed at solving the problem of "identifying the same person across cameras or time intervals." The core technique is training a feature extraction network (typically based on ResNet or Vision Transformer) that maps person images into compact appearance embedding vectors, such that the same person's embeddings remain close across different poses, lighting conditions, and occlusions, while different people's vectors remain far apart. ReID faces several core challenges in practice: clothing changes (features shift dramatically when the same person changes outfits), severe occlusion (only partial body visible), and viewpoint variation (significant appearance differences between front and back views). In the scenario discussed here, the quality of the appearance embedding extracted when the user selects themselves in the first frame directly determines the accuracy of subsequent tracking and cross-session recognition.

The Engineering Implications of Letting Users "Select Themselves"

When a user manually draws a bounding box around themselves in the first frame, they are essentially giving the system an initial anchor point. The system then needs to extract an appearance embedding of that person and continuously match it across subsequent frames. This embedding vector itself can also become part of the cross-session memory — in future uploads, the system can automatically identify the user without requiring re-selection.

Long-Term Memory Component: A Differentiating Architecture Powered by pgvector

What truly sets this system apart from ordinary video analysis tools is the long-term memory design. The author's choice of PostgreSQL + pgvector is a reasonable starting point.

Layered Storage Structure

It's recommended to split memory into two layers:

• Structured observation layer: Each VLM analysis result stored in JSON/relational table format (timestamps, scene labels, quantitative metrics, etc.), enabling precise queries and aggregate statistics.
• Vector semantic layer: Embed the text descriptions or frame features of observations and store them in pgvector for "retrieving historically similar behavioral segments."

pgvector is an open-source extension for PostgreSQL that adds vector data types and similarity search capabilities to the relational database. It supports storing high-dimensional floating-point vectors (typically 768-dimensional or 1536-dimensional embeddings) and implements Approximate Nearest Neighbor (ANN) search through IVFFlat or HNSW indexes. The core principle of vector retrieval is: first map unstructured data like text and images into points in a high-dimensional vector space using embedding models (such as OpenAI's text-embedding-3-small), where semantically similar content is closer in vector space. At query time, the query is also mapped to a vector, and the closest historical records are found via cosine similarity or Euclidean distance. The biggest advantage of pgvector is that it eliminates the need for a separate vector database — developers can perform both relational queries and semantic retrieval within the same PostgreSQL instance, significantly reducing architectural complexity.

Memory Retrieval Implementation Approach

When a new video arrives, the system first generates a vector for the current observation, performs a similarity search in pgvector to retrieve the user's relevant historical observations, and feeds both new and historical material into the VLM or feedback generation logic. This achieves "personalized feedback based on historical data."

Interestingly, there's no need to introduce a standalone vector database (such as Pinecone or Weaviate) at the MVP stage. pgvector can handle small-to-medium-scale requirements while eliminating the extra data synchronization and operational overhead. Migration can wait until data volume truly scales up.

Open-Source Tools and Reference Resources Worth Exploring

In response to the author's search for reference implementations, the following directions are worth investigating:

• Tracking layer: The Ultralytics ecosystem (YOLO + built-in trackers), the official ByteTrack repository, and mmtracking.
• VLM analysis: Start with managed APIs (such as the vision capabilities of GPT-4o, Claude, and Gemini) for quick validation, avoiding the operational burden of self-hosting large models.
• Architecture references: AWS official "Video on Demand" and media analysis samples, as well as reference architectures for asynchronous batch processing based on SQS + ECS.

Architecture Assessment: Risk Points and Optimization Recommendations

Overall, the author's architectural direction has no obvious flaws and is reasonable as an MVP. However, several points are worth flagging:

1. The difficulty of object tracking is significantly underestimated. It is likely the biggest technical risk in the entire project. Prioritize technical validation here.
2. VLM call costs and latency. Sending every frame to a VLM is prohibitively expensive. Use tracking to filter keyframes first, then selectively analyze. In a video analysis pipeline, deciding which frames are "worth sending to the VLM" is a critical engineering decision. A 2-minute video at 30fps contains 3,600 frames. If every frame were sent to a VLM (using GPT-4o as an example, each image costs roughly $0.01–$0.05), API costs alone could reach $36–$180 — clearly unacceptable. Common keyframe selection strategies include: fixed-interval sampling (e.g., 1 frame per second), scene change detection (identifying shot transitions via inter-frame differences or histogram changes), and tracking confidence changes (sampling when the tracked target's pose or action changes significantly). A smarter approach combines object tracking results to extract frames only when the target person performs key actions or undergoes state changes — controlling costs while ensuring the VLM receives the highest information-density input.
3. Don't over-optimize the memory layer prematurely. pgvector is sufficient for the MVP — no need to introduce a dedicated vector database.
4. Idle costs for GPU resources. Make sure to configure auto-scaling based on queue depth to avoid GPU instances idling and burning money.

All in all, this "S3 → SQS → GPU Worker → Tracking → VLM → pgvector → Retrieval → Analysis" pipeline is a pragmatic and actionable starting point. The real challenge lies not in the architecture diagram itself, but in the robustness of object tracking and the relevance of long-term memory retrieval — these two aspects are what ultimately determine the product experience.