SAP-C03 Practice Set: Operational Excellence and Automation

Practice Operational Excellence and Automation for the AWS Solutions Architect – Professional (new) (SAP-C03) exam with the 5 questions below. It makes up about 20% of the exam. Every option is explained, so a wrong pick still teaches something.

AWS Solutions Architect – Professional (new) (SAP-C03) · Operational Excellence and Automation · 5 questions with answers and explanations

🧭 Before you start

Operational excellence here covers classic workloads and AI systems alike. Scenarios show a monitoring gap (restarting containers, agents with no shared view, a model whose labels arrive late) or a release that hurt answer quality, and ask how to see the problem or stop it automatically. Correct answers add the missing signal or an automated gate; a person watching dashboards is rarely the intended fix.

Keep in mind

• Service-level averages can hide what a single task or container is doing. When restarts point to resource pressure, look for the feature that exposes task and container metrics.
• Separate input checks from accuracy checks. Drift in incoming data can be watched right away, while quality against ground truth waits for labels and needs its own schedule.
• Treat generative AI quality like a test result: an evaluation job on a fixed golden dataset can fail a pipeline stage, and required expert sign-off calls for human evaluation with your own work team.

Question 1 of 5

An Amazon ECS service on AWS Fargate runs tasks with three containers each. Tasks are restarting several times a day, and the service's CloudWatch metrics show memory utilization of only 60 percent. The team suspects one container is reaching its memory limit but cannot see per-task or per-container metrics. Which action gives the needed visibility with the least effort?

AUse ECS Exec to open a shell in each running container and run the top command periodically to record memory usage before each restart occurs

BEnable AWS X-Ray tracing in the application code so that traces record container memory usage for each request that the service handles

CEnable CloudWatch Container Insights with enhanced observability for the cluster, and review task and container memory metrics on its dashboards

DTurn on EC2 detailed monitoring for the cluster so that one-minute memory metrics are published for each task and each container in the service

Show answer

✅ Answer: C

Difficulty: applied · Topic: Amazon ECS, AWS Fargate, Amazon CloudWatch

Default ECS metrics
Without extra setup, ECS publishes service-level CPUUtilization and MemoryUtilization to CloudWatch: an average across the service's tasks compared with their reservations. A single container that hits its own limit can be hidden by that average.

Container Insights for ECS
Container Insights adds metrics at cluster, service, task and, with enhanced observability, container level, including memory used versus the limit, CPU and network. It works for Fargate and EC2 launch types and is turned on per cluster or as an account default (containerInsights setting).
The data is stored as performance log events in CloudWatch Logs, so you can also query it with Logs Insights.

Example
The dashboard shows the worker container at 100% of its 512 MiB limit just before each restart while the other two containers sit near 30%, which explains the 60% service average.

Why each option works or fails
• ✗ A. ECS Exec with top: manual sampling misses the moment before a restart, does not scale across tasks, and produces no stored metrics or history.
• ✗ B. X-Ray tracing: traces show request paths and latency between services; they do not record container memory usage.
• ✓ C. Container Insights with enhanced observability (correct): it adds per-task and per-container memory metrics on Fargate with a cluster setting and no code, exposing the container that hits its limit.
• ✗ D. EC2 detailed monitoring: Fargate has no EC2 instances in the customer's account, and EC2 metrics are per instance, not per task or container.

Question 2 of 5

A hospital group plans to use an Amazon Bedrock model to draft patient discharge instructions. Its clinical governance board requires that licensed nurses on staff judge clinical appropriateness and tone of two candidate models before launch, and patient data must not be shown to anyone outside the organization. Which approach meets these requirements?

ARun a programmatic model evaluation job with the toxicity and robustness metrics, and treat the absence of toxic responses as proof of clinical appropriateness and tone

BRun an LLM-as-a-judge evaluation job with a custom metric for clinical appropriateness, and have the judge model's scores signed off by the board in place of any nurse review

CSend the drafts to a public crowd workforce through Amazon Augmented AI, so that many independent raters score clinical appropriateness quickly and at a low cost per item

DRun a human-based model evaluation job in Bedrock that compares the two models, with a private work team of the hospital's own nurses and rating methods such as Likert scales

Show answer

✅ Answer: D

Difficulty: applied · Topic: Amazon Bedrock

When humans must judge
Automated metrics and judge models scale well, but some decisions need people: clinical appropriateness, legal tone, brand voice, or any case where a governance body requires named experts to sign off.

Human-based evaluation in Amazon Bedrock
• You create a human-based model evaluation job for one model or to compare two models on the same prompts.
• You bring your own work team: a private workforce of people you invite (for example the hospital's nurses), so the prompts and outputs are seen only by them.
• You define the metrics (for example "clinical appropriateness", "tone") and a rating method per metric: thumbs up or down, Likert scale for a single response or for comparing two responses, choice buttons or ordinal ranking.
• Results are summarized per model and metric so the board can compare candidates.

Public workforces
The Amazon Mechanical Turk public workforce reached end of support in 2026, and a public crowd would in any case break the rule that patient data stays inside the organization. Bedrock human-based evaluation jobs use your own private work team.

Why each option works or fails
• ✗ A. Toxicity and robustness metrics: the absence of toxic language says nothing about whether discharge instructions are clinically appropriate or well pitched for patients.
• ✗ B. Judge model instead of nurses: an LLM-as-a-judge custom metric can support the review, but the board explicitly requires licensed nurses to make the judgment, so model scores cannot replace them.
• ✗ C. Public crowd through Augmented AI: anonymous crowd workers are not licensed nurses, and sending patient drafts outside the organization breaks the data rule; the Mechanical Turk public workforce is also no longer available.
• ✓ D. Human-based evaluation with a private nurse team (correct): it is the only option where the required experts make the judgment, the two models are compared on the same prompts, and data is shown only to the hospital's own staff.

Question 3 of 5

A payments company serves a fraud model from a SageMaker AI real-time endpoint with data capture enabled. Confirmed fraud labels arrive about 30 days after each transaction. The risk team wants an early warning when incoming transaction features shift away from the training data, and later a measured accuracy figure when labels exist. Which TWO actions meet these requirements? (Select TWO)

ARun SageMaker Clarify bias analysis once on the training data before deployment, because a balanced training set prevents feature drift from occurring in production

BEnable Amazon Bedrock model invocation logging for the endpoint and run a Logs Insights query each week that compares the latest inputs with the training dataset

CCreate a CloudWatch alarm on the endpoint's ModelLatency metric, because feature drift makes the model run more slowly and so latency rises before accuracy falls

DCreate a data quality monitoring schedule from a baseline of the training data, so captured requests are compared with its statistics and violations reach CloudWatch

ECreate a model quality monitoring schedule that merges the captured predictions with the ground-truth labels uploaded to Amazon S3 once the fraud outcomes are confirmed

Show answer

✅ Answer: D, E

Difficulty: advanced · Topic: Amazon SageMaker AI, Amazon CloudWatch

Two kinds of drift
• Data drift: the inputs change (new merchant types, different amounts). It can be detected immediately because it needs only the inputs.
• Model quality drift: the predictions get worse. Measuring it needs the true outcome (ground truth), which here arrives about 30 days later.

SageMaker AI Model Monitor
Model Monitor works on data captured from an endpoint (requests and responses written to S3) and runs on a schedule:
• Data quality monitor: a baseline job computes statistics and constraints (statistics.json, constraints.json) from the training data; each run compares captured inputs against them and reports violations, such as a shifted distribution or missing values.
• Model quality monitor: merges captured predictions with ground-truth labels you upload to S3 and computes metrics such as accuracy, precision, recall or AUC against a baseline.
• Bias drift and feature attribution drift monitors use SageMaker Clarify.
All monitors can publish CloudWatch metrics, so alarms and EventBridge rules can start an investigation or a retraining pipeline.

Why each option works or fails
• ✗ A. One-time Clarify run: a pre-deployment bias check says nothing about how production data changes later; drift has to be monitored continuously.
• ✗ B. Bedrock invocation logging: it records Bedrock model calls and has nothing to do with a SageMaker AI endpoint; a hand-written weekly query would also replace the built-in statistical comparison.
• ✗ C. ModelLatency alarm: latency measures how long the container takes to respond; a fraud model gets no slower when transaction patterns shift, so drift would go unnoticed.
• ✓ D. Data quality monitor (correct): it compares live inputs with the training baseline on every run, giving the early warning without waiting for labels.
• ✓ E. Model quality monitor with delayed labels (correct): once the confirmed outcomes are uploaded, it joins them to the captured predictions and reports the real accuracy against the baseline.

Question 4 of 5

A platform team hosts twelve AI agents built with different open-source frameworks on Amazon Bedrock AgentCore Runtime. Leadership wants one place to compare agents by session latency, token usage and tool-call error rate, and to drill into a slow session to see which model or tool call took the time. Which approach meets these requirements with the least custom code?

AEnable CloudWatch Transaction Search, instrument each agent with AWS Distro for OpenTelemetry, and use AgentCore Observability in CloudWatch to compare agents and inspect traces

BLog the AgentCore API calls with AWS CloudTrail in each account and build an Amazon Athena view that reconstructs every agent session from the API calls made by the agents

CAsk each agent team to write latency and token figures to a shared DynamoDB table at the end of each session, and build a BI dashboard on a nightly export of that table

DAttach SageMaker AI Model Monitor schedules to the agents' runtime endpoints, so that latency, token and tool errors are reported as monitoring violations per agent

Show answer

✅ Answer: A

Difficulty: advanced · Topic: Amazon Bedrock AgentCore, Amazon CloudWatch, AWS Distro for OpenTelemetry

What agent monitoring needs
Model metrics alone do not describe an agent. One agent request is a chain of model calls and tool calls, and the questions operators ask are about the chain:
• How long do sessions take end to end, and which step is slow?
• How many tokens does each agent spend per session?
• How often do tool calls fail or time out?

Amazon Bedrock AgentCore Observability
AgentCore Observability is built on Amazon CloudWatch and OpenTelemetry:
• Agents on AgentCore Runtime emit runtime metrics automatically.
• For traces and spans, enable CloudWatch Transaction Search and instrument the agents with AWS Distro for OpenTelemetry (ADOT); common agent frameworks emit OpenTelemetry spans for model and tool calls, so the framework choice does not matter.
• The CloudWatch generative AI observability pages show agents, sessions, traces and spans with latency, token usage and error data, so you can compare agents and drill from a slow session to the span that caused it.
CloudWatch alarms can then watch these metrics like any others.

Why each option works or fails
• ✓ A. Transaction Search, ADOT and AgentCore Observability (correct): standard instrumentation gives every agent the same metrics and traces in CloudWatch, providing the cross-agent comparison and the drill-down to individual model and tool spans without custom pipelines.
• ✗ B. CloudTrail and Athena: CloudTrail records API calls (who invoked which runtime and when), not the internal steps, timings and token counts of each agent session, so sessions cannot be rebuilt from it.
• ✗ C. Custom DynamoDB reporting: twelve teams writing their own figures creates inconsistent data and custom code everywhere, and end-of-session totals cannot show which step inside a session was slow.
• ✗ D. Model Monitor on agents: Model Monitor analyzes data captured from SageMaker AI endpoints against a baseline; it does not monitor AgentCore Runtime agents or their tool calls.

Question 5 of 5

A company releases prompt-template and model changes for a Bedrock-based support assistant through AWS CodePipeline. Last month a new template went live and reduced answer correctness, which was found only after customer complaints. The company wants the pipeline to stop any release whose quality scores fall below the current production baseline. What should the solutions architect add?

AA post-deployment CloudWatch alarm on InvocationClientErrors and InvocationLatency that rolls the release back when either metric breaches its threshold for an hour

BA stage that runs a Bedrock evaluation job on a fixed golden dataset, reads the scores from the job output in S3, and fails the stage if a score is below the baseline

CAn Amazon Bedrock guardrail with denied topics and content filters on the new version, failing the stage if the guardrail blocks any test prompt during deployment

DA manual approval stage where a product owner chats with the new version in staging for ten minutes and approves the release if the answers look reasonable

Show answer

✅ Answer: B

Difficulty: advanced · Topic: Amazon Bedrock, AWS CodePipeline, AWS CodeBuild

Quality gates for generative AI releases
Traditional pipelines gate on unit tests and error rates. For generative AI, a change to a prompt template, model version or retrieval setting can keep every request "successful" while making the answers worse. The gate must therefore measure answer quality.

Building the gate with Bedrock Evaluations
• Keep a versioned golden dataset (JSONL in S3) of representative prompts with reference answers.
• In the pipeline, a stage (for example AWS CodeBuild, or a Step Functions workflow) calls CreateEvaluationJob for the candidate configuration, typically an LLM-as-a-judge job with metrics such as correctness and faithfulness.
• The stage waits for the job to complete, reads the result files from the S3 output location and compares each metric with the stored production baseline.
• If a score falls below the threshold, the stage fails and the release stops before customers see it. If it passes, the new scores can become the next baseline.
Example: baseline correctness 0.86, candidate 0.79, threshold "no drop greater than 0.02": the pipeline stops and links the evaluation report in the failure message.

Why each option works or fails
• ✗ A. Error and latency alarms: a worse template still returns valid, fast responses, so client errors and latency stay normal while quality falls, and the check runs only after customers are affected.
• ✓ B. Automated evaluation stage (correct): it scores every candidate release on the same golden dataset with the same metrics and blocks regressions automatically, catching the kind of correctness drop that slipped through last month.
• ✗ C. Guardrail as a gate: guardrails filter harmful or off-topic content at runtime; they do not score correctness, so a less accurate template passes untouched.
• ✗ D. Ten-minute manual chat: a few hand-picked conversations are not a repeatable measurement and would likely miss a drop in correctness across many question types.

🎯 Takeaways

• ECS service metrics are averages; enable Container Insights (enhanced observability) to see task and container level resource use.
• When governance requires domain experts to judge outputs, use a Bedrock human-based evaluation job with your own private work team.
• Watch inputs now with a Model Monitor data quality schedule and accuracy later with a model quality schedule that merges delayed ground-truth labels.
• Monitor agent performance with AgentCore Observability: Transaction Search plus OpenTelemetry (ADOT) instrumentation gives per-agent metrics and step-level traces in CloudWatch.
• Gate generative AI releases on quality: run a Bedrock evaluation job against a golden dataset in the pipeline and fail the stage when scores fall below the baseline.

📊 Set profile

Exam: SAP-C03 · 75 questions · 180 minutes · passing score 720
This set: 3 advanced, 2 applied · 4 scenario, 1 troubleshooting · 1 with more than one answer
Topics: Amazon ECS, AWS Fargate, Amazon CloudWatch, Amazon Bedrock, Amazon SageMaker AI, Amazon Bedrock AgentCore, AWS Distro for OpenTelemetry, AWS CodePipeline, AWS CodeBuild

🔗 Related practice sets

• AWS Solutions Architect – Professional (SAP-C03): 5 Practice Questions on Security, Compliance, and Governance
• 5 Questions to Test Your Cost-Optimized Architecture Design Knowledge (SAP-C03)
• Can You Answer These 5 SAP-C03 Questions on Resilience, Migration, and Business Continuity?
• All SAP-C03 practice sets
• Every exam on this blog

📚 Keep studying SAP-C03

More SAP-C03 questions: https://hyunhp.github.io/cert-bank/#exam=SAP-C03
Study guide: https://hyunhp.github.io/cert-bank/study.html#exam=SAP-C03

These are original practice questions, not real exam questions, and cert-bank is not affiliated with any certification vendor.

Original practice questions written for this blog, not exam dumps.