Testing, Validation, and Troubleshooting: 5 AIP-C01 Questions with Full Explanations
5 questions, one domain: Testing, Validation, and Troubleshooting from the AWS Generative AI Developer – Professional (AIP-C01) exam. It makes up about 11% of the exam. Pick your answer, then check it against the explanation.
AWS Generative AI Developer – Professional (AIP-C01) · Testing, Validation, and Troubleshooting · 5 questions with answers and explanations
📊 Set profile
Exam: AIP-C01 · 75 questions · 180 minutes · passing score 750
This set: 3 applied, 2 advanced · 3 scenario, 2 troubleshooting
Topics: AWS X-Ray, Amazon API Gateway, AWS Lambda, Amazon Bedrock, Amazon DynamoDB
🧭 Reading Testing, Validation, and Troubleshooting questions
Testing questions ask how to prove a GenAI system works and how to find what broke when it does not. Expect scenarios about scoring open-ended answers, evaluating RAG, catching invented facts and explaining a drop after a change. Correct options measure the specific part in question; wrong ones rely on aggregate numbers, text overlap or guesses about which stage is slow.
Watch for these while you work
• Evaluate RAG in two halves: whether the retriever found relevant context, and whether the generated answer stays faithful to it.
• Measure before tuning. Tracing per stage, or metrics per tool, shows where a slowdown or failure actually starts.
• For open-ended answers with no single reference, a rubric-based model judge works at scale, as long as humans check its scores from time to time.
Question 1 of 5
A GenAI application orchestrates a multi-step pipeline: an API Gateway endpoint invokes a Lambda function that first queries a Bedrock Knowledge Base for retrieval, then calls Bedrock InvokeModel for generation, then writes the result to DynamoDB. Users report that overall response time has crept up over several weeks, but it is unclear which stage of the pipeline is responsible. What is the most effective way to pinpoint the slow stage?
Show answer
✅ Answer: B
Difficulty: advanced · Topic: AWS X-Ray, Amazon API Gateway, AWS Lambda, Amazon Bedrock, Amazon DynamoDB
Distributed tracing
A request that passes through several services has one total latency but many parts. AWS X-Ray records a trace for each request: a segment for each service and subsegments for downstream calls, each with start time, duration and errors.
• Enable tracing on the API Gateway stage and active tracing on the Lambda function.
• Instrument the AWS SDK clients in the function (X-Ray SDK or AWS Distro for OpenTelemetry), so calls to Bedrock Knowledge Bases (Retrieve), Bedrock Runtime (InvokeModel) and DynamoDB (PutItem) each appear as subsegments.
Reading the result
The trace timeline shows, for example, retrieval 180 ms, generation 2.4 s, DynamoDB write 12 ms. Comparing traces from several weeks ago with today's, or using trace analytics and the service map, shows which subsegment grew. Only then do you know whether to tune retrieval, the model call or the database.
Note: the AWS X-Ray SDKs and the X-Ray daemon entered maintenance mode on 2026-02-25 (security fixes only), and AWS recommends OpenTelemetry (AWS Distro for OpenTelemetry) for new instrumentation. X-Ray itself, active tracing and the trace console are unchanged, and exam questions may still name the X-Ray SDK.
Why each option works or fails
• ✗ A. Raise reserved concurrency: concurrency controls how many invocations run at once. It does not make any single stage faster and gives no information about which stage slowed down.
• ✓ B. X-Ray across API Gateway, Lambda and downstream calls (correct): tracing breaks the end-to-end time into per-stage durations, so the stage whose latency crept up is identified directly instead of guessed.
• ✗ C. Assume a model update and switch models: this guesses at the cause; if retrieval or the database is the slow stage, switching models changes quality and cost without fixing latency.
• ✗ D. Reduce DynamoDB read capacity: the pipeline writes results, reading capacity is not the issue, and lowering capacity can only add throttling. It diagnoses nothing.
Question 2 of 5
A team needs to evaluate open-ended answers for helpfulness at scale when no single reference answer exists. Which method fits?
Show answer
✅ Answer: A
Difficulty: applied · Topic: Amazon Bedrock
Evaluating answers with no single right wording
For open-ended tasks (advice, explanations, summaries) many different answers can be good. Metrics that compare text to one reference, such as exact match or word overlap, punish valid rephrasing.
LLM-as-a-judge uses a capable model to grade each answer against a written rubric, for example:
• Helpfulness 1 to 5: does it address the user's actual need?
• Accuracy: are the claims correct and supported?
• Clarity and tone.
Amazon Bedrock Evaluations offers model-as-a-judge jobs with built-in and custom metrics. Because judges can be biased (for example favoring longer answers), teams periodically compare judge scores with human ratings on a sample and refine the rubric when they disagree.
Why each option works or fails
• ✓ A. Rubric-based LLM judge with human review (correct): a judge model can score thousands of open-ended answers consistently against clear criteria, and periodic human review keeps the judge calibrated and trustworthy.
• ✗ B. Prefer the longest answers: length rewards padding and repetition; a concise answer is often more helpful, so this metric can move opposite to quality.
• ✗ C. Count product names: keyword counts measure mentions, not whether the answer solves the user's problem, and they are easy to game.
• ✗ D. Exact string matching: open-ended answers vary in wording, so exact matching marks almost every correct answer as wrong, and there is no single reference answer anyway.
Question 3 of 5
A team building a Retrieval Augmented Generation application on Amazon Bedrock Knowledge Bases wants an evaluation that separately reports whether the retriever returned passages relevant to the user's query and whether the generated answer is actually supported by the retrieved passages. Which pair of RAG evaluation metrics addresses these two concerns respectively?
Show answer
✅ Answer: A
Difficulty: applied · Topic: Amazon Bedrock
RAG has two stages to evaluate
A RAG answer can go wrong in two separate places:
• Retrieval: the retriever returns passages that do not match the question. Metrics such as context relevance (and context coverage) score whether the retrieved passages are relevant to and sufficient for the query.
• Generation: the model writes an answer that goes beyond or contradicts the passages. Faithfulness (groundedness) scores whether every claim in the answer is supported by the retrieved context.
Amazon Bedrock evaluation jobs for Knowledge Bases can report retrieval-only metrics and retrieve-and-generate metrics, including these.
Why separate them
Example: low context relevance with high faithfulness means the model faithfully summarized the wrong passages, so fix chunking or embeddings. High context relevance with low faithfulness means retrieval is fine but the model adds unsupported claims, so fix the prompt or model.
Why each option works or fails
• ✓ A. Context relevance and faithfulness (correct): the first scores whether the retriever returned passages relevant to the query, the second whether the answer is supported by those passages, exactly the two concerns in that order.
• ✗ B. BLEU and token count: BLEU measures word overlap with a reference answer and token count measures length; neither looks at retrieved passages or whether the answer is supported by them.
• ✗ C. Temperature and top-p: these are generation settings chosen before inference, not evaluation metrics that report quality.
• ✗ D. Toxicity and latency: toxicity measures harmful language and latency measures speed; both are valid metrics, but neither evaluates retrieval relevance or grounding.
Question 4 of 5
An agent's task completion rate has dropped from 91 percent to 74 percent after a new action group, a Lambda function wrapping a payment API, was added. Reviewing Bedrock Agents trace output, the team sees the agent frequently selects the correct action group but the Lambda function returns malformed responses that the agent cannot parse, causing it to abandon the task. Which evaluation approach would have surfaced this specific failure before it affected production task completion?
Show answer
✅ Answer: D
Difficulty: advanced · Topic: Amazon Bedrock, AWS Lambda
Where agent failures hide
An Amazon Bedrock agent calls an action group, usually an AWS Lambda function, and expects a response in a specific shape: fields such as messageVersion and a response object that names the action group and the API path or function and carries the response body. If the Lambda returns malformed or unexpected content, the agent cannot use the result, may retry, and often gives up.
Aggregate metrics such as overall task completion only show that something got worse. To find what got worse, break results down per component.
Per-tool metrics
For each action group, record over a test set (and in production):
• success rate: calls that returned a well-formed, usable result
• error rate: exceptions, timeouts, non-success status codes
• unparsable-output rate: responses that did not match the expected schema
Bedrock Agents traces (the orchestration trace shows each invocation input and observation) provide the raw data. Running this per-tool evaluation before release would show the new payment action group failing while the others stay healthy.
Note: Amazon Bedrock Agents was renamed Bedrock Agents Classic and closed to new customers on 2026-07-30; existing customers can keep using it, and AWS points new agent builds to Amazon Bedrock AgentCore. Exam questions may still name Bedrock Agents; per-tool evaluation applies on either platform.
Why each option works or fails
• ✗ A. Final response toxicity score: toxicity measures harmful language in answers. Malformed Lambda output causes abandoned tasks, not toxic text, so the score would stay flat.
• ✗ B. Tokens generated per turn: longer or shorter reasoning is at most an indirect side effect and does not tell you which tool failed or why.
• ✗ C. Increase the session timeout: the agent does not fail because the session expires; it fails because the tool response cannot be parsed. More time only gives it more time to fail, and this is not an evaluation approach at all.
• ✓ D. (correct): Per-action-group success versus error or unparsable-output rates: it isolates each tool, so the new payment action group's malformed responses show up as a spike in unparsable output even while tool selection is correct.
Question 5 of 5
A financial services company built a Bedrock-based application that summarizes quarterly earnings call transcripts. Before launch, the team needs an evaluation framework that specifically catches cases where the summary states a numeric figure (e.g., revenue growth) that does not appear in the source transcript. Which evaluation approach best targets this specific failure mode?
Show answer
✅ Answer: C
Difficulty: applied · Topic: Amazon Bedrock
Faithfulness versus other metrics
A summary has a faithfulness (or groundedness) problem when it states something the source does not support. For financial summaries the most dangerous version is a wrong or invented number, such as "revenue grew 12%" when the transcript says 8%.
How faithfulness is measured
• The evaluator breaks the summary into individual claims.
• It checks each claim, including every figure, against the source transcript.
• Unsupported claims lower the score, and flagged claims can be reviewed.
In Amazon Bedrock, evaluation jobs that use an LLM as a judge offer faithfulness-style metrics, and a team can add a custom metric that specifically checks numbers. A simple programmatic check can also extract figures from the summary and confirm each appears in the transcript.
Why each option works or fails
• ✗ A. Response latency: latency measures speed, not correctness; and factual support can be evaluated programmatically with claim-level checks and LLM-as-a-judge metrics.
• ✗ B. Toxicity metric only: toxicity detects harmful or offensive language, and earnings summaries rarely contain it; it would not notice a wrong revenue figure at all.
• ✓ C. Faithfulness metric against the transcript (correct): it checks every claim, including numeric figures, for support in the source, so an invented or altered number is flagged, which is exactly the failure mode the team must catch.
• ✗ D. BLEU against one reference: BLEU measures word n-gram overlap with a reference text. A summary with the wrong number can still score high, and a correct summary phrased differently can score low.
🎯 What to remember
• Measure before you optimize: distributed tracing shows per-stage latency so you fix the stage that actually slowed down.
• Open-ended quality at scale is scored by a rubric-driven LLM judge checked by humans, not by length, keyword or exact-match metrics.
• Evaluate RAG in two parts: context relevance for the retriever and faithfulness for the generator.
• After adding a tool, measure success and error rates per tool; aggregate metrics show that quality dropped, per-tool metrics show where.
• To catch invented facts or numbers in a summary, measure faithfulness to the source, not overlap, toxicity or speed.
🔗 Keep practicing on this blog
• 5 Questions to Test Your Implementation and Integration Knowledge (AIP-C01)
• 5 Questions to Test Your AI Safety, Security, and Governance Knowledge (AIP-C01)
• AWS Generative AI Developer – Professional (AIP-C01): 5 Practice Questions on Operational Efficiency and Optimization
• All AIP-C01 practice sets
• Every exam on this blog
📚 Keep studying AIP-C01
More AIP-C01 questions: https://hyunhp.github.io/cert-bank/#exam=AIP-C01
Study guide: https://hyunhp.github.io/cert-bank/study.html#exam=AIP-C01
These are original practice questions, not real exam questions, and cert-bank is not affiliated with any certification vendor.
Original practice questions written for this blog, not exam dumps.
Comments
Post a Comment