5 Questions to Test Your Model Selection, Observability, and Performance Knowledge (PLTR-AIE)

5 questions, one domain: Model Selection, Observability, and Performance from the Palantir AI Engineer – Associate (PLTR-AIE) exam. Pick your answer, then check it against the explanation.

Palantir AI Engineer – Associate (PLTR-AIE) · Model Selection, Observability, and Performance · 5 questions with answers and explanations

🧭 What these questions look for

Choosing a model and keeping it healthy in production are the two halves of this domain. Situations include language coverage, context size, rising cost, slower answers and a product owner asking for proof. Correct options tie each choice to a stated requirement and confirm it with evaluation or AIP Observability data, while weaker ones pick the biggest model or guess without evidence.

Three things to look for

• Let the requirement pick the model. If quality must hold for every language or input type, the evaluation suite should be split the same way.
• When a live system drifts with no change on the team's side, traces and telemetry over time show which stage moved, so start there.
• Caching only fits answers that do not depend on the user or context. The cache key needs the content version so stale answers can be cleared.

Question 1 of 5

A workflow needs to process documents in Japanese, Korean and English with equal quality. The reference sheet lists one model with strong multilingual capability at medium cost and another with top English quality but weak support for Asian languages at lower cost. Which should be chosen and how verified?

ABoth models, with the English one used for all documents and the multilingual one used only when the English model's output looks wrong to a reviewer

BThe cheaper English-focused model, since English is the most common language in the corpus and the cost saving is significant over a year

CWhichever model the provider recommends for document processing, since providers understand their own models' strengths better than any customer

DThe multilingual model, verified with a language-stratified evaluation suite so quality per language is measured against the requirement

Show answer

✅ Answer: D

Difficulty: applied · Topic: model catalog, AIP Evals

Capability must match the task's languages
The exam guide asks you to select models from reference characteristics against task requirements. "Equal quality in three languages" is a capability requirement that the cheaper model does not meet.

Verification
• Stratified evaluation: cases per language with per-language metrics.
• Compare each language's score with the threshold; the model must pass all three.
• Cost is compared only among models that meet the requirement.
Example: the multilingual model scores 92/90/91 across the three; the English model scores 94/71/68, failing two.

Why each option works or fails
• ✗ A. Reviewer-triggered fallback: manual, inconsistent, and the weak model still produces first drafts in two languages.
• ✗ B. Cheaper model: fails the Japanese and Korean requirement; cost does not override capability.
• ✗ C. Provider recommendation: generic advice is not an evaluation on this corpus.
• ✓ D. The multilingual model with per-language evaluation (correct): it meets the stated requirement and the verification proves it.

🎯 Key idea: Choose the model that meets every language's requirement, verified with a language-stratified suite.

Question 2 of 5

Users report that an agent's answers have become slower and less accurate over the past week, but nobody changed the agent. Which source of evidence should the engineer consult first?

AA fresh evaluation run of the agent, comparing the score with last month's, which tells whether the degradation is real or imagined

BThe model provider's public status page, since degradation on the provider side is the most common cause of a slow week

CAIP Observability: session traces, execution history, latency and token telemetry over the week, to see which stage changed

DThe agent's system prompt, re-reading it line by line to find instructions that could cause slow or inaccurate answers

Show answer

✅ Answer: C

Difficulty: applied · Topic: AIP Observability

Observability first
The exam guide lists AIP Observability, execution history, session traces, distributed traces, logs, token telemetry and cost signals as the evidence for operating AI systems. "Nobody changed the agent" does not mean nothing changed: the corpus, the Ontology data, the model's version at the provider, or the questions users ask can all shift.

Reading the week
• Latency breakdown per stage: did retrieval get slower (bigger corpus, more chunks) or the model call?
• Token telemetry: longer contexts over time suggest retrieval is returning more or longer chunks.
• Tool calls: more calls per session point to changed tool behaviour or new question types.
• Error and retry rates, and which model version served requests.
Example: traces show retrieved context doubled after a new document set was indexed, raising both latency and noise; the fix is in retrieval, not the prompt.

Why each option works or fails
• ✗ A. Evaluation run: a score confirms degradation on fixed cases but does not say which stage drifted or why; the traces do, and they cover real traffic.
• ✗ B. Provider status page: a possible cause, but the traces will show whether model latency changed; starting from an external page skips the system's own evidence.
• ✓ C. Observability data over the period (correct): it shows which stage changed and when, which is the only way to diagnose a drift that no deliberate change explains.
• ✗ D. Re-read the prompt: the prompt did not change, so reading it cannot explain a change that happened during the week.

🎯 Key idea: When a live AI system drifts, start from traces and telemetry to find the stage that changed.

Question 3 of 5

An agent answers the same 50 frequently asked policy questions thousands of times a day, alongside a long tail of unique questions. Costs are high. Which optimization is appropriate?

ACache answers for recurring context-independent questions, keyed on normalized question and policy version; unique questions still run fully

BCache every answer the agent ever gives and serve the cached one whenever the new question is somewhat similar

CLower the model's temperature so that repeated questions generate identical answers, which the provider bills at a discounted rate for repeated output

DRefuse to answer the frequently asked questions and point users to the policy document

Show answer

✅ Answer: A

Difficulty: applied · Topic: AIP Agent Studio

Caching where it is safe
The exam guide lists caching among optimizations, with the caveat "where safe". Safe caching requires that the answer depends only on the cache key: the same question, the same policy version, and no user-specific context.

Design
• Normalize the question (case, punctuation, canonical phrasing); map to a cached answer keyed with the policy version.
• Invalidate on policy change.
• Exclude questions that depend on the user (their balance, their contract).
• Measure: hit rate, cost saved, and quality of cached answers in the evaluation suite.
Example: 60% of traffic hits the cache; cost falls by more than half with no change to answer quality.

Why each option works or fails
• ✓ A. (correct): Keyed cache for recurring context-independent questions with invalidation: it cuts cost on the repetitive share while keeping correctness and freshness.
• ✗ B. Cache everything by similarity: "somewhat similar" questions can need different answers, and user-specific answers would leak between users.
• ✗ C. Temperature discount: providers do not bill by output determinism; identical answers still cost a model call.
• ✗ D. Refuse FAQs: the agent's main value is removed to save cost.

🎯 Key idea: Cache answers only for context-independent, recurring questions, keyed on question and content version, with invalidation.

Question 4 of 5

Which selection criterion most directly justifies choosing a model with a larger context window for a workflow?

AThe workflow must reason over long inputs that cannot be chunked without losing cross-references, and evaluation confirms the gain

BEngineers prefer not to implement chunking, and a larger window lets them inject whole documents without any extra work

CLarger context windows are always better for every task, so the model with the largest window should be chosen by default

DThe model's name sounds newer and more advanced than the current one, which suggests it will perform better on everything

Show answer

✅ Answer: A

Difficulty: applied · Topic: model catalog

Context window as a requirement
The exam guide lists context limits among selection criteria. A larger window is justified by inputs that genuinely need it, confirmed by evaluation.

Reasoning
• Task: cross-document reasoning where chunking breaks references.
• Evaluation: accuracy gain on long-input cases.
• Cost: longer contexts cost more; accept if the gain warrants.
Example: contract comparison accuracy rises from 71% to 90% with the larger window; the extra cost is accepted for that workflow only.

Why each option works or fails
• ✓ A. (correct): Long-input tasks where chunking loses meaning, validated by evaluation: the direct justification.
• ✗ B. Developer convenience: avoiding chunking work is not a task requirement, and longer contexts cost more on every call.
• ✗ C. Always better: longer contexts cost more and can dilute attention.
• ✗ D. Newer name: a name is not a capability; selection rests on requirements and evaluation.

🎯 Key idea: Choose larger context windows for tasks that need them, confirmed by evaluation, not by default.

Question 5 of 5

A product owner asks for proof that a new AI feature is "working well" after two weeks in production. Which evidence from observability answers this properly?

AVolume, success and failure rates, latency distribution, cost trends, tool-call patterns, user feedback, and the suite score versus baseline

BThe provider's monthly invoice showing increased usage

CA screenshot of one impressive conversation

DThe number of distinct users who opened the feature during the two weeks, since adoption by users is the clearest sign of success

Show answer

✅ Answer: A

Difficulty: applied · Topic: AIP Observability, AIP Evals

Evidence of a working AI system
The exam guide asks engineers to operate AI systems with observability (traces, execution history, token telemetry, cost signals) and feedback, and to monitor real behaviour and regressions. "Working well" is a multi-metric claim.

The evidence set
• Volume and success/failure rates.
• Latency (p50, p95) and trends.
• Cost per request and total, with token telemetry.
• Tool-call patterns and error categories.
• User feedback and overturn rates.
• Evaluation score of the deployed version versus baseline.
Example: 12,000 requests, 1.2% failures, p95 3.1 s, cost within budget, 4% thumbs-down, suite at 91%: a defensible "working well".

Why each option works or fails
• ✓ A. A multi-metric view with a baseline comparison (correct): it substantiates the claim across quality, reliability, cost and user response.
• ✗ B. Invoice: spend shows that the feature was used, not whether the usage went well; a broken feature that retries endlessly also raises the invoice.
• ✗ C. One screenshot: a single good conversation is an anecdote; it says nothing about failure rates, latency, cost or the thousands of other conversations.
• ✗ D. Adoption count: people may keep using a feature that answers badly because they have no alternative; adoption is not quality.

🎯 Key idea: Demonstrate an AI feature's health with observability metrics, feedback and evaluation compared with baseline.

📊 This set at a glance

Exam: PLTR-AIE · 95 questions · 180 minutes
This set: 5 applied · 2 scenario, 1 comparison, 1 trade-off, 1 troubleshooting
Topics: model catalog, AIP Evals, AIP Observability, AIP Agent Studio

🔗 More from this exam

• All PLTR-AIE practice sets
• Every exam on this blog

📚 Beyond this set

More PLTR-AIE questions: https://hyunhp.github.io/cert-bank/#exam=PLTR-AIE
Study guide: https://hyunhp.github.io/cert-bank/study.html#exam=PLTR-AIE

These are original practice questions, not real exam questions, and cert-bank is not affiliated with any certification vendor.

Original practice questions written for this blog, not exam dumps.