Module 17: Evaluation, Security, Governance, Ethics, and Professional Responsibility

Learning Objectives

  • Design evaluation programs for engineering AI systems using model testing, red teaming, user testing, and deterministic engineering checks.
  • Build representative evaluation datasets with known-good, edge, failure, adversarial, and unauthorized-action cases.
  • Defend agentic workflows against prompt injection, excessive agency, tool poisoning, privilege escalation, data exfiltration, and memory poisoning.
  • Apply privacy, governance, reproducibility, incident-response, and human-approval controls proportional to consequence.
  • Preserve licensed professional responsibility and applicable Philippine legal, regulatory, and project obligations.

Hallucination

A hallucination is an AI-generated statement, citation, calculation, interpretation, or artifact that is unsupported, fabricated, or inconsistent with the available evidence even though it may be expressed confidently and fluently.

Automation Bias

Automation bias is the tendency to over-rely on an automated recommendation or output, especially when the system appears authoritative, efficient, or technically sophisticated.

Prompt Injection

Prompt injection is an attempt embedded in user input, documents, webpages, files, or connected data to manipulate an AI system into following unintended instructions, disclosing information, or using tools in ways that conflict with the authorized task.

AI Incident

An AI incident is an event in which an AI-assisted workflow produces, propagates, or nearly propagates an erroneous, unsafe, unauthorized, misleading, privacy-compromising, or otherwise unacceptable engineering outcome.

Engineering AI Risk Categories

Civil engineering AI risk includes:

  • technical risk: incorrect calculations, assumptions, models, transformations, or interpretations;
  • source risk: fabricated citations, stale editions, inapplicable standards, or unverified requirements;
  • data risk: poor quality, bias, missing records, wrong units, bad provenance, or unrepresentative samples;
  • privacy risk: disclosure of personal, commercial, or confidential project information;
  • security risk: malicious files, prompt injection, excessive permissions, credential exposure, or unauthorized tool use;
  • governance risk: unclear ownership, review, retention, change control, or approval responsibility;
  • professional risk: allowing AI output to substitute for required competent engineering judgment, inspection, certification, or accountability.
Engineering AI Risk-Management Workflow

A lifecycle for trustworthy AI-assisted civil engineering work.

Risk management is a lifecycle: governance and context come first, then measurement, controls, incident response, and improvement.

Engineering AI Risk-Management WorkflowA lifecycle for trustworthy AI-assisted civil engineering work.. Govern: define policy, roles, and permitted use → Map engineering context, data, consequence, and stakeholders; Map engineering context, data, consequence, and stakeholders → Measure performance, errors, bias, privacy, and security risk; Measure performance, errors, bias, privacy, and security risk → Apply controls, permissions, verification, and approval gates; Apply controls, permissions, verification, and approval gates → Failure, near miss, or material change?; Failure, near miss, or material change? — No → Continue monitored use; Failure, near miss, or material change? — Yes → Contain, investigate, correct, and retest; Contain, investigate, correct, and retest → Map engineering context, data, consequence, and stakeholders

Govern: define policy, roles, and permitted use → Map engineering context, data, consequence, and stakeholders; Map engineering context, data, consequence, and stakeholders → Measure performance, errors, bias, privacy, and security risk; Measure performance, errors, bias, privacy, and security risk → Apply controls, permissions, verification, and approval gates; Apply controls, permissions, verification, and approval gates → Failure, near miss, or material change?; Failure, near miss, or material change? — No → Continue monitored use; Failure, near miss, or material change? — Yes → Contain, investigate, correct, and retest; Contain, investigate, correct, and retest → Map engineering context, data, consequence, and stakeholders

  • Govern: define policy, roles, and permitted use: terminator
  • Map engineering context, data, consequence, and stakeholders: process
  • Measure performance, errors, bias, privacy, and security risk: process
  • Apply controls, permissions, verification, and approval gates: process
  • Failure, near miss, or material change?: decision
  • Contain, investigate, correct, and retest: process
  • Continue monitored use: terminator

NIST Risk-Management Perspective

The NIST AI Risk Management Framework organizes AI risk management around Govern, Map, Measure, and Manage. The Generative AI Profile extends that perspective to risks intensified by generative systems, such as confabulation, information integrity, privacy, security, and misuse.

For civil engineering organizations, the key lesson is to manage AI as a lifecycle system: define governance before use, map the engineering context and consequences, measure performance and failure modes, and manage risk through controls, monitoring, escalation, and continuous improvement.

ARIA Engineering Evaluation Matrix

Production assurance combines technical model tests, adversarial testing, real-user evidence, and deterministic engineering checks.

Engineering AI risk spans more than hallucination; organizations should review technical, source, data, privacy, security, governance, and professional dimensions.

Model testing
Red teaming
User testing
Deterministic engineering checks
Release criteria
Regression evaluation

Mandatory AI Output Verification Pattern

Before this output is accepted:
1. Identify all factual claims that affect the engineering conclusion.
2. Identify every assumption, missing input, and source dependency.
3. Recompute consequential numerical results independently.
4. Verify standards, clauses, and citations from authoritative sources.
5. Check units, signs, datums, boundary conditions, and governing cases.
6. Compare the result with known physical behavior, field evidence, or historical data.
7. Record unresolved uncertainty and evidence gaps.
8. Confirm that tool permissions and data handling were appropriate.
9. Require qualified human approval for consequential decisions.
Agent Security Trust-Boundary Stack

Untrusted content must not acquire higher privileges as it moves through an agentic workflow.

Verification controls should increase with consequence and permission scope, especially when AI output can trigger real-world actions.

1
Untrusted users, files, webpages, and tool output
2
Validated model context and bounded memory
3
Schema-constrained tools and scoped credentials
4
Approval gates for privileged or irreversible actions
5
Retained security and engineering evidence

AI Does Not Replace Professional Engineering Responsibility

AI may assist a licensed or otherwise qualified professional, but it does not possess professional licensure, duty of care, accountability, site knowledge, or legal authority to assume the responsible engineer's obligations. AI-generated calculations, drawings, reports, or recommendations remain decision-support artifacts until appropriately verified and accepted by the responsible professional.

Philippine Civil Engineering Practice Context — September 24, 2026

For work performed in the Philippines:

  • Republic Act No. 544, as amended by R.A. No. 1582, remains the Civil Engineering Law published by the Professional Regulation Commission and defines the regulated practice of civil engineering.
  • The Professional Regulatory Board of Civil Engineering remains the relevant PRC board for the profession. AI use does not transfer regulated professional responsibility from the registered civil engineer or other legally responsible party.
  • Project conclusions must still be based on the governing Philippine law, adopted code, agency issuance, contract document, specification, and project-approved standard applicable to that work. Do not assume that a familiar foreign standard or a model's training data is the governing authority.
  • Republic Act No. 10173, the Data Privacy Act of 2012, applies to processing personal data. Project teams should classify and minimize personal information before using external AI systems.
  • In an August 11, 2026 public notice, the National Privacy Commission reiterated that a real person's face and likeness are personal information and that AI-generated processing or sharing of such likenesses must comply with lawful-processing requirements.

For professional or legal questions, verify the current PRC, National Privacy Commission, government-agency, and project-specific requirements before acting.

AI Assistance Versus Professional Authority

The boundary applies in Philippine practice as well as general engineering governance.

Professional responsibility and lawful data handling remain human and organizational obligations; AI output cannot create those authorities.

AI-assisted support
  • Organize evidence
  • Screen and summarize
  • Draft calculations or reports
  • Flag uncertainty
Human / legal responsibility
  • Licensed professional judgment
  • Sign, seal, certify, or approve where required
  • Lawful personal-data processing
  • Final safety and project decisions

Do Not Upload Sensitive Project Data by Default

Before supplying project files, determine whether the account, platform, organization policy, contract, client requirement, applicable privacy law, and data classification permit the information to be processed. Use data minimization and remove secrets, credentials, unnecessary personal information, proprietary values, identifiable faces or likenesses when not required, and sensitive infrastructure details where appropriate.

Prompt Injection and Untrusted Source Content

When AI can read files, webpages, emails, or connected sources, content inside those sources should be treated as evidence, not automatically as instructions. A defensive workflow should:

  • establish the authorized task outside the source material;
  • ignore embedded instructions that attempt to redirect the task or expand permissions;
  • flag suspicious instructions for human review;
  • prevent source content from authorizing writes, sends, deletions, or credential access;
  • restrict tools and connected data according to least privilege;
  • preserve the source location of suspicious content for incident review.

Reproducibility and Auditability

For material AI-assisted engineering work, retain enough information to reproduce and defend the result when policy requires it:

  • input data, file IDs, and revisions;
  • governing references and access dates;
  • prompt, template, or workflow version;
  • model/tool context where material;
  • generated scripts, formulas, or transformations;
  • intermediate artifacts and exception logs;
  • human corrections and independent checks;
  • unresolved limitations;
  • final approval, reviewer, and date.

The objective is not to archive every casual interaction. It is to preserve the evidence necessary to retrace consequential engineering work.

Bias and Representativeness Controls

AI-assisted rankings, predictions, or prioritization can be biased by the underlying engineering data. Review whether:

  • some locations, assets, populations, or failure types are inspected more often than others;
  • missing data are concentrated in particular groups or regions;
  • historical maintenance or reporting practices distort apparent risk;
  • proxy variables unintentionally encode socioeconomic, geographic, organizational, or other unrelated effects;
  • the validation dataset represents the conditions in which the workflow will be used;
  • engineering exposure variables are included where required.

Bias review should focus on the decision process and data-generation process, not merely on the model output.

Risk-Based Use Policy

A practical organization can classify AI use by consequence:

  • Green: formatting, learning, brainstorming, low-risk drafting from verified facts.
  • Amber: data analysis, schedule review, research synthesis, spreadsheet generation, document extraction, and other decision-support tasks requiring defined verification.
  • Red: safety-critical design approval, code certification, contractual decisions, acceptance of nonconforming work, emergency instructions, or actions with major financial, legal, or public-welfare consequences. AI may assist, but qualified human authority and independent review remain mandatory.

Escalate the control level when uncertainty, tool permissions, data sensitivity, or downstream consequence increases.

AI Incident Review and Continuous Improvement

When an AI-assisted workflow fails or nearly fails:

  1. Contain: stop the workflow or downstream use.
  2. Preserve evidence: retain prompt/workflow version, sources, outputs, logs, and approvals.
  3. Classify the failure: model error, source error, data problem, prompt/workflow defect, permission failure, review failure, or governance gap.
  4. Assess consequence: determine actual and potential engineering impact.
  5. Correct the immediate output or process.
  6. Improve the control: update prompts, validation rules, stop conditions, permissions, training, or review procedures.
  7. Retest: verify that the revised workflow catches the failure mode.

The purpose is to strengthen the engineering system rather than merely obtain a better one-time answer.

AI Incident Improvement Loop

Near misses and failures become inputs to stronger controls.

Incident review should improve the workflow control system, not merely regenerate a better answer.

Continuous risk improvement
Contain
Preserve evidence
Classify cause
Correct control
Retest and monitor
↺ repeat when checks fail

Worked Use Cases and Reusable Prompts

The companion Worked Examples includes prompts for fabricated-citation detection, confidential-data handling, prompt-injection defense, bias audits, professional-responsibility boundaries, reproducibility records, consequence-based verification, and AI incident review.

NIST ARIA: Evaluate the Whole Application

The September 18, 2026 NIST ARIA Evaluation Planning Manual emphasizes a holistic evaluation approach that combines three evidence streams:

  • Model testing: controlled technical tests against defined tasks and metrics.
  • Red teaming: adversarial attempts to induce unsafe, misleading, unauthorized, or otherwise unacceptable behavior.
  • User testing: observation of how real users understand, rely on, correct, or misuse the system.

For civil engineering, add deterministic domain checks such as numerical validation, source verification, unit checks, physical plausibility, code/standard traceability, and safe-stop behavior.

Build an Engineering Evaluation Dataset

A useful evaluation set should include:

  • representative normal cases with verified expected results;
  • boundary cases near design, operational, or classification thresholds;
  • missing, stale, contradictory, and malformed evidence;
  • unit, sign, coordinate, revision, and schema traps;
  • ambiguous multimodal evidence;
  • tool failures and timeouts;
  • indirect prompt injection embedded in documents, websites, emails, or tool responses;
  • attempts to obtain unauthorized reads, writes, sends, deletions, or approvals;
  • cases where the correct output is uncertainty, refusal, or escalation.

Version the dataset and rerun it when the model, prompt, skill, tool, source corpus, or workflow changes materially.

Agent Security Requires Enforced Boundaries

Prompt instructions alone are not a security boundary. Enforce least privilege outside the model through scoped credentials, read-only tools, allowlists, schema validation, network restrictions, sandboxing, approval gates, and transactional writes. A compromised or confused model should still be unable to perform actions that its workflow does not require.

Tool and MCP Outputs Are Untrusted Input

A connected tool can return malicious or misleading content. Tool poisoning exploits the gap between trusting a tool at connection time and processing its runtime responses. Keep external and privileged tools in separate trust zones, constrain return schemas where possible, validate results before they re-enter the model context, and prevent untrusted tool output from authorizing higher-privilege actions.

Measure Excessive Agency Explicitly

An agent has excessive agency when its functionality, permissions, or autonomy exceed what the objective requires. Evaluation should verify that the system cannot access unnecessary tools, cannot perform writes when read-only access is sufficient, cannot expand its own permissions, and stops for approval before irreversible or consequential actions.

LLM-Based Evaluation Is Supporting Evidence, Not Ground Truth

Model-based graders can scale qualitative review, but they can share blind spots with the system being evaluated and can be sensitive to rubric wording. Calibrate them against human-reviewed examples and deterministic checks. For engineering correctness, do not replace authoritative calculations, physical measurements, or source verification with a judge model merely because the score is convenient.

Key Takeaways
  • Engineering AI risk includes technical error, source failure, data problems, privacy, security, bias, governance, and professional responsibility.
  • Verification effort and approval controls should increase with consequence, uncertainty, and tool permissions.
  • Prompt injection and untrusted source content matter whenever AI can access files, apps, webpages, or external systems.
  • Reproducibility and incident review should preserve the evidence needed to defend and improve material engineering work.
  • AI can assist professional engineering but cannot assume licensure, certification, or accountable engineering judgment.
  • Philippine users must also check current PRC professional requirements, the Data Privacy Act, National Privacy Commission guidance, governing agency issuances, and project-specific standards.

References