CI・段階配信・supply chainを結果証拠で閉じる

到達目標

1. 必須checkの欠落、失敗、未知を成功へ倒さないfail-closed CIを設計できる

   • 失敗を閉じるCI、段階配信、来歴、ロールバックの実行証拠
   • 未知check、digest不一致、canary悪化を含むrelease判断

2. artifact digest、subject、builderを結びprovenanceの存在と信頼を区別できる

   • 失敗を閉じるCI、段階配信、来歴、ロールバックの実行証拠
   • provenanceが存在することと検証済みであることの差を説明する5分発表

3. 段階配信の観測からadvanceとrollbackを決め、実行後のsystem outcomeまで検証できる

   • 失敗を閉じるCI、段階配信、来歴、ロールバックの実行証拠
   • canary error rateだけを変えたadvance・rollback再評価

能力の進行

1. recognize

   CI成功、artifact同一性、provenance検証、配信結果を別の証拠として識別できる

   証拠: 失敗を閉じるCI、段階配信、来歴、ロールバックの実行証拠

2. explain

   command successだけではsystem outcomeを証明できない理由を説明できる

   証拠: provenanceが存在することと検証済みであることの差を説明する5分発表

3. apply

   必須check、digest、builder、canary指標から配信判断を再現できる

   証拠: 失敗を閉じるCI、段階配信、来歴、ロールバックの実行証拠

4. diagnose

   provenance presence、green表示、rollback commandを結果証拠と誤認する経路を反証できる

   証拠: 未知check、digest不一致、canary悪化を含むrelease判断

5. lead

   停止条件、rollback責任者、DORA five metrics、改善ループを合意できる

   証拠: canary error rateだけを変えたadvance・rollback再評価

なぜ重要か

delivery pipelineがexit code 0を返しても、利用者が正しいversionを安全に使えているとは限らない。required checkの取りこぼし、別artifactの配信、信頼しないbuilderが作ったprovenance、canary悪化、rollback後も残る障害は、command successの外側にあるsystem outcomeである。

SLSA v1.2はApproved仕様としてsupply-chain integrityを段階化し、NIST SSDF 1.1は安全なsoftware development practiceを組織の工程へ統合する。DORA continuous deliveryは変更を低riskでon demandにreleaseできる能力であり、単にdeployment回数を増やすことではない。

メンタルモデル

配信を、sourceからsystem outcomeへ証拠を引き渡すstate machineとして扱う。各境界は未知を成功へ倒さないfail-closed gateにし、次段へ進む条件と戻る条件を対で定義する。

DORAのcurrent five metricsはchange lead time、deployment frequency、failed deployment recovery time、change fail rate、deployment rework rateである。指標はrankingのためではなく、throughputとinstabilityを同時に見て改善仮説を検証するために使う。

動く例で考える

同じartifactでcanary観測だけを変える

前提

lesson-defined simulated fixtureを使う。digest、check、canary、rollback healthはproduction観測やvendor保証ではない。

入力

required check 3件、artifact bytes、provenance subject、trusted builder、baseline canary error rate 0.01、停止閾値0.02を与える。

操作

CIをfail-closedで検証し、bytesからdigestを導き、subjectとbuilderを検証する。canary error rateだけを0.05へ変えて同じ判断関数を再実行する。

観測

baselineはadvance、transferはrollbackになる。rollback commandが0でも、post-rollback error rateがtarget以下かを別に検証する。

結論

command successとsystem outcomeは同義ではない。配信完了は、同一性、来歴、段階観測、rollback outcomeまで結ばれた証拠で判断する。

python3.13 - <<'PY'
import hashlib
import json

HARNESS = "delivery_safety_lab_v1"
REQUIRED_CHECKS = ["unit", "security", "integration"]
CI_CHECKS = [
    {"name": "unit", "state": "passed"},
    {"name": "security", "state": "passed"},
    {"name": "integration", "state": "passed"},
]
ARTIFACT_BYTES = "curriculum-release-24"
TRUSTED_BUILDERS = ["builder://curriculum/release"]
DORA_METRICS = [
    "change_lead_time",
    "deployment_frequency",
    "failed_deployment_recovery_time",
    "change_fail_rate",
    "deployment_rework_rate",
]
BASELINE_CANARY = {
    "canary_error_rate": 0.01,
    "maximum_error_rate": 0.02,
}
TRANSFER_CANARY = {
    "canary_error_rate": 0.05,
    "maximum_error_rate": 0.02,
}
ROLLBACK_COMMAND_INPUT = {
    "command_exit_code": 0,
}

verification_trace = []

def is_number(value):
    return type(value) in {int, float} and type(value) is not bool

def validate_raw_delivery_inputs():
    if (
        type(REQUIRED_CHECKS) is not list
        or not REQUIRED_CHECKS
        or not all(
            type(name) is str and name
            for name in REQUIRED_CHECKS
        )
        or len(REQUIRED_CHECKS) != len(set(REQUIRED_CHECKS))
        or type(CI_CHECKS) is not list
        or not CI_CHECKS
        or type(ARTIFACT_BYTES) is not str
        or not ARTIFACT_BYTES
        or type(TRUSTED_BUILDERS) is not list
        or not TRUSTED_BUILDERS
        or not all(
            type(builder) is str and builder
            for builder in TRUSTED_BUILDERS
        )
        or len(TRUSTED_BUILDERS) != len(set(TRUSTED_BUILDERS))
        or type(DORA_METRICS) is not list
        or not DORA_METRICS
        or not all(
            type(metric) is str and metric
            for metric in DORA_METRICS
        )
        or len(DORA_METRICS) != len(set(DORA_METRICS))
        or type(BASELINE_CANARY) is not dict
        or type(TRANSFER_CANARY) is not dict
        or type(ROLLBACK_COMMAND_INPUT) is not dict
    ):
        raise AssertionError(
            "delivery-input-invariant: invalid raw input collection"
        )
    if (
        any(
            type(check) is not dict
            or set(check) != {"name", "state"}
            or type(check["name"]) is not str
            or not check["name"]
            or type(check["state"]) is not str
            or not check["state"]
            for check in CI_CHECKS
        )
        or len(CI_CHECKS)
        != len({check["name"] for check in CI_CHECKS})
        or set(ROLLBACK_COMMAND_INPUT) != {"command_exit_code"}
        or type(ROLLBACK_COMMAND_INPUT["command_exit_code"]) is not int
    ):
        raise AssertionError(
            "delivery-input-invariant: invalid raw input schema"
        )
    for canary in (BASELINE_CANARY, TRANSFER_CANARY):
        if (
            set(canary)
            != {"canary_error_rate", "maximum_error_rate"}
            or not all(
                is_number(value) and value >= 0
                for value in canary.values()
            )
        ):
            raise AssertionError(
                "delivery-transfer-invariant: invalid canary schema"
            )

def validate_delivery_input_schema(delivery_input):
    valid = (
        type(delivery_input) is dict
        and set(delivery_input)
        == {"artifact_bytes", "provenance", "canary"}
        and type(delivery_input["artifact_bytes"]) is str
        and bool(delivery_input["artifact_bytes"])
        and type(delivery_input["provenance"]) is dict
        and set(delivery_input["provenance"])
        == {"subject_digest", "builder_id"}
        and all(
            type(value) is str and value
            for value in delivery_input["provenance"].values()
        )
        and type(delivery_input["canary"]) is dict
        and set(delivery_input["canary"])
        == {"canary_error_rate", "maximum_error_rate"}
        and all(
            is_number(value) and value >= 0
            for value in delivery_input["canary"].values()
        )
    )
    if not valid:
        raise AssertionError(
            "delivery-transfer-invariant: invalid delivery input schema"
        )

def changed_paths(baseline, transferred, prefix=""):
    if type(baseline) is dict and type(transferred) is dict:
        paths = []
        for key in sorted(set(baseline) | set(transferred)):
            path = key if not prefix else prefix + "." + key
            if key not in baseline or key not in transferred:
                paths.append(path)
            else:
                paths.extend(
                    changed_paths(
                        baseline[key],
                        transferred[key],
                        path,
                    )
                )
        return paths
    return [] if baseline == transferred else [prefix]

def validate_delivery_transfer(baseline, transferred):
    validate_delivery_input_schema(baseline)
    validate_delivery_input_schema(transferred)
    paths = changed_paths(baseline, transferred)
    if paths != ["canary.canary_error_rate"]:
        raise AssertionError(
            "delivery-transfer-invariant: transfer changed wrong path"
        )
    return paths

def verify_ci(required, observed):
    if (
        type(required) is not list
        or not required
        or not all(type(name) is str and name for name in required)
        or type(observed) is not list
        or any(
            type(check) is not dict
            or set(check) != {"name", "state"}
            or type(check["name"]) is not str
            or not check["name"]
            or type(check["state"]) is not str
            or not check["state"]
            for check in observed
        )
    ):
        raise AssertionError(
            "delivery-ci-invariant: invalid check schema"
        )
    observed_by_name = {
        check["name"]: check["state"]
        for check in observed
    }
    all_observed = all(name in observed_by_name for name in required)
    all_passed = (
        all_observed
        and all(observed_by_name[name] == "passed" for name in required)
    )
    return {
        "failure_policy": "fail-closed",
        "all_required_checks_observed": all_observed,
        "all_required_checks_passed": all_passed,
        "required": list(required),
        "observed": list(observed),
    }

def verify_provenance(provenance, expected_digest):
    if (
        type(provenance) is not dict
        or set(provenance) != {"subject_digest", "builder_id"}
        or not all(
            type(value) is str and value
            for value in provenance.values()
        )
        or type(expected_digest) is not str
        or not expected_digest
    ):
        raise AssertionError(
            "delivery-provenance-invariant: invalid provenance schema"
        )
    subject_matches = (
        provenance["subject_digest"] == expected_digest
    )
    builder_trusted = (
        provenance["builder_id"] in TRUSTED_BUILDERS
    )
    verification_trace.extend(
        ["subject-digest-match", "trusted-builder"]
    )
    return subject_matches and builder_trusted

def decide_delivery(ci, provenance_ok, observation):
    gates_passed = (
        ci["all_required_checks_passed"]
        and provenance_ok
    )
    canary_healthy = (
        observation["canary_error_rate"]
        <= observation["maximum_error_rate"]
    )
    return "advance" if gates_passed and canary_healthy else "rollback"

def observe_rollback(fixture):
    if (
        type(fixture) is not dict
        or set(fixture)
        != {
            "command_exit_code",
            "before_error_rate",
            "after_error_rate",
            "target_error_rate",
        }
        or type(fixture["command_exit_code"]) is not int
        or not all(
            is_number(fixture[field]) and fixture[field] >= 0
            for field in (
                "before_error_rate",
                "after_error_rate",
                "target_error_rate",
            )
        )
    ):
        raise AssertionError(
            "delivery-outcome-invariant: invalid rollback schema"
        )
    command_completed = fixture["command_exit_code"] == 0
    service_restored = (
        fixture["after_error_rate"]
        <= fixture["target_error_rate"]
        and fixture["after_error_rate"]
        < fixture["before_error_rate"]
    )
    return {
        "executed": command_completed,
        "system_outcome_verified": service_restored,
        "service_restored": service_restored,
        "before_error_rate": fixture["before_error_rate"],
        "after_error_rate": fixture["after_error_rate"],
    }

def main():
    validate_raw_delivery_inputs()
    artifact_digest = hashlib.sha256(
        ARTIFACT_BYTES.encode("utf-8")
    ).hexdigest()
    PROVENANCE = {
        "subject_digest": artifact_digest,
        "builder_id": TRUSTED_BUILDERS[0],
    }
    baseline_delivery_inputs = {
        "artifact_bytes": ARTIFACT_BYTES,
        "provenance": dict(PROVENANCE),
        "canary": dict(BASELINE_CANARY),
    }
    transferred_delivery_inputs = {
        "artifact_bytes": ARTIFACT_BYTES,
        "provenance": dict(PROVENANCE),
        "canary": dict(TRANSFER_CANARY),
    }
    delivery_changed_paths = validate_delivery_transfer(
        baseline_delivery_inputs,
        transferred_delivery_inputs,
    )
    baseline_artifact_digest = hashlib.sha256(
        baseline_delivery_inputs["artifact_bytes"].encode("utf-8")
    ).hexdigest()
    transferred_artifact_digest = hashlib.sha256(
        transferred_delivery_inputs["artifact_bytes"].encode("utf-8")
    ).hexdigest()
    ROLLBACK_DRILL = {
        "command_exit_code": (
            ROLLBACK_COMMAND_INPUT["command_exit_code"]
        ),
        "before_error_rate": (
            transferred_delivery_inputs["canary"][
                "canary_error_rate"
            ]
        ),
        "after_error_rate": (
            baseline_delivery_inputs["canary"]["canary_error_rate"]
        ),
        "target_error_rate": (
            baseline_delivery_inputs["canary"]["maximum_error_rate"]
        ),
    }
    changed_fields = [
        path.removeprefix("canary.")
        for path in delivery_changed_paths
    ]

    ci = verify_ci(REQUIRED_CHECKS, CI_CHECKS)
    provenance_verified = verify_provenance(PROVENANCE, artifact_digest)
    assert (
        provenance_verified
        and verification_trace
        == ["subject-digest-match", "trusted-builder"]
    ), "delivery-provenance-invariant"
    provenance_present = bool(PROVENANCE)
    subject_matches = (
        PROVENANCE["subject_digest"] == artifact_digest
    )
    builder_trusted = (
        PROVENANCE["builder_id"] in TRUSTED_BUILDERS
    )
    required_provenance_evidence = list(verification_trace)
    presence_alone_is_trusted = (
        len(required_provenance_evidence) == 0
    )

    baseline_decision = decide_delivery(
        ci,
        provenance_verified,
        BASELINE_CANARY,
    )
    transferred_decision = decide_delivery(
        ci,
        provenance_verified,
        TRANSFER_CANARY,
    )
    rollback = observe_rollback(ROLLBACK_DRILL)
    assert baseline_decision == "advance", "delivery-gate-invariant"
    assert transferred_decision == "rollback", "delivery-canary-invariant"
    assert rollback["service_restored"], "delivery-outcome-invariant"

    report = {
        "harness": HARNESS,
        "ci": ci,
        "artifact": {
            "bytes_label": ARTIFACT_BYTES,
            "digest": artifact_digest,
        },
        "provenance_verification": {
            "present": provenance_present,
            "subject_digest_matches": subject_matches,
            "builder_trusted": builder_trusted,
            "verified": provenance_verified,
            "presence_alone_is_trusted": presence_alone_is_trusted,
            "verification_trace": required_provenance_evidence,
        },
        "staged_delivery": {
            "observation": dict(BASELINE_CANARY),
            "decision": baseline_decision,
        },
        "rollback_outcome": rollback,
        "dora_five_metrics": list(DORA_METRICS),
        "canary_error_rate_transfer": {
            "changed_assumption": "canary-error-rate",
            "changed_fields": changed_fields,
            "same_artifact": (
                baseline_artifact_digest
                == transferred_artifact_digest
            ),
            "same_provenance": (
                baseline_delivery_inputs["provenance"]
                == transferred_delivery_inputs["provenance"]
            ),
            "baseline_artifact_digest": baseline_artifact_digest,
            "transferred_artifact_digest": (
                transferred_artifact_digest
            ),
            "baseline_provenance": (
                baseline_delivery_inputs["provenance"]
            ),
            "transferred_provenance": (
                transferred_delivery_inputs["provenance"]
            ),
            "baseline_decision": baseline_decision,
            "transferred_decision": transferred_decision,
            "baseline_observation": (
                baseline_delivery_inputs["canary"]
            ),
            "transferred_observation": (
                transferred_delivery_inputs["canary"]
            ),
        },
        "fixture_metadata": {
            "kind": "simulated",
            "provenance": "lesson-defined delivery evidence fixture",
            "limitations": (
                "not production telemetry, signing, pricing, "
                "or vendor assurance"
            ),
            "synthetic_or_observed_explicit": True,
        },
        "runtime_bound": {
            "records": (
                len(CI_CHECKS)
                + len(PROVENANCE)
                + len(BASELINE_CANARY)
                + len(ROLLBACK_DRILL)
            ),
            "subprocesses": 0,
            "maximum_iterations": len(CI_CHECKS),
        },
        "external_network_used": False,
        "command_success_distinction": {
            "command_completed": (
                ROLLBACK_DRILL["command_exit_code"] == 0
            ),
            "system_outcome_checked": rollback[
                "system_outcome_verified"
            ],
            "command_success_equals_system_outcome": (
                ROLLBACK_DRILL["command_exit_code"] == 0
                and not rollback["system_outcome_verified"]
            ),
            "outcome_evidence": {
                "before": rollback["before_error_rate"],
                "after": rollback["after_error_rate"],
            },
        },
        "mastery_evidence": {
            "lab_steps": [
                {
                    "step": index,
                    "evidence": evidence,
                }
                for index, evidence in enumerate(
                    [
                        ci,
                        artifact_digest,
                        baseline_decision,
                        rollback,
                        transferred_decision,
                    ],
                    start=1,
                )
            ],
            "assessments": [
                {
                    "assessment": 1,
                    "evidence": ci["failure_policy"],
                },
                {
                    "assessment": 2,
                    "evidence": required_provenance_evidence,
                },
            ],
            "rubric_dimensions": [
                "technical-correctness",
                "judgment",
                "evidence",
                "communication",
            ],
            "transfer": {
                "task": (
                    "canary error rateだけを変え、同じartifactと"
                    "provenanceで段階配信のadvance・rollback判断を"
                    "再評価する"
                ),
                "changed_assumption": "canary-error-rate",
                "evidence": {
                    "baseline": baseline_decision,
                    "transferred": transferred_decision,
                },
            },
        },
    }
    print(json.dumps(report, ensure_ascii=False, sort_keys=True))

try:
    main()
except AssertionError as error:
    raise SystemExit(str(error)) from None
PY

トレードオフと失敗モード

• 誤診: CIがgreenなので配信対象も正しく安全である。反証: required check集合、artifact digest、provenance subject、trusted builderを別々に検証する。
• 誤診: rollback commandが成功したのでincidentは解消した。反証: 利用者可視error rateと対象versionを再観測し、system outcomeがtargetへ戻ったことを確認する。

知識チェック

1. required checkのunknownをsuccessへ倒すと、どの証拠境界が壊れるか。
2. SLSA v1.2 Approvedのprovenance presence aloneを拒否するために何を照合するか。
3. DORA five metricsを一つだけ最大化すると起こりうる局所最適を説明せよ。
4. command successとsystem outcomeを分けるrollback証拠を設計せよ。

出典と次の学習

supply-chain integrityはSLSA Specification Version 1.2 Approved、安全な開発工程はNIST SSDF 1.1、低riskな配信能力はDORA continuous delivery、評価はDORA current five metricsを基準にした。仕様名やmetricsはversionと更新日を固定して解釈する。

次は、core15の利用者可視SLIをcanaryの停止条件へ結び、artifact、provenance、decision、rollback outcomeを一つの監査可能なevidence bundleとして再現する。

実践ラボ

検証可能な段階配信evidence bundleを作る

提出成果物: 失敗を閉じるCI、段階配信、来歴、ロールバックの実行証拠

1. 必須CI checkの観測状態とfail-closed規則を固定する
2. artifact bytesからdigestを導きprovenance subjectとtrusted builderを検証する
3. baseline canary error rateを停止閾値と比較しadvanceを判断する
4. rollbackを実行したfixtureでservice restorationを別観測として検証する
5. canary error rateだけを変えて同じartifactとprovenanceの判断を再評価する

説明して理解を確かめる

5分で、SLSA v1.2 Approved、NIST SSDF 1.1、DORA continuous deliveryとfive metricsを使い、署名またはprovenanceの存在だけでは信頼できない理由を説明する。

アセスメント

1. 問い: required checkが一件unknownだが他はgreenである。配信可否と追加証拠を答えよ。

   期待する証拠: fail-closed規則、観測済みcheck集合、unknownの解消、承認境界

2. 問い: provenance fileはあるがsubject digestがartifactと異なる。なぜ拒否し、何を再検証するか。

   期待する証拠: artifact digest、subject digest、builder identity、provenance presence aloneの不十分性

別問題へ転用する

canary error rateだけを変え、同じartifactとprovenanceで段階配信のadvance・rollback判断を再評価する

復習スケジュール

1. 1日後

   CIのunknownをfail-closedにする理由は何か

2. 7日後

   provenance presence aloneが信頼証拠にならない理由は何か

3. 30日後

   rollback command success後にどのsystem outcomeを確認するか

4. 90日後

   CIのunknownをfail-closedにする理由は何か

評価ルーブリック

出典

以下の外部資料は利用者が選択したときだけ開きます。

• SLSA Specification Version 1.2 (Approved) (standard)
• DORA's software delivery performance metrics (updated January 5, 2026) (primary)
• DORA Capability: Continuous delivery (accessed 2026-07-31) (primary)
• NIST SP 800-218: Secure Software Development Framework Version 1.1 (February 2022) (standard)