Compare commits

..
16 Commits
Author SHA1 Message Date
jiang 01bad116c1 fix(scada): align location capability and investigation guidance
Generic Container CI/CD / test-build-publish (push) Successful in 5m25s
Server CI/CD v2 / build-test-publish-and-deploy (push) Successful in 5m26s
2026-11-07 20:14:24 +08:00
jiang bf8ea45d40 test: support importlib mode for calibration fixtures
Generic Container CI/CD / test-build-publish (push) Successful in 5m17s
Server CI/CD v2 / build-test-publish-and-deploy (push) Successful in 5m17s
2026-11-07 19:54:42 +08:00
jiang dc83cf5c88 fix(scada): verify first sample after field completion
Generic Container CI/CD / test-build-publish (push) Failing after 47s
Server CI/CD v2 / build-test-publish-and-deploy (push) Failing after 47s
2026-11-07 19:50:31 +08:00
jiang ff8aba4964 feat(scada): add calibrated monitoring and incident recovery 2026-11-07 19:44:58 +08:00
jiang 26fa5d1909 feat(scada): add project monitoring replay
Generic Container CI/CD / test-build-publish (push) Successful in 3m39s
Server CI/CD v2 / build-test-publish-and-deploy (push) Successful in 3m39s
2026-11-04 14:27:21 +08:00
jiang f25423dec4 feat(benchmark): add tjwater2 burst calibration design 2026-11-03 17:26:38 +08:00
jiang e48ccfba9e test: update MCP catalog count for node coverage
Generic Container CI/CD / test-build-publish (push) Successful in 3m35s
Server CI/CD v2 / build-test-publish-and-deploy (push) Successful in 3m35s
2026-11-02 15:26:14 +08:00
jiang f2cfce8192 feat: add node coverage and improve investigation presentation
Generic Container CI/CD / test-build-publish (push) Failing after 1m50s
Server CI/CD v2 / build-test-publish-and-deploy (push) Failing after 1m51s
2026-11-02 15:23:34 +08:00
jiang 05183aff93 feat: extend operational investigation workflows
Generic Container CI/CD / test-build-publish (push) Successful in 3m53s
Server CI/CD v2 / build-test-publish-and-deploy (push) Successful in 3m52s
2026-10-30 17:30:19 +08:00
jiang 63af998045 feat(mcp): queue long-running calculations 2026-10-29 18:25:15 +08:00
jiang 2865811f1b chore(config): align server environment templates 2026-10-27 17:17:08 +08:00
jiang af1dc11681 docs: record deployed API port 2026-10-23 16:12:26 +08:00
jiang 8673429ec4 docs: clarify peripheral proxy upstream 2026-10-23 16:11:34 +08:00
jiang 6d31f456e6 feat: mount peripheral API in main server
Generic Container CI/CD / test-build-publish (push) Successful in 2m59s
Server CI/CD v2 / build-test-publish-and-deploy (push) Successful in 2m59s
2026-10-23 16:05:56 +08:00
jiang cddfc8647d docs: update server operations guide
Generic Container CI/CD / test-build-publish (push) Successful in 5m20s
Server CI/CD v2 / build-test-publish-and-deploy (push) Successful in 5m20s
2026-10-23 14:29:44 +08:00
jiang 1d061f6099 feat: add project-scoped operations
Generic Container CI/CD / test-build-publish (push) Failing after 6s
Server CI/CD v2 / build-test-publish-and-deploy (push) Failing after 6s
2026-10-22 18:40:20 +08:00
229 changed files with 25801 additions and 638 deletions
+105 -71
View File
@@ -1,127 +1,161 @@
# TJWater Server 环境变量配置模板
# 复制此文件为 .env 并填写实际值
# CI/CD: 生产环境变量由 Dev 主机的受控 backend.env 注入,不要将完整 .env 保存为 Gitea 仓库密钥。
ENVIRONMENT="production"
NETWORK_NAME="tjwater"
# ============================================
# 敏感配置加密 (必填)
# ============================================
# NextTJWater Server 环境变量模板
# 复制为 .env 后按启用的功能填写;不要提交 .env 或任何真实凭据。
# 生产配置由部署主机的受控 next-backend.env 注入,不随镜像或 CI/CD 同步本地 .env。
# 进程环境变量优先;修改 Compose env_file 后需重新创建容器。
# Fernet 格式,生产环境必须替换为独立密钥
# 生成方式: python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"
# 用于项目数据库 DSN、GeoServer 管理密码等敏感配置
# 基础运行
PROJECT_NAME="TJWater Server"
ENVIRONMENT="production"
API_V1_STR="/api/v1"
NETWORK_NAME="tjwater"
# Fernet 格式;用于加密项目数据库 DSN、GeoServer 管理密码等敏感配置。
# 生成方式:python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"
DATABASE_ENCRYPTION_KEY="replace-with-generated-fernet-key"
# ============================================
# 数据库配置 (PostgreSQL)
# ============================================
# 业务数据库(PostgreSQL)
DB_NAME="tjwater"
DB_HOST="localhost"
DB_PORT="5432"
DB_USER="tjwater"
DB_PASSWORD="password"
# ============================================
# 数据库配置 (TimescaleDB)
# ============================================
# 时序数据库(TimescaleDB)
TIMESCALEDB_DB_NAME="tjwater"
TIMESCALEDB_DB_HOST="localhost"
TIMESCALEDB_DB_PORT="5433"
TIMESCALEDB_DB_USER="tjwater"
TIMESCALEDB_DB_PASSWORD="password"
# 实时快照替换事务的解压预算;仅事务内生效,禁止0(无限制)。
# 实时快照替换事务的解压预算;仅事务内生效,禁止设为 0(无限制)。
TIMESCALEDB_REALTIME_MAX_DECOMPRESSED_TUPLES=1000000
# ============================================
# 元数据数据库配置 (Metadata DB)
# ============================================
# 元数据数据库
METADATA_DB_NAME="system_hub"
METADATA_DB_HOST="localhost"
METADATA_DB_PORT="5432"
METADATA_DB_USER="tjwater"
METADATA_DB_PASSWORD="password"
METADATA_DB_POOL_SIZE=5
METADATA_DB_MAX_OVERFLOW=10
# Per-project synchronous connection pools
PROJECT_PG_CACHE_SIZE="16"
PROJECT_TS_CACHE_SIZE="16"
PROJECT_PG_POOL_MIN_SIZE="0"
PROJECT_PG_POOL_SIZE="4"
PROJECT_PG_MAX_OVERFLOW="2"
PROJECT_TS_POOL_MIN_SIZE="0"
PROJECT_TS_POOL_MAX_SIZE="4"
# 项目数据库连接池、缓存与模板
PROJECT_PG_CACHE_SIZE=16
PROJECT_TS_CACHE_SIZE=16
PROJECT_PG_POOL_MIN_SIZE=0
PROJECT_PG_POOL_SIZE=4
PROJECT_PG_MAX_OVERFLOW=2
PROJECT_TS_POOL_MIN_SIZE=0
PROJECT_TS_POOL_MAX_SIZE=4
WNDB_SCHEMA_TEMPLATE_DB_NAME="tjwater_v2_schema_template"
TIMESCALEDB_SCHEMA_TEMPLATE_DB_NAME="tjwater_v2_timescale_template"
WNDB_TEMP_DB_MAX_COUNT="8"
WNDB_TEMP_DB_MAX_COUNT=8
# ============================================
# GeoServer 项目供应
# ============================================
GEOSERVER_URL="http://localhost:8080/geoserver"
GEOSERVER_USERNAME="admin"
GEOSERVER_PASSWORD="password"
# 留空时分别复用 DB_HOST / DB_PORT / DB_USER / DB_PASSWORD。
# 留空时分别复用 DB_HOST、DB_PORT、DB_USER、DB_PASSWORD。
# GeoServer 在容器内运行时,GEOSERVER_DB_HOST 通常应填写数据库服务名。
GEOSERVER_DB_HOST=""
GEOSERVER_DB_PORT=""
GEOSERVER_DB_USER=""
GEOSERVER_DB_PASSWORD=""
# 30 天客户端缓存,单位为秒。
GEOSERVER_CLIENT_CACHE_SECONDS="2592000"
# 客户端缓存时间,单位为秒;默认 30 天。
GEOSERVER_CLIENT_CACHE_SECONDS=2592000
# ============================================
# Keycloak JWT (可选)
# ============================================
KEYCLOAK_PUBLIC_KEY="-----BEGIN PUBLIC KEY-----\n...\n-----END PUBLIC KEY-----"
KEYCLOAK_ALGORITHM=RS256
# Keycloak JWT 校验
# 未启用认证时可留空;生产环境应注入完整公钥和预期 audience。
KEYCLOAK_PUBLIC_KEY=""
KEYCLOAK_ALGORITHM="RS256"
KEYCLOAK_AUDIENCE="account"
KEYCLOAK_ACCESS_TOKEN_MAX_AGE_SECONDS=900
# ============================================
# 工作台事件同步
# ============================================
# 认证接口必须使用明确来源,逗号分隔;不要配置为 *。
CORS_ALLOWED_ORIGINS="http://127.0.0.1:5173,http://localhost:5173"
# 浏览器跨域访问(主 API)
# 必须填写明确来源(协议、域名或 IP、端口),英文逗号分隔;不允许 * 或网段正则。
CORS_ALLOWED_ORIGINS=http://127.0.0.1:5173,http://localhost:5173,http://127.0.0.1:15173,http://localhost:15173
# 工作台事件
WORKBENCH_EVENTS_ENABLED=true
WORKBENCH_EVENT_RETENTION_DAYS=7
# 每个项目启动一个 worker 实例。长期运行优先使用 OAuth 客户端凭据。
# 天地图服务
# 地理编码、逆地理编码和地名搜索共用同一令牌。
TIANDITU_TOKEN=""
TIANDITU_GEOCODER_URL="https://api.tianditu.gov.cn/geocoder"
TIANDITU_GEOCODER_TIMEOUT_SECONDS=30
TIANDITU_SEARCH_URL="https://api.tianditu.gov.cn/v2/search"
TIANDITU_SEARCH_TIMEOUT_SECONDS=15
# 调查任务 worker(独立进程)
# 单项目填写 PROJECT_ID;多项目填写逗号分隔的 PROJECT_IDS,后者优先。
# worker 与外围工单同步器共用项目范围,服务身份必须拥有每个项目的权限。
OPERATIONAL_WORKER_PROJECT_ID=""
OPERATIONAL_WORKER_PROJECT_IDS=""
OPERATIONAL_WORKER_SERVER_URL="http://127.0.0.1:8000"
OPERATIONAL_WORKER_AGENT_URL="http://127.0.0.1:8787"
OPERATIONAL_WORKER_RECONCILE_SECONDS=30
OPERATIONAL_WORKER_MAX_CONCURRENCY=4
# 可选 ISO 8601 时间下界,必须包含时区,例如 2026-10-01T00:00:00+08:00。
OPERATIONAL_WORKER_TASK_CREATED_FROM=""
# 长期运行使用 OAuth 客户端凭据;固定 Bearer 仅用于受控开发或短期运行。
OPERATIONAL_WORKER_TOKEN_URL=""
# 仅 ENVIRONMENT=development 时覆盖 TOKEN_URL,其他环境忽略。
OPERATIONAL_WORKER_DEV_TOKEN_URL=""
OPERATIONAL_WORKER_CLIENT_ID=""
OPERATIONAL_WORKER_CLIENT_SECRET=""
OPERATIONAL_WORKER_TOKEN_REFRESH_SECONDS=300
OPERATIONAL_WORKER_BEARER=""
# 调查数据适配器
OPERATIONAL_DATA_SERVER_URL="http://127.0.0.1:8000"
OPERATIONAL_DATA_PERIPHERAL_URL=""
OPERATIONAL_DATA_TIMEOUT_SECONDS=15
# 仅供 Server 在已批准的兼容委托路径中使用,不传给 Agent。
OPERATIONAL_DATA_DELEGATION_BEARER=""
# 调查动作适配器
OPERATIONAL_ACTION_PERIPHERAL_URL=""
OPERATIONAL_ACTION_COMMAND_GATEWAY_URL=""
OPERATIONAL_ACTION_TIMEOUT_SECONDS=10
# ============================================
# Tianditu shared API key and service URLs
# ============================================
TIANDITU_TOKEN="your-tianditu-token"
TIANDITU_GEOCODER_URL="https://api.tianditu.gov.cn/geocoder"
TIANDITU_GEOCODER_TIMEOUT_SECONDS=30
# 外围样板(挂载于 /peripheral;故障注入默认关闭)
# 独立启动时用于调用主 API 验证身份;容器内按网络填写主服务地址。
PERIPHERAL_SAMPLE_NEXT_SERVER_URL="http://127.0.0.1:8000"
# 以下两个默认库选择只供种子、导出等脚本使用,HTTP 请求仍按已授权项目路由。
PERIPHERAL_SAMPLE_DB_NAME="tjwater_v2"
PERIPHERAL_SAMPLE_PROJECT_CODE="tjwater_v2"
# 子应用独立运行时的浏览器来源列表,使用 JSON 格式;挂载时还受主 API CORS 配置控制。
PERIPHERAL_SAMPLE_CORS_ORIGINS='["http://127.0.0.1:5173","http://localhost:5173"]'
PERIPHERAL_SAMPLE_ENABLE_FAULTS=false
PERIPHERAL_SAMPLE_TIMEOUT_SECONDS=5
# 地名搜索与地理编码共用 TIANDITU_TOKEN;也可沿用 TJWATER_TIANDITU_TOKEN。
TIANDITU_SEARCH_URL="https://api.tianditu.gov.cn/v2/search"
TIANDITU_SEARCH_TIMEOUT_SECONDS=15
# MCP query resource, off until OAuth and every project BizDB are provisioned.
# 仅 ENVIRONMENT=development 时覆盖 worker 取令牌地址;留空沿用 OPERATIONAL_WORKER_TOKEN_URL。
OPERATIONAL_WORKER_DEV_TOKEN_URL=
# MCP 查询服务;OAuth 和所有项目业务库就绪前保持关闭。
TJWATER_MCP_ENABLED=false
TJWATER_MCP_ISSUER_URI=
# 仅 ENVIRONMENT=development,可填写与 issuer 完全相同的内网 HTTP IPv4 issuer。
# 正式环境必须留空;此例外不放行远程 HTTP MCP resource。
TJWATER_MCP_DEV_HTTP_ISSUER_URI=
TJWATER_MCP_RESOURCE_URI=
TJWATER_MCP_AUDIENCE=tjwater-mcp
TJWATER_MCP_HOST_CLIENT_ID=tjwater-agent-mcp
TJWATER_MCP_ALLOWED_HOSTS=["localhost:*","127.0.0.1:*"]
TJWATER_MCP_ALLOWED_ORIGINS=[]
# 普通查询预算;计算工具由 Agent 用 timeout_seconds 按次指定(默认120秒)。
TJWATER_MCP_QUERY_TIMEOUT_SECONDS=30
# 计算等待上限,默认1小时;管理员可配置120至86400秒,实际上限通过 tools/list 发布。
TJWATER_MCP_ISSUER_URI=""
# 仅 ENVIRONMENT=development:可与 ISSUER_URI 同值并指向内网 HTTP IPv4 issuer。
# 正式环境必须留空;该例外不允许远程 HTTP MCP resource。
TJWATER_MCP_DEV_HTTP_ISSUER_URI=""
TJWATER_MCP_RESOURCE_URI=""
TJWATER_MCP_AUDIENCE="tjwater-mcp"
TJWATER_MCP_HOST_CLIENT_ID="tjwater-agent-mcp"
# 列表使用 JSON 格式。
TJWATER_MCP_ALLOWED_HOSTS='["127.0.0.1:*","localhost:*"]'
TJWATER_MCP_ALLOWED_ORIGINS='[]'
TJWATER_MCP_CONTEXT_TTL_SECONDS=600
# 单次 HTTP 返回预算;由目录元数据公布给宿主,Agent 默认等待 180 秒。
TJWATER_MCP_QUERY_TIMEOUT_SECONDS=120
# HTTP 请求预算上限;Agent 加 60 秒余量并封顶 270 秒,外层 MCP 总时限 300 秒。
TJWATER_MCP_MAX_REQUEST_TIMEOUT_SECONDS=240
# 后台计算独立执行时限为 120–86400 秒,不受工具 timeout_seconds 控制。
TJWATER_MCP_MAX_CALCULATION_TIMEOUT_SECONDS=3600
# 独立计算队列消费者;从 MetaDB 动态发现活动项目,不会在 API lifespan 内启动。
# 先在每个项目 BizDB 应用 026、027 和 030 迁移,再启动计算服务。
# 所有活动项目合计并发,独立于 Uvicorn --workers 4;单实例锁防止重复消费者扩容。
CALCULATION_WORKER_MAX_CONCURRENCY=2
CALCULATION_WORKER_POLL_SECONDS=2
CALCULATION_WORKER_LEASE_SECONDS=30
CALCULATION_WORKER_HEARTBEAT_SECONDS=5
# 持久化本机恢复账本;容器模板已挂载独立 volume,不能当缓存删除。
CALCULATION_WORKER_STATE_DIR="temp/calculation-jobs"
+2 -2
View File
@@ -10,11 +10,11 @@ jobs:
build-test-publish-and-deploy:
uses: OrgTJWater/ci-templates/.gitea/workflows/container-cd.yml@68c8a9855391baa31d31523674f5812cd24ec604
with:
image_name: gitea.waternetwork.cn/orgtjwater/tjwater-backend
image_name: gitea.waternetwork.cn/orgtjwater/next-tjwater-backend
dockerfile: Dockerfile
build_context: .
test_target: test
deploy_service: backend
deploy_service: next-backend
deploy_host: 192.168.1.114
secrets:
REGISTRY_USERNAME: ${{ secrets.REGISTRY_USERNAME }}
+1 -1
View File
@@ -12,7 +12,7 @@ Use the existing conda environment when available:
```bash
conda run -n server python -m pytest tests/unit tests/auth -q
conda run -n server uvicorn app.main:app --host 0.0.0.0 --port 8000 --reload
conda run -n server python scripts/run_server.py --reload
docker build -t tjwater-server:local .
docker compose -f infra/docker/docker-compose.yml config
```
+1 -1
View File
@@ -23,7 +23,7 @@ COPY app ./app
COPY contracts ./contracts
COPY infra ./infra
COPY resources ./resources
COPY scripts/run_operational_worker.py scripts/sync_peripheral_work_orders.py ./scripts/
COPY scripts/run_operational_worker.py scripts/run_calculation_worker.py scripts/sync_peripheral_work_orders.py ./scripts/
COPY samples/peripheral ./samples/peripheral
RUN python -c "from pathlib import Path; from zipfile import ZipFile; model_dir = Path('app/algorithms/pipe_health_prediction/model'); zip_path = model_dir / 'my_survival_forest_model_quxi.zip'; joblib_name = 'my_survival_forest_model_quxi.joblib'; joblib_path = model_dir / joblib_name; assert zip_path.exists(), f'Model archive not found: {zip_path}'; archive = ZipFile(zip_path); archive.extract(joblib_name, model_dir); archive.close(); assert joblib_path.exists(), f'Model file not extracted: {joblib_path}'" && \
rm -f app/algorithms/pipe_health_prediction/model/my_survival_forest_model_quxi.zip
+99 -10
View File
@@ -1,6 +1,6 @@
# TJWaterServerBinary 内部后端
# NextTJWater Server
`TJWaterServerBinary` 是 TJWater 内部版 Python 后端,基于 FastAPI 提供认证、项目、管网、模拟、爆管、漏损、SCADA 和地图服务集成能力。该仓库用于内部开发和完整功能维护。
`next-tjwater-server` 是 NextTJWater 供水系统的 FastAPI 后端,提供认证、项目、管网、模拟、爆管、漏损、SCADA、地图服务、工况任务和受控 MCP 能力。业务数据和时序数据按已认证项目路由,后台工作进程也必须使用明确的项目范围。
## 技术栈
@@ -30,11 +30,19 @@ infra/docker/ Docker Compose 编排
## 本地开发
先从样板创建本地配置,再按启用的功能填写:
```bash
cp .env.example .env
```
`.env` 只用于本地或部署主机运行,不得提交。生产配置由部署主机的受控环境文件注入,不将完整应用 `.env` 保存为 Gitea Actions 密钥。
推荐使用已有 conda 环境:
```bash
conda run -n server python -m pytest tests/unit tests/auth -q
conda run -n server uvicorn app.main:app --host 0.0.0.0 --port 8000 --reload
conda run -n server python scripts/run_server.py --reload
```
如需要进入环境:
@@ -45,15 +53,52 @@ conda activate server
`GET /health` 是无需鉴权的 HTTP 服务存活检查,返回 `200` 和 `{"status":"ok"}`,不查询数据库。
`GET /ready` 检查应用启动状态、元数据库 `SELECT 1` 及已启用的 MCP 传输生命周期;
全部就绪返回 `200`,否则返回 `503`,元数据库探测预算为2秒。MCP 关闭时标记为
全部就绪返回 `200`,否则返回 `503`,元数据库探测预算为 2 秒。MCP 关闭时标记为
`disabled`,不阻止就绪。返回结果不包含数据库地址、凭据或原始异常,禁止缓存。
就绪状态不等同于 OAuth、用户权限或真实工具调用成功;指定项目的业务库和时序库仍通过
带身份与 `X-Project-Id` 的 `GET /api/v1/projects/current/database-health` 检查。
## 环境配置
`.env.example` 是唯一应提交的环境配置样板,主要分为以下几组:
- 元数据库、项目业务库、TimescaleDB 及项目数据库连接池。
- GeoServer 项目供应、Keycloak JWT 校验、CORS 和工作台事件。
- 天地图地理编码和地名搜索。
- 调查任务 worker、外围工单同步、数据与动作适配器。
- 可选的 MCP resource、OAuth issuer、宿主和超时预算。
基础数据库变量用于启动和项目供应。运行时的项目请求仍以元数据库中的 `biz_data` 和 `iot_data` 路由为准,不会因为存在默认数据库变量而回退到固定项目。
CORS 通过后端 `.env` 或部署进程环境变量中的 `CORS_ALLOWED_ORIGINS` 配置,进程环境变量优先。
必须使用英文逗号分隔的明确页面来源(协议、域名或 IP、端口,不带路径),不能填写 `*`。
未设置时默认只允许本机 `5173`、`15173` 开发来源。正式部署按实际浏览器地址填写,例如:
```dotenv
CORS_ALLOWED_ORIGINS=https://next-demo.waternetwork.cn,http://192.168.1.133,https://192.168.1.133,http://192.168.1.110,https://192.168.1.110
```
CORS 匹配浏览器页面来源,不是 Nginx 连接 IP 或域名解析结果;非默认端口须逐项列出。
保留跨域凭据支持及 `Authorization`、`X-Project-Id` 请求头,接口认证和权限校验继续生效。
修改后需重启进程;使用 Compose `env_file` 注入时需重新创建容器才能读取新值。
## 业务 Skill 与闭环
[爆管事件处理流程](docs/burst-response-workflow.md)覆盖工单接收、资产关联、隔离阀门、影响说明、审批处置和反馈闭环。Server 按任务 workflow 与回放来源分发独立 Skill,版本以实际返回为准:
| workflow | Skill | 调查重点 |
| --- | --- | --- |
| burst_report(工单) | [爆管事件处理](resources/work-order-skills/burst-event-handling/SKILL.md) | 资产关联、隔离方案、影响说明、处置及恢复闭环 |
| scada_anomaly/历史回放 | [SCADA 异常调查](resources/work-order-skills/scada-anomaly/SKILL.md) | 冻结数据诊断、候选定位与隔离影响分析 |
| water_quality_complaint | [水质投诉](resources/work-order-skills/water-quality-complaint/SKILL.md) | 投诉关联、监测与采样、候选影响、处置及复测回访 |
| general | [预约跟进](resources/work-order-skills/appointment-follow-up/SKILL.md) | 审批与回执、现场履约、反馈复核及与原事件的闭环区别 |
JSONL 整理遵守会话文件权限,后台图表和表格通过既有报告 `ui_blocks` 展示;Skill 不增加审批、执行或原工单关单权限。预约投影不自动启动 Agent 调查,普通协作与后台执行分别遵守其原有工具范围。
## 常用命令
外围接入样板是独立进程,复用 PostgreSQL,不自动建表;启动、授权边界和契约见 [外围样板说明](docs/peripheral-sample.md)。
外围样板提供 synthetic 测试工单和测试数据的版本化 CRUD;工单经同步器投影为 `workbench.operational_tasks` 中的外围来源任务。固定演示时窗和造数方式见 [合成样本说明](docs/peripheral-generated-data.md),运行态读取边界见 [工作台运行态说明](docs/workbench-runtime.md)。这些数据不代表现场事实,也不会触发设备控制。
外围接入样板作为 `/peripheral` 子应用随主 API 进程启动,复用 PostgreSQL 且不自动建表;授权边界和契约见 [外围样板说明](docs/peripheral-sample.md)。
外围系统提供业务工单和反馈的版本化 CRUD,工单来源固定为 `system_data`,经同步器投影为 `workbench.operational_tasks` 中的外围来源任务。系统上报与已核实事实分别记录,现场处置通过审批流程执行。独立测试数据和监测演示的固定时窗及造数方式见 [合成样本说明](docs/peripheral-generated-data.md),运行态读取边界见 [工作台运行态说明](docs/workbench-runtime.md)。
```bash
conda run -n server python -m pytest tests -q
@@ -67,6 +112,14 @@ docker compose -f infra/docker/docker-compose.yml config
- `docker build`:构建后端镜像。
- `docker compose config`:检查 compose 配置和变量展开。
开发时使用 `python scripts/run_server.py --reload` 启用进程交接式重载:父进程持有监听端口,
候选进程完成 ASGI 初始化、通过元数据库就绪检查并接管监听后,才通知旧进程停止接收新连接。
旧 HTTP 请求和 SSE 连接继续由旧进程完成,期间新请求由新进程响应;不迁移已建立的 TCP 连接,
也不通过强制取消旧请求来加快重载。候选启动失败或超出 `--startup-timeout`(默认 60 秒)时,
保留旧进程继续服务。快速连续修改最多保留两个正在排空的旧进程,其余修改合并后再重载。
默认监视 `app/` 和 `samples/peripheral/` 下的 Python 文件,可重复传入 `--reload-dir` 指定目录。
直接执行 `uvicorn ... --reload` 仍使用其原有先停后启流程;已经运行的旧启动命令不会自动切换到新机制。
## 开发规范
- Python 文件、函数、变量、Pydantic 字段、JSON body 字段和 query 参数使用 `snake_case`。
@@ -84,8 +137,8 @@ docker compose -f infra/docker/docker-compose.yml config
- 元数据、业务库和 TimescaleDB 可以部署在同一主机,也可以分别部署。
在线重算会原子替换一个时刻的节点、管段基线。历史分块已经压缩时,写入事务使用
`TIMESCALEDB_REALTIME_MAX_DECOMPRESSED_TUPLES` 设置解压预算,默认100万条;配置必须
为正整数,不允许0。该设置仅在实时快照写入事务内生效,不改变数据库全局设置或普通读取。
`TIMESCALEDB_REALTIME_MAX_DECOMPRESSED_TUPLES` 设置解压预算,默认 100 万条;配置必须
为正整数,不允许 0。该设置仅在实时快照写入事务内生效,不改变数据库全局设置或普通读取。
预算按需根据受影响压缩批次的规模调整,超过预算仍会失败并回滚当前快照;此前完成的其他
时刻不会一起回滚。同步、异步及单表批量快照入口使用同一预算。
@@ -117,6 +170,19 @@ conda run -n server python scripts/publish_fengxian_map_layers.py --database tjw
发布命令需要配置 `GEOSERVER_URL`、`GEOSERVER_USERNAME` 和 `GEOSERVER_PASSWORD`。其中 7 个基础管网图层仍由项目供应流程创建,奉贤发布脚本追加 9 个原始 GIS 图层和 1 个水质监测点图层。
## 工况任务工作进程
调查任务 worker 和外围工单同步器是独立进程,不随 API 进程自动启动:
```bash
conda run -n server python scripts/run_operational_worker.py
conda run -n server python scripts/sync_peripheral_work_orders.py
```
`OPERATIONAL_WORKER_PROJECT_IDS` 是逗号分隔的显式项目 UUID 列表,非空时优先于单项目兼容变量 `OPERATIONAL_WORKER_PROJECT_ID`。如需覆盖所有已启用项目,应将这些项目的 UUID 全部明确写入列表。进程不会自动枚举项目,配置列表也不会自动授权;服务身份必须是每个目标项目的有效成员。
每个项目保持独立的事件游标、会话、消息游标和 OAuth 缓存。`OPERATIONAL_WORKER_MAX_CONCURRENCY` 是每个项目的上限,而不是全局上限。更改项目列表后,需要同时重启 worker 和外围同步器。完整配置、授权和重试边界见 [工况工作进程说明](docs/operational-task-workers.md)。
## 测试与发布
提交前根据改动范围运行最小有效测试:
@@ -131,7 +197,15 @@ conda run -n server python -m pytest tests/unit tests/auth -q
docker build -t tjwater-server:local .
```
Gitea 包工作流位于 `.gitea/workflows/package.yml`,通常由 tag 触发构建、推送镜像并通知部署 webhook。
Dockerfile 的 `test` 阶段会执行 Python 编译检查、应用导入、OpenAPI 检查和完整 pytest。
Gitea Actions 工作流位于 `.gitea/workflows/package.yml`,可手动触发,也会在推送 `v*` tag 时运行。工作流构建 `test` 阶段,推送不可变的 `sha-*` 镜像,然后通过受限 SSH 账号对 `next-backend` 执行候选验证。只有 tag 工作流会在验证通过后提升正式版本。仓库需要以下 Actions secrets:
- `REGISTRY_USERNAME`
- `REGISTRY_PASSWORD`
- `DEV_DEPLOY_SSH_KEY`
这些密钥只用于镜像仓库和受限开发机部署,不替代后端应用的运行环境配置。
## 安全规则
@@ -139,4 +213,19 @@ Gitea 包工作流位于 `.gitea/workflows/package.yml`,通常由 tag 触发
## 产品 MCP 查询
Server 可选启用 MCP 2026-07-28 的25项项目查询及10项受控分析,默认关闭。它使用专用 OAuth resource Bearer、宿主绑定上下文及现有项目/任务权限,调查查询返回前持久化证据。启用需更新依赖、应用项目 SQL 迁移和配置 Keycloak token exchange;这些生产操作尚未执行。详见 [实施与启用](docs/mcp-query-implementation.md) 和 [协议/旧 CLI 能力对照](docs/mcp-query-design.md)。
Server 可选启用 MCP 2026-07-28,默认关闭。当前目录包含 67 项类型化工具,其中 42 项查询、12 项计算、13 项预览式写入。MCP 使用专用 OAuth resource Bearer、宿主绑定上下文和现有项目或任务权限,不提供自由命令执行。
写入工具默认只执行 `dry_run`。真实写入必须使用同一 `request_id`,带上预览返回的 `expected_digest`,并通过宿主原生许可和项目权限检查。超时或结果不明时先查询操作状态,不盲目重试。实施、启用和授权边界见 [MCP 实施与启用](docs/mcp-query-implementation.md) 和 [协议与旧 CLI 能力对照](docs/mcp-query-design.md)。
## 后台计算队列
长计算通过项目数据库队列和独立计算 Worker 执行,API 保持4个进程,HTTP超时不延长。接入需要新增数据库迁移和独立服务,详见[计算队列部署与MCP契约](docs/calculation-queue.md)。
后台调查通过 `POST /api/v1/operational-tasks/{task_id}/questions` 保存结构化问题,并沿现有任务通知展示。当前 run 可在没有最终报告时正常结束,任务进入等待人工;具备 `operational_task.write` 权限的用户通过 `questions/{question_id}/answer` 回答,原任务排队新一轮。回答保存为交互记录,不自动转为现场证据或审批。问题支持最多三个单选、多选或自定义回答,版本校验和请求幂等保护重复提交。任务详情返回最近 100 组提问,完整历史由 MCP 的 `get_task_context(section="questions")` 分页读取。
部署任务提问功能前,在各项目业务库执行 `resources/sql/028_task_questions.sql`,再同步 Server、Agent 与前端版本。迁移不修改既有调查记录;更新代码不会自动迁移数据库或重启服务。
外围工单来源统一为系统数据。部署对应版本前,在各项目业务库执行 `resources/sql/029_peripheral_work_orders_system_data.sql`,再同步 Server、同步器、Agent 与前端版本。迁移会修正既有外围源工单、修订记录、任务快照和外围反馈证据,不修改测试数据或模拟监测数据。
多 Agent 后台调查按实例归属运行。部署对应版本前,在每个启用调查 Worker 的项目 BizDB 执行 `resources/sql/030_operational_run_agent_ownership.sql`,再同步 Server、Agent 和 Worker,并为各 Agent 配置唯一且重启稳定的 `AGENT_INSTANCE_ID`。迁移只增加运行归属列和活跃运行索引;旧运行不猜测归属,待其结束或租约恢复后,新一代运行开始记录实例。
+8
View File
@@ -3,6 +3,14 @@ import os
import pandas as pd
def calculation_temp_directory(kind: str) -> str:
"""Keep native outputs inside the supervisor-owned job directory."""
root = os.environ.get("TJWATER_CALCULATION_JOB_DIR", "temp")
path = os.path.abspath(os.path.join(root, kind))
os.makedirs(path, exist_ok=True)
return path
def fill_time_gaps(
data: pd.DataFrame,
time_col: str = "time",
@@ -8,15 +8,14 @@ import sys
import pandas as pd
import wntr
from app.algorithms._utils import _cleanup_temp_files
from app.algorithms._utils import _cleanup_temp_files, calculation_temp_directory
_PIPE2LEAKNODE = None
_SIGNATURE_WORKER_DATA = {}
def _make_temp_prefix(tag):
temp_dir = os.path.abspath(os.path.join("temp", "burst_location"))
os.makedirs(temp_dir, exist_ok=True)
temp_dir = calculation_temp_directory("burst_location")
safe_tag = str(tag).replace(os.sep, "_").replace(" ", "_")
return os.path.join(temp_dir, f"{safe_tag}_{os.getpid()}")
@@ -15,7 +15,7 @@ from pymoo.optimize import minimize as pymoo_minimize
from pymoo.termination.default import DefaultSingleObjectiveTermination
from app.algorithms._utils import _cleanup_temp_files
from app.algorithms._utils import _cleanup_temp_files, calculation_temp_directory
_worker_data: dict[str, Any] = {}
DEFAULT_N_WORKERS = max(1, min(cpu_count() - 1, 4))
@@ -92,7 +92,7 @@ def _worker_evaluate(raw_ratios: np.ndarray) -> float:
demand_obj.base_value = original_val + per_node_leak
modifications.append((demand_obj, original_val))
temp_dir = os.path.abspath(os.path.join("temp", "leakage"))
temp_dir = calculation_temp_directory("leakage")
os.makedirs(temp_dir, exist_ok=True)
prefix = os.path.join(temp_dir, f"temp_{os.getpid()}")
@@ -569,7 +569,7 @@ class LeakageProblem(Problem):
modifications.append((demand_obj, original_val))
# 结果保存在根目录的temp/leakage文件夹中
temp_dir = os.path.abspath(os.path.join("temp", "leakage"))
temp_dir = calculation_temp_directory("leakage")
os.makedirs(temp_dir, exist_ok=True)
prefix = os.path.join(temp_dir, f"temp_{os.getpid()}")
@@ -1,10 +1,11 @@
"""Topology-only valve isolation search."""
from collections import defaultdict, deque
from typing import Any, Iterable
from typing import Any, Iterable, Literal
VALVE_LINK_TYPE = "valve"
AccidentElementType = Literal["auto", "link", "node"]
def _parse_link_entry(link_entry: str) -> tuple[str, str, str, str]:
@@ -14,12 +15,37 @@ def _parse_link_entry(link_entry: str) -> tuple[str, str, str, str]:
return parts[0], parts[1], parts[2], parts[3]
def _reachable_nodes(
adjacency: dict[str, list[tuple[str, str]]],
start_nodes: Iterable[str],
blocked_links: set[str],
) -> set[str]:
visited = set(start_nodes)
queue = deque(visited)
while queue:
for neighbor, link_id in adjacency.get(queue.popleft(), ()):
if link_id not in blocked_links and neighbor not in visited:
visited.add(neighbor)
queue.append(neighbor)
return visited
def valve_isolation_analysis(
link_entries: Iterable[str],
accident_elements: str | list[str],
disabled_valves: list[str] | None = None,
*,
accident_element_type: AccidentElementType = "auto",
water_source_nodes: Iterable[str] | None = None,
closed_links: Iterable[str] = (),
) -> dict[str, Any]:
"""Determine boundary valves and affected nodes from a topology snapshot."""
"""Find nearest valve boundaries, then check source connectivity.
Legacy untyped IDs prefer links; explicit node mode handles cross-type ID
collisions. Supply impact is a before/after connectivity difference, not
a hydraulic pressure or demand calculation. Without source metadata only
the provisional boundary search is available.
"""
disabled_valves_set = set(disabled_valves or [])
target_elements = (
@@ -27,8 +53,13 @@ def valve_isolation_analysis(
if isinstance(accident_elements, str)
else accident_elements
)
if not target_elements:
raise ValueError("At least one accident element is required")
if accident_element_type not in {"auto", "link", "node"}:
raise ValueError(f"Invalid accident element type: {accident_element_type}")
closed_links_set = set(closed_links)
pipe_adj: dict[str, set[str]] = defaultdict(set)
adjacency: dict[str, list[tuple[str, str]]] = defaultdict(list)
all_valves: dict[str, tuple[str, str]] = {}
link_lookup: dict[str, tuple[str, str, str]] = {}
node_set: set[str] = set()
@@ -39,46 +70,31 @@ def valve_isolation_analysis(
node_set.update((node1, node2))
if link_type_name == VALVE_LINK_TYPE:
all_valves[link_id] = (node1, node2)
else:
pipe_adj[node1].add(node2)
pipe_adj[node2].add(node1)
if link_id not in closed_links_set:
adjacency[node1].append((node2, link_id))
adjacency[node2].append((node1, link_id))
unknown_disabled = disabled_valves_set - all_valves.keys()
if unknown_disabled:
raise ValueError(f"Disabled valve was not found in topology: {sorted(unknown_disabled)}")
start_nodes: set[str] = set()
for element in target_elements:
if element in node_set:
start_nodes.add(element)
elif element in link_lookup:
if accident_element_type != "node" and element in link_lookup:
node1, node2, _ = link_lookup[element]
start_nodes.update((node1, node2))
elif accident_element_type != "link" and element in node_set:
start_nodes.add(element)
else:
raise ValueError(f"Accident element {element} was not found in topology")
extra_adj: dict[str, list[str]] = defaultdict(list)
boundary_valves: dict[str, tuple[str, str]] = {}
for valve_id, (node1, node2) in all_valves.items():
if valve_id in disabled_valves_set:
extra_adj[node1].append(node2)
extra_adj[node2].append(node1)
else:
boundary_valves[valve_id] = (node1, node2)
affected_nodes: set[str] = set()
queue = deque(start_nodes)
while queue:
node = queue.popleft()
if node in affected_nodes:
continue
affected_nodes.add(node)
queue.extend(pipe_adj.get(node, set()) - affected_nodes)
queue.extend(
neighbor
for neighbor in extra_adj.get(node, ())
if neighbor not in affected_nodes
)
closable_valves = set(all_valves) - disabled_valves_set - closed_links_set
affected_nodes = _reachable_nodes(adjacency, start_nodes, closable_valves)
must_close_valves: list[str] = []
optional_valves: list[str] = []
for valve_id, (node1, node2) in boundary_valves.items():
for valve_id in closable_valves:
node1, node2 = all_valves[valve_id]
node1_affected = node1 in affected_nodes
node2_affected = node2 in affected_nodes
if node1_affected and node2_affected:
@@ -89,14 +105,37 @@ def valve_isolation_analysis(
must_close_valves.sort()
optional_valves.sort()
isolatable = bool(must_close_valves)
failure_reason = None if isolatable else "no_boundary_valves"
sources = set(water_source_nodes or ())
source_check_performed = bool(sources)
supply_affected_nodes = None
if water_source_nodes is not None:
if not sources:
isolatable = False
failure_reason = "water_sources_unavailable"
else:
after_closure = _reachable_nodes(adjacency, sources, set(must_close_valves))
if affected_nodes & after_closure:
isolatable = False
failure_reason = "connected_to_water_source"
if isolatable:
before_closure = _reachable_nodes(adjacency, sources, set())
supply_affected_nodes = sorted(before_closure - after_closure)
result: dict[str, Any] = {
"accident_elements": target_elements,
"accident_element_type": accident_element_type,
"disabled_valves": disabled_valves,
"affected_nodes": sorted(affected_nodes) if isolatable else [],
"affected_node_count": len(affected_nodes),
"must_close_valves": must_close_valves,
"optional_valves": optional_valves,
"isolatable": isolatable,
"source_check_performed": source_check_performed,
"isolation_failure_reason": failure_reason,
"supply_affected_nodes": supply_affected_nodes,
"supply_affected_node_count": (
len(supply_affected_nodes) if supply_affected_nodes is not None else None
),
}
if len(target_elements) == 1:
result["accident_element"] = target_elements[0]
+4
View File
@@ -48,6 +48,8 @@ def _project_response(project: models.Project) -> AdminProjectResponse:
description=project.description,
gs_workspace=project.gs_workspace,
map_extent=project.map_extent,
map_config=getattr(project, "map_config", {}) or {},
settings=getattr(project, "settings", {}) or {},
status=project.status,
created_at=project.created_at,
updated_at=project.updated_at,
@@ -274,6 +276,8 @@ async def create_admin_project(
description=payload.description,
gs_workspace=payload.gs_workspace,
map_extent=payload.map_extent,
map_config=payload.map_config.model_dump(exclude_unset=True),
settings=payload.settings.model_dump(exclude_unset=True),
status=payload.status,
creator_user_id=current_user.id,
)
+2
View File
@@ -44,6 +44,8 @@ async def get_project_metadata(
description=project.description,
gs_workspace=project.gs_workspace,
map_extent=project.map_extent,
map_config=getattr(project, "map_config", {}) or {},
settings=getattr(project, "settings", {}) or {},
status=project.status,
project_role=ctx.project_role,
)
+2
View File
@@ -319,6 +319,8 @@ async def provision_project(
description=project.description,
gs_workspace=project.gs_workspace,
map_extent=project.map_extent,
map_config=getattr(project, "map_config", {}) or {},
settings=getattr(project, "settings", {}) or {},
status=project.status,
created_at=project.created_at,
updated_at=project.updated_at,
+87 -4
View File
@@ -7,6 +7,7 @@ from uuid import UUID
from fastapi import APIRouter, Depends, Header, HTTPException, Query, status
from psycopg import AsyncConnection
from app.auth.metadata_dependencies import get_metadata_repository
from app.auth.permissions import (
OPERATIONAL_TASK_EXECUTE,
@@ -22,10 +23,13 @@ from app.auth.project_dependencies import (
ProjectContext,
get_project_context,
get_project_pg_connection,
get_project_timescale_connection,
)
from app.domain.domain_errors import DomainError
from app.domain.schemas.workbench_runtime import WorkbenchDateTime
from app.domain.schemas.operational_data import TaskDataQuery, TaskDataSnapshot
from app.domain.schemas.task_question import TaskQuestionCreate, TaskQuestionReply, TaskQuestionRequest
from app.services.task_questions import create_question, answer_question
from app.domain.schemas.operational_task import (
ActionDecisionCommand,
ActionExecuteCommand,
@@ -74,6 +78,7 @@ from app.domain.schemas.operational_task import (
TaskPresentationEventCommand,
TaskPresentationEventResult,
TaskPresentationCursor,
TaskPresentationHistoryPage,
)
from app.infra.db.postgresql.operational_tasks import OperationalTaskRepository
from app.infra.integrations.operational_actions import (
@@ -87,6 +92,31 @@ from app.services.operational_tasks import OperationalTaskService
router = APIRouter()
@router.post("/operational-tasks/recovery/reconcile")
async def reconcile_task_recoveries(
command: RecoverExpiredLeasesCommand,
ctx: ProjectContext = Depends(require_permission(OPERATIONAL_TASK_EXECUTE)),
conn: AsyncConnection = Depends(get_project_pg_connection),
timescale=Depends(get_project_timescale_connection),
metadata=Depends(get_metadata_repository),
) -> dict[str, int]:
from app.domain.schemas.project_settings import ProjectSettings
from app.services.operational_recovery import reconcile_recoveries
if SCADA_VIEW not in permissions_for_context(ctx):
raise HTTPException(403, "恢复监测需要 SCADA 查看权限")
project = await metadata.get_project_by_id(ctx.project_id)
if project is None:
raise HTTPException(404, "当前项目不存在")
try:
resumed = await reconcile_recoveries(_service(ctx, conn), timescale,
ProjectSettings.model_validate(project.settings or {}), limit=command.limit,
created_from=command.task_created_from)
return {"resumed": resumed}
except DomainError as exc:
_raise_domain(exc)
def _service(ctx: ProjectContext, conn: AsyncConnection) -> OperationalTaskService:
return OperationalTaskService(
OperationalTaskRepository(conn, ctx.project_id),
@@ -521,6 +551,21 @@ async def get_operational_task_presentation_cursor(
_raise_domain(exc)
@router.get("/operational-tasks/{task_id}/presentation-events", response_model=TaskPresentationHistoryPage)
async def get_operational_task_presentation_history(
task_id: UUID,
after: int = Query(default=0, ge=0),
through: int | None = Query(default=None, ge=0),
limit: int = Query(default=100, ge=1, le=200),
ctx: ProjectContext = Depends(require_permission(OPERATIONAL_TASK_VIEW)),
conn: AsyncConnection = Depends(get_project_pg_connection),
):
try:
return await _service(ctx, conn).presentation_history(task_id, after=after, through=through, limit=limit)
except DomainError as exc:
_raise_domain(exc)
@router.post("/operational-tasks/{task_id}/presentation-events", response_model=TaskPresentationEventResult)
async def publish_operational_task_presentation_event(
task_id: UUID,
@@ -549,10 +594,24 @@ async def query_operational_task_data(
if not await service.can_read_source(task_id, query.source):
raise DomainError("PERMISSION_DENIED", "Task has no approved access to this data source", status=403)
source_permission = SCADA_VIEW if query.source.startswith("scada_") else SIMULATION_VIEW
snapshot = await read_task_data(
context=ctx, query=query, credential=authorization,
delegated=source_permission not in _task_permissions(ctx),
)
from app.services.scada_replay_queries import replay_snapshot_rows
replay_context = await service.replay_context(task_id)
if replay_context:
if query.source != "scada_readings":
raise HTTPException(403, "回放调查数据接口仅开放本轮 SCADA 快照,请通过 MCP 查询设备目录")
rows = await replay_snapshot_rows(conn, ctx.project_id, replay_context, query.device_ids,
query.start, query.end, 1001)
from app.services.content_digest import digest
from datetime import datetime, timezone
snapshot = TaskDataSnapshot(
source="scada_readings", source_ref=f"scada_replay:{replay_context['replay_id']}:{digest(rows)}",
queried_at=datetime.now(timezone.utc), origin="synthetic", data=rows[:1000], truncated=len(rows) > 1000,
)
else:
snapshot = await read_task_data(
context=ctx, query=query, credential=authorization,
delegated=source_permission not in _task_permissions(ctx),
)
evidence, stored_snapshot = await service.record_data_snapshot(task_id, snapshot)
return TaskDataQueryResult(evidence=evidence, snapshot=stored_snapshot)
except TaskDataError as exc:
@@ -771,3 +830,27 @@ async def withdraw_operational_task_external_source(
return await _service(ctx, conn).withdraw_external_source(task_id, command)
except DomainError as exc:
_raise_domain(exc)
@router.post("/operational-tasks/{task_id}/questions", response_model=TaskQuestionRequest)
async def create_operational_task_question(
task_id: UUID, command: TaskQuestionCreate,
ctx: ProjectContext = Depends(require_permission(OPERATIONAL_TASK_EXECUTE)),
conn: AsyncConnection = Depends(get_project_pg_connection),
):
try:
return await create_question(_service(ctx, conn), task_id, command)
except DomainError as exc:
_raise_domain(exc)
@router.post("/operational-tasks/{task_id}/questions/{question_id}/answer", response_model=OperationalTaskDetail)
async def answer_operational_task_question(
task_id: UUID, question_id: UUID, command: TaskQuestionReply,
ctx: ProjectContext = Depends(require_permission(OPERATIONAL_TASK_WRITE)),
conn: AsyncConnection = Depends(get_project_pg_connection),
):
try:
return await answer_question(_service(ctx, conn), task_id, question_id, command)
except DomainError as exc:
_raise_domain(exc)
+47 -6
View File
@@ -1,19 +1,60 @@
from fastapi import APIRouter, HTTPException
from fastapi import APIRouter, Depends, HTTPException
from pydantic import ValidationError
from app.domain.schemas.place_search import PlaceSearchRequest, PlaceSearchResponse
from app.auth.project_dependencies import ProjectContext, get_metadata_repository, get_project_context
from app.infra.db.metadb.repositories.metadata_repository import MetadataRepository
from app.domain.schemas.project_map_config import PlaceSearchConfig, ProjectMapConfig
from app.domain.schemas.place_search import PlaceSearchItem, PlaceSearchRequest, PlaceSearchResponse, ReverseGeocodingRequest
from app.services.place_search import PlaceSearchError, search_tianditu_places
from app.services.reverse_geocoding import reverse_geocode_tianditu
router = APIRouter()
async def get_place_search_config(
ctx: ProjectContext = Depends(get_project_context),
metadata_repo: MetadataRepository = Depends(get_metadata_repository),
) -> PlaceSearchConfig:
project = await metadata_repo.get_project_by_id(ctx.project_id)
if project is None:
raise HTTPException(status_code=404, detail="Project not found")
try:
config = ProjectMapConfig.model_validate(project.map_config or {}).place_search
except ValidationError:
raise HTTPException(status_code=503, detail="当前项目地名搜索配置无效") from None
if config is None or not config.enabled:
raise HTTPException(status_code=503, detail="当前项目暂未配置地名搜索")
return config
@router.post(
"/place-search-requests",
response_model=PlaceSearchResponse,
summary="Search Fengxian places",
description="调用天地图地名搜索,固定上海市奉贤区范围,只返回前 10 个可定位候选,不提供分页。",
summary="Search project places",
description="调用天地图地名搜索,按当前项目 map_config.place_search 的行政区范围,只返回前 10 个可定位候选,不提供分页。",
)
async def tianditu_place_search(request: PlaceSearchRequest) -> PlaceSearchResponse:
async def tianditu_place_search(
request: PlaceSearchRequest,
config: PlaceSearchConfig = Depends(get_place_search_config),
) -> PlaceSearchResponse:
try:
return await search_tianditu_places(request)
return await search_tianditu_places(request, config=config)
except PlaceSearchError as exc:
raise HTTPException(status_code=exc.status_code, detail=exc.detail) from exc
@router.post(
"/reverse-geocoding-requests",
response_model=PlaceSearchItem,
summary="Reverse geocode a project map point",
description="查询 EPSG:4326 地图取点的地名和地址,复用当前项目的地名搜索配置并保留输入坐标。",
)
async def tianditu_reverse_geocode(
request: ReverseGeocodingRequest,
config: PlaceSearchConfig = Depends(get_place_search_config),
) -> PlaceSearchItem:
try:
return await reverse_geocode_tianditu(request, config=config)
except PlaceSearchError as exc:
raise HTTPException(status_code=exc.status_code, detail=exc.detail) from exc
+2
View File
@@ -34,6 +34,8 @@ async def get_project_info_endpoint(
description=project_detail.description,
gs_workspace=project_detail.gs_workspace,
map_extent=project_detail.map_extent,
map_config=getattr(project_detail, "map_config", {}) or {},
settings=getattr(project_detail, "settings", {}) or {},
status=project_detail.status,
project_role="viewer", # Default role for public access
)
+115
View File
@@ -0,0 +1,115 @@
"""SCADA replay controls. Manual steps are processed by the command request."""
from uuid import UUID
from datetime import datetime
from fastapi import APIRouter, Depends, Query, HTTPException
from app.auth.metadata_dependencies import get_metadata_repository
from app.auth.permissions import SCADA_VIEW, SCADA_CLEAN, require_permission, permissions_for_context
from app.auth.project_dependencies import ProjectContext, get_project_pg_connection, get_project_timescale_connection
from app.domain.schemas.scada_replay import (
ScadaReplayCapabilities, ScadaReplayCreate, ScadaReplayCommand, ScadaReplaySnapshot,
ScadaReplayEventsPage, ScadaReplayReadings,
)
from app.infra.db.postgresql.scada_replays import ScadaReplayRepository
from app.domain.schemas.project_settings import ProjectSettings
from app.services.scada_replay import ScadaReplayService, snapshot
router = APIRouter(prefix="/scada-replays", tags=["SCADA Replay"])
async def repository(
ctx: ProjectContext = Depends(require_permission(SCADA_VIEW)),
connection=Depends(get_project_pg_connection),
):
return ScadaReplayRepository(connection, ctx.project_id)
async def project_settings(
ctx: ProjectContext = Depends(require_permission(SCADA_VIEW)),
metadata_repo=Depends(get_metadata_repository),
) -> ProjectSettings:
project = await metadata_repo.get_project_by_id(ctx.project_id)
if project is None:
raise HTTPException(404, "当前项目不存在")
return ProjectSettings.model_validate(project.settings or {})
@router.get("/capabilities", response_model=ScadaReplayCapabilities)
async def replay_capabilities(repo=Depends(repository), ctx=Depends(require_permission(SCADA_VIEW)),
settings=Depends(project_settings)):
available = await repo.available()
current = await repo.current() if available else None
monitoring = settings.scada_monitoring
monitoring_ready = bool(
monitoring
and len(monitoring.pressure_device_ids) >= 5
and (monitoring.decision_policy == "weighted" or monitoring.outlet_flow_device_ids)
)
if monitoring_ready:
monitoring_reason = None
elif monitoring is None:
monitoring_reason = "项目尚未配置压力监测角色及组合判定规则"
elif len(monitoring.pressure_device_ids) < 5:
monitoring_reason = "爆管侦测至少需要 5 个压力监测设备"
else:
monitoring_reason = "项目尚未配置出厂边界流量设备"
return ScadaReplayCapabilities(
available=available, can_write=SCADA_CLEAN in permissions_for_context(ctx),
reason=None if available else "需先部署 SCADA 回放存储结构",
monitoring_interval_seconds=settings.scada_monitoring_interval_seconds,
burst_monitoring_ready=monitoring_ready,
burst_monitoring_reason=monitoring_reason,
monitoring_config=monitoring,
current=snapshot(current) if current else None,
)
@router.post("", response_model=ScadaReplaySnapshot, status_code=201)
async def create_replay(command: ScadaReplayCreate, repo=Depends(repository),
ctx=Depends(require_permission(SCADA_CLEAN)), settings=Depends(project_settings)):
return await ScadaReplayService(repo, ctx).create(
command,
monitoring_interval_seconds=settings.scada_monitoring_interval_seconds,
monitoring_config=settings.scada_monitoring,
)
@router.get("/{replay_id}", response_model=ScadaReplaySnapshot)
async def get_replay(replay_id: UUID, repo=Depends(repository)):
await ScadaReplayService(repo, None).ensure_storage()
return snapshot(await repo.get(replay_id))
@router.post("/{replay_id}/commands", response_model=ScadaReplaySnapshot)
async def replay_command(replay_id: UUID, command: ScadaReplayCommand, repo=Depends(repository),
ctx=Depends(require_permission(SCADA_CLEAN)),
timescale=Depends(get_project_timescale_connection)):
service = ScadaReplayService(repo, ctx, timescale)
result = await service.command(replay_id, command)
if command.action in {"step", "monitor", "retry"} and result.step_pending:
return await service.process_pending_step(replay_id)
return result
@router.get("/{replay_id}/events", response_model=ScadaReplayEventsPage)
async def replay_events(replay_id: UUID, offset: int = Query(0, ge=0), limit: int = Query(100, ge=1, le=200),
repo=Depends(repository)):
await ScadaReplayService(repo, None).ensure_storage()
await repo.get(replay_id)
items = await repo.events(replay_id, offset, limit + 1)
return ScadaReplayEventsPage(items=items[:limit], next_offset=offset + limit if len(items) > limit else None)
@router.get("/{replay_id}/observations", response_model=ScadaReplayReadings)
async def replay_observations(replay_id: UUID, through: datetime, repo=Depends(repository)):
await ScadaReplayService(repo, None).ensure_storage()
document = await repo.get(replay_id)
if (through.tzinfo is None or not document.get("delivered_until")
or through > datetime.fromisoformat(document["delivered_until"])
or through < datetime.fromisoformat(document["start_time"])):
raise HTTPException(422, "回看仅允许本轮已读取的时间区间")
return ScadaReplayReadings(
items=await repo.latest_outputs(replay_id, through),
through=through,
)
+16 -4
View File
@@ -161,6 +161,9 @@ def valve_isolation_endpoint(
network: str = Query(..., description="管网名称(或数据库名称)"),
accident_element: List[str] = Query(..., description="发生事故的管段/节点ID列表"),
disabled_valves: List[str] = Query(None, description="已故障的阀门ID列表(可选)"),
accident_element_type: Literal["auto", "link", "node"] = Query(
"auto", description="事故要素类型;auto优先管段再节点;分析同名节点须指定node,纯管段分析可指定link",
),
):
"""
阀门隔离分析
@@ -168,13 +171,19 @@ def valve_isolation_endpoint(
- **network**: 管网名称(或数据库名称)
- **accident_element**: 发生事故的管段/节点ID列表
- **disabled_valves**: 已故障的阀门ID列表(可选)
- **accident_element_type**: auto优先管段;link仅匹配管段;node仅匹配节点
返回隔离方案,包括:
- must_close_valves: 必须关闭的阀门列表
- optional_valves: 可选关闭的阀门列表
- affected_nodes: 受影响的节点列表;不可隔离时为空列表
- affected_node_count: 受影响的节点总数
- isolatable: 是否可以有效隔离
- affected_nodes / affected_node_count: 最近阀门围成的直接隔离区;不可隔离时列表为空,计数保留
- supply_affected_nodes / supply_affected_node_count: 按建议关阀后新增失去水源连接的节点;未验证或不可隔离时为null
- source_check_performed: 是否已执行水源连通检查
- isolation_failure_reason: 不可隔离原因;可隔离时为null
- isolatable: 存在关阀边界,且直接隔离区不再连通任何水源
使用模型初始开关状态进行无向拓扑分析,水池按潜在水源处理。
不包含时变控制、水池蓄水量或压力流量计算,不是现场停水范围的水力证明。
"""
# result = {
# "accident_element": "P461309",
@@ -193,7 +202,10 @@ def valve_isolation_endpoint(
# "must_close_valves": ["210521658", "V12974", "V12986", "V12993"],
# "optional_valves": [],
# }
result = analyze_valve_isolation(network, accident_element, disabled_valves)
result = analyze_valve_isolation(
network, accident_element, disabled_valves,
accident_element_type=accident_element_type,
)
return result
+92
View File
@@ -1,9 +1,13 @@
from fastapi import APIRouter, Depends, HTTPException, Query, Path, Body
from typing import List
from datetime import datetime
from uuid import UUID
from psycopg import AsyncConnection
from pydantic import BaseModel, Field, field_validator
from app.domain.schemas.scada import ScadaLatestReadingsResponse, ScadaRawPage
from app.auth.project_dependencies import get_project_context, ProjectContext
from app.services.scada_raw import read_raw_page, validate_window
from app.infra.db.postgresql.scada import ScadaInfoRepository
from app.infra.db.timescaledb.repositories.scada import ScadaRepository
from .dependencies import get_postgres_connection, get_timescale_connection
@@ -12,6 +16,46 @@ router = APIRouter()
SCADA_BATCH_MAX_ITEMS = 10_000
@router.get("/timeseries/scada-readings/raw", response_model=ScadaRawPage,
summary="按时间顺序分页读取 SCADA 原始值")
async def get_scada_raw(
start_time: datetime, end_time: datetime, device_ids: str,
limit: int = Query(500, ge=1, le=500),
cursor: str | None = Query(None, max_length=2048),
replay_id: UUID | None = None,
conn: AsyncConnection = Depends(get_timescale_connection),
postgres_conn: AsyncConnection = Depends(get_postgres_connection),
ctx: ProjectContext = Depends(get_project_context),
) -> ScadaRawPage:
try:
validate_window(start_time, end_time)
except ValueError as exc:
raise HTTPException(422, str(exc)) from exc
ids = list(dict.fromkeys(item.strip() for item in device_ids.split(",") if item.strip()))
if not ids or len(ids) > 500:
raise HTTPException(422, "请选择 1 至 500 个 SCADA 设备")
existing = await ScadaInfoRepository.get_existing_device_ids(postgres_conn, ids)
if set(ids) != existing:
raise HTTPException(422, "查询包含当前项目未配置的 SCADA 设备")
try:
through = None
if replay_id is not None:
from app.infra.db.postgresql.scada_replays import ScadaReplayRepository
from app.services.scada_replay import ScadaReplayService
repo = ScadaReplayRepository(postgres_conn, ctx.project_id)
await ScadaReplayService(repo, ctx).ensure_storage()
replay = await repo.get(replay_id)
if not set(ids) <= set(replay["config"]["device_ids"]):
raise HTTPException(403, "设备超出回放范围")
if start_time < datetime.fromisoformat(replay["start_time"]) or end_time > datetime.fromisoformat(replay["end_time"]):
raise HTTPException(422, "查询超出选定回放区间")
through = datetime.fromisoformat(replay["virtual_now"])
return await read_raw_page(conn, project_id=str(ctx.project_id), device_ids=ids,
start=start_time, end=end_time, limit=limit, cursor=cursor, through=through)
except ValueError as exc:
raise HTTPException(422, str(exc)) from exc
class ScadaReadingBatchItem(BaseModel):
time: datetime
device_id: str = Field(min_length=1)
@@ -96,6 +140,54 @@ async def get_scada_by_ids_time_range(
)
@router.get(
"/timeseries/scada-readings/latest",
response_model=ScadaLatestReadingsResponse,
summary="获取SCADA设备最新数据",
)
async def get_latest_scada_readings(
limit: int = Query(100, ge=1, le=500, description="返回设备数上限"),
offset: int = Query(0, ge=0, description="设备目录偏移量"),
conn: AsyncConnection = Depends(get_timescale_connection),
postgres_conn: AsyncConnection = Depends(get_postgres_connection),
) -> ScadaLatestReadingsResponse:
"""Read one latest persisted row per project device without applying a time window."""
devices = await ScadaInfoRepository.get_scadas(postgres_conn)
selected_devices = devices[offset : offset + limit]
latest_by_device = await ScadaRepository.get_latest_scada_readings(
conn,
[device["device_id"] for device in selected_devices],
)
items = []
for device in selected_devices:
reading = latest_by_device.get(device["device_id"])
items.append(
{
"device_id": device["device_id"],
"device_type": device["device_type"],
"measurement_unit": device["measurement_unit"],
"transmission_mode": device["transmission_mode"],
"transmission_frequency": device["transmission_frequency"],
"observed_at": reading["time"] if reading else None,
"monitored_value": reading["monitored_value"] if reading else None,
"cleaned_value": reading["cleaned_value"] if reading else None,
}
)
observed_times = [item["observed_at"] for item in items if item["observed_at"] is not None]
return ScadaLatestReadingsResponse.model_validate(
{
"items": items,
"total": len(devices),
"limit": limit,
"offset": offset,
"observed_count": len(observed_times),
"latest_observed_at": max(observed_times) if observed_times else None,
}
)
@router.get(
"/timeseries/scada-readings/fields", summary="按设备ID、字段和时间范围查询SCADA数据"
)
+2
View File
@@ -61,8 +61,10 @@ from app.auth.permissions import (
require_method_permission,
require_permission,
)
from app.api.v1.endpoints import scada_replay
api_router = APIRouter()
api_router.include_router(scada_replay.router)
webgis_access = Depends(
require_method_permission(
+20 -2
View File
@@ -2,7 +2,10 @@ from datetime import datetime
from typing import Literal
from uuid import UUID
from pydantic import BaseModel, ConfigDict, Field, model_validator
from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator
from app.domain.schemas.project_map_config import ProjectMapConfig, ProjectMapConfigUpdate
from app.domain.schemas.project_settings import ProjectSettings
BusinessRole = Literal["admin", "service", "user"]
@@ -70,15 +73,21 @@ class ProjectMemberResponse(BaseModel):
class AdminProjectCreateRequest(BaseModel):
settings: ProjectSettings = Field(default_factory=ProjectSettings)
name: str = Field(..., min_length=1, max_length=100)
code: str = Field(..., min_length=1, max_length=50)
description: str | None = None
gs_workspace: str = Field(..., min_length=1, max_length=100)
map_extent: dict | None = None
map_config: ProjectMapConfig = Field(default_factory=ProjectMapConfig)
status: ProjectStatus = "active"
class AdminProjectUpdateRequest(BaseModel):
class AdminProjectUpdateRequest(ProjectMapConfigUpdate):
settings: ProjectSettings | None = Field(
default=None,
description="按顶层键合并项目配置;省略的键保持原值,嵌套对象整体替换",
)
name: str | None = Field(default=None, min_length=1, max_length=100)
code: str | None = Field(default=None, min_length=1, max_length=50)
description: str | None = None
@@ -86,14 +95,23 @@ class AdminProjectUpdateRequest(BaseModel):
map_extent: dict | None = None
status: ProjectStatus | None = None
@field_validator("settings")
@classmethod
def reject_null_settings(cls, value: ProjectSettings | None) -> ProjectSettings:
if value is None:
raise ValueError("settings must be an object")
return value
class AdminProjectResponse(BaseModel):
settings: ProjectSettings = Field(default_factory=ProjectSettings)
project_id: UUID
name: str
code: str
description: str | None = None
gs_workspace: str
map_extent: dict | None = None
map_config: ProjectMapConfig = Field(default_factory=ProjectMapConfig)
status: str
created_at: datetime
updated_at: datetime
+146
View File
@@ -0,0 +1,146 @@
"""Validated, local presentation blocks. Component props come from the frontend catalog."""
from __future__ import annotations
import json
import math
import re
from datetime import datetime
from pathlib import Path
from typing import Annotated, Any, Literal
from jsonschema import Draft7Validator
from pydantic import BaseModel, ConfigDict, Field, field_validator, model_serializer, model_validator
CHART_SAMPLE_TIME_PATTERN = (
r"^(?:[1-9][0-9]{3}|0[1-9][0-9]{2}|00[1-9][0-9]|000[1-9])-[0-9]{2}-[0-9]{2}T(?:[01][0-9]|2[0-3]):[0-5][0-9]:[0-5][0-9]"
r"(?:\.[0-9]+)?(?:Z|[+-](?:[01][0-9]|2[0-3]):[0-5][0-9])$"
)
def validate_chart_sample_time(value: str) -> None:
"""Match the bounded frontend RFC3339 spelling and an actual calendar date."""
if len(value) > 64 or re.fullmatch(CHART_SAMPLE_TIME_PATTERN, value) is None:
raise ValueError("Chart sample times require RFC3339 seconds and a timezone")
if datetime.fromisoformat(value).tzinfo is None:
raise ValueError("Chart sample times require a timezone")
_catalog = json.loads(
(Path(__file__).resolve().parents[3] / "contracts/answer-ui-catalog.json").read_text()
)
_validators = {name: Draft7Validator(schema) for name, schema in _catalog["props"].items()}
Key = Annotated[str, Field(pattern=r"^[a-zA-Z0-9_-]{1,80}$")]
class UiModel(BaseModel):
model_config = ConfigDict(extra="forbid")
class AnswerUiElement(UiModel):
type: str
props: dict[str, Any]
children: list[Key] = Field(max_length=48)
class AnswerUiSpec(UiModel):
root: Key
elements: dict[Key, AnswerUiElement]
class AnswerUiBlock(UiModel):
version: Literal["answer-ui@1"]
block_id: Key
phase: Literal["process", "final"] | None = None
fallback_text: Annotated[str, Field(min_length=1, max_length=4000)]
spec: AnswerUiSpec
@field_validator("phase", mode="before")
@classmethod
def validate_explicit_phase(cls, value):
if value is None:
raise ValueError("UI phase must be process or final when supplied")
return value
@model_serializer(mode="wrap")
def serialize_optional_phase(self, handler):
data = handler(self)
if self.phase is None:
data.pop("phase", None)
return data
@model_validator(mode="after")
def validate_components(self):
spec = self.spec
if not 1 <= len(spec.elements) <= 48:
raise ValueError("UI requires 1 to 48 elements")
# Match the existing JavaScript serialized character limit, including UTF-16.
serialized = json.dumps(self.model_dump(), ensure_ascii=False, separators=(",", ":"))
if len(serialized.encode("utf-16-le")) // 2 > 100000:
raise ValueError("UI block is too large")
for element in spec.elements.values():
validator = _validators.get(element.type)
if validator is None or not validator.is_valid(element.props):
raise ValueError("Unknown UI component or invalid properties")
if element.children and element.type not in {"Stack", "Section"}:
raise ValueError("Leaf components cannot contain children")
props = element.props
if element.type == "Chart":
if any(len(series["data"]) != len(props["x_data"]) for series in props["series"]):
raise ValueError("Chart series must match the axis")
if any(value is not None and not math.isfinite(value) for series in props["series"] for value in series["data"]):
raise ValueError("Chart values must be finite")
for series in props["series"]:
times = series.get("sample_times")
if times is not None:
if len(times) != len(series["data"]):
raise ValueError("Chart sample times must match the axis")
for value, at in zip(series["data"], times, strict=True):
if at is not None:
if value is None:
raise ValueError("Chart sample times require an observed value")
validate_chart_sample_time(at)
if element.type == "Table" and any(len(row) != len(props["columns"]) for row in props["rows"]):
raise ValueError("Table row width must match its columns")
if element.type == "FollowUpForm":
names = [field["name"] for field in props["fields"]]
if len(names) != len(set(names)):
raise ValueError("Form field names must be unique")
visited: set[str] = set()
def visit(key: str, depth: int):
if depth > 12 or key in visited or key not in spec.elements:
raise ValueError("UI must be a bounded tree without duplicate or missing references")
visited.add(key)
for child in spec.elements[key].children:
visit(child, depth + 1)
visit(spec.root, 0)
if len(visited) != len(spec.elements):
raise ValueError("UI contains unreachable elements")
return self
class ReportUiBlock(UiModel):
block: AnswerUiBlock
evidence_refs: list[Annotated[str, Field(min_length=1, max_length=128)]] = Field(min_length=1, max_length=100)
@model_validator(mode="after")
def validate_report_phase(self):
if self.block.phase == "process":
raise ValueError("Report UI blocks must contain final results")
return self
def validate_report_ui_blocks(blocks: list[ReportUiBlock], evidence_refs: list[str]) -> None:
ids = [item.block.block_id for item in blocks]
if len(ids) != len(set(ids)):
raise ValueError("Report UI block ids must be unique")
serialized = json.dumps([item.model_dump() for item in blocks], ensure_ascii=False, separators=(",", ":"))
if len(serialized.encode("utf-8")) > 400000:
raise ValueError("Report UI exceeds 400 KB")
for item in blocks:
if not set(item.evidence_refs) <= set(evidence_refs):
raise ValueError("UI evidence must be included in report evidence")
for element in item.block.spec.elements.values():
if element.type == "Evidence" and element.props.get("evidence_id") not in item.evidence_refs:
raise ValueError("UI evidence component must reference its block evidence")
+6 -1
View File
@@ -1,16 +1,21 @@
from typing import Optional
from uuid import UUID
from pydantic import BaseModel
from pydantic import BaseModel, Field
from app.domain.schemas.project_map_config import ProjectMapConfig
from app.domain.schemas.project_settings import ProjectSettings
class ProjectMetaResponse(BaseModel):
settings: ProjectSettings = Field(default_factory=ProjectSettings)
project_id: UUID
name: str
code: str
description: Optional[str] = None
gs_workspace: str
map_extent: Optional[dict] = None
map_config: ProjectMapConfig = Field(default_factory=ProjectMapConfig)
status: str
project_role: str
+34
View File
@@ -16,6 +16,39 @@ TaskDataSource = Literal[
McpSnapshotSource = Literal[
'get_mutation_status',
'query_nearby_elements',
'list_regions',
'get_region',
'get_region_nodes',
'get_node_connections',
'query_network_extent',
'list_major_nodes',
'get_model_configuration',
'list_model_configurations',
'search_project_places',
'reverse_geocode',
'get_project_info',
'export_project_model',
'get_sensor_placement_candidate',
'export_sensor_placement',
'list_task_schedules',
'get_task_run_history',
'calculate_demand_allocation',
'run_burst_simulation',
'edit_model_element',
'edit_model_configuration',
'clean_scada_readings',
'write_scada_readings',
'delete_scada_readings',
'save_simulation_scenario',
'create_sensor_placement',
'update_sensor_placement',
'create_operational_task',
'create_task_schedule',
'update_task_schedule',
'retry_operational_task',
'cancel_task_run',
'run_simulation',
'run_contaminant_simulation',
'analyze_valve_closure',
@@ -25,6 +58,7 @@ McpSnapshotSource = Literal[
'run_burst_detection',
'run_burst_location',
'analyze_valve_isolation',
'generate_node_coverage',
'run_leakage_identification',
'get_element_properties',
'list_elements',
+154 -9
View File
@@ -5,8 +5,12 @@ import re
from typing import Annotated, Literal
from uuid import UUID
from pydantic import Field, JsonValue, model_validator
from pydantic import Field, JsonValue, model_serializer, model_validator
from app.domain.schemas.answer_ui import (
CHART_SAMPLE_TIME_PATTERN, ReportUiBlock, validate_chart_sample_time, validate_report_ui_blocks,
)
from app.domain.schemas.task_question import TaskQuestionRequest
from app.domain.schemas.shared import NonEmptyStr, PositiveInt, StrictBool, StrictModel
from app.domain.schemas.operational_data import TaskDataSnapshot, TaskDataSource
from app.domain.schemas.workbench_runtime import (
@@ -39,12 +43,19 @@ ExecutionStatus = Literal[
]
class ScadaReplayTaskContext(StrictModel):
replay_id: UUID
knowledge_cutoff: WorkbenchDateTime
batch_sequence: PositiveInt
class TaskSource(StrictModel):
kind: TaskSourceKind
external_id: NonEmptyStr | None = None
external_revision: PositiveInt | None = None
event_kind: NonEmptyStr | None = None
deleted_at: WorkbenchDateTime | None = None
replay_context: ScadaReplayTaskContext | None = None
class TaskTrigger(StrictModel):
@@ -217,8 +228,12 @@ class InvestigationDiagramNodeRef(StrictModel):
evidence_refs: list[NonEmptyStr] = Field(default_factory=list)
action_refs: list[UUID] = Field(default_factory=list)
status: Literal[
"pending", "running", "completed", "waiting_human", "failed", "skipped", "cancelled"
] | None = None
"pending", "running", "completed", "waiting", "waiting_human", "failed", "skipped", "cancelled"
] | None = Field(default=None, description=(
"Evidence-confirmed node state; waiting means prerequisites/external/system result, "
"waiting_human requires manual authorization/approval/feedback. "
"Unknown stays omitted/null; host controls colors."
))
map_targets: list[InvestigationMapTarget] = Field(default_factory=list, max_length=100)
@@ -239,6 +254,21 @@ class InvestigationDiagram(StrictModel):
return self
def _undeclared_diagram_node_refs(diagram: InvestigationDiagram | None) -> list[str]:
if diagram is None:
return []
missing: list[str] = []
for node in diagram.node_refs:
escaped = re.escape(node.node_id)
if not re.search(
rf"(?<![A-Za-z0-9_-]){escaped}(?=\s*(?:[\[({{@;]|:::|--|==|-\.|$))",
diagram.mermaid,
re.MULTILINE,
):
missing.append(node.node_id)
return missing
class InvestigationReport(StrictModel):
"""Read model: absent historical facts remain unknown, never inferred."""
@@ -253,9 +283,11 @@ class InvestigationReport(StrictModel):
stop_reason: NonEmptyStr | None = None
data_gaps: list[InvestigationDataGap] | None = None
diagram: InvestigationDiagram | None = None
ui_blocks: list["ReportUiBlock"] = Field(default_factory=list, max_length=8)
@model_validator(mode="after")
def validate_result(self):
validate_report_ui_blocks(self.ui_blocks, self.evidence_refs)
if self.outcome is None:
if self.stage is not None or self.stop_reason is not None or self.data_gaps is not None:
raise ValueError("structured report fields must be supplied together")
@@ -338,9 +370,19 @@ class AgentArtifactsCommand(StrictModel):
progress: InvestigationProgress | None = None
@model_validator(mode="after")
def require_artifact(self):
def validate_artifacts(self):
if not self.evidence and self.plan is None and self.report is None and self.progress is None:
raise ValueError("at least one artifact must be supplied")
for diagram in (
self.report.diagram if self.report is not None else None,
self.progress.diagram if self.progress is not None else None,
):
missing = _undeclared_diagram_node_refs(diagram)
if missing:
raise ValueError(
"diagram node references must use ids declared in Mermaid source: "
+ ", ".join(missing)
)
return self
@@ -499,7 +541,7 @@ class ExternalWorkOrderSnapshot(StrictModel):
scenario_code: NonEmptyStr | None = None
feedback_requirements: list[NonEmptyStr] = Field(default_factory=list)
activity: list[WorkOrderActivity] = Field(default_factory=list)
origin: Literal["synthetic"]
origin: Literal["system_data"]
updated_at: WorkbenchDateTime
@model_validator(mode="before")
@@ -518,6 +560,7 @@ class OperationalTaskRun(StrictModel):
status: OperationalTaskRunStatus
revision: PositiveInt
agent_session_id: NonEmptyStr | None = None
agent_instance_id: NonEmptyStr | None = None
lease_expires_at: WorkbenchDateTime | None = None
heartbeat_at: WorkbenchDateTime | None = None
started_at: WorkbenchDateTime | None = None
@@ -547,6 +590,7 @@ class OperationalTaskSummary(StrictModel):
execution_request_id: UUID
title: NonEmptyStr
summary: NonEmptyStr
handling_context: dict[str, JsonValue] = Field(default_factory=dict)
source: TaskSource
trigger: TaskTrigger
scheduled_for: WorkbenchDateTime | None
@@ -567,6 +611,7 @@ class OperationalTaskPermissions(StrictModel):
can_decide_commands: bool
can_execute_commands: bool
can_decide_access: bool = False
can_answer_questions: bool = False
class OperationalTaskDetail(OperationalTaskSummary):
@@ -576,6 +621,7 @@ class OperationalTaskDetail(OperationalTaskSummary):
evidence: list[EvidenceRecord]
actions: list[HumanApprovalAction]
access_requests: list[TaskAccessRequest] = Field(default_factory=list)
questions: list[TaskQuestionRequest] = Field(default_factory=list)
permissions: OperationalTaskPermissions
work_order_snapshot: ExternalWorkOrderSnapshot | None = None
@@ -602,6 +648,7 @@ class AgentRunCommand(StrictModel):
request_id: UUID
expected_revision: PositiveInt
source_session_id: NonEmptyStr
agent_instance_id: NonEmptyStr | None = None
class AgentRunFailureCommand(AgentRunCommand):
@@ -648,15 +695,42 @@ class AgentArtifactsResult(StrictModel):
class TaskSafeChartSeries(StrictModel):
name: Annotated[str, Field(max_length=128)]
data: Annotated[list[Annotated[float, Field(strict=True, allow_inf_nan=False)]], Field(min_length=1, max_length=100)]
type: Literal["line", "bar", "scatter"] | None = None
data: Annotated[list[Annotated[float, Field(strict=True, allow_inf_nan=False)] | None], Field(min_length=1, max_length=100)]
sample_times: Annotated[list[Annotated[str, Field(
strict=True, max_length=64,
pattern=CHART_SAMPLE_TIME_PATTERN,
)] | None], Field(min_length=1, max_length=100)] | None = None
@model_serializer(mode="wrap")
def serialize_optional_fields(self, handler):
return {key: value for key, value in handler(self).items() if value is not None}
@model_validator(mode="after")
def validate_sample_times(self):
if self.sample_times is not None:
if len(self.sample_times) != len(self.data):
raise ValueError("chart sample_times must align with data")
for value, at in zip(self.data, self.sample_times, strict=True):
if at is not None:
if value is None:
raise ValueError("chart sample_times require an observed value")
validate_chart_sample_time(at)
return self
class TaskSafeChart(StrictModel):
title: Annotated[str, Field(max_length=200)]
chart_type: Literal["line", "bar"]
x_axis_name: Annotated[str, Field(max_length=128)] | None = None
y_axis_name: Annotated[str, Field(max_length=128)] | None = None
x_data: Annotated[list[Annotated[str, Field(max_length=128)]], Field(min_length=1, max_length=100)]
series: Annotated[list[TaskSafeChartSeries], Field(min_length=1, max_length=8)]
@model_serializer(mode="wrap")
def serialize_optional_fields(self, handler):
return {key: value for key, value in handler(self).items() if value is not None}
@model_validator(mode="after")
def validate_alignment(self):
if any(len(series.data) != len(self.x_data) for series in self.series):
@@ -668,7 +742,59 @@ class TaskHighlightFeatures(StrictModel):
action: Literal["highlight_features"]
layer: TaskMapLayer
feature_ids: Annotated[list[TaskFeatureId], Field(min_length=1, max_length=100)]
effect: Literal["steady", "pulse"] = "steady"
effect: Literal["steady", "pulse"]
class TaskLocateFeatures(StrictModel):
action: Literal["locate_features"]
layer: TaskMapLayer
feature_ids: Annotated[list[TaskFeatureId], Field(min_length=1, max_length=100)]
class TaskZoomToMap(StrictModel):
action: Literal["zoom_to_map"]
center: tuple[
Annotated[float, Field(strict=True, ge=-180, le=180, allow_inf_nan=False)],
Annotated[float, Field(strict=True, ge=-90, le=90, allow_inf_nan=False)],
]
zoom: Annotated[float, Field(strict=True, ge=0, le=24, allow_inf_nan=False)]
class TaskScadaAnalysisItem(StrictModel):
device_id: TaskFeatureId
level: Literal["high", "medium", "low", "unrated"]
class TaskRenderScadaAnalysis(StrictModel):
action: Literal["render_scada_analysis"]
items: Annotated[list[TaskScadaAnalysisItem], Field(min_length=1, max_length=100)]
@model_validator(mode="after")
def validate_unique_devices(self):
if len({item.device_id for item in self.items}) != len(self.items):
raise ValueError("SCADA analysis device IDs must be unique")
return self
class TaskClearScadaAnalysis(StrictModel):
action: Literal["clear_scada_analysis"]
class TaskOverlayGeometry(StrictModel):
type: Literal["Point", "MultiPoint", "LineString", "MultiLineString", "Polygon", "MultiPolygon"]
coordinates: list[JsonValue]
class TaskOverlayFeature(StrictModel):
type: Literal["Feature"]
geometry: TaskOverlayGeometry
properties: dict[str, JsonValue] | None
id: Annotated[str, Field(strict=True)] | Annotated[float, Field(strict=True, allow_inf_nan=False)] | None = None
class TaskOverlayData(StrictModel):
type: Literal["FeatureCollection"]
features: Annotated[list[TaskOverlayFeature], Field(min_length=1, max_length=1000)]
class TaskClearHighlights(StrictModel):
@@ -693,6 +819,7 @@ class TaskSetOverlay(StrictModel):
title: Annotated[str, Field(min_length=1, max_length=200)]
mode: Literal["geometry", "heatmap"]
evidence_id: NonEmptyStr
data: TaskOverlayData | None = None
class TaskClearOverlay(StrictModel):
@@ -701,7 +828,8 @@ class TaskClearOverlay(StrictModel):
TaskPresentationCommand = Annotated[
TaskHighlightFeatures | TaskClearHighlights | TaskOpenChart | TaskCloseChart | TaskSetOverlay | TaskClearOverlay,
TaskHighlightFeatures | TaskLocateFeatures | TaskZoomToMap | TaskRenderScadaAnalysis
| TaskClearScadaAnalysis | TaskClearHighlights | TaskOpenChart | TaskCloseChart | TaskSetOverlay | TaskClearOverlay,
Field(discriminator="action"),
]
@@ -733,6 +861,23 @@ class TaskPresentationCursor(StrictModel):
event_cursor: int = Field(ge=0)
class TaskPresentationHistoryItem(StrictModel):
event_seq: PositiveInt
run_id: UUID
execution_generation: PositiveInt
occurred_at: WorkbenchDateTime
# Stored records are validated against the current command contract by the
# reader; an obsolete record must not prevent reading the rest of the page.
command: dict[str, JsonValue]
class TaskPresentationHistoryPage(StrictModel):
task_id: UUID
event_cursor: int = Field(ge=0)
items: list[TaskPresentationHistoryItem]
next_after: int | None = Field(default=None, ge=0)
class TaskDataQueryResult(StrictModel):
evidence: EvidenceRecord
snapshot: TaskDataSnapshot
@@ -813,7 +958,7 @@ class OperationalTaskNotification(StrictModel):
description: NonEmptyStr
required_permission: NonEmptyStr | None = None
lifecycle_status: Literal["active", "resolved"] = "active"
subject_type: Literal["task", "run", "access_request", "action"] = "task"
subject_type: Literal["task", "run", "access_request", "action", "question"] = "task"
subject_id: UUID | None = None
report_reference: InvestigationReportReference | None = None
resolved_at: WorkbenchDateTime | None = None
+9 -4
View File
@@ -1,5 +1,3 @@
from typing import Literal
from pydantic import BaseModel, ConfigDict, Field
MAX_PLACE_SEARCH_RESULTS = 10
@@ -11,6 +9,13 @@ class PlaceSearchRequest(BaseModel):
keyword: str = Field(..., min_length=1, max_length=120)
class ReverseGeocodingRequest(BaseModel):
model_config = ConfigDict(extra="forbid")
longitude: float = Field(ge=-180, le=180, allow_inf_nan=False)
latitude: float = Field(ge=-90, le=90, allow_inf_nan=False)
class PlaceSearchItem(BaseModel):
id: str = Field(..., min_length=1)
name: str = Field(..., min_length=1)
@@ -18,11 +23,11 @@ class PlaceSearchItem(BaseModel):
longitude: float = Field(..., ge=-180, le=180, allow_inf_nan=False)
latitude: float = Field(..., ge=-90, le=90, allow_inf_nan=False)
category: str | None = None
district: Literal["奉贤区"] = "奉贤区"
district: str = Field(min_length=1)
class PlaceSearchResponse(BaseModel):
items: list[PlaceSearchItem] = Field(..., max_length=MAX_PLACE_SEARCH_RESULTS)
total: int = Field(..., ge=0, le=MAX_PLACE_SEARCH_RESULTS)
truncated: bool = False
region: Literal["奉贤区"] = "奉贤区"
region: str = Field(min_length=1)
+29
View File
@@ -0,0 +1,29 @@
from typing import Literal
from pydantic import BaseModel, ConfigDict, Field, model_validator
class PlaceSearchConfig(BaseModel):
model_config = ConfigDict(extra="forbid", str_strip_whitespace=True)
enabled: bool = False
provider: Literal["tianditu"] = "tianditu"
admin_code: str = Field(pattern=r"^(?:156)?[0-9]{6}$")
region: str = Field(min_length=1, max_length=100)
class ProjectMapConfig(BaseModel):
# Preserve existing map options such as legend when editing search settings.
model_config = ConfigDict(extra="allow")
place_search: PlaceSearchConfig | None = None
class ProjectMapConfigUpdate(BaseModel):
map_config: ProjectMapConfig | None = None
@model_validator(mode="after")
def reject_null_map_config(self):
if "map_config" in self.model_fields_set and self.map_config is None:
raise ValueError("map_config must be an object")
return self
+64
View File
@@ -0,0 +1,64 @@
from math import isclose
from typing import Literal
from pydantic import BaseModel, ConfigDict, Field, JsonValue, model_validator
class ScadaMonitoringWeights(BaseModel):
model_config = ConfigDict(extra="forbid", allow_inf_nan=False)
burst_detection: float = Field(default=0.5, ge=0, le=1)
pressure: float = Field(default=0.4, ge=0, le=1)
outlet_flow: float = Field(default=0.1, ge=0, le=1)
@model_validator(mode="after")
def normalized(self):
if not isclose(self.burst_detection + self.pressure + self.outlet_flow, 1.0, abs_tol=1e-9):
raise ValueError("SCADA 三项监测权重之和必须为 1")
return self
class ScadaMonitoringSettings(BaseModel):
"""Project-owned device roles used by the manual SCADA monitoring demo."""
model_config = ConfigDict(extra="forbid", allow_inf_nan=False)
rule_version: str = Field(
default="burst-monitor-v2",
min_length=1,
max_length=80,
pattern=r"^[a-zA-Z0-9_.-]+$",
)
pressure_device_ids: list[str] = Field(default_factory=list, max_length=20)
outlet_flow_device_ids: list[str] = Field(default_factory=list, max_length=20)
decision_policy: Literal["all_signals", "weighted"] = "weighted"
weights: ScadaMonitoringWeights = Field(default_factory=ScadaMonitoringWeights)
alert_threshold: float = Field(default=0.8, gt=0, le=1)
@model_validator(mode="after")
def distinct_roles(self):
for values in (self.pressure_device_ids, self.outlet_flow_device_ids):
if len(set(values)) != len(values) or any(not value.strip() for value in values):
raise ValueError("SCADA 监测角色中的设备标识不能为空或重复")
if set(self.pressure_device_ids) & set(self.outlet_flow_device_ids):
raise ValueError("同一设备不能同时作为压力与出厂边界流量")
return self
class ProjectSettings(BaseModel):
"""Extensible, non-secret project parameters stored in MetaDB as JSON."""
model_config = ConfigDict(extra="allow", allow_inf_nan=False)
__pydantic_extra__: dict[str, JsonValue] = Field(init=False)
scada_monitoring_interval_seconds: int | None = Field(
default=None,
strict=True,
ge=60,
le=31_536_000,
description="SCADA 主动监测演示步长,单位秒;null 表示未配置",
)
scada_monitoring: ScadaMonitoringSettings | None = Field(
default=None,
description="主动监测使用的项目设备角色;不创建定时任务",
)
+48 -1
View File
@@ -1,5 +1,7 @@
from pydantic import BaseModel
from datetime import datetime
from typing import Literal
from pydantic import BaseModel, ConfigDict, Field
class ScadaDeviceResponse(BaseModel):
"""Project SCADA metadata keyed by the canonical device identifier."""
@@ -17,3 +19,48 @@ class ScadaDeviceResponse(BaseModel):
y: float | None = None
longitude: float | None = None
latitude: float | None = None
class ScadaLatestReadingResponse(BaseModel):
"""Latest persisted observation for one project SCADA device."""
model_config = ConfigDict(extra="forbid", allow_inf_nan=False)
device_id: str
device_type: str
measurement_unit: str
transmission_mode: str
transmission_frequency: str
observed_at: datetime | None = None
monitored_value: float | None = None
cleaned_value: float | None = None
class ScadaLatestReadingsResponse(BaseModel):
"""Bounded device page with each device's latest persisted observation."""
model_config = ConfigDict(extra="forbid", allow_inf_nan=False)
items: list[ScadaLatestReadingResponse]
total: int = Field(ge=0)
limit: int = Field(ge=1, le=500)
offset: int = Field(ge=0)
observed_count: int = Field(ge=0)
latest_observed_at: datetime | None = None
class ScadaRawReading(BaseModel):
model_config = ConfigDict(extra="forbid", allow_inf_nan=False)
time: datetime
device_id: str
monitored_value: float | None
quality: Literal["unvalidated", "missing", "invalid"] = "unvalidated"
class ScadaRawPage(BaseModel):
model_config = ConfigDict(extra="forbid", allow_inf_nan=False)
items: list[ScadaRawReading]
next_cursor: str | None
scanned_through: datetime
+169
View File
@@ -0,0 +1,169 @@
"""Persisted replay controls and explicit, unit-aware demonstration rules."""
from datetime import datetime, timedelta
from typing import Literal
from uuid import UUID
from pydantic import BaseModel, ConfigDict, Field, model_validator
from app.domain.schemas.project_settings import ScadaMonitoringSettings
class ReplayModel(BaseModel):
model_config = ConfigDict(extra="forbid", allow_inf_nan=False)
class ScadaReplayRule(ReplayModel):
rule_id: str = Field(min_length=1, max_length=80, pattern=r"^[a-zA-Z0-9_-]+$")
device_ids: list[str] = Field(min_length=1, max_length=500)
kind: Literal["relative_change", "bounds"]
direction: Literal["both", "increase", "decrease"] = "both"
threshold: float | None = Field(None, gt=0)
lower: float | None = None
upper: float | None = None
unit: str = Field(min_length=1, max_length=40)
metric_name: str = Field(default="", max_length=80)
minimum_samples: int = Field(3, ge=1, le=96)
window_samples: int = Field(4, ge=1, le=96)
consecutive_samples: int = Field(1, ge=1, le=20)
recovery_samples: int = Field(2, ge=1, le=20)
minimum_devices: int = Field(1, ge=1, le=500)
@model_validator(mode="after")
def parameters(self):
if len(set(self.device_ids)) != len(self.device_ids):
raise ValueError("规则内设备不能重复")
if self.minimum_devices > len(self.device_ids):
raise ValueError("同时异常设备数超过规则设备数")
if self.kind == "relative_change" and (self.threshold is None or self.minimum_samples > self.window_samples):
raise ValueError("变化率规则需要阈值和足够的历史窗口")
if self.kind == "bounds" and self.lower is None and self.upper is None:
raise ValueError("上下限规则至少配置一个阈值")
if self.lower is not None and self.upper is not None and self.lower >= self.upper:
raise ValueError("规则下限必须小于上限")
return self
class ScadaQualityRange(ReplayModel):
lower: float | None = None
upper: float | None = None
@model_validator(mode="after")
def bounds(self):
if self.lower is not None and self.upper is not None and self.lower >= self.upper:
raise ValueError("仪表量程下限必须小于上限")
return self
class ScadaReplayCreate(ReplayModel):
request_id: UUID
start_time: datetime
end_time: datetime
device_ids: list[str] = Field(min_length=1, max_length=500)
rules: list[ScadaReplayRule] = Field(min_length=1, max_length=30)
quality_ranges: dict[str, ScadaQualityRange] = Field(default_factory=dict)
@model_validator(mode="after")
def configuration(self):
if (self.start_time.tzinfo is None or self.end_time.tzinfo is None
or not timedelta(0) < self.end_time - self.start_time <= timedelta(days=31)):
raise ValueError("请选择带时区、最长 31 天的有效时间区间")
ids = set(self.device_ids)
if len(ids) != len(self.device_ids) or any(not value.strip() for value in ids):
raise ValueError("设备标识不能为空或重复")
if len({rule.rule_id for rule in self.rules}) != len(self.rules):
raise ValueError("规则标识不能重复")
if any(not set(rule.device_ids) <= ids for rule in self.rules) or not set(self.quality_ranges) <= ids:
raise ValueError("规则及量程设备必须属于本次回放")
return self
class ScadaReplayCommand(ReplayModel):
request_id: UUID
expected_revision: int = Field(ge=1)
action: Literal["step", "monitor", "stop", "retry"]
at_time: datetime | None = None
@model_validator(mode="after")
def monitoring_time(self):
if self.action == "monitor":
if self.at_time is None or self.at_time.tzinfo is None:
raise ValueError("选定时刻监测需要提供带时区的 at_time")
elif self.at_time is not None:
raise ValueError("只有选定时刻监测可以指定 at_time")
return self
class ScadaMonitoringIncident(ReplayModel):
incident_id: UUID
status: Literal["active", "resolved"]
action: Literal["raised", "ongoing", "resolved"]
started_at: datetime
last_observed_at: datetime
recovery_count: int = Field(ge=0, le=2)
class ScadaMonitoringResult(ReplayModel):
monitoring_id: UUID
period_start: datetime
period_end: datetime
knowledge_cutoff: datetime
batch_sequence: int = Field(ge=1)
rule_version: str = Field(min_length=1, max_length=80)
status: Literal["normal", "attention", "insufficient_data", "suspected_burst"]
detail: str = Field(min_length=1, max_length=1000)
task_id: UUID | None = None
pressure: dict
outlet_flow: dict
burst_detection: dict
weighted: dict | None = None
incident: ScadaMonitoringIncident | None = None
role_gaps: list[str] = Field(default_factory=list)
anomaly_signals: list[dict] = Field(default_factory=list)
class ScadaReplaySnapshot(ReplayModel):
replay_id: UUID
project_id: UUID
revision: int
status: Literal["ready", "paused", "completed", "stopped", "failed"]
start_time: datetime
end_time: datetime
monitoring_interval_seconds: int | None = Field(None, ge=60, le=31_536_000)
step_pending: bool = False
monitoring_count: int = 0
last_monitoring: ScadaMonitoringResult | None = None
monitoring_config: ScadaMonitoringSettings | None = None
monitoring_calibration: dict | None = None
virtual_now: datetime
delivered_until: datetime | None = None
processed_until: datetime | None = None
processed_count: int = 0
anomaly_count: int = 0
devices: list[dict]
rules: list[ScadaReplayRule]
latest: list[dict]
active_events: list[dict]
error: str | None = None
created_at: datetime
updated_at: datetime
class ScadaReplayCapabilities(ReplayModel):
available: bool
can_write: bool
reason: str | None = None
monitoring_interval_seconds: int | None = Field(None, ge=60, le=31_536_000)
burst_monitoring_ready: bool = False
burst_monitoring_reason: str | None = None
monitoring_config: ScadaMonitoringSettings | None = None
current: ScadaReplaySnapshot | None = None
class ScadaReplayEventsPage(ReplayModel):
items: list[dict]
next_offset: int | None
class ScadaReplayReadings(ReplayModel):
items: list[dict]
through: datetime
+3 -3
View File
@@ -5,7 +5,7 @@ from typing import Literal
from uuid import UUID
from app.domain.schemas.shared import NonEmptyStr, PositiveInt, StrictModel
from app.domain.schemas.workbench_runtime import WorkbenchAlertSource, WorkbenchDateTime
from app.domain.schemas.workbench_runtime import WorkbenchAlertSource, WorkbenchScadaReplaySource, WorkbenchDateTime
class TaskAlertCommand(StrictModel):
@@ -14,7 +14,7 @@ class TaskAlertCommand(StrictModel):
event_type: Literal["raised", "acknowledged", "resolved"]
occurred_at: WorkbenchDateTime
condition_id: NonEmptyStr | None
source: WorkbenchAlertSource
source: WorkbenchAlertSource | WorkbenchScadaReplaySource
title: NonEmptyStr
description: NonEmptyStr
severity: Literal["warning", "critical"]
@@ -29,7 +29,7 @@ class TaskAlertState(StrictModel):
revision: PositiveInt
producer_id: NonEmptyStr
condition_id: NonEmptyStr | None
source: WorkbenchAlertSource
source: WorkbenchAlertSource | WorkbenchScadaReplaySource
title: NonEmptyStr
description: NonEmptyStr
occurred_at: WorkbenchDateTime
+75
View File
@@ -0,0 +1,75 @@
"""Durable investigation questions; answers are user input, not field evidence."""
from typing import Annotated, Literal
from uuid import UUID
from pydantic import Field, model_validator
from app.domain.schemas.shared import StrictModel, PositiveInt
from app.domain.schemas.workbench_runtime import WorkbenchDateTime
Id = Annotated[str, Field(min_length=1, max_length=128)]
class TaskQuestionOption(StrictModel):
id: Id
label: Annotated[str, Field(min_length=1, max_length=200)]
class TaskQuestion(StrictModel):
question_id: Id
title: Annotated[str, Field(min_length=1, max_length=200)]
body: Annotated[str, Field(min_length=1, max_length=2000)]
options: list[TaskQuestionOption] = Field(max_length=12)
mode: Literal["single", "multiple"]
allow_custom: bool
required: bool
@model_validator(mode="after")
def valid_options(self):
if len({o.id for o in self.options}) != len(self.options):
raise ValueError("duplicate option ids")
if not self.options and not self.allow_custom:
raise ValueError("question must offer an answer input")
return self
class TaskQuestionAnswer(StrictModel):
question_id: Id
selected_option_ids: list[Id] = Field(max_length=12)
custom_text: Annotated[str, Field(max_length=2000)]
class TaskQuestionCreate(StrictModel):
request_id: UUID
source_session_id: Id
run_id: UUID
questions: list[TaskQuestion] = Field(min_length=1, max_length=3)
@model_validator(mode="after")
def unique_questions(self):
if len({q.question_id for q in self.questions}) != len(self.questions):
raise ValueError("duplicate question ids")
if len(self.model_dump_json().encode()) > 32768:
raise ValueError("question definition too large")
return self
class TaskQuestionReply(StrictModel):
request_id: UUID
expected_version: PositiveInt
answers: list[TaskQuestionAnswer] = Field(max_length=3)
class TaskQuestionRequest(StrictModel):
question_request_id: UUID
task_id: UUID
run_id: UUID
execution_generation: PositiveInt
source_session_id: Id
questions: list[TaskQuestion]
status: Literal["pending", "answered", "cancelled"]
version: PositiveInt
created_at: WorkbenchDateTime
answered_at: WorkbenchDateTime | None = None
answered_by: str | None = None
answers: list[TaskQuestionAnswer] = Field(default_factory=list)
+9 -2
View File
@@ -49,10 +49,17 @@ class WorkbenchAlertSource(StrictModel):
run_id: NonEmptyStr
class WorkbenchScadaReplaySource(StrictModel):
kind: Literal["scada_replay"]
replay_id: NonEmptyStr
rule_id: NonEmptyStr
batch_sequence: int = Field(ge=1)
class WorkbenchRuntimeAlert(StrictModel):
id: NonEmptyStr
condition_id: NonEmptyStr | None
source: WorkbenchAlertSource
source: WorkbenchAlertSource | WorkbenchScadaReplaySource
title: NonEmptyStr
description: NonEmptyStr
occurred_at: WorkbenchDateTime
@@ -62,7 +69,7 @@ class WorkbenchRuntimeAlert(StrictModel):
class WorkbenchPersistedAlert(StrictModel):
alert_id: NonEmptyStr
condition_id: NonEmptyStr | None
source: WorkbenchAlertSource
source: WorkbenchAlertSource | WorkbenchScadaReplaySource
title: NonEmptyStr
description: NonEmptyStr
occurred_at: WorkbenchDateTime
+97
View File
@@ -0,0 +1,97 @@
"""Database resources owned by the standalone calculation worker."""
from __future__ import annotations
from contextlib import asynccontextmanager
from uuid import UUID
from sqlalchemy import text
async def discover_project_ids() -> tuple[str, ...]:
"""Return every active PostgreSQL-backed project from authoritative metadata."""
from sqlalchemy import select
from app.infra.db.metadb.database import SessionLocal
from app.infra.db.metadb import models
async with SessionLocal() as session:
rows = await session.scalars(
select(models.Project.id)
.join(
models.ProjectDatabase,
models.ProjectDatabase.project_id == models.Project.id,
)
.where(
models.Project.status == "active",
models.ProjectDatabase.db_role == "biz_data",
models.ProjectDatabase.db_type == "postgresql",
)
.order_by(models.Project.id)
)
return tuple(str(project_id) for project_id in rows.all())
@asynccontextmanager
async def queue_repository(project_id):
"""Resolve the current project route and open its calculation queue."""
from app.core.config import settings
from app.infra.db.dynamic_manager import project_connection_manager
from app.infra.db.metadb.database import SessionLocal
from app.infra.db.metadb.repositories.metadata_repository import MetadataRepository
from app.infra.db.postgresql.calculation_jobs import CalculationJobRepository
project_uuid = UUID(str(project_id))
async with SessionLocal() as session:
metadata = MetadataRepository(session)
project = await metadata.get_project_by_id(project_uuid)
if not project or project.status != "active":
raise ValueError("Calculation project is not active")
routing = await metadata.get_project_db_routing(project_uuid, "biz_data")
if not routing or routing.db_type != "postgresql":
raise ValueError("Project business PostgreSQL routing is unavailable")
await session.commit()
pool_min_size = (
routing.pool_min_size
if routing.pool_min_size is not None
else settings.PROJECT_PG_POOL_MIN_SIZE
)
pool_max_size = (
routing.pool_max_size
if routing.pool_max_size is not None
else settings.PROJECT_PG_POOL_SIZE
)
async with project_connection_manager.pg_connection(
project_uuid,
"biz_data",
routing.dsn,
pool_min_size,
pool_max_size,
) as connection:
yield CalculationJobRepository(connection, project_id)
class CalculationSupervisorLease:
"""Keep the singleton advisory lock alive on one metadata connection."""
def __init__(self, connection):
self.connection = connection
async def ping(self) -> None:
await self.connection.execute(text("SELECT 1"))
await self.connection.commit()
@asynccontextmanager
async def calculation_supervisor_lease():
"""Acquire and release the session-level calculation supervisor lock."""
from app.infra.db.metadb.database import engine
async with engine.connect() as connection:
acquired = await connection.scalar(text("SELECT pg_try_advisory_lock(1748531781, 2)"))
await connection.commit()
if not acquired:
raise RuntimeError("Another calculation supervisor is already active")
try:
yield CalculationSupervisorLease(connection)
finally:
await connection.execute(text("SELECT pg_advisory_unlock(1748531781, 2)"))
await connection.commit()
+2
View File
@@ -43,6 +43,8 @@ class Project(Base):
description: Mapped[str | None] = mapped_column(Text, nullable=True)
gs_workspace: Mapped[str] = mapped_column(String(100), unique=True)
map_extent: Mapped[dict | None] = mapped_column(JSONB, nullable=True)
map_config: Mapped[dict] = mapped_column(JSONB, nullable=False, default=dict, server_default="{}")
settings: Mapped[dict] = mapped_column(JSONB, nullable=False, default=dict, server_default="{}")
status: Mapped[str] = mapped_column(String(20), default="active")
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), default=datetime.utcnow
@@ -1,4 +1,4 @@
from dataclasses import dataclass
from dataclasses import dataclass, field
from datetime import datetime, timezone
from typing import Optional, List
from uuid import UUID, uuid4
@@ -66,6 +66,8 @@ class ProjectDetail:
gs_workspace: str
map_extent: Optional[dict]
status: str
map_config: dict = field(default_factory=dict)
settings: dict = field(default_factory=dict)
@dataclass(frozen=True)
@@ -221,6 +223,8 @@ class MetadataRepository:
map_extent: dict | None,
status: str,
creator_user_id: UUID | None = None,
map_config: dict | None = None,
settings: dict | None = None,
) -> models.Project:
project = models.Project(
id=uuid4(),
@@ -230,6 +234,8 @@ class MetadataRepository:
gs_workspace=gs_workspace,
map_extent=map_extent,
status=status,
map_config=map_config or {},
settings=settings or {},
created_at=_utcnow(),
updated_at=_utcnow(),
)
@@ -260,6 +266,7 @@ class MetadataRepository:
timescale_dsn: str,
pool_min_size: int = 1,
pool_max_size: int = 4,
settings: dict | None = None,
) -> models.Project:
"""Atomically expose a fully provisioned project and both DB routes."""
business_secret = _encrypt_database_secret(business_dsn)
@@ -272,6 +279,8 @@ class MetadataRepository:
gs_workspace=gs_workspace,
map_extent=map_extent,
status="active",
map_config={},
settings=settings or {},
created_at=_utcnow(),
updated_at=_utcnow(),
)
@@ -327,6 +336,10 @@ class MetadataRepository:
):
if field in updates:
setattr(project, field, updates[field])
if "map_config" in updates:
project.map_config = {**(project.map_config or {}), **updates["map_config"]}
if "settings" in updates:
project.settings = {**(project.settings or {}), **updates["settings"]}
project.updated_at = _utcnow()
await self.session.commit()
await self.session.refresh(project)
@@ -343,6 +356,8 @@ class MetadataRepository:
description=project.description,
gs_workspace=project.gs_workspace,
map_extent=project.map_extent,
map_config=project.map_config or {},
settings=project.settings or {},
status=project.status,
)
+144
View File
@@ -0,0 +1,144 @@
"""Durable project calculation queue. Every worker update is generation fenced."""
from uuid import uuid4
from fastapi import HTTPException
from psycopg.types.json import Jsonb
from app.mcp.errors import CalculationCapacityExceeded
# Explicit metadata projection prevents status polling from decoding large JSON results.
JOB_COLUMNS = "job_id,project_id,owner_user_id,request_key,tool,parameters,binding,timeout_seconds,status,error,lease_generation,lease_owner,lease_expires_at,created_at,updated_at,started_at,finished_at,result_expires_at"
PUBLIC_JOB_COLUMNS = "job_id,status,tool,error,created_at,updated_at,started_at,finished_at,result_expires_at"
class CalculationJobRepository:
def __init__(self, connection, project_id):
self.connection = connection
self.project_id = str(project_id)
async def maintain(self):
await self.connection.execute(
"""UPDATE workbench.calculation_jobs SET result=NULL
WHERE project_id=%s AND result IS NOT NULL AND result_expires_at<=clock_timestamp()""",
(self.project_id,),
)
async def enqueue(self, *, owner_user_id, request_key, tool, parameters, binding, timeout_seconds,
max_active_jobs=100):
async with self.connection.transaction():
return await self._enqueue(owner_user_id=owner_user_id, request_key=request_key,
tool=tool, parameters=parameters, binding=binding, timeout_seconds=timeout_seconds,
max_active_jobs=max_active_jobs)
async def _enqueue(self, *, owner_user_id, request_key, tool, parameters, binding,
timeout_seconds, max_active_jobs):
# Serialize admission within the caller's transaction, across every API process.
await self.connection.execute(
"SELECT pg_advisory_xact_lock(hashtextextended(%s, 1748531781))", (self.project_id,))
await self.maintain()
cursor = await self.connection.execute(
f"SELECT {JOB_COLUMNS} FROM workbench.calculation_jobs WHERE project_id=%s AND owner_user_id=%s AND request_key=%s",
(self.project_id, owner_user_id, request_key),
)
existing = await cursor.fetchone()
if existing:
row = dict(existing)
if (row['tool'] != tool
or {key: value for key, value in row['parameters'].items() if key != 'timeout_seconds'}
!= {key: value for key, value in parameters.items() if key != 'timeout_seconds'}
or row['binding'] != binding):
raise HTTPException(409, 'Calculation request_key was used with different inputs')
return row
cursor = await self.connection.execute(
"""SELECT count(*) AS active FROM workbench.calculation_jobs WHERE project_id=%s
AND status IN ('queued','running','cancel_requested')""", (self.project_id,))
if (await cursor.fetchone())['active'] >= max_active_jobs:
raise CalculationCapacityExceeded()
cursor = await self.connection.execute(
f"""INSERT INTO workbench.calculation_jobs
(job_id,project_id,owner_user_id,request_key,tool,parameters,binding,timeout_seconds)
VALUES (%s,%s,%s,%s,%s,%s,%s,%s)
RETURNING {JOB_COLUMNS}""",
(uuid4(), self.project_id, owner_user_id, request_key, tool,
Jsonb(parameters), Jsonb(binding), timeout_seconds),
)
return dict(await cursor.fetchone())
async def get(self, job_id, owner_user_id=None, *, include_result=False):
await self.maintain()
columns = JOB_COLUMNS + (",result" if include_result else "")
cursor = await self.connection.execute(
f"""SELECT {columns} FROM workbench.calculation_jobs WHERE project_id=%s AND job_id=%s
AND (%s::uuid IS NULL OR owner_user_id=%s)""",
(self.project_id, job_id, owner_user_id, owner_user_id),
)
row = await cursor.fetchone()
return dict(row) if row else None
async def list(self, *, owner_user_id, binding, limit, offset, status=None):
await self.maintain()
cursor = await self.connection.execute(
f"""SELECT {PUBLIC_JOB_COLUMNS} FROM workbench.calculation_jobs WHERE project_id=%s AND owner_user_id=%s
AND binding @> %s AND (%s::text IS NULL OR status=%s) ORDER BY created_at DESC,job_id LIMIT %s OFFSET %s""",
(self.project_id, owner_user_id, Jsonb(binding), status, status, limit, offset),
)
return [dict(row) for row in await cursor.fetchall()]
async def cancel(self, job_id, owner_user_id):
await self.connection.execute(
"""UPDATE workbench.calculation_jobs SET status=CASE WHEN status='queued' THEN 'cancelled'
ELSE 'cancel_requested' END,finished_at=CASE WHEN status='queued' THEN clock_timestamp() ELSE finished_at END,updated_at=clock_timestamp()
WHERE project_id=%s AND job_id=%s AND owner_user_id=%s AND status IN ('queued','running')""",
(self.project_id, job_id, owner_user_id),
)
return await self.get(job_id, owner_user_id)
async def recover(self):
await self.maintain()
# A crashed write is never automatically replayed.
await self.connection.execute(
"""UPDATE workbench.calculation_jobs SET status='interrupted',
error='{"code":"WORKER_INTERRUPTED"}'::jsonb,finished_at=clock_timestamp(),updated_at=clock_timestamp()
WHERE project_id=%s AND status IN ('running','cancel_requested')
AND lease_expires_at < clock_timestamp()""", (self.project_id,),
)
async def claim(self, owner, lease_seconds):
await self.maintain()
cursor = await self.connection.execute(
"""UPDATE workbench.calculation_jobs j SET status='running',started_at=clock_timestamp(),lease_owner=%s,
lease_generation=j.lease_generation+1,
lease_expires_at=clock_timestamp()+%s*interval '1 second',updated_at=clock_timestamp()
WHERE j.job_id=(SELECT job_id FROM workbench.calculation_jobs
WHERE project_id=%s AND status='queued' ORDER BY created_at,job_id
FOR UPDATE SKIP LOCKED LIMIT 1) RETURNING j.*""",
(owner, lease_seconds, self.project_id),
)
row = await cursor.fetchone()
return dict(row) if row else None
async def heartbeat(self, row, owner, lease_seconds):
cursor = await self.connection.execute(
"""UPDATE workbench.calculation_jobs SET lease_expires_at=clock_timestamp()+%s*interval '1 second',
updated_at=clock_timestamp() WHERE project_id=%s AND job_id=%s AND lease_owner=%s
AND lease_generation=%s AND lease_expires_at>clock_timestamp()
AND status IN ('running','cancel_requested') RETURNING status""",
(lease_seconds, self.project_id, row['job_id'], owner, row['lease_generation']),
)
result = await cursor.fetchone()
return result['status'] if result else None
async def finish(self, row, owner, status, *, result=None, error=None, result_retention_seconds=604800):
cursor = await self.connection.execute(
"""UPDATE workbench.calculation_jobs SET status=%s,result=%s,error=%s,finished_at=clock_timestamp(),
result_expires_at=clock_timestamp()+%s*interval '1 second',updated_at=clock_timestamp()
WHERE project_id=%s AND job_id=%s AND lease_owner=%s AND lease_generation=%s
AND lease_expires_at>clock_timestamp() AND status IN ('running','cancel_requested')
AND (status!='cancel_requested' OR %s IN ('cancelled','interrupted')
OR (%s='failed' AND %s='CLEANUP_FAILED')) RETURNING job_id""",
(status, Jsonb(result) if result is not None else None, Jsonb(error) if error else None, result_retention_seconds,
self.project_id, row['job_id'], owner, row['lease_generation'], status, status, (error or {}).get('code')),
)
return await cursor.fetchone() is not None
+7
View File
@@ -89,3 +89,10 @@ class McpQueryRepository:
"SELECT pg_advisory_xact_lock(hashtextextended(%s,0))",
(f"mcp:{self.project_id}:{scope_key}:{call_id}",),
)
async def finish_receipt(self, scope_key: str, call_id: str, result: dict):
async with self.conn.cursor() as cur:
await cur.execute(
"UPDATE workbench.mcp_query_receipts SET result=%s WHERE project_id=%s AND scope_key=%s AND call_id=%s",
(Jsonb(result), self.project_id, scope_key, call_id),
)
+18
View File
@@ -6,6 +6,24 @@ from psycopg import AsyncConnection
class NetworkAssetRepository:
@staticmethod
async def get_isolation_model(conn: AsyncConnection) -> list[dict[str, Any]]:
"""Read source nodes, link state and pipe lengths in one model snapshot."""
from app.infra.db.postgresql.read_queries import read_rows
return await read_rows(conn, """
SELECT l.id, l.link_type::text AS kind,
l.start_node_id, l.end_node_id, p.length, p.diameter,
COALESCE(s.status::text, p.status::text, '') AS status
FROM network.links l
LEFT JOIN network.pipes p ON p.link_id = l.id
LEFT JOIN network.link_initial_settings s ON s.link_id = l.id
UNION ALL
SELECT n.id, n.node_type::text, NULL, NULL, NULL, NULL, ''
FROM network.nodes n
WHERE n.node_type::text IN ('reservoir', 'tank')
""")
@staticmethod
async def get_pipes_by_ids(
conn: AsyncConnection,
+113 -10
View File
@@ -19,6 +19,7 @@ class OperationalTaskRepository:
def __init__(self, connection: AsyncConnection, project_id):
self.connection = connection
self.project_id = str(project_id)
self._agent_ownership_available: bool | None = None
@asynccontextmanager
async def transaction(self):
@@ -75,6 +76,48 @@ class OperationalTaskRepository:
status=409,
)
async def pending_calculation_jobs(
self, task_id: str, run_id: str, execution_request_id: str,
) -> list[str]:
"""Only the exact original run blocks final artifacts and completion."""
cursor = await self.connection.execute(
"""SELECT job_id FROM workbench.calculation_jobs
WHERE project_id=%s AND binding->>'task_id'=%s
AND binding->>'run_id'=%s AND binding->>'execution_request_id'=%s
AND status IN ('queued','running','cancel_requested')
ORDER BY created_at,job_id LIMIT 100""",
(self.project_id, task_id, run_id, execution_request_id),
)
return [str(row["job_id"]) for row in await cursor.fetchall()]
async def list_questions(self, task_id: str) -> list[dict]:
cursor = await self.connection.execute(
"""SELECT document FROM workbench.operational_task_questions
WHERE project_id=%s AND task_id=%s
ORDER BY document->>'created_at' DESC, question_request_id DESC LIMIT 100""",
(self.project_id, task_id),
)
return [row["document"] for row in await cursor.fetchall()]
async def get_question(self, task_id: str, question_id: str) -> dict | None:
cursor = await self.connection.execute(
"""SELECT document FROM workbench.operational_task_questions
WHERE project_id=%s AND task_id=%s AND question_request_id=%s""",
(self.project_id, task_id, question_id),
)
row = await cursor.fetchone()
return row["document"] if row else None
async def save_question(self, task_id: str, document: dict) -> None:
# All writers hold the owning task advisory lock.
await self.connection.execute(
"""INSERT INTO workbench.operational_task_questions
(project_id,task_id,question_request_id,document) VALUES (%s,%s,%s,%s)
ON CONFLICT (project_id,question_request_id) DO UPDATE SET document=excluded.document
WHERE operational_task_questions.task_id=excluded.task_id""",
(self.project_id, task_id, document["question_request_id"], Jsonb(document)),
)
async def get_task(self, task_id: str) -> dict | None:
cursor = await self.connection.execute(
"""SELECT document FROM workbench.operational_tasks
@@ -87,7 +130,8 @@ class OperationalTaskRepository:
async def get_active_run(self, task_id: str) -> dict | None:
cursor = await self.connection.execute(
"""SELECT run_id,task_id,generation,execution_request_id,status,revision,
agent_session_id,lease_expires_at,heartbeat_at,started_at,finished_at,
agent_session_id,to_jsonb(operational_task_runs)->>'agent_instance_id' AS agent_instance_id,
lease_expires_at,heartbeat_at,started_at,finished_at,
failure_code,failure_detail,dispatch_attempts,next_dispatch_at,
created_at,updated_at
FROM workbench.operational_task_runs
@@ -102,7 +146,8 @@ class OperationalTaskRepository:
async def get_run(self, run_id: str) -> dict | None:
cursor = await self.connection.execute(
"""SELECT run_id,task_id,generation,execution_request_id,status,revision,
agent_session_id,lease_expires_at,heartbeat_at,started_at,finished_at,
agent_session_id,to_jsonb(operational_task_runs)->>'agent_instance_id' AS agent_instance_id,
lease_expires_at,heartbeat_at,started_at,finished_at,
failure_code,failure_detail,dispatch_attempts,next_dispatch_at,
created_at,updated_at
FROM workbench.operational_task_runs
@@ -129,13 +174,23 @@ class OperationalTaskRepository:
async def require_execution_storage(self) -> None:
"""Reject claims before changing state if the inbox cannot be used."""
cursor = await self.connection.execute(
"SELECT to_regclass('workbench.agent_messages') IS NOT NULL AS available"
"""SELECT to_regclass('workbench.agent_messages') IS NOT NULL
AND to_regclass('workbench.operational_task_questions') IS NOT NULL AS available,
EXISTS (
SELECT 1 FROM pg_attribute
WHERE attrelid=to_regclass('workbench.operational_task_runs')
AND attname='agent_instance_id' AND NOT attisdropped
) AS agent_ownership_available"""
)
if not (await cursor.fetchone())["available"]:
readiness = await cursor.fetchone()
self._agent_ownership_available = bool(readiness.get("agent_ownership_available"))
if not readiness["available"] or not self._agent_ownership_available:
raise DomainError(
"DEPENDENCY_UNAVAILABLE",
"Agent message storage is not provisioned; apply "
"resources/sql/024_agent_messages.sql to this project database before starting runs",
"Agent execution storage is not provisioned; apply "
"resources/sql/024_agent_messages.sql, resources/sql/028_task_questions.sql, "
"and resources/sql/030_operational_run_agent_ownership.sql "
"to this project database before starting runs",
status=503,
)
@@ -176,14 +231,26 @@ class OperationalTaskRepository:
)
async def update_run(self, run: dict, expected_revision: int) -> None:
if self._agent_ownership_available is None:
cursor = await self.connection.execute(
"""SELECT EXISTS (
SELECT 1 FROM pg_attribute
WHERE attrelid=to_regclass('workbench.operational_task_runs')
AND attname='agent_instance_id' AND NOT attisdropped
) AS available"""
)
self._agent_ownership_available = bool((await cursor.fetchone())["available"])
owner_assignment = "agent_instance_id=%s," if self._agent_ownership_available else ""
owner_values = (run.get("agent_instance_id"),) if self._agent_ownership_available else ()
cursor = await self.connection.execute(
"""UPDATE workbench.operational_task_runs SET
status=%s,revision=%s,agent_session_id=%s,lease_expires_at=%s,
f"""UPDATE workbench.operational_task_runs SET
status=%s,revision=%s,agent_session_id=%s,{owner_assignment}lease_expires_at=%s,
heartbeat_at=%s,started_at=%s,finished_at=%s,failure_code=%s,
failure_detail=%s,dispatch_attempts=%s,next_dispatch_at=%s,updated_at=%s
WHERE project_id=%s AND run_id=%s AND revision=%s""",
(
run["status"], run["revision"], run.get("agent_session_id"),
*owner_values,
run.get("lease_expires_at"), run.get("heartbeat_at"),
run.get("started_at"), run.get("finished_at"), run.get("failure_code"),
run.get("failure_detail"), run.get("dispatch_attempts", 0),
@@ -200,7 +267,8 @@ class OperationalTaskRepository:
) -> list[dict]:
result = await self.connection.execute(
"""SELECT run_id,task_id,generation,execution_request_id,status,revision,
agent_session_id,lease_expires_at,heartbeat_at,started_at,finished_at,
agent_session_id,to_jsonb(operational_task_runs)->>'agent_instance_id' AS agent_instance_id,
lease_expires_at,heartbeat_at,started_at,finished_at,
failure_code,failure_detail,dispatch_attempts,next_dispatch_at,
created_at,updated_at
FROM workbench.operational_task_runs
@@ -229,7 +297,8 @@ class OperationalTaskRepository:
) -> list[dict]:
cursor = await self.connection.execute(
"""SELECT run_id,task_id,generation,execution_request_id,status,revision,
agent_session_id,lease_expires_at,heartbeat_at,started_at,finished_at,
agent_session_id,to_jsonb(operational_task_runs)->>'agent_instance_id' AS agent_instance_id,
lease_expires_at,heartbeat_at,started_at,finished_at,
failure_code,failure_detail,dispatch_attempts,next_dispatch_at,
created_at,updated_at
FROM workbench.operational_task_runs
@@ -286,6 +355,18 @@ class OperationalTaskRepository:
async def current_event_cursor(self) -> int:
return await WorkbenchEventRepository(self.connection, self.project_id).current_cursor()
async def list_presentation_events(self, task_id: str, after: int, through: int, limit: int) -> list[dict]:
cursor = await self.connection.execute(
"""SELECT event_seq,occurred_at,metadata
FROM workbench.domain_event_log
WHERE project_id=%s AND topic='operational.presentation.requested'
AND aggregate_type='operational_task_presentation' AND aggregate_id=%s
AND event_seq>%s AND event_seq<=%s
ORDER BY event_seq LIMIT %s""",
(self.project_id, task_id, after, through, limit),
)
return [dict(row) for row in await cursor.fetchall()]
async def get_task_by_create_request(self, request_id: str) -> dict | None:
cursor = await self.connection.execute(
"""SELECT document FROM workbench.operational_tasks
@@ -630,6 +711,28 @@ class OperationalTaskRepository:
row = await cursor.fetchone()
return dict(row) if row else None
async def recovery_candidates(self, limit: int, created_from=None) -> list[dict]:
cursor = await self.connection.execute(
"""SELECT document FROM workbench.operational_tasks
WHERE project_id=%s AND task_type='event_investigation'
AND status NOT IN ('cancelled','failed')
AND document#>>'{handling_context,phase}'='waiting_scada'
AND document#>>'{source,replay_context}' IS NULL
AND (%s::timestamptz IS NULL OR created_at>=%s)
ORDER BY COALESCE((document#>>'{handling_context,last_checked_at}')::timestamptz,created_at),task_id LIMIT %s""",
(self.project_id, created_from, created_from, limit),
)
return [row["document"] for row in await cursor.fetchall()]
async def list_investigation_alerts(self, task_id: str) -> list[dict]:
cursor = await self.connection.execute(
"""SELECT document FROM workbench.alerts WHERE project_id=%s
AND status IN ('active','acknowledged')
AND document->>'condition_id'=%s ORDER BY alert_id""",
(self.project_id, f"operational-task:{task_id}"),
)
return [row["document"] for row in await cursor.fetchall()]
async def upsert_feedback(self, task_id: str, feedback: dict, accepted_at) -> None:
await self.connection.execute(
"""INSERT INTO workbench.operational_task_external_feedback
+2
View File
@@ -312,6 +312,7 @@ async def task_context_summary(conn: AsyncConnection, *, project_id: UUID, task_
'current_report_revision',t.current_report_revision,
'title_preview',left(t.document->>'title',160),'preview_only',true,
'detail',jsonb_build_object('section','task')),
'handling_context',COALESCE(t.document->'handling_context','{}'::jsonb),
'latest_report',(SELECT REPORT_SUMMARY FROM workbench.operational_task_reports r
WHERE r.project_id=t.project_id AND r.task_id=t.task_id ORDER BY r.revision DESC LIMIT 1),
'selected_report',CASE WHEN %s::int IS NULL THEN NULL ELSE
@@ -337,6 +338,7 @@ async def task_context_page(
"evidence": ("operational_task_evidence", "evidence_id"),
"actions": ("operational_task_actions", "action_id"),
"access_requests": ("operational_task_access_requests", "access_request_id"),
"questions": ("operational_task_questions", "question_request_id"),
}
documents = {"task": "operational_tasks", "report": "operational_task_reports", "plan": "operational_task_plans"}
if section in {"access_requests", "progress"}:
+207
View File
@@ -0,0 +1,207 @@
"""Short-transaction storage for replay control and resumable batches."""
from contextlib import asynccontextmanager
from datetime import datetime, timezone
from fastapi import HTTPException
from psycopg.types.json import Jsonb
class ScadaReplayRepository:
def __init__(self, connection, project_id):
self.connection, self.project_id = connection, str(project_id)
self.lock_key = f"scada-replay:{self.project_id}"
async def available(self) -> bool:
async with self.connection.transaction():
cur = await self.connection.execute(
"SELECT to_regclass('peripheral_sample.scada_replays') IS NOT NULL "
"AND to_regclass('peripheral_sample.scada_replay_batches') IS NOT NULL AS ready"
)
return (await cur.fetchone())["ready"]
@asynccontextmanager
async def control(self):
async with self.connection.transaction():
await self.connection.execute("SELECT pg_advisory_xact_lock(hashtextextended(%s,0))", (self.lock_key,))
yield
@asynccontextmanager
async def processing(self):
# Session lock outlives short transactions, never an open SQL transaction
# across Timescale IO. A dead connection automatically releases ownership.
async with self.connection.transaction():
cur = await self.connection.execute("SELECT pg_try_advisory_lock(hashtextextended(%s,0)) AS owned", (self.lock_key,))
owned = (await cur.fetchone())["owned"]
try:
yield owned
finally:
if owned:
async with self.connection.transaction():
await self.connection.execute("SELECT pg_advisory_unlock(hashtextextended(%s,0))", (self.lock_key,))
async def get(self, replay_id) -> dict:
async with self.connection.transaction():
cur = await self.connection.execute(
"SELECT document FROM peripheral_sample.scada_replays WHERE project_id=%s AND replay_id=%s",
(self.project_id, str(replay_id)),
)
row = await cur.fetchone()
if not row:
raise HTTPException(404, "当前项目不存在此 SCADA 回放")
return row["document"]
async def current(self) -> dict | None:
async with self.connection.transaction():
cur = await self.connection.execute(
"SELECT document FROM peripheral_sample.scada_replays WHERE project_id=%s"
" ORDER BY created_at DESC LIMIT 1", (self.project_id,),
)
row = await cur.fetchone()
return row["document"] if row else None
async def by_request(self, request_id) -> dict | None:
async with self.connection.transaction():
cur = await self.connection.execute(
"SELECT document FROM peripheral_sample.scada_replays WHERE project_id=%s AND request_id=%s",
(self.project_id, str(request_id)),
)
row = await cur.fetchone()
return row["document"] if row else None
async def active_other(self, replay_id) -> bool:
async with self.connection.transaction():
cur = await self.connection.execute(
"SELECT 1 FROM peripheral_sample.scada_replays WHERE project_id=%s AND replay_id<>%s "
"AND status IN ('ready','paused') LIMIT 1",
(self.project_id, str(replay_id)),
)
return await cur.fetchone() is not None
async def insert(self, document: dict):
await self.connection.execute(
"INSERT INTO peripheral_sample.scada_replays(project_id,replay_id,request_id,status,document) "
"VALUES(%s,%s,%s,%s,%s)",
(self.project_id, document["replay_id"], document["config"]["request_id"], document["status"], Jsonb(document)),
)
async def save(self, document: dict):
document["updated_at"] = datetime.now(timezone.utc).isoformat()
await self.connection.execute(
"UPDATE peripheral_sample.scada_replays SET status=%s,document=%s,updated_at=now() "
"WHERE project_id=%s AND replay_id=%s",
(document["status"], Jsonb(document), self.project_id, document["replay_id"]),
)
async def pending_batch(self, replay_id) -> dict | None:
async with self.connection.transaction():
cur = await self.connection.execute(
"SELECT document FROM peripheral_sample.scada_replay_batches "
"WHERE project_id=%s AND replay_id=%s AND sequence>0 AND stage<>'detected' ORDER BY sequence LIMIT 1",
(self.project_id, str(replay_id)),
)
row = await cur.fetchone()
return row["document"] if row else None
async def save_batch(self, replay_id, batch: dict):
await self.connection.execute(
"INSERT INTO peripheral_sample.scada_replay_batches(project_id,replay_id,sequence,stage,document) "
"VALUES(%s,%s,%s,%s,%s) ON CONFLICT(project_id,replay_id,sequence) "
"DO UPDATE SET stage=excluded.stage,document=excluded.document",
(self.project_id, str(replay_id), batch["sequence"], batch["stage"], Jsonb(batch)),
)
async def clear_unpublished_baseline(self, replay_id) -> None:
"""Discard only an incomplete attempt, retaining every published snapshot."""
await self.connection.execute(
"DELETE FROM peripheral_sample.scada_replay_batches "
"WHERE project_id=%s AND replay_id=%s AND sequence<0 "
"AND stage='clean_prepared' AND document->>'kind'='baseline'",
(self.project_id, str(replay_id)),
)
async def publish_baseline(self, replay_id) -> None:
"""Publish with the replay's baseline metadata in the caller's transaction."""
await self.connection.execute(
"UPDATE peripheral_sample.scada_replay_batches "
"SET stage='detected',document=jsonb_set(document,'{stage}','\"detected\"'::jsonb) "
"WHERE project_id=%s AND replay_id=%s AND sequence<0 "
"AND stage='clean_prepared' AND document->>'kind'='baseline'",
(self.project_id, str(replay_id)),
)
async def events(self, replay_id, offset=0, limit=100) -> list[dict]:
async with self.connection.transaction():
cur = await self.connection.execute(
"SELECT e.value AS event FROM peripheral_sample.scada_replay_batches b "
"CROSS JOIN LATERAL jsonb_array_elements(COALESCE(b.document->'events','[]'::jsonb)) e "
"WHERE b.project_id=%s AND b.replay_id=%s ORDER BY b.sequence DESC LIMIT %s OFFSET %s",
(self.project_id, str(replay_id), limit, offset),
)
return [r["event"] for r in await cur.fetchall()]
async def monitoring_outputs(
self,
replay_id,
device_ids: list[str],
start: datetime,
end: datetime,
before_sequence: int,
) -> list[dict]:
"""Read only committed replay outputs available before the current batch."""
if not device_ids:
return []
async with self.connection.transaction():
cur = await self.connection.execute(
"SELECT r.value AS reading FROM peripheral_sample.scada_replay_batches b "
"CROSS JOIN LATERAL jsonb_array_elements(COALESCE(b.document->'outputs','[]'::jsonb)) r "
"WHERE b.project_id=%s AND b.replay_id=%s AND b.stage='detected' AND b.sequence<%s "
"AND r.value->>'device_id'=ANY(%s) "
"AND (r.value->>'time')::timestamptz>=%s AND (r.value->>'time')::timestamptz<=%s "
"ORDER BY (r.value->>'time')::timestamptz,r.value->>'device_id'",
(self.project_id, str(replay_id), before_sequence, device_ids, start, end),
)
return [row["reading"] for row in await cur.fetchall()]
async def detected_outputs(
self,
replay_id,
device_ids: list[str],
start: datetime,
end: datetime,
before_sequence: int,
*,
limit: int | None = None,
offset: int = 0,
) -> list[dict]:
"""Read immutable detector outputs available to a bounded replay investigation."""
statement = (
"SELECT r.value AS reading FROM peripheral_sample.scada_replay_batches b "
"CROSS JOIN LATERAL jsonb_array_elements(COALESCE(b.document->'outputs','[]'::jsonb)) r "
"WHERE b.project_id=%s AND b.replay_id=%s AND b.sequence<=%s AND b.stage='detected' "
"AND r.value->>'device_id'=ANY(%s) "
"AND (r.value->>'time')::timestamptz>=%s AND (r.value->>'time')::timestamptz<=%s "
"ORDER BY (r.value->>'time')::timestamptz,r.value->>'device_id'"
)
params: tuple = (
self.project_id, str(replay_id), before_sequence, device_ids, start, end,
)
if limit is not None:
statement += " LIMIT %s OFFSET %s"
params += (limit, offset)
async with self.connection.transaction():
cur = await self.connection.execute(statement, params)
return [row["reading"] for row in await cur.fetchall()]
async def latest_outputs(self, replay_id, through: datetime, limit: int = 500) -> list[dict]:
"""Return the latest committed cleaned output for each replay device."""
async with self.connection.transaction():
cur = await self.connection.execute(
"SELECT DISTINCT ON(r.value->>'device_id') r.value AS reading "
"FROM peripheral_sample.scada_replay_batches b "
"CROSS JOIN LATERAL jsonb_array_elements(COALESCE(b.document->'outputs','[]'::jsonb)) r "
"WHERE b.project_id=%s AND b.replay_id=%s AND b.sequence>0 AND b.stage IN ('cleaned','detected') "
"AND (r.value->>'time')::timestamptz<=%s "
"ORDER BY r.value->>'device_id',(r.value->>'time')::timestamptz DESC LIMIT %s",
(self.project_id, str(replay_id), through, limit),
)
return [row["reading"] for row in await cur.fetchall()]
+8 -9
View File
@@ -12,20 +12,19 @@ async def lock_event_transaction(connection: AsyncConnection) -> None:
PostgreSQL sequence allocation is not commit order. All producers must use
this same transaction lock so a committed higher cursor cannot hide an
earlier, still-uncommitted event. Other BizDBs remain rollout opt-out.
earlier, still-uncommitted event. The lock is local to the connected BizDB
and creates no schema or data in projects without an event log.
"""
if getattr(getattr(connection, "info", None), "dbname", None) == "tjwater_v2":
await connection.execute(
"SELECT pg_advisory_xact_lock(hashtextextended(%s,0))", (EVENT_ORDER_LOCK,)
)
await connection.execute(
"SELECT pg_advisory_xact_lock(hashtextextended(%s,0))", (EVENT_ORDER_LOCK,)
)
def lock_event_transaction_sync(connection: Connection) -> None:
"""Synchronous counterpart using the identical PostgreSQL lock key."""
if getattr(getattr(connection, "info", None), "dbname", None) == "tjwater_v2":
connection.execute(
"SELECT pg_advisory_xact_lock(hashtextextended(%s,0))", (EVENT_ORDER_LOCK,)
)
connection.execute(
"SELECT pg_advisory_xact_lock(hashtextextended(%s,0))", (EVENT_ORDER_LOCK,)
)
class WorkbenchEventRepository:
+128 -1
View File
@@ -1,5 +1,5 @@
from typing import List, Any
from datetime import datetime
from datetime import datetime, timedelta
from collections import defaultdict
from psycopg import AsyncConnection, Connection, sql
from psycopg.rows import dict_row
@@ -9,6 +9,91 @@ class ScadaRepository:
VALUE_FIELDS = frozenset({"monitored_value", "cleaned_value"})
RESULT_COLUMNS = "time, device_id, monitored_value, cleaned_value"
@staticmethod
async def begin_replay_baseline_snapshot(conn: AsyncConnection) -> None:
"""Set the caller's source transaction to a repeatable read-only snapshot."""
await conn.execute("SET TRANSACTION ISOLATION LEVEL REPEATABLE READ, READ ONLY")
@staticmethod
async def get_warmup_values(
conn: AsyncConnection,
device_ids: list[str],
before: datetime,
per_device_limit: int,
) -> list[dict]:
"""Return the latest raw values before a replay starts for detector warmup."""
async with conn.cursor() as cur:
await cur.execute(
"SELECT d.device_id,h.monitored_value FROM unnest(%s::text[]) d(device_id) "
"CROSS JOIN LATERAL (SELECT monitored_value,time FROM scada.measurements "
"WHERE device_id=d.device_id AND time<%s ORDER BY time DESC LIMIT %s) h "
"ORDER BY d.device_id,h.time",
(device_ids, before, per_device_limit),
)
return [dict(row) for row in await cur.fetchall()]
@staticmethod
async def get_raw_page(
conn: AsyncConnection, device_ids: list[str], start: datetime,
end: datetime, through: datetime, limit: int,
after: tuple[datetime, str] | None = None, *, frame: bool = False,
) -> list[dict]:
"""Read raw only, with a stable composite key and a bounded frame/page."""
predicate = "device_id=ANY(%s) AND time>=%s AND time<%s AND time<=%s"
params: tuple = (device_ids, start, end, through)
if after is not None:
predicate += " AND (time,device_id)>(%s,%s)"
params += after
if frame:
statement = (
"WITH next_time AS (SELECT min(time) AS time FROM scada.measurements WHERE "
+ predicate + ") SELECT m.time,m.device_id,m.monitored_value,"
"md5(COALESCE(m.monitored_value::text,'NULL')) AS source_hash "
"FROM scada.measurements m WHERE m.device_id=ANY(%s) "
"AND m.time=(SELECT time FROM next_time) ORDER BY m.time,m.device_id LIMIT %s"
)
params += (device_ids, limit)
else:
statement = (
"SELECT time,device_id,monitored_value,"
"md5(COALESCE(monitored_value::text,'NULL')) AS source_hash "
"FROM scada.measurements WHERE " + predicate
+ " ORDER BY time,device_id LIMIT %s"
)
params += (limit,)
async with conn.cursor() as cur:
await cur.execute(statement, params)
return [dict(row) for row in await cur.fetchall()]
@staticmethod
async def apply_replay_cleaning(conn: AsyncConnection, rows: list[dict]) -> None:
"""Guard every raw version; update corrected values without inserting raw."""
if not rows:
return
async with conn.transaction():
async with conn.cursor() as cur:
await cur.execute(
"SELECT m.time,m.device_id,md5(COALESCE(m.monitored_value::text,'NULL')) AS source_hash "
"FROM scada.measurements m JOIN unnest(%s::timestamptz[],%s::text[]) "
"b(time,device_id) USING(time,device_id) FOR UPDATE OF m",
([r["time"] for r in rows], [r["device_id"] for r in rows]),
)
existing = {(r["time"].isoformat(), r["device_id"]): r["source_hash"]
for r in await cur.fetchall()}
if len(existing) != len(rows) or any(
existing.get((datetime.fromisoformat(r["time"]).isoformat(), r["device_id"]))
!= r["source_hash"] for r in rows
):
raise ValueError("SOURCE_CHANGED: SCADA 原始值在处理期间发生变化")
await cur.execute(
"UPDATE scada.measurements m SET cleaned_value=b.value "
"FROM unnest(%s::timestamptz[],%s::text[],%s::double precision[]) "
"b(time,device_id,value) WHERE m.time=b.time AND m.device_id=b.device_id "
"AND m.cleaned_value IS DISTINCT FROM b.value",
([r["time"] for r in rows], [r["device_id"] for r in rows],
[r["cleaned_value"] for r in rows]),
)
@staticmethod
async def insert_scada_batch(conn: AsyncConnection, data: List[dict]):
if not data:
@@ -46,6 +131,48 @@ class ScadaRepository:
)
return await cur.fetchall()
@staticmethod
async def get_recovery_scada_readings(
conn: AsyncConnection, device_ids: list[str], incident_at: datetime,
returned_at: datetime, through: datetime, *, baseline_limit: int = 100_001,
) -> list[dict]:
"""Read the causal reference and exactly the first stored frame after T."""
if not device_ids or through <= returned_at:
return []
async with conn.cursor() as cur:
await cur.execute(
"WITH next_frame AS (SELECT min(time) AS time FROM scada.measurements "
"WHERE device_id=ANY(%s) AND time>%s AND time<=%s), "
"reference AS (SELECT time,device_id,monitored_value,cleaned_value "
"FROM scada.measurements WHERE device_id=ANY(%s) AND time>=%s AND time<%s "
"ORDER BY time,device_id LIMIT %s) "
"SELECT time,device_id,monitored_value,cleaned_value FROM reference UNION ALL "
"SELECT m.time,m.device_id,m.monitored_value,m.cleaned_value FROM scada.measurements m "
"JOIN next_frame f ON m.time=f.time WHERE m.device_id=ANY(%s) ORDER BY time,device_id",
(device_ids, returned_at, through, device_ids, incident_at - timedelta(days=15),
incident_at, baseline_limit, device_ids),
)
return [dict(row) for row in await cur.fetchall()]
@staticmethod
async def get_latest_scada_readings(
conn: AsyncConnection,
device_ids: List[str],
) -> dict[str, dict[str, Any]]:
"""Return at most one latest row for each requested canonical device ID."""
if not device_ids:
return {}
async with conn.cursor() as cur:
await cur.execute(
f"SELECT DISTINCT ON (device_id) {ScadaRepository.RESULT_COLUMNS} "
"FROM scada.measurements WHERE device_id = ANY(%s) "
"ORDER BY device_id, time DESC",
(device_ids,),
)
rows = await cur.fetchall()
return {str(row["device_id"]): dict(row) for row in rows}
@staticmethod
def get_scada_by_ids_time_range_sync(
conn: Connection,
+1 -1
View File
@@ -84,7 +84,7 @@ class Output:
def units(self) -> dict[str, str]:
f_us = ["CFS", "GPM", "MGD", "IMGD", "AFD", "LPS", "LPM", "MLD", "CMH", "CMD"]
p_us = ["PSI", "MTR", "KPA"]
p_us = ["PSI", "KPA", "MTR"]
q_us = ["NONE", "MGL", "UGL", "HOURS", "PRCNT"]
f, p, q = ctypes.c_int(1), ctypes.c_int(2), ctypes.c_int(3)
f_u, p_u, q_u = ctypes.c_int(), ctypes.c_int(), ctypes.c_int()
@@ -52,7 +52,7 @@ class SampleWorkOrderReceipt(BaseModel):
work_order_id: UUID
project_id: UUID
created_at: datetime
origin: Literal["synthetic"]
origin: Literal["system_data"]
status: Literal[
"draft", "open", "in_progress", "awaiting_reply", "completed", "cancelled"
]
@@ -194,8 +194,10 @@ class HttpOperationalActionDispatcher:
raise ValueError("invalid external project or timestamp")
receipt = ActionReceipt(
adapter="peripheral_sample", receipt_id=str(order.work_order_id),
# This is the creation action's completion, not field repair.
submitted_at=order.created_at, completed_at=order.created_at,
external_status=order.status, detail="合成外围工单已创建,现场工作进度由外围系统继续同步",
external_status=order.status,
detail="外围工单已创建,现场工作进度由外围系统继续同步",
)
return ActionDispatchOutcome("succeeded", receipt.detail, receipt)
result = CommandGatewayReceipt.model_validate(value)
+26 -1
View File
@@ -5,6 +5,7 @@ import hashlib
import json
from datetime import datetime, timezone
from pathlib import Path
from typing import Literal
import httpx
from pydantic import Field
@@ -33,6 +34,23 @@ class TaskDataError(Exception):
self.detail = detail
def peripheral_records_origin(items: object) -> Literal["system_data", "synthetic"]:
"""Classify a record collection; mixed collections retain per-item provenance."""
if not isinstance(items, list) or any(
not isinstance(item, dict) or item.get("origin") not in {"system_data", "synthetic"}
for item in items
):
raise TaskDataError(502, "外围记录缺少明确来源,不能作为已核实证据")
return "synthetic" if items and all(item["origin"] == "synthetic" for item in items) else "system_data"
def normalize_peripheral_work_orders(items: object) -> list[dict[str, object]]:
"""Expose peripheral work orders only as project-owned system data."""
if not isinstance(items, list) or any(not isinstance(item, dict) for item in items):
raise TaskDataError(502, "外围工单格式无效,不能作为已核实证据")
return [{**item, "origin": "system_data"} for item in items]
def _matches_project(value: object, project_id: str) -> bool:
if isinstance(value, list):
return all(_matches_project(item, project_id) for item in value)
@@ -147,7 +165,14 @@ async def read_task_data(
data = _select_demo_observations(data, query, str(context.project_id))
elif not _matches_project(data, str(context.project_id)):
raise TaskDataError(502, "数据源返回了其他项目的数据")
origin = "synthetic" if peripheral else "system_data"
origin = "system_data"
if query.source == "peripheral_work_orders":
items = data.get("items") if isinstance(data, dict) else None
data = {**data, "items": normalize_peripheral_work_orders(items)}
elif query.source == "peripheral_test_data":
items = data.get("items") if isinstance(data, dict) else None
origin = peripheral_records_origin(items)
data = {**data, "provenance_scope": "per_record", "source_origins": sorted({item["origin"] for item in items})}
historical_replay = False
if query.source in {"peripheral_points", "peripheral_observations"}:
source_origin = data.get("origin") if isinstance(data, dict) else None
+2
View File
@@ -14,6 +14,7 @@ from app.infra.db.metadb.database import close_metadata_engine
from app.infra.db.timescaledb.sync_pool import close_all_timescale_pools
from app.native.wndb.core.connection import close_all_project_pools
from app.core.config import settings
from samples.peripheral.api import create_app as create_peripheral_app
# 导入审计中间件
from app.infra.audit.middleware import AuditMiddleware
@@ -80,6 +81,7 @@ install_problem_details_handlers(app)
from app.api.v1.endpoints.mcp_queries import router as mcp_context_router
app.state.mcp_runtime = mcp_runtime
app.include_router(mcp_context_router, prefix="/api/v1")
app.mount("/peripheral", create_peripheral_app(), name="peripheral")
# Legacy routers without version prefix
# app.include_router(api_router)
+50 -19
View File
@@ -4,23 +4,53 @@ from datetime import datetime, timedelta
from typing import Annotated, Literal
from uuid import UUID
from pydantic import Field, model_validator
from pydantic import BaseModel, ConfigDict, Field, model_validator
from app.mcp.models import Id, StrictInput
MAX_CALCULATION_TIMEOUT_SECONDS = 86400
MAX_REQUEST_TIMEOUT_SECONDS = 240
class CalculationInput(StrictInput):
timeout_seconds: int = Field(
default=120, ge=1, le=MAX_CALCULATION_TIMEOUT_SECONDS, strict=True,
description="本次计算等待预算,秒;由Agent按模型规模和任务时长选择,受工具目录中的服务端上限约束;与模拟duration不同,超时不保证已启动的计算停止或回滚",
default=120, ge=1, le=MAX_REQUEST_TIMEOUT_SECONDS, strict=True,
description="本次HTTP请求响应等待预算,秒,默认120秒;后台执行时限由作业系统管理,受工具目录中的请求上限约束;后台任务通过状态和结果工具查询",
)
class NodeCoverageGroup(BaseModel):
model_config = ConfigDict(extra="forbid")
group_id: Id = Field(max_length=80)
title: str = Field(min_length=1, max_length=200)
node_ids: list[Id] = Field(min_length=1, max_length=10000)
class NodeCoverageInput(CalculationInput):
groups: list[NodeCoverageGroup] = Field(min_length=1, max_length=8,
description="已核实的模型节点集合;各组为需要比较的业务或空间集合,可重叠,位置通过模型节点关联获取")
concave_ratio: float = Field(default=0.25, ge=0, le=1, allow_inf_nan=False,
description="凹包比例;越接近1越平滑接近凸包,参数只决定示意形状")
buffer_m: float = Field(default=25, gt=0, le=1000, allow_inf_nan=False,
description="展示缓冲距离,米;用于几何展示")
split_by_connectivity: bool = Field(default=False, strict=True,
description="需要按模型连接分组时设true;地理分布比较默认false,供水关系按拓扑核验")
excluded_link_ids: list[Id] = Field(default_factory=list, max_length=200,
description="连通分组时额外视为断开的管段/阀门,例如已有隔离建议中的关阀ID;仅影响绘图分组")
@model_validator(mode="after")
def bounded_groups(self):
if len({group.group_id for group in self.groups}) != len(self.groups):
raise ValueError("coverage group IDs must be distinct")
if len({node for group in self.groups for node in group.node_ids}) > 10000:
raise ValueError("coverage supports at most 10000 distinct nodes")
if self.excluded_link_ids and not self.split_by_connectivity:
raise ValueError("excluded_link_ids requires split_by_connectivity")
return self
class BurstDetectionInput(CalculationInput):
target_time: datetime = Field(description="带时区的侦测时刻;读取此前15天同周期压力数据,不是单小时定位")
target_time: datetime = Field(description="带时区的侦测时刻;读取此前15天同周期压力数据,返回压力异常诊断")
sensor_nodes: list[Id] = Field(min_length=5, max_length=20, description="已核实的压力监测节点ID,至少5个完整传感器")
sampling_interval_minutes: int = Field(default=15, ge=1, le=60)
@@ -37,14 +67,14 @@ class BurstDetectionInput(CalculationInput):
class BurstLocationInput(CalculationInput):
data_source: Literal["monitoring", "simulation"] = "monitoring"
simulation_run_id: UUID | None = Field(default=None, description="已有分析运行UUID,不是方案名称;simulation模式只读取已有异常结果及模型基线")
burst_leakage: float = Field(gt=0, allow_inf_nan=False, description="有依据的假定漏水流量,WNTR SI单位m³/s;不是已知爆管管段,不能随意猜测")
pressure_scada_ids: list[Id] | None = Field(default=None, min_length=1, max_length=20, description="压力监测设备关联的模型节点ID,不是设备查询编号;省略时自动读取当前项目SCADA配置中的压力测点,不需要猜测节点ID")
flow_scada_ids: list[Id] | None = Field(default=None, min_length=1, max_length=20, description="流量监测设备关联的模型要素ID,不是设备查询编号")
simulation_run_id: UUID | None = Field(default=None, description="已有分析运行UUID;simulation模式只读取已有异常结果及模型基线")
burst_leakage: float | None = Field(default=None, gt=0, allow_inf_nan=False, description="水力定位必需的假定漏水流量,WNTR SI单位m³/s;使用有依据的估计或明确给定的情景假设,保留估计方法与输入来源。仅 Server 明确开放的候选定位路径可省略,该路径直接返回候选管段与相似度;其他项目和时段省略将被拒绝")
pressure_scada_ids: list[Id] | None = Field(default=None, min_length=1, max_length=20, description="压力监测设备关联的模型节点ID,查询编号通过设备映射转换;省略时读取项目配置的压力测点")
flow_scada_ids: list[Id] | None = Field(default=None, min_length=1, max_length=20, description="流量监测设备关联的模型要素ID,由设备映射取得")
scada_burst_start: datetime
scada_burst_end: datetime
scada_normal_start: datetime | None = Field(default=None, description="monitoring模式正常对照窗开始,须与结束时间成对提供;均省略时用爆管窗前一天相同时段,不代表该时段已确认正常")
scada_normal_end: datetime | None = Field(default=None, description="monitoring模式正常对照窗结束,须与开始时间成对提供;simulation模式不填写,使用同时间窗模型基线")
scada_normal_start: datetime | None = Field(default=None, description="monitoring模式正常对照窗开始,须与结束时间成对提供;均省略时用爆管窗前一天相同时段,使用前须核对正常对照有效性")
scada_normal_end: datetime | None = Field(default=None, description="monitoring模式正常对照窗结束,须与开始时间成对提供;simulation模式省略此项,使用同时间窗模型基线")
@model_validator(mode="after")
def valid_windows(self):
@@ -69,14 +99,15 @@ class BurstLocationInput(CalculationInput):
class ValveIsolationInput(CalculationInput):
accident_element: list[Id] = Field(min_length=1, max_length=20, description="经核实或明确作为候选的事故管段/节点ID")
disabled_valves: list[Id] = Field(default_factory=list, max_length=200, description="已知无法关闭的阀门;未提供不代表现场确认全部可用")
accident_element_type: Literal["auto", "link", "node"] = Field(default="auto", description="事故要素类型;auto优先识别管段,再识别节点;节点与管段重号时,节点分析须指定node,纯管段分析可指定link")
disabled_valves: list[Id] = Field(default_factory=list, max_length=200, description="已知无法关闭的阀门;现场可操作性按执行前核验记录确认")
class LeakageIdentificationInput(CalculationInput):
scada_start: datetime
scada_end: datetime
sensor_nodes: list[Id] = Field(min_length=1, max_length=20)
q_sum: float = Field(gt=0, allow_inf_nan=False, description="有依据的总漏损流量约束,m³/s,不得使用无依据默认值")
q_sum: float = Field(gt=0, allow_inf_nan=False, description="有依据的总漏损流量约束,m³/s,保留数据来源与估算方法")
start_time: float = Field(ge=0, lt=24, allow_inf_nan=False, description="模型起算小时;需与SCADA时间窗及模型时钟对齐")
timestep: int = Field(default=15, ge=5, le=60, description="计算步长,分钟;需与观测采样对应")
dma_count: int = Field(ge=1, le=20)
@@ -98,7 +129,7 @@ class LeakageIdentificationInput(CalculationInput):
class ScenarioInput(CalculationInput):
start_time: datetime
duration: int = Field(default=900, ge=60, le=86400, description="模拟时长,秒;首次优先短时间窗")
output_node_ids: list[Id] = Field(min_length=1, max_length=20, description="需要返回结果的节点;不把局部结果解释为全网无影响")
output_node_ids: list[Id] = Field(min_length=1, max_length=20, description="需要返回结果的节点;结论范围限定为返回对象")
output_link_ids: list[Id] = Field(default_factory=list, max_length=20)
@model_validator(mode="after")
@@ -109,18 +140,18 @@ class ScenarioInput(CalculationInput):
class ContaminantSimulationInput(ScenarioInput):
source: Id = Field(description="假设污染源节点;这是正向传播模拟,不是自动反演污染源")
source: Id = Field(description="假设污染源节点;这是正向传播模拟,结果按给定源解释")
concentration: float = Field(gt=0, allow_inf_nan=False, description="假设源浓度,mg/L")
pattern: Id | None = Field(default=None, description="已存在的污染源时间模式ID;不填为持续恒定源")
pattern: Id | None = Field(default=None, description="已存在的污染源时间模式ID;省略时采用持续恒定源")
class ValveClosureInput(ScenarioInput):
valves: list[Id] = Field(min_length=1, max_length=20, description="在临时模型中关闭的阀门,不下发现场控制")
valves: list[Id] = Field(min_length=1, max_length=20, description="在临时模型中关闭的阀门,现场操作另走审批流程")
class FlushingInput(ScenarioInput):
drainage_node_id: Id
flushing_flow_m3h: float = Field(gt=0, allow_inf_nan=False, description="明确的冲洗流量,m³/h;不使用无依据的自动排放量")
flushing_flow_m3h: float = Field(gt=0, allow_inf_nan=False, description="明确的冲洗流量,m³/h;提供流量来源与估算依据")
closed_valves: list[Id] = Field(default_factory=list, max_length=20)
@@ -129,7 +160,7 @@ PumpSeries = Annotated[list[NonnegativeValue], Field(min_length=1, max_length=96
class PressureRegulationInput(ScenarioInput):
pump_control: dict[Id, PumpSeries] = Field(min_length=1, max_length=20, description="已登记泵ID及运行序列;工频值为0/1,变频值为频率Hz,每项对应模型水力步长,不自动优化")
pump_control: dict[Id, PumpSeries] = Field(min_length=1, max_length=20, description="已登记泵ID及运行序列;工频值为0/1,变频值为频率Hz,每项对应模型水力步长,按给定序列评估")
tank_init_level: dict[Id, Annotated[float, Field(gt=0, allow_inf_nan=False)]] = Field(default_factory=dict, max_length=20, description="水箱初始水位,须在模型允许范围内")
+49 -2
View File
@@ -4,9 +4,9 @@ import asyncio
from threading import BoundedSemaphore
from typing import Any
from fastapi import HTTPException
from pydantic import BaseModel
from app.mcp.scenario_service import SCENARIO_TOOLS, calculate_scenario
from app.mcp.errors import CalculationCapacityExceeded
# These services synchronously read SCADA/model series or write online results.
# Topology-only isolation and model-based sensor placement need only BizDB.
@@ -21,10 +21,25 @@ _slots = BoundedSemaphore(2)
def _calculate(name: str, parameters: dict[str, Any], network: str, username: str) -> dict[str, Any]:
if not _slots.acquire(blocking=False):
raise HTTPException(503, "Calculation capacity is occupied")
raise CalculationCapacityExceeded()
try:
if name in SCENARIO_TOOLS:
return calculate_scenario(name, parameters, network=network, username=username)
if name == "calculate_demand_allocation":
from app.services import demand_allocation
target = parameters.pop("target")
limit, offset = parameters.pop("limit"), parameters.pop("offset")
demand = parameters["demand"]
if target == "nodes":
result = demand_allocation.calculate_demand_to_nodes(network, demand, parameters["node_ids"])
elif target == "region":
result = demand_allocation.calculate_demand_to_region(network, demand, parameters["region_id"])
else:
result = demand_allocation.calculate_demand_to_network(network, demand)
keys = sorted(result)
return {"items": [{"node_id": key, "demand": result[key]} for key in keys[offset:offset + limit]],
"total": len(keys), "limit": limit, "offset": offset,
"has_more": offset + limit < len(keys), "persisted": False}
if name == "optimize_sensor_placement":
from app.services.sensor_placement import optimize_sensor_placement_by_kmeans, optimize_sensor_placement_by_sensitivity
parameters = dict(parameters)
@@ -56,6 +71,10 @@ def _calculate(name: str, parameters: dict[str, Any], network: str, username: st
from app.services.valve_isolation import analyze_valve_isolation
return analyze_valve_isolation(network=network, **parameters)
if name == "generate_node_coverage":
from app.services.node_coverage import generate_node_coverage
return generate_node_coverage(network=network, **parameters)
raise ValueError("Calculation is not registered")
finally:
_slots.release()
@@ -67,3 +86,31 @@ async def calculate(name: str, args: BaseModel, *, network: str, username: str)
return await asyncio.to_thread(
_calculate, name, args.model_dump(exclude={"context_handle", "timeout_seconds"}, exclude_none=True), network, username,
)
async def bounded_calculation(function, *args, **kwargs):
"""Use the same native-work capacity for persistent calculation tools."""
def run():
if not _slots.acquire(blocking=False):
raise CalculationCapacityExceeded()
try:
return function(*args, **kwargs)
finally:
_slots.release()
return await asyncio.to_thread(run)
async def calculate_replay_burst_location(
parameters: dict[str, Any], *, network: str, username: str,
) -> dict[str, Any]:
"""Run location with Server-built frozen replay series and no live SCADA query."""
from app.services.burst_location import run_burst_location_by_network
return await bounded_calculation(
run_burst_location_by_network,
network=network,
username=username,
scheme_name=None,
final_candidates_csv_path=None,
**parameters,
)
+60 -16
View File
@@ -32,15 +32,20 @@ from app.mcp.catalog import TOOLS
from app.mcp.config import McpSettings
from app.mcp.contexts import resolve_context, verify_task
from app.mcp.models import QueryResult
from app.mcp.display_time import display_query_result, display_time_text
from app.mcp.scada_catalog import compact_scada_catalog
from app.mcp.errors import CalculationCapacityExceeded
from app.mcp.scope import project_scope
from app.mcp.query_service import execute_query, task_service, authorize_query
from app.infra.db.postgresql.mcp_queries import McpQueryRepository
from app.domain.schemas.operational_data import TaskDataSnapshot
from app.domain.domain_errors import DomainError
logger = logging.getLogger(__name__)
def tool_error(code, message):
message = display_time_text(message)
return types.CallToolResult(
isError=True,
content=[types.TextContent(type="text", text=message)],
@@ -200,8 +205,12 @@ class McpQueryApplication:
if tool is None:
return None
schema = tool.model.model_json_schema()
if tool.computation:
schema["properties"]["timeout_seconds"]["maximum"] = self.settings.max_calculation_timeout_seconds
if tool.computation or tool.mutating:
schema["properties"]["timeout_seconds"]["maximum"] = self.settings.max_request_timeout_seconds
schema["properties"]["timeout_seconds"]["default"] = min(
self.settings.query_timeout_seconds,
self.settings.max_request_timeout_seconds,
)
return schema
async def list_tools(self, ctx, params):
@@ -211,6 +220,10 @@ class McpQueryApplication:
for name in sorted(TOOLS):
tool = TOOLS[name].protocol_tool()
tool.input_schema.update(self.tool_input_schema(name))
tool.meta = {**(tool.meta or {}), "io.tjwater/request-timeout": {
"defaultSeconds": self.settings.query_timeout_seconds,
"maxSeconds": self.settings.max_request_timeout_seconds,
}}
tools.append(tool)
return types.ListToolsResult(
tools=tools,
@@ -289,11 +302,13 @@ class McpQueryApplication:
return tool_error(
"INVALID_QUERY", "Query parameters do not match the tool schema"
)
if (tool.computation or tool.mutating) and "timeout_seconds" not in args.model_fields_set:
args.timeout_seconds = min(self.settings.query_timeout_seconds, self.settings.max_request_timeout_seconds)
timeout_seconds = (
args.timeout_seconds if tool.computation else self.settings.query_timeout_seconds
args.timeout_seconds if tool.computation or tool.mutating else self.settings.query_timeout_seconds
)
if tool.computation and timeout_seconds > self.settings.max_calculation_timeout_seconds:
return tool_error("INVALID_QUERY", "Calculation timeout exceeds the server limit")
if (tool.computation or tool.mutating) and timeout_seconds > self.settings.max_request_timeout_seconds:
return tool_error("INVALID_QUERY", "HTTP request timeout exceeds the server limit")
request = ctx.request
project = request.headers.get("x-project-id", "") if request else ""
meta = ctx.meta or {}
@@ -345,20 +360,29 @@ class McpQueryApplication:
)
result = QueryResult.model_validate(previous["result"])
else:
result = (
await self.executor(
scope, binding, tool, args, full_snapshot=True
from app.mcp.calculation_jobs import should_queue, submit_calculation
queued = should_queue(tool, args, project_code=getattr(scope.project, "project_code", None))
if queued:
result = await submit_calculation(
scope, binding, tool, args, call_id,
job_timeout_seconds=self.settings.max_calculation_timeout_seconds,
)
else:
result = (
await self.executor(
scope, binding, tool, args, full_snapshot=True
)
if include_full
else await self.executor(scope, binding, tool, args)
)
if include_full
else await self.executor(scope, binding, tool, args)
)
# Never serialize an unbounded response into a model message or database receipt.
if len(result.model_dump_json().encode()) > 8 * 1024 * 1024:
return tool_error(
"RESULT_TOO_LARGE",
"Narrow the query or request a smaller page",
)
if binding.task_id and name not in {
if binding.task_id and not queued and name not in {
"get_calculation_job", "get_calculation_result", "cancel_calculation_job", "list_calculation_jobs",
"get_task_context",
"get_task_evidence",
}:
@@ -380,12 +404,16 @@ class McpQueryApplication:
result.source_ref = "mcp:" + str(
uuid5(NAMESPACE_URL, scope_key + ":" + call_id)
)
snapshot_data = result.model_dump(mode="json")
from app.mcp.extended_service import ARTIFACT_TOOLS
if name in ARTIFACT_TOOLS:
snapshot_data["data"]["file"].pop("data", None)
snapshot = TaskDataSnapshot(
source=name,
source_ref=result.source_ref,
queried_at=result.queried_at,
origin=result.origin,
data=result.model_dump(mode="json"),
data=snapshot_data,
truncated=result.truncated,
)
evidence, _ = await task_service(
@@ -398,10 +426,17 @@ class McpQueryApplication:
parameters,
result.model_dump(mode="json"),
)
output = result.model_dump(mode="json")
output = display_query_result(result.model_dump(mode="json"))
content = []
from app.mcp.extended_service import ARTIFACT_TOOLS
if (not include_full and name == "list_scada_devices"
and len(json.dumps(output, ensure_ascii=False).encode()) > 12000):
preview = compact_scada_catalog(output, budget=12000)
if preview is not None:
output = preview
if (
not include_full
and name not in ARTIFACT_TOOLS
and len(json.dumps(output, ensure_ascii=False).encode()) > 12000
):
evidence_id = result.evidence_id or (
@@ -415,6 +450,10 @@ class McpQueryApplication:
"resource_uri": uri,
"preview": "The complete query snapshot is saved in task evidence",
}
from app.mcp.presentation_preview import saved_presentation_preview
presentation = saved_presentation_preview(result)
if presentation is not None:
output["data"]["presentation"] = display_query_result(presentation)
output["truncated"] = True
output["data_gaps"] = [
*result.data_gaps,
@@ -440,12 +479,17 @@ class McpQueryApplication:
),
)
return types.CallToolResult(content=content, structuredContent=output)
except DomainError as exc:
return tool_error(exc.code, exc.message)
except CalculationCapacityExceeded as exc:
return tool_error(exc.code, str(exc.detail))
except HTTPException as exc:
codes = {
401: "AUTH_REQUIRED",
403: "QUERY_CONTEXT_DENIED",
404: "NOT_FOUND",
409: "QUERY_CONTEXT_STALE",
410: "RESULT_EXPIRED",
413: "RESULT_TOO_LARGE",
422: "INVALID_QUERY",
503: "SOURCE_UNAVAILABLE",
@@ -454,7 +498,7 @@ class McpQueryApplication:
codes.get(exc.status_code, "QUERY_FAILED"),
str(exc.detail)
if isinstance(exc.detail, str)
and exc.status_code in {403, 404, 409, 413, 422}
and exc.status_code in {403, 404, 409, 410, 413, 422}
else "Query is unavailable",
)
except (UndefinedTable, InvalidSchemaName):
@@ -463,7 +507,7 @@ class McpQueryApplication:
"MCP project storage has not been provisioned",
)
except TimeoutError:
return tool_error("QUERY_TIMEOUT", "Query execution timed out")
return tool_error("QUERY_TIMEOUT", "HTTP request timed out")
except ValueError:
return tool_error(
"INVALID_QUERY", "Invalid query value or incompatible source data"
+209
View File
@@ -0,0 +1,209 @@
"""Durable MCP calculation submission and owner-scoped result retrieval.
Only safe server-owned binding fields are persisted. OAuth tokens and context
handles are never used to resume a calculation.
"""
from datetime import datetime, timezone
from uuid import uuid5, NAMESPACE_URL
from fastapi import HTTPException
from app.mcp.models import BoundContext, QueryResult, PageInfo
from app.infra.db.postgresql.calculation_jobs import CalculationJobRepository
QUEUED_TOOLS = frozenset({
"run_burst_location", "run_leakage_identification", "run_burst_detection",
"run_simulation", "run_burst_simulation", "run_contaminant_simulation",
"analyze_valve_closure", "analyze_flushing", "analyze_pressure_regulation",
"optimize_sensor_placement", "save_simulation_scenario", "create_sensor_placement",
})
CONTROL_TOOLS = frozenset({"get_calculation_job", "get_calculation_result", "cancel_calculation_job", "list_calculation_jobs"})
BINDING_FIELDS = {"project_id", "subject", "client_id", "native_session_id", "purpose", "parent_session_id", "task_id", "run_id", "execution_request_id", "tool_cap", "permission_cap", "task_scope_cap"}
def durable_binding(binding: BoundContext) -> dict:
return binding.model_dump(mode="json", include=BINDING_FIELDS)
def restore_binding(document: dict) -> BoundContext:
# Expiration governs admission via a context handle; durable work has its own
# lifecycle and must revalidate identity, membership and run on every use.
return BoundContext.model_validate({**document, "context_handle": "durable-calculation", "expires_at": datetime.max.replace(tzinfo=timezone.utc)})
def should_queue(tool, args, *, project_code: str | None = None) -> bool:
if tool.name == "run_burst_location":
from app.services.burst_location_demo import is_location_demo
if is_location_demo(project_code, args):
return False
return tool.name in QUEUED_TOOLS and (not tool.mutating or not args.dry_run)
def envelope(source: str, data, *, job_id="list", page=None) -> QueryResult:
return QueryResult(source=source, source_ref="calculation:" + str(job_id), queried_at=datetime.now(timezone.utc), semantics="task", data=data, page=page, truncated=bool(page and (page.offset or page.has_more)),
data_gaps=["Calculation status and queue receipts are not analytical evidence."])
def public_job(row: dict) -> dict:
fields = {"job_id", "status", "tool", "created_at", "started_at", "finished_at", "updated_at", "error", "result_expires_at"}
from fastapi.encoders import jsonable_encoder
expiry = row.get("result_expires_at")
available = row.get("status") == "succeeded" and (expiry is None or expiry > datetime.now(timezone.utc))
return {**jsonable_encoder({key: row[key] for key in fields if key in row}),
"result_available": available, "poll_after_ms": 1500}
async def submit_calculation(scope, binding, tool, args, call_id: str, *, job_timeout_seconds: int = 3600) -> QueryResult:
from app.mcp.query_service import authorize_query
await authorize_query(scope, binding, tool, args)
if binding.purpose == "task_reader":
raise HTTPException(403, "Read-only task agents cannot start calculations")
if tool.mutating:
from app.mcp.mutation_service import (
preview_change, check_digest, mutation_arguments, retryable_capacity_rejection,
)
from app.infra.db.postgresql.mcp_queries import McpQueryRepository
mutation_repo = McpQueryRepository(scope.connection, scope.project.project_id)
mutation_key = "mutation:" + binding.subject + ":" + binding.native_session_id
approved_parameters = {"tool": tool.name, "arguments": mutation_arguments(args, preview=False)}
async with scope.connection.transaction():
await mutation_repo.lock_receipt(mutation_key, str(args.request_id))
previous = await mutation_repo.receipt(mutation_key, str(args.request_id))
if previous:
if previous["parameters"] != approved_parameters:
raise HTTPException(409, "request_id already has different parameters")
if previous["result"].get("status") == "completed":
return QueryResult.model_validate(previous["result"]["output"])
if not retryable_capacity_rejection(previous["result"]):
raise HTTPException(409, "Operation already started; inspect the existing calculation or mutation status")
check_digest(args, await preview_change(scope, binding, tool.name, args))
parameters = args.model_dump(mode="json", exclude={"context_handle", "timeout_seconds"})
request_id = str(args.request_id) if tool.mutating else call_id
key = str(uuid5(NAMESPACE_URL, ":".join([binding.subject, binding.client_id, binding.native_session_id, str(binding.task_id), str(binding.run_id), request_id])))
repo = CalculationJobRepository(scope.connection, scope.project.project_id)
if binding.task_id:
from app.mcp.query_service import task_service
from app.mcp.contexts import verify_task
task_repo = task_service(scope).repo
# Share the operational transaction lock with finalization. Admission
# cannot race a pending-job check and then publish work into a closed run.
async with task_repo.transaction():
await task_repo.lock(f"task:{binding.task_id}")
await task_repo.lock(f"run:{binding.run_id}")
await verify_task(scope, binding)
row = await repo.enqueue(
owner_user_id=scope.project.user_id, request_key=key, tool=tool.name,
parameters=parameters, binding=durable_binding(binding), timeout_seconds=job_timeout_seconds,
)
else:
row = await repo.enqueue(
owner_user_id=scope.project.user_id, request_key=key, tool=tool.name,
parameters=parameters, binding=durable_binding(binding), timeout_seconds=job_timeout_seconds,
)
await scope.connection.commit()
return envelope(tool.name, public_job(row), job_id=row["job_id"])
def check_binding(binding, row):
original = row["binding"]
if any(str(original.get(key)) != str(getattr(binding, key)) for key in ("subject", "client_id", "native_session_id", "task_id", "run_id", "execution_request_id")):
raise HTTPException(403, "Calculation belongs to a different session or task run")
async def authorize_original(scope, row):
from app.mcp.catalog import TOOLS
from app.mcp.query_service import authorize_query
from app.mcp.contexts import verify_task
original = restore_binding(row["binding"])
tool = TOOLS[row["tool"]]
# Legacy queue rows stored the old HTTP timeout. It does not govern durable execution.
parameters = {key: value for key, value in row["parameters"].items() if key != "timeout_seconds"}
args = tool.model.model_validate({**parameters, "context_handle": "durable-calculation-context-handle"})
if (str(scope.project.user_id) != str(row["owner_user_id"])
or str(scope.project.project_id) != str(original.project_id)):
raise HTTPException(403, "Calculation owner is no longer authorized")
if original.task_id:
await verify_task(scope, original)
await authorize_query(scope, original, tool, args)
return original, tool, args
async def read_calculation_job(scope, binding, tool, args) -> QueryResult:
repo = CalculationJobRepository(scope.connection, scope.project.project_id)
if tool.name == "list_calculation_jobs":
identity_scope = binding.model_dump(mode="json", include={
"subject", "client_id", "native_session_id", "task_id", "run_id", "execution_request_id",
})
rows = await repo.list(owner_user_id=scope.project.user_id, binding=identity_scope, limit=args.limit + 1, offset=args.offset, status=args.status)
return envelope(tool.name, [public_job(row) for row in rows[:args.limit]], page=PageInfo(limit=args.limit, offset=args.offset, has_more=len(rows) > args.limit))
row = await repo.get(args.job_id, owner_user_id=scope.project.user_id,
include_result=tool.name == "get_calculation_result")
if row is None:
raise HTTPException(404, "Calculation was not found")
check_binding(binding, row)
await authorize_original(scope, row)
if tool.name == "cancel_calculation_job":
if binding.purpose == "task_reader":
raise HTTPException(403, "Read-only task agents cannot cancel calculations")
row = await repo.cancel(args.job_id, owner_user_id=scope.project.user_id)
await scope.connection.commit()
if tool.name != "get_calculation_result" or row["status"] != "succeeded":
return envelope(tool.name, public_job(row), job_id=row["job_id"])
expiry = row.get("result_expires_at")
if row.get("result") is None or (expiry is not None and expiry <= datetime.now(timezone.utc)):
raise HTTPException(410, "Calculation result retention period has expired")
result = QueryResult.model_validate(row["result"])
result = await attach_calculation_evidence(scope, row, result)
value = result.data
for part in args.data_path:
if isinstance(value, dict) and part in value:
value = value[part]
else:
raise HTTPException(422, "data_path must select an existing object field")
if isinstance(value, list):
result.data = value[args.offset:args.offset + args.limit]
result.page = PageInfo(limit=args.limit, offset=args.offset, total=len(value), has_more=args.offset + args.limit < len(value))
result.truncated = result.truncated or args.offset > 0 or result.page.has_more
elif args.offset:
raise HTTPException(422, "offset requires a collection data_path")
else:
result.data = value
if len(result.model_dump_json().encode()) > 11000:
raise HTTPException(413, "Select a narrower data_path or smaller result page")
return result
async def execute_calculation_job(scope, row) -> QueryResult:
"""Worker entry point; success is persisted by the fenced supervisor."""
from app.mcp.query_service import execute_query
original, tool, args = await authorize_original(scope, row)
result = await execute_query(scope, original, tool, args)
return result
async def attach_calculation_evidence(scope, row, result):
from app.mcp.query_service import task_service
from app.mcp.contexts import verify_task
original, tool, _ = await authorize_original(scope, row)
if original.task_id:
from app.domain.schemas.operational_task import TaskDataSnapshot
from app.infra.db.postgresql.mcp_queries import McpQueryRepository
repo = McpQueryRepository(scope.connection, scope.project.project_id)
key, call_id = "calculation:" + str(row["job_id"]), "result"
# Result lookup can leave an implicit SELECT transaction open. End it so
# evidence and its receipt commit together before returning a response.
await scope.connection.commit()
task_repo = task_service(scope).repo
# Match operational finalization's lock order: event lock, then task.
async with task_repo.transaction():
await repo.lock_receipt(key, call_id)
previous = await repo.receipt(key, call_id)
if previous:
return QueryResult.model_validate(previous["result"])
await task_repo.lock(f"task:{original.task_id}")
await verify_task(scope, original)
result.source_ref = key
evidence, _ = await task_service(scope).record_data_snapshot(original.task_id, TaskDataSnapshot(
source=tool.name, source_ref=key, queried_at=result.queried_at, origin=result.origin,
data=result.model_dump(mode="json"), truncated=result.truncated))
result.evidence_id = evidence.evidence_id
await repo.save_receipt(key, call_id, {"job_id": str(row["job_id"])}, result.model_dump(mode="json"))
return result
+21
View File
@@ -0,0 +1,21 @@
"""Bounded inputs for durable computation control."""
from typing import Literal
from uuid import UUID
from pydantic import Field
from app.mcp.models import StrictInput, PageInput, Id
CalculationStatus = Literal["queued", "running", "cancel_requested", "succeeded", "failed", "cancelled", "interrupted", "timed_out"]
class CalculationJobInput(StrictInput):
job_id: UUID
class CalculationResultInput(CalculationJobInput):
data_path: list[Id] = Field(default_factory=list, max_length=8)
limit: int = Field(default=20, ge=1, le=500)
offset: int = Field(default=0, ge=0, le=1000000)
class CalculationJobsInput(PageInput):
status: CalculationStatus | None = None
+77 -22
View File
@@ -4,7 +4,9 @@ from dataclasses import dataclass
from pydantic import BaseModel
from mcp import types
from app.mcp import models as m
from app.mcp.analysis_models import BurstDetectionInput, BurstLocationInput, ValveIsolationInput, LeakageIdentificationInput
from app.mcp import calculation_models as jobs
from app.mcp import extended_models as e
from app.mcp.analysis_models import BurstDetectionInput, BurstLocationInput, ValveIsolationInput, LeakageIdentificationInput, NodeCoverageInput
from app.mcp.analysis_models import ScenarioInput, ContaminantSimulationInput, ValveClosureInput, FlushingInput, PressureRegulationInput, SensorPlacementInput
@@ -17,41 +19,54 @@ class QueryTool:
semantics: str
task_only: bool = False
computation: bool = False
mutating: bool = False
def protocol_tool(self) -> types.Tool:
if self.name == "run_simulation":
if self.mutating:
suffix = "。默认dry_run只预览;执行必须携带预览摘要、相同request_id并通过宿主原生许可。由项目成员的普通聊天执行写入;失败或超时先核实状态,再按原请求接续。"
elif self.name == "run_simulation":
suffix = "。任务调用复用证据留存。"
elif self.computation:
suffix = "。仅计算建议,不保存方案、不控制设备;任务调用复用证据留存。"
suffix = "。返回分析建议并留存任务证据;正式方案与现场操作按各自业务流程执行。"
elif self.name == "query_nearby_elements":
suffix = "。仅查询当前已授权项目的模型管网,按距离和元素ID稳定排序;按需分页,不自动遍历全量;空间邻近不代表管网连通或供水关系。"
suffix = "。仅查询当前已授权项目的模型管网,按距离和元素ID稳定排序;按核查目标分页;连接与供水关系由拓扑分析核验。"
elif self.name == "list_scada_devices":
suffix = (
"。默认每页100个,最多500个;page.has_more=true时继续递增offset读取后续页面。"
"大目录内联保留本页全部设备的编号、类型和单位,模型映射及坐标用get_scada_device核验。"
)
else:
suffix = (
"。先限定对象和时间范围,再读取明细;集合默认20条,按需逐页扩大,"
"不要自动遍历全量;截断或非零offset不能证明完整范围无异常。"
"按当前目标读取必要页面,并说明截断和分页覆盖范围。"
)
return types.Tool(
name=self.name,
description=self.description + suffix,
description=self.description + suffix + (
"。返回queued/running和job_id时,使用get_calculation_job查询状态、get_calculation_result读取成功结果,排队回执用于跟踪;直接返回分析结果时立即读取。"
if self.name in {"run_burst_location", "run_leakage_identification", "run_burst_detection", "run_simulation", "run_burst_simulation", "run_contaminant_simulation", "analyze_valve_closure", "analyze_flushing", "analyze_pressure_regulation", "optimize_sensor_placement", "save_simulation_scenario", "create_sensor_placement"}
else ""
),
inputSchema=self.model.model_json_schema(),
outputSchema=m.QueryResult.model_json_schema(),
annotations=types.ToolAnnotations(
readOnlyHint=False, destructiveHint=self.name == "run_simulation", openWorldHint=False
readOnlyHint=not (self.computation or self.mutating or self.name == "cancel_calculation_job"), destructiveHint=self.mutating or self.name == "run_simulation", openWorldHint=self.name in {"search_project_places", "reverse_geocode"}
),
)
_specs = [
("run_simulation", "按水力步长在在线业务模型上执行指定时间段的实时模拟,不复制数据库;更新该时段的模型条件及实时基线,返回指定节点和管段结果;duration单位为秒,不创建方案或正式分析运行,结果不是实测", ScenarioInput, "simulation.run", "analysis", False, True),
("run_contaminant_simulation", "在隔离模型中模拟假设污染源的浓度传播;返回指定节点/管段结果,不反演污染源,不代表实测污染范围", ContaminantSimulationInput, "simulation.run", "analysis", False, True),
("analyze_valve_closure", "在隔离模型中模拟关阀的水力影响,返回指定节点/管段结果;不操作真实阀门", ValveClosureInput, "simulation.run", "analysis", False, True),
("analyze_flushing", "在隔离模型中评估指定排放节点、冲洗流量和关阀组合的水力结果;不是现场冲洗或消毒合格证明", FlushingInput, "simulation.run", "analysis", False, True),
("analyze_pressure_regulation", "在隔离模型中评估给定调泵序列和水箱初始水位的压力影响;不自动优化调度,不实际控制泵", PressureRegulationInput, "simulation.run", "analysis", False, True),
("optimize_sensor_placement", "使用灵敏度或K-means算法推荐压力监测点;仅返回候选节点,不保存正式方案,不保证污染溯源可观测性", SensorPlacementInput, "optimization.run", "analysis", False, True),
("run_leakage_identification", "基于SCADA压力与模型进行DMA漏损分配识别,需有依据的总漏损流量和时间对齐;返回区域估计,不是已确认漏点。内部执行遗传算法与水力计算", LeakageIdentificationInput, "burst.run", "analysis", False, True),
("run_burst_detection", "爆管异常侦测:使用至少5个完整压力传感器此前15天数据;只提示压力异常,不定位、不证明爆管", BurstDetectionInput, "burst.run", "analysis", False, True),
("run_burst_location", "爆管候选定位:对比异常与正常压力,可辅助流量;需有依据的漏水量假设。压力节点可省略,自动读取项目SCADA压力测点;monitoring正常窗可成对省略,默认前一天同时间段,须说明该对照假设。simulation须有已有分析运行UUID并读取同窗模型基线,不是实测;定位内部会执行水力计算,不创建新的事件方案", BurstLocationInput, "burst.run", "analysis", False, True),
("analyze_valve_isolation", "按事故候选要素分析拓扑隔离边界、建议关闭阀门和受影响节点;不是污染扩散或水力影响范围,不执行关阀,须现场核实", ValveIsolationInput, "simulation.run", "analysis", False, True),
("run_simulation", "按水力步长在在线业务模型上执行指定时间段的实时模拟;更新该时段的模型条件及实时基线,返回指定节点和管段结果;duration单位为秒,输出为在线模型结果,正式分析成果通过方案工具保存", ScenarioInput, "simulation.run", "analysis", False, True),
("run_contaminant_simulation", "在隔离模型中模拟假设污染源的浓度传播;返回指定节点/管段结果,结果标注假设来源及模型预测范围", ContaminantSimulationInput, "simulation.run", "analysis", False, True),
("analyze_valve_closure", "在隔离模型中模拟关阀的水力影响,返回指定节点/管段结果;现场阀门操作按业务审批执行", ValveClosureInput, "simulation.run", "analysis", False, True),
("analyze_flushing", "在隔离模型中评估指定排放节点、冲洗流量和关阀组合的水力结果;现场效果由执行回执和复测核验", FlushingInput, "simulation.run", "analysis", False, True),
("analyze_pressure_regulation", "在隔离模型中评估给定调泵序列和水箱初始水位的压力影响;输出为给定方案评估,现场调度按审批执行", PressureRegulationInput, "simulation.run", "analysis", False, True),
("optimize_sensor_placement", "使用灵敏度或K-means算法推荐压力监测点;返回候选节点,正式部署另行保存并核验观测能力", SensorPlacementInput, "optimization.run", "analysis", False, True),
("run_leakage_identification", "基于SCADA压力与模型进行DMA漏损分配识别,需有依据的总漏损流量和时间对齐;返回待现场核验的区域估计。内部执行遗传算法与水力计算", LeakageIdentificationInput, "burst.run", "analysis", False, True),
("run_burst_detection", "爆管异常侦测:使用至少5个完整压力传感器此前15天数据;返回压力异常信号,候选管段由定位工具分析", BurstDetectionInput, "burst.run", "analysis", False, True),
("run_burst_location", "爆管候选定位:对比异常与正常压力,可辅助流量;依据现有异常形成候选,水力算法使用有依据的burst_leakage作为输入。任务上下文location_capability.available=true时可省略漏量,Server直接返回候选管段与相似度,随后继续地图展示、隔离边界与影响分析。压力节点可省略,自动读取项目配置;monitoring正常窗省略时默认前一天同时间段,须说明假设;回放必须显式提供冻结正常窗。simulation须有已有分析运行UUID及同窗模型基线,结果标注模型来源", BurstLocationInput, "burst.run", "analysis", False, True),
("analyze_valve_isolation", "按事故候选要素分析拓扑隔离边界、建议关闭阀门和受影响节点;候选分析无需等待现场确诊。输出为拓扑隔离与供水失源范围;水力影响另用关阀模拟评估,现场操作前核实阀门映射及可用性", ValveIsolationInput, "simulation.run", "analysis", False, True),
("generate_node_coverage", "为当前项目已核实的模型节点集合生成空间覆盖示意,适用于候选分布、监测覆盖比较、区域成员、隔离或模拟结果节点等,支持各类项目分析。支持最多8组、合计10000个唯一节点,从真实GIS查询坐标;监测设备、投诉或地址须先关联模型节点。可按连接分组,记录凹包/缓冲参数、缺失节点和display_only;结果为节点空间覆盖示意,保留分组及几何覆盖范围。返回result.map_payload及coverage,任务中用set_overlay引用本次evidence_id、省略data展示;地图通过展示工具发布", NodeCoverageInput, "webgis.view", "analysis", False, True),
(
"get_element_properties",
"读取当前项目的单个管网元素属性",
@@ -61,14 +76,14 @@ _specs = [
),
(
"list_elements",
"分页读取管网元素,不自动读取全网",
"按查询目标分页读取管网元素",
m.ElementPageInput,
"webgis.view",
"asset",
),
(
"query_nearby_elements",
"查询坐标附近的模型元素,默认pipe;center为EPSG:4326 [经度,纬度],可直接使用geocode.center;radius_m默认200米、最大10000米,limit默认5、最大100,支持offset。按点到完整几何的WGS84椭球最短距离筛选并排序,返回element_id、distance_m和属性;不查询原始GIS管线、不自动扩大半径、不解析道路交叉点",
"查询坐标附近的模型元素,默认pipe;center为EPSG:4326 [经度,纬度],可直接使用geocode.center;radius_m默认200米、最大10000米,limit默认5、最大100,支持offset。按点到完整几何的WGS84椭球最短距离筛选并排序,返回element_id、distance_m和属性;查询范围为当前模型管网;原始GIS、范围调整与歧义地点按各自入口核验",
m.NearbyElementsInput,
"webgis.view",
"asset",
@@ -90,7 +105,7 @@ _specs = [
(
"list_scada_devices",
"分页读取SCADA设备和物理单位",
m.PageInput,
m.ScadaDeviceListInput,
"scada.view",
"asset",
),
@@ -104,7 +119,7 @@ _specs = [
),
(
"query_model_baseline_series",
"读取最多20个元素的模型基线,不能称为实测",
"读取最多20个元素的模型基线,结果标注模型来源",
m.BaselineSeriesInput,
"simulation.view",
"model_baseline",
@@ -204,7 +219,7 @@ _specs = [
),
(
"query_peripheral_data",
"查询有限外围数据源;synthetic是合成证据",
"查询外围业务工单及监测数据;外围工单属于系统数据,监测数据按实际来源保留来源信息",
m.PeripheralInput,
"simulation.view",
"peripheral",
@@ -218,4 +233,44 @@ _specs = [
True,
),
]
_specs += [
("list_regions", "分页查询当前项目区域/DMA列表", m.PageInput, "webgis.view", "asset"),
("get_region", "查询当前项目区域边界,返回EPSG:4326几何;供水关系按拓扑核验", e.RegionInput, "webgis.view", "asset"),
("get_region_nodes", "分页查询区域配置的成员节点,未配置时使用区域几何筛选", e.RegionInput, "webgis.view", "asset"),
("get_node_connections", "按目标节点分页查询直接连接的管线", e.NodeConnectionsInput, "webgis.view", "asset"),
("query_network_extent", "按EPSG:4326矩形范围分页查询当前项目模型几何;GIS缺失时返回当前项目的具体缺口", e.ExtentInput, "webgis.view", "asset"),
("list_major_nodes", "按关联管道最小管径查询骨干节点,经纬度EPSG:4326", e.MajorNodesInput, "webgis.view", "asset"),
("get_model_configuration", "读取模式、曲线、控制规则、需水量、水质及反应等模型配置;集合按页返回", e.ModelConfigurationInput, "webgis.view", "asset"),
("list_model_configurations", "分页读取模式或曲线ID", e.ConfigurationListInput, "webgis.view", "asset"),
("search_project_places", "按项目配置的行政区搜索地名,最多10个候选,依赖天地图配置", e.PlaceSearchInput, "webgis.view", "asset"),
("reverse_geocode", "把EPSG:4326地图取点转为项目范围内地名,保留输入坐标", e.ReverseGeocodeInput, "webgis.view", "asset"),
("get_project_info", "读取当前项目概况与地图范围,公开字段限于项目业务概况", m.StrictInput, "webgis.view", "asset"),
("export_project_model", "从当前项目数据库导出INP或ChangeSet压缩文件,由宿主保存到会话文件", e.ProjectExportInput, "webgis.view", "asset"),
("get_sensor_placement_candidate", "读取监测点候选节点工程属性", e.SensorCandidateInput, "optimization.run", "asset"),
("export_sensor_placement", "导出已有监测点方案的Excel工程清单,由宿主保存到会话文件", e.SensorExportInput, "optimization.run", "analysis"),
("list_task_schedules", "分页读取当前项目定时任务配置,执行状态由调度系统管理", m.PageInput, "operational_task.view", "task"),
("get_task_run_history", "分页读取指定项目任务的运行历史", e.TaskHistoryInput, "operational_task.view", "task"),
("get_mutation_status", "读取当前身份和会话内的写入执行状态;started只表示已启动,超时后先核实目标数据,再按原请求接续", e.MutationStatusInput, "webgis.view", "task"),
("calculate_demand_allocation", "按管长权重分配总需水量到节点、区域或全网,返回分配建议,模型变更通过业务写入执行,按页返回", e.DemandAllocationInput, "webgis.view", "analysis", False, True),
("run_burst_simulation", "在隔离模型中模拟假设管道破口面积的水力影响;返回给定情景的模拟结果,正式方案通过方案工具保存", e.BurstSimulationInput, "simulation.run", "analysis", False, True),
("edit_model_element", "预览或增删改模型节点、管段、泵阀、水池和区域;变更范围为业务模型,现场设备操作按对应流程执行", e.ModelElementMutationInput, "webgis.edit", "asset", False, False, True),
("edit_model_configuration", "预览或修改模型需水量、模式、曲线、控制规则、水质及模拟选项", e.ConfigurationMutationInput, "webgis.edit", "asset", False, False, True),
("clean_scada_readings", "预览或清洗明确设备与24小时内时间窗的数据,写入cleaned_value", e.ScadaCleaningInput, "scada.clean", "observed", False, False, True),
("write_scada_readings", "预览或补录/修正SCADA数据,要求真实来源依据;观测来源按真实记录核验", e.ScadaWriteInput, "scada.clean", "observed", False, False, True),
("delete_scada_readings", "预览或删除明确设备与24小时内时间窗的SCADA记录", e.ScadaDeleteInput, "scada.clean", "observed", False, False, True),
("save_simulation_scenario", "预览或重新计算并保存正式隔离场景成果;有真实计算开销,观测数据按真实来源记录", e.SavedScenarioInput, "simulation.run", "analysis", False, False, True),
("create_sensor_placement", "预览或计算并保存正式压力监测点选址方案", e.SensorCreateInput, "optimization.run", "analysis", False, False, True),
("update_sensor_placement", "预览或修改本人有权编辑的监测点方案,保留旧点位乐观校验", e.SensorUpdateInput, "optimization.run", "analysis", False, False, True),
("create_operational_task", "预览或创建项目任务,Agent运行终态由执行系统管理", e.TaskCreateInput, "operational_task.write", "task", False, False, True),
("create_task_schedule", "预览或创建定时任务配置", e.ScheduleCreateInput, "operational_task.write", "task", False, False, True),
("update_task_schedule", "预览或启停定时任务配置,校验版本", e.ScheduleUpdateInput, "operational_task.write", "task", False, False, True),
("retry_operational_task", "预览或重新入队已终止任务;后续会话由任务调度系统创建", e.TaskRetryInput, "operational_task.write", "task", False, False, True),
("cancel_task_run", "预览或取消指定任务运行,校验运行版本", e.TaskCancelInput, "operational_task.write", "task", False, False, True),
]
_specs += [
("get_calculation_job", "读取当前身份与会话的持久计算状态;排队回执用于跟踪,分析依据读取成功结果", jobs.CalculationJobInput, "webgis.view", "task"),
("get_calculation_result", "读取成功计算的原始结果;data_path选择嵌套集合,limit/offset按需分页", jobs.CalculationResultInput, "webgis.view", "task"),
("cancel_calculation_job", "请求取消本人当前会话的计算;运行中取消由独立计算进程处理,须继续核实最终状态", jobs.CalculationJobInput, "webgis.view", "task"),
("list_calculation_jobs", "分页读取本人当前会话及任务范围的计算状态", jobs.CalculationJobsInput, "webgis.view", "task"),
]
TOOLS = {row[0]: QueryTool(*row) for row in _specs}
+4 -1
View File
@@ -21,7 +21,8 @@ class McpSettings(BaseSettings):
)
allowed_origins: list[str] = Field(default_factory=list)
context_ttl_seconds: int = Field(default=600, ge=30, le=600)
query_timeout_seconds: int = Field(default=30, ge=1, le=60)
query_timeout_seconds: int = Field(default=120, ge=1, le=240)
max_request_timeout_seconds: int = Field(default=240, ge=1, le=240)
max_calculation_timeout_seconds: int = Field(default=3600, ge=120, le=86400)
model_config = SettingsConfigDict(
env_prefix="TJWATER_MCP_",
@@ -32,6 +33,8 @@ class McpSettings(BaseSettings):
@model_validator(mode="after")
def validate_resource(self):
if self.query_timeout_seconds > self.max_request_timeout_seconds:
raise ValueError("query timeout exceeds the HTTP request timeout limit")
if self.dev_http_issuer_uri:
dev_url = urlsplit(self.dev_http_issuer_uri)
try:
+7 -1
View File
@@ -15,6 +15,7 @@ from app.mcp.models import BindInput, BoundContext
from app.mcp.scope import QueryScope
from app.mcp.config import McpSettings
from mcp.server.auth.provider import AccessToken
from app.mcp.work_order_skills import skill_name_for_task
def task_repository(scope: QueryScope) -> OperationalTaskRepository:
@@ -30,6 +31,9 @@ async def verify_task(scope: QueryScope, binding: BoundContext) -> None:
task = await repo.get_task(str(binding.task_id))
if not task:
raise HTTPException(404, "Task is not in this project")
binding.replay_context = task.get("source", {}).get("replay_context")
if skill_name_for_task(task.get("workflow"), binding.replay_context) == "burst-event-handling":
binding.tool_cap = [name for name in binding.tool_cap if name != "run_burst_location"]
if binding.purpose == "task_collaboration":
return
if OPERATIONAL_TASK_EXECUTE not in permissions:
@@ -69,7 +73,9 @@ async def bind_context(
project_id=scope.project.project_id,
subject=principal.subject,
client_id=principal.client_id,
tool_cap=sorted(name for name, tool in TOOLS.items() if command.purpose != "task_reader" or not tool.computation),
tool_cap=sorted(name for name, tool in TOOLS.items()
if (command.purpose != "task_reader" or not (tool.computation or tool.mutating or name == "cancel_calculation_job"))
and (not tool.mutating or command.purpose == "ordinary_chat")),
permission_cap=sorted(permissions_for_context(scope.project)),
native_session_id=command.native_session_id,
purpose=command.purpose,
+63
View File
@@ -0,0 +1,63 @@
"""Project model-facing timestamps after saving the original query evidence."""
from datetime import datetime, timedelta, timezone
import re
from typing import Any
DISPLAY_TIME_ZONE = "Asia/Shanghai"
DISPLAY_OFFSET = timezone(timedelta(hours=8))
_INSTANT = re.compile(
r"\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?P<fraction>\.\d+)?(?:Z|[+-]\d{2}:\d{2})"
)
_PROSE_FIELDS = frozenset({
"title", "summary", "detail", "description", "notice", "label", "message",
"current_conclusion", "conclusion_boundary", "recommendations", "pending_items",
"missing_data", "required_data", "impact", "scope", "fallback_text", "text",
})
_OPAQUE_FIELDS = frozenset({"id", "cursor", "url", "uri", "path", "sha256", "instructions", "code", "sql"})
_OPAQUE_SUFFIXES = ("_id", "_ids", "_ref", "_refs", "_cursor", "_digest", "_sha256", "_url", "_uri", "_path")
def _opaque(key: str) -> bool:
return key in _OPAQUE_FIELDS or key.endswith(_OPAQUE_SUFFIXES)
def _display_instant(value: str) -> str:
match = _INSTANT.fullmatch(value)
if match is None:
return value
try:
local = datetime.fromisoformat(value).astimezone(DISPLAY_OFFSET)
except (ValueError, OverflowError):
return value
fraction = match.group("fraction") or ""
return (f"{local.year:04d}-{local.month:02d}-{local.day:02d}T"
f"{local.hour:02d}:{local.minute:02d}:{local.second:02d}{fraction}+08:00")
def display_time_text(value: str) -> str:
"""Convert explicit instants in prose while retaining links and quoted code."""
parts = re.split(r"(https?://\S+|tjwater://\S+|`[^`]*`)", value)
for index in range(0, len(parts), 2):
parts[index] = re.sub(
r"(?<![A-Za-z0-9_/+:.-])" + _INSTANT.pattern + r"(?![A-Za-z0-9_:+/-])",
lambda match: _display_instant(match.group()), parts[index],
)
return "".join(parts)
def display_query_result(value: Any, *, field: str = "") -> Any:
"""Keep instants, fractions and identifiers intact; change display offsets."""
if _opaque(field):
return value
if isinstance(value, dict):
path = value.get("selected_path") or []
opaque_selection = isinstance(path, list) and bool(path) and _opaque(str(path[-1]))
return {key: item if key == "value" and opaque_selection else display_query_result(item, field=key)
for key, item in value.items()}
if isinstance(value, list):
return [display_query_result(item, field=field) for item in value]
if isinstance(value, str):
if field in _PROSE_FIELDS:
return display_time_text(value)
return _display_instant(value) if len(value) <= 128 else value
return value
+16
View File
@@ -0,0 +1,16 @@
"""Errors with a known execution boundary for MCP calculations."""
from fastapi import HTTPException
class CalculationCapacityExceeded(HTTPException):
"""Raised by admission control before the native function is invoked."""
code = "CALCULATION_CAPACITY_EXCEEDED"
def __init__(self) -> None:
super().__init__(
503,
"Calculation capacity is occupied; calculation not started. "
"Retry the same request after an active calculation finishes.",
)
+289
View File
@@ -0,0 +1,289 @@
"""Finite project services exposed through MCP; no routes, SQL or actor overrides."""
from datetime import datetime, timedelta
from typing import Annotated, Literal
from uuid import UUID
from pydantic import BaseModel, ConfigDict, Field, JsonValue, model_validator, create_model
from app.mcp.models import Id, PageInput, StrictInput
from app.mcp.analysis_models import (
CalculationInput, ScenarioInput, ContaminantSimulationInput, ValveClosureInput,
FlushingInput, PressureRegulationInput,
)
from app.domain.schemas.operational_task import (
OperationalTaskCreate, OperationalTaskScheduleCreate, OperationalTaskScheduleUpdate,
OperationalTaskRetryCommand, AgentRunCancelCommand,
)
from app.domain.schemas.sensor_placement import SensorPlacementUpdateRequest
class RegionInput(PageInput):
region_id: Id
class NodeConnectionsInput(PageInput):
node_id: Id
class ExtentInput(PageInput):
element_type: Literal["junction", "pipe", "valve", "pump", "tank", "reservoir"] = "pipe"
bounds: tuple[float, float, float, float] = Field(description="EPSG:4326 [west,south,east,north];经度范围满足west<=east")
@model_validator(mode="after")
def valid_bounds(self):
west, south, east, north = self.bounds
if not (-180 <= west < east <= 180 and -85 <= south < north <= 85):
raise ValueError("Invalid EPSG:4326 bounds")
if east - west > 1 or north - south > 1:
raise ValueError("Extent must be at most one degree per axis")
return self
class MajorNodesInput(PageInput):
min_diameter_mm: float = Field(gt=0, le=10000, allow_inf_nan=False)
ConfigKind = Literal[
"pattern", "curve", "control", "rule", "quality", "source", "reaction",
"pipe-reaction", "tank-reaction", "mixing", "emitter", "demand",
"time", "energy", "pump-energy", "option",
]
class ModelConfigurationInput(PageInput):
kind: ConfigKind
object_id: Id | None = None
data_path: list[Id] = Field(default_factory=list, max_length=8, description="选择嵌套字段,再按limit/offset分页读取")
@model_validator(mode="after")
def valid_target(self):
needs_id = {"pattern", "curve", "quality", "source", "pipe-reaction", "tank-reaction", "mixing", "emitter", "demand", "pump-energy"}
if (self.kind in needs_id) != (self.object_id is not None):
raise ValueError("object_id is required only for object-scoped configuration")
return self
class ConfigurationListInput(PageInput):
kind: Literal["pattern", "curve"]
class PlaceSearchInput(StrictInput):
keyword: str = Field(min_length=1, max_length=120)
class ReverseGeocodeInput(StrictInput):
longitude: float = Field(ge=-180, le=180, allow_inf_nan=False)
latitude: float = Field(ge=-90, le=90, allow_inf_nan=False)
class ProjectExportInput(StrictInput):
format: Literal["inp", "change_set"] = "inp"
version: Literal["2", "3"] = "2"
class SensorCandidateInput(StrictInput):
node_id: Id
class SensorExportInput(StrictInput):
run_id: UUID
class TaskHistoryInput(PageInput):
target_task_id: UUID
class MutationStatusInput(StrictInput):
request_id: UUID
class DemandAllocationInput(CalculationInput):
target: Literal["nodes", "region", "network"]
demand: float = Field(gt=0, allow_inf_nan=False, description="模型流量单位的总需水量;调用前查询模型单位")
node_ids: list[Id] = Field(default_factory=list, max_length=200)
region_id: Id | None = None
limit: int = Field(default=20, ge=1, le=200)
offset: int = Field(default=0, ge=0, le=10000)
@model_validator(mode="after")
def valid_target(self):
if (self.target == "nodes") != bool(self.node_ids) or (self.target == "region") != (self.region_id is not None):
raise ValueError("Provide node_ids for nodes or region_id for region only")
if len(self.node_ids) != len(set(self.node_ids)):
raise ValueError("node_ids must be distinct")
return self
class BurstSimulationInput(ScenarioInput):
burst_pipe_ids: list[Id] = Field(min_length=1, max_length=20)
orifice_areas_cm2: list[Annotated[float, Field(gt=0, allow_inf_nan=False)]] = Field(min_length=1, max_length=20, description="破口面积,单位cm²;保留面积来源及估算依据")
@model_validator(mode="after")
def paired_bursts(self):
if len(self.burst_pipe_ids) != len(self.orifice_areas_cm2) or len(set(self.burst_pipe_ids)) != len(self.burst_pipe_ids):
raise ValueError("Distinct pipes and one orifice area per pipe are required")
return self
class MutationInput(CalculationInput):
request_id: UUID = Field(description="操作幂等键;预览与执行使用相同值,失败后先核实状态")
dry_run: bool = Field(default=True, strict=True, description="默认只预览;执行需dry_run=false及预览返回的expected_digest")
expected_digest: str | None = Field(default=None, pattern=r"^[a-f0-9]{64}$")
@model_validator(mode="after")
def require_preview(self):
if not self.dry_run and self.expected_digest is None:
raise ValueError("Execution requires the preview's expected_digest")
command = getattr(self, "command", None)
if command is not None and getattr(command, "request_id", self.request_id) != self.request_id:
raise ValueError("Nested command request_id must match the operation")
return self
class ModelElementMutationInput(MutationInput):
operation: Literal["create", "update", "delete"]
element_type: Literal["junction", "pipe", "valve", "pump", "tank", "reservoir", "region"]
element_id: Id
properties: dict[Id, JsonValue] = Field(default_factory=dict, max_length=40, description="仅允许get_network_schema声明的可写字段;id由element_id绑定")
@model_validator(mode="after")
def valid_properties(self):
if (self.operation == "delete") == bool(self.properties):
raise ValueError("Create/update require properties; delete takes none")
if "id" in self.properties or "type" in self.properties:
raise ValueError("Element identity is bound separately")
return self
class ConfigurationMutationInput(MutationInput):
operation: Literal["create", "update", "delete"] = "update"
kind: ConfigKind
object_id: Id | None = None
properties: dict[Id, JsonValue] = Field(default_factory=dict, max_length=40)
@model_validator(mode="after")
def valid_configuration(self):
ModelConfigurationInput(context_handle=self.context_handle, kind=self.kind, object_id=self.object_id)
if self.operation != "update" and self.kind not in {"pattern", "curve", "source", "mixing"}:
raise ValueError("Only pattern, curve, source and mixing support create/delete")
if (self.operation == "delete") == bool(self.properties):
raise ValueError("Create/update require properties; delete takes none")
if "id" in self.properties:
raise ValueError("Object identity is bound separately")
return self
class ScadaCleaningInput(MutationInput):
start: datetime
end: datetime
device_ids: list[Id] = Field(min_length=1, max_length=20, description="显式列出本次处理的实际设备ID")
@model_validator(mode="after")
def valid_window(self):
if self.start.tzinfo is None or self.end.tzinfo is None or not timedelta(0) < self.end - self.start <= timedelta(days=1):
raise ValueError("SCADA writes require explicit timezone and at most 24 hours")
if len(set(self.device_ids)) != len(self.device_ids):
raise ValueError("device_ids must be distinct")
return self
class ScadaRecord(BaseModel):
model_config = ConfigDict(extra="forbid")
device_id: Id
time: datetime
monitored_value: float | None = Field(default=None, allow_inf_nan=False)
cleaned_value: float | None = Field(default=None, allow_inf_nan=False)
@model_validator(mode="after")
def valid_record(self):
if self.time.tzinfo is None or not ({"monitored_value", "cleaned_value"} & self.model_fields_set):
raise ValueError("Record needs an explicit timezone and a value field")
return self
class ScadaWriteInput(MutationInput):
operation: Literal["insert", "update"]
records: list[ScadaRecord] = Field(min_length=1, max_length=500)
provenance: str = Field(min_length=1, max_length=2000, description="补录/修正观测的真实来源与依据,模型结果按其来源另行记录")
@model_validator(mode="after")
def bounded_records(self):
keys = [(r.device_id, r.time) for r in self.records]
if len(set(keys)) != len(keys):
raise ValueError("Duplicate device/time records")
if (max(r.time for r in self.records) - min(r.time for r in self.records)).total_seconds() > 86400:
raise ValueError("At most 24 hours per write")
return self
class ScadaDeleteInput(ScadaCleaningInput):
reason: str = Field(min_length=1, max_length=2000)
class SensorCreateInput(MutationInput):
name: str = Field(min_length=1, max_length=64)
method: Literal["sensitivity", "kmeans"]
sensor_count: int = Field(ge=1, le=20)
min_diameter: int = Field(default=0, ge=0, le=5000)
class SensorUpdateInput(MutationInput):
run_id: UUID
changes: SensorPlacementUpdateRequest
def _payload(model):
return create_model(model.__name__ + "Payload", __config__=ConfigDict(extra="forbid"), **{
name: (field.annotation, field) for name, field in model.model_fields.items()
if name not in {"context_handle", "timeout_seconds"}
})
BurstPayload = _payload(BurstSimulationInput)
ContaminantPayload = _payload(ContaminantSimulationInput)
ValvePayload = _payload(ValveClosureInput)
FlushingPayload = _payload(FlushingInput)
PressurePayload = _payload(PressureRegulationInput)
class SavedScenarioInput(MutationInput):
name: str = Field(min_length=1, max_length=64)
scenario: Literal["burst", "contaminant", "valve_closure", "flushing", "pressure_regulation"]
parameters: BurstPayload | ContaminantPayload | ValvePayload | FlushingPayload | PressurePayload
@model_validator(mode="after")
def matching_scenario(self):
expected = {"burst": BurstSimulationInput, "contaminant": ContaminantSimulationInput, "valve_closure": ValveClosureInput, "flushing": FlushingInput, "pressure_regulation": PressureRegulationInput}[self.scenario]
expected.model_validate({**self.parameters.model_dump(), "context_handle": self.context_handle})
return self
class TaskCreateInput(MutationInput):
command: OperationalTaskCreate
@model_validator(mode="after")
def unclaimed_event(self):
if self.command.agent_session_id is not None or self.command.task_type != "event_investigation" or self.command.source.kind not in {"operator", "agent"}:
raise ValueError("Only unclaimed operator/agent event investigations may be created")
return self
class ScheduleCreateInput(MutationInput):
command: OperationalTaskScheduleCreate
class ScheduleUpdateInput(MutationInput):
schedule_id: UUID
command: OperationalTaskScheduleUpdate
class TaskRetryInput(MutationInput):
target_task_id: UUID
command: OperationalTaskRetryCommand
class TaskCancelInput(MutationInput):
target_task_id: UUID
run_id: UUID
command: AgentRunCancelCommand
+224
View File
@@ -0,0 +1,224 @@
"""Adapters for bounded, project-owned queries and downloadable exports."""
import asyncio
import base64
import hashlib
import io
import json
import zipfile
from datetime import datetime, timezone
from uuid import uuid4
from typing import Any, TYPE_CHECKING
from fastapi import HTTPException
from fastapi.encoders import jsonable_encoder
from psycopg import sql
from pydantic import BaseModel
from app.infra.db.postgresql.read_queries import ELEMENT_VIEWS, read_rows
from app.mcp.models import PageInfo, QueryResult, BoundContext
from app.mcp.scope import QueryScope
if TYPE_CHECKING:
from app.mcp.catalog import QueryTool
# Both function names and object identities are selected from a finite registry.
CONFIG_FUNCTIONS = {
"pattern": ("pattern", "id"), "curve": ("curve", "id"),
"control": ("control", None), "rule": ("rule", None),
"quality": ("quality", "node"), "source": ("source", "node"),
"reaction": ("reaction", None), "pipe-reaction": ("pipe_reaction", "pipe"),
"tank-reaction": ("tank_reaction", "tank"), "mixing": ("mixing", "tank"),
"emitter": ("emitter", "junction"), "demand": ("demand", "junction"),
"time": ("time", None), "energy": ("energy", None),
"pump-energy": ("pump_energy", "pump"), "option": ("option_v3", None),
}
EXTENDED_READ_TOOLS = {
"list_regions", "get_region", "get_region_nodes", "get_node_connections",
"query_network_extent", "list_major_nodes", "get_model_configuration",
"list_model_configurations", "search_project_places", "reverse_geocode",
"get_project_info", "export_project_model", "get_sensor_placement_candidate",
"export_sensor_placement", "list_task_schedules", "get_task_run_history", "get_mutation_status",
}
ARTIFACT_TOOLS = {"export_project_model", "export_sensor_placement"}
def configuration(network: str, kind: str, object_id: str | None = None) -> Any:
from app.services import tjnetwork
function, identity = CONFIG_FUNCTIONS[kind]
args = (network, object_id) if identity else (network,)
return getattr(tjnetwork, "get_" + function)(*args)
def artifact(filename: str, payload: bytes, mime_type: str) -> dict[str, Any]:
if len(payload) > 5 * 1024 * 1024:
raise HTTPException(413, "Export exceeds the 5 MiB compressed file budget")
return {"file": {
"name": filename, "mime_type": mime_type, "size": len(payload),
"sha256": hashlib.sha256(payload).hexdigest(),
"encoding": "base64", "data": base64.b64encode(payload).decode("ascii"),
}}
def project_export(network: str, args: BaseModel) -> dict[str, Any]:
from app.services import tjnetwork
from app.services.project_inp import temporary_project_inp
output = io.BytesIO()
with zipfile.ZipFile(output, "w", zipfile.ZIP_DEFLATED) as archive:
if args.format == "inp":
with temporary_project_inp(network, purpose="mcp_export", version=args.version) as path:
archive.write(path, arcname="model.inp")
else:
changes = tjnetwork.export_inp(network, args.version)
if changes.operations:
changes.operations[0]["vertex"] = json.dumps(tjnetwork.get_all_vertices(network))
changes.operations[0]["scada"] = json.dumps(tjnetwork.get_all_scada_info(network))
archive.writestr("model.json", json.dumps(jsonable_encoder(changes.operations), ensure_ascii=False))
return artifact("model.zip", output.getvalue(), "application/zip")
def sensor_export(network: str, args: BaseModel, user: Any, project: Any) -> dict[str, Any]:
from app.services.sensor_placement import get_sensor_placement_run, build_sensor_placement_workbook
run = get_sensor_placement_run(network, args.run_id)
workbook = build_sensor_placement_workbook(
network=network, scheme=run, sensor_location=run["sensor_locations"], adjustment_status={},
)
return artifact("sensor-placement.xlsx", workbook.getvalue(), "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet")
async def execute_extended(scope: QueryScope, binding: BoundContext, tool: "QueryTool", args: BaseModel) -> QueryResult:
name, conn, network = tool.name, scope.connection, scope.project.project_code
page, gaps, truncated = None, [], False
def paginate(items, *, already_paged=False):
nonlocal page, truncated
if already_paged:
more, selected, total = len(items) > args.limit, items[:args.limit], None
else:
total = len(items)
more = args.offset + args.limit < total
selected = items[args.offset:args.offset + args.limit]
page = PageInfo(limit=args.limit, offset=args.offset, total=total, has_more=more)
truncated = more or args.offset > 0
return selected
if name == "list_regions":
rows = await read_rows(conn, "SELECT id,region_type FROM gis.regions ORDER BY id LIMIT %s OFFSET %s", (args.limit + 1, args.offset))
data = paginate(rows, already_paged=True)
elif name == "get_region":
rows = await read_rows(conn, "SELECT id,region_type,ST_AsGeoJSON(ST_Transform(boundary,4326))::json AS geometry FROM gis.regions WHERE id=%s", (args.region_id,))
if not rows:
raise HTTPException(404, "Region not found in the current project")
data = {**rows[0], "crs": "EPSG:4326"}
elif name == "get_region_nodes":
rows = await read_rows(conn, """WITH configured AS (
SELECT node_id FROM gis.region_nodes WHERE region_id=%s
), members AS (
SELECT node_id FROM configured UNION ALL
SELECT n.node_id FROM gis.node_geometries n JOIN gis.regions r
ON r.id=%s AND ST_Intersects(n.geom,r.boundary)
WHERE NOT EXISTS (SELECT 1 FROM configured)
) SELECT node_id AS id FROM members ORDER BY node_id LIMIT %s OFFSET %s""",
(args.region_id, args.region_id, args.limit + 1, args.offset))
data = paginate(rows, already_paged=True)
gaps = ["Uses configured region membership, or geometry if none is configured; membership does not prove hydraulic connectivity"]
elif name == "get_node_connections":
rows = await read_rows(conn, """SELECT id,link_type,start_node_id,end_node_id FROM network.links
WHERE start_node_id=%s OR end_node_id=%s ORDER BY id LIMIT %s OFFSET %s""", (args.node_id, args.node_id, args.limit + 1, args.offset))
data = paginate(rows, already_paged=True)
elif name == "query_network_extent":
statement = sql.SQL("""SELECT v.id,to_jsonb(v)-'geom' AS properties,
ST_AsGeoJSON(ST_Transform(v.geom,4326))::json AS geometry FROM gis.{} v
WHERE ST_Intersects(v.geom,ST_Transform(ST_MakeEnvelope(%s,%s,%s,%s,4326),3857))
ORDER BY v.id LIMIT %s OFFSET %s""").format(sql.Identifier(ELEMENT_VIEWS[args.element_type]))
rows = await read_rows(conn, statement, (*args.bounds, args.limit + 1, args.offset))
data = {"items": paginate(rows, already_paged=True), "crs": "EPSG:4326", "source": "current_project_model"}
gaps = ["Only current-project model elements with valid stored geometry are covered; missing GIS geometry is not evidence of an empty network"]
elif name == "list_major_nodes":
rows = await read_rows(conn, """SELECT n.id,ST_X(ST_Transform(n.geom,4326)) AS longitude,
ST_Y(ST_Transform(n.geom,4326)) AS latitude FROM gis.junctions n
WHERE EXISTS (SELECT 1 FROM gis.pipes p WHERE p.diameter >= %s
AND (p.node1=n.id OR p.node2=n.id)) ORDER BY n.id LIMIT %s OFFSET %s""", (args.min_diameter_mm, args.limit + 1, args.offset))
data = {"items": paginate(rows, already_paged=True), "crs": "EPSG:4326"}
gaps = ["Only junctions with stored geometry and qualifying model pipes are covered; no other project's GIS is queried"]
elif name == "get_model_configuration":
data = await asyncio.to_thread(configuration, network, args.kind, args.object_id)
for key in args.data_path:
if not isinstance(data, dict) or key not in data:
raise HTTPException(404, "Configuration path not found")
data = data[key]
if isinstance(data, dict):
data = {"items": paginate([{"key": key, "value": value} for key, value in sorted(data.items())]), "data_path": args.data_path}
elif isinstance(data, list):
data = {"items": paginate(data), "data_path": args.data_path}
else:
data = {"value": data, "data_path": args.data_path}
elif name == "list_model_configurations":
from app.services import tjnetwork
rows = await asyncio.to_thread(getattr(tjnetwork, "get_" + args.kind + "s"), network)
data = paginate(sorted(rows or []))
elif name in {"search_project_places", "reverse_geocode", "get_project_info"}:
project = await scope.metadata.get_project_by_id(scope.project.project_id)
if project is None:
raise HTTPException(404, "Project not found")
if name == "get_project_info":
data = {key: getattr(project, key, None) for key in ("name", "code", "description", "map_extent", "status")}
data["project_id"] = str(scope.project.project_id)
else:
from app.domain.schemas.project_map_config import ProjectMapConfig
from app.domain.schemas.place_search import PlaceSearchRequest, ReverseGeocodingRequest
from app.services.place_search import search_tianditu_places, PlaceSearchError
from app.services.reverse_geocoding import reverse_geocode_tianditu
config = ProjectMapConfig.model_validate(project.map_config or {}).place_search
if config is None or not config.enabled:
raise HTTPException(503, "Current project place search is not configured")
await scope.metadata.session.commit()
try:
data = (await search_tianditu_places(PlaceSearchRequest(keyword=args.keyword), config=config)
if name == "search_project_places" else await reverse_geocode_tianditu(
ReverseGeocodingRequest(longitude=args.longitude, latitude=args.latitude), config=config))
truncated = bool(getattr(data, "truncated", False))
except PlaceSearchError as exc:
raise HTTPException(exc.status_code, exc.detail) from exc
elif name == "export_project_model":
data = await asyncio.to_thread(project_export, network, args)
elif name == "get_sensor_placement_candidate":
from app.services.sensor_placement import get_sensor_placement_candidate
data = await asyncio.to_thread(get_sensor_placement_candidate, network, args.node_id)
elif name == "export_sensor_placement":
data = await asyncio.to_thread(sensor_export, network, args, scope.user, scope.project)
elif name == "get_mutation_status":
from app.infra.db.postgresql.mcp_queries import McpQueryRepository
repo = McpQueryRepository(conn, scope.project.project_id)
previous = await repo.receipt("mutation:" + binding.subject + ":" + binding.native_session_id, str(args.request_id))
if previous is None:
data = {"status": "not_started", "request_id": str(args.request_id)}
else:
output = previous["result"].get("output", {})
data = {"request_id": str(args.request_id), "status": previous["result"]["status"],
"tool": previous["parameters"]["tool"], "source_ref": output.get("source_ref"),
"queried_at": output.get("queried_at")}
if previous["result"]["status"] == "rejected":
from app.mcp.mutation_service import retryable_capacity_rejection
data["retryable"] = retryable_capacity_rejection(previous["result"])
data["error"] = previous["result"].get("error")
gaps = ["started does not prove success or rollback; verify target state before any new operation"]
elif name in {"list_task_schedules", "get_task_run_history"}:
from app.mcp.query_service import task_service
service = task_service(scope)
if name == "list_task_schedules":
data = paginate(await service.list_schedules())
else:
rows = await service.task_runs(args.target_task_id, limit=args.limit + 1, offset=args.offset)
data = paginate(rows, already_paged=True)
else:
raise ValueError("Unknown extended query")
return QueryResult(source=name, source_ref="mcp:" + uuid4().hex,
queried_at=datetime.now(timezone.utc), semantics=tool.semantics,
data=jsonable_encoder(data), page=page, truncated=truncated, data_gaps=gaps)
+9 -1
View File
@@ -21,8 +21,12 @@ class PageInput(StrictInput):
offset: int = Field(default=0, ge=0, le=10000)
class ScadaDeviceListInput(PageInput):
limit: int = Field(default=100, ge=1, le=500)
class TaskContextInput(PageInput):
section: Literal["summary", "task", "report", "evidence", "actions", "progress", "access_requests", "plan", "run", "skill"] = "summary"
section: Literal["summary", "task", "report", "evidence", "actions", "progress", "access_requests", "plan", "run", "skill", "questions"] = "summary"
report_revision: int | None = Field(default=None, ge=1)
data_path: list[Id] = Field(default_factory=list, max_length=8)
@@ -78,6 +82,8 @@ class SchemaInput(StrictInput):
"pump-energy",
"option",
"scada-device",
"region", "pattern", "curve", "control", "rule", "quality", "source",
"reaction", "pipe-reaction", "tank-reaction", "mixing", "emitter", "demand",
]
@@ -253,6 +259,7 @@ class PageInfo(BaseModel):
class QueryResult(BaseModel):
model_config = ConfigDict(extra="forbid")
display_time_zone: Literal["Asia/Shanghai"] = "Asia/Shanghai"
source: str
source_ref: str
queried_at: datetime
@@ -289,5 +296,6 @@ class BoundContext(BaseModel):
task_id: UUID | None = None
run_id: UUID | None = None
execution_request_id: UUID | None = None
replay_context: dict | None = None
expires_at: datetime
revoked_at: datetime | None = None
+339
View File
@@ -0,0 +1,339 @@
"""Previewed writes with project permissions and a durable no-retry journal."""
import asyncio
import hashlib
import json
import math
from datetime import datetime, timezone
from uuid import uuid4
from typing import Any, TYPE_CHECKING
from fastapi import HTTPException
from fastapi.encoders import jsonable_encoder
from psycopg import sql
from pydantic import BaseModel
from app.infra.db.postgresql.mcp_queries import McpQueryRepository
from app.infra.db.postgresql.read_queries import read_rows, scada_devices
from app.mcp.extended_service import CONFIG_FUNCTIONS, configuration
from app.mcp.errors import CalculationCapacityExceeded
from app.mcp.models import QueryResult, BoundContext
from app.mcp.scope import QueryScope
if TYPE_CHECKING:
from app.mcp.catalog import QueryTool
def digest(value: Any) -> str:
return hashlib.sha256(json.dumps(jsonable_encoder(value), sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode()).hexdigest()
def check_digest(args: BaseModel, preview: dict) -> None:
if not args.dry_run and args.expected_digest != digest(preview):
raise HTTPException(409, "Preview changed; request a new dry_run before execution")
def mutation_arguments(args: BaseModel, *, preview: bool) -> dict[str, Any]:
excluded = {"context_handle", "timeout_seconds"}
if preview:
excluded.update({"dry_run", "expected_digest"})
payload = args.model_dump(mode="json", exclude=excluded)
if hasattr(args, "records"):
# An omitted cleaned_value means keep it during update; explicit null
# means clear it. Preserve that difference in previews and replay keys.
payload["records"] = [record.model_dump(mode="json", exclude_unset=True) for record in args.records]
return payload
def check_properties(schema: dict, properties: dict, *, creating: bool = False) -> None:
def finite(value):
if isinstance(value, float) and not math.isfinite(value):
raise HTTPException(422, "Model values must be finite")
if isinstance(value, dict):
for child in value.values():
finite(child)
elif isinstance(value, list):
for child in value:
finite(child)
finite(properties)
for key, value in properties.items():
spec = schema.get(key)
if spec is None or spec.get("readonly"):
raise HTTPException(422, "Unknown or read-only model property: " + key)
if value is None and not spec.get("optional", False):
raise HTTPException(422, "Required model property cannot be null: " + key)
expected = spec.get("type")
if value is not None:
valid = {"str": isinstance(value, str), "float": isinstance(value, (int, float)) and not isinstance(value, bool),
"int": isinstance(value, int) and not isinstance(value, bool), "bool": isinstance(value, bool),
"list": isinstance(value, list), "str_list": isinstance(value, list) and all(isinstance(v, str) for v in value),
"float_list": isinstance(value, list) and all(isinstance(v, (int, float)) and not isinstance(v, bool) for v in value),
"tuple_list": isinstance(value, list)}.get(expected, True)
if not valid:
raise HTTPException(422, "Invalid model property type: " + key)
if isinstance(value, list) and len(value) > 500:
raise HTTPException(422, "Model property exceeds 500 items")
if isinstance(value, list) and isinstance(spec.get("element"), dict):
for item in value:
if not isinstance(item, dict):
raise HTTPException(422, "Model list item must be an object")
check_properties(spec["element"], item, creating=True)
if creating:
required = {key for key, spec in schema.items() if not spec.get("readonly") and not spec.get("optional")}
if not required.issubset(properties):
raise HTTPException(422, "Missing required model properties: " + ",".join(sorted(required - properties.keys())))
def model_change(network: str, name: str, args: BaseModel, *, execute: bool) -> dict[str, Any]:
from app.services import tjnetwork
from app.native.wndb.core.database import ChangeSet, model_mutation_transaction
# Preview revalidation and the actual write share the same model mutation
# lock; another WNDB writer cannot change the target between them.
with model_mutation_transaction(network):
if name == "edit_model_element":
kind, identity, object_id = args.element_type, "id", args.element_id
current = getattr(tjnetwork, "get_" + kind)(network, object_id)
schema = getattr(tjnetwork, "get_" + kind + "_schema")(network)
else:
kind, identity = CONFIG_FUNCTIONS[args.kind]
object_id = args.object_id
current = configuration(network, args.kind, object_id)
schema = getattr(tjnetwork, "get_" + kind + "_schema")(network)
check_properties(schema, args.properties, creating=args.operation == "create")
if identity and args.operation == "create" and current:
raise HTTPException(409, "Object already exists")
if identity and args.operation != "create" and not current:
raise HTTPException(404, "Model object not found")
preview = {"tool": name, "operation": args.operation, "kind": kind,
"object_id": object_id, "before": current, "properties": args.properties}
if not execute:
return preview
check_digest(args, preview)
prefix = {"create": "add_", "update": "set_", "delete": "delete_"}[args.operation]
values = ({identity: object_id} if identity else {}) | args.properties
if kind == "region" and "boundary" in values:
values["boundary"] = [tuple(point) for point in values["boundary"]]
result = getattr(tjnetwork, prefix + kind)(network, ChangeSet(values))
if not result.operations:
raise HTTPException(409, "Model rejected the change; no operations were applied")
return {"operations": result.operations, "field_device_control": False}
async def scada_snapshot(scope: QueryScope, args: BaseModel, *, connection: Any = None, lock: bool = False) -> dict[str, Any]:
ids = sorted({r.device_id for r in args.records}) if hasattr(args, "records") else sorted(set(args.device_ids))
for device_id in ids:
if not await scada_devices(scope.connection, device_id=device_id, limit=1):
raise HTTPException(404, "SCADA device not found in the current project")
async def read(ts):
suffix = " FOR UPDATE" if lock else ""
if hasattr(args, "records"):
rows = []
for record in args.records:
rows.extend(await read_rows(ts, "SELECT time,device_id,monitored_value,cleaned_value FROM scada.measurements WHERE device_id=%s AND time=%s" + suffix, (record.device_id, record.time)))
else:
rows = await read_rows(ts, """SELECT time,device_id,monitored_value,cleaned_value FROM scada.measurements
WHERE device_id=ANY(%s) AND time >= %s AND time <= %s ORDER BY device_id,time LIMIT 10001""" + suffix, (ids, args.start, args.end))
if len(rows) > 10000:
raise HTTPException(413, "Narrow the SCADA window to at most 10000 records")
return {"count": len(rows), "devices": ids, "state_digest": digest(rows)}
if connection is not None:
return await read(connection)
async with scope.timescale() as ts:
return await read(ts)
async def preview_change(scope: QueryScope, binding: BoundContext, name: str, args: BaseModel) -> dict[str, Any]:
network = scope.project.project_code
if name in {"edit_model_element", "edit_model_configuration"}:
from app.mcp.analysis_service import bounded_calculation
return await bounded_calculation(model_change, network, name, args, execute=False)
payload = mutation_arguments(args, preview=True)
state = None
if name in {"clean_scada_readings", "write_scada_readings", "delete_scada_readings"}:
state = await scada_snapshot(scope, args)
elif name in {"update_sensor_placement"}:
from app.services.sensor_placement import get_sensor_placement_run, can_edit_sensor_placement
state = await asyncio.to_thread(get_sensor_placement_run, network, args.run_id)
if not can_edit_sensor_placement(scope.user, state):
raise HTTPException(403, "Cannot edit another user's sensor placement")
elif name in {"retry_operational_task", "cancel_task_run"}:
from app.mcp.query_service import task_service
state = (await task_service(scope).detail(args.target_task_id)).model_dump(mode="json")
elif name == "update_task_schedule":
from app.mcp.query_service import task_service
schedules = await task_service(scope).list_schedules()
state = next((s.model_dump(mode="json") for s in schedules if s.schedule_id == args.schedule_id), None)
if state is None:
raise HTTPException(404, "Schedule not found in the current project")
return {"tool": name, "parameters": payload, "before": state}
def saved_scenario(network: str, username: str, args: BaseModel) -> Any:
from app.services import simulation_scenarios as scenarios
from app.services.simulation import query_corresponding_element_id_and_query_id
parameters = args.parameters.model_dump(exclude_none=True)
parameters.pop("output_node_ids")
parameters.pop("output_link_ids")
common = dict(name=network, modify_pattern_start_time=parameters.pop("start_time").isoformat(),
modify_total_duration=parameters.pop("duration"), scheme_name=args.name, persist_results=True)
if args.scenario == "burst":
return scenarios.burst_analysis(**common, username=username,
burst_ID=parameters["burst_pipe_ids"], burst_size=parameters["orifice_areas_cm2"])
if args.scenario == "contaminant":
return scenarios.contaminant_simulation(**common, username=username,
source=parameters["source"], concentration=parameters["concentration"], source_pattern=parameters.get("pattern"))
if args.scenario == "valve_closure":
return scenarios.valve_close_analysis(**common, username=username,
modify_valve_opening={valve: 0.0 for valve in parameters["valves"]})
if args.scenario == "flushing":
from app.native.wndb.model.options import get_option
if get_option(network).get("UNITS") not in {"LPS", "CMH"}:
raise ValueError("Flushing supports LPS or CMH models only")
return scenarios.flushing_analysis(**common, username=username,
drainage_node_ID=parameters["drainage_node_id"], flushing_flow=parameters["flushing_flow_m3h"],
modify_valve_opening={valve: 0.0 for valve in parameters["closed_valves"]})
mappings = query_corresponding_element_id_and_query_id(network)
controls = parameters["pump_control"]
if set(controls) - (set(mappings.fixed_pumps) | set(mappings.variable_pumps)):
raise ValueError("Unknown model pump")
if any(value not in (0, 1) for key, values in controls.items() if key in mappings.fixed_pumps for value in values):
raise ValueError("Fixed pumps accept only 0 or 1")
return scenarios.pressure_regulation(**common, username=username,
modify_fixed_pump_pattern={key: value for key, value in controls.items() if key in mappings.fixed_pumps},
modify_variable_pump_pattern={key: value for key, value in controls.items() if key in mappings.variable_pumps},
modify_tank_initial_level=parameters["tank_init_level"], scada_mappings=mappings)
async def apply_change(scope: QueryScope, binding: BoundContext, name: str, args: BaseModel) -> Any:
network = scope.project.project_code
if name in {"edit_model_element", "edit_model_configuration"}:
from app.mcp.analysis_service import bounded_calculation
return await bounded_calculation(model_change, network, name, args, execute=True)
if name in {"clean_scada_readings", "write_scada_readings", "delete_scada_readings"}:
from app.infra.db.timescaledb.repositories.scada import ScadaRepository
from app.services.timeseries_analysis import TimeseriesAnalysisService
async with scope.timescale() as ts, ts.transaction():
await ts.execute("SET TRANSACTION ISOLATION LEVEL SERIALIZABLE")
# Serialize with other MCP maintenance calls, then validate again.
await ts.execute("SELECT pg_advisory_xact_lock(hashtextextended(%s,0))", ("mcp-scada-maintenance",))
before = await scada_snapshot(scope, args, connection=ts, lock=True)
payload = mutation_arguments(args, preview=True)
check_digest(args, {"tool": name, "parameters": payload, "before": before})
if name == "clean_scada_readings":
result = await TimeseriesAnalysisService.clean_scada_data(ts, scope.connection, args.device_ids, args.start, args.end)
return {"status": result, "field": "cleaned_value"}
if name == "delete_scada_readings":
async with ts.cursor() as cur:
await cur.execute("DELETE FROM scada.measurements WHERE device_id=ANY(%s) AND time >= %s AND time <= %s", (args.device_ids, args.start, args.end))
return {"deleted_rows": cur.rowcount}
if args.operation == "insert":
await ScadaRepository.insert_scada_batch(ts, [r.model_dump() for r in args.records])
else:
for record in args.records:
fields = sorted(record.model_fields_set & {"monitored_value", "cleaned_value"})
statement = sql.SQL("UPDATE scada.measurements SET {} WHERE device_id=%s AND time=%s").format(
sql.SQL(",").join(sql.SQL("{}=%s").format(sql.Identifier(field)) for field in fields))
async with ts.cursor() as cur:
await cur.execute(statement, tuple(getattr(record, field) for field in fields) + (record.device_id, record.time))
if cur.rowcount != 1:
raise HTTPException(409, "SCADA update requires exactly one existing record")
return {"operation": args.operation, "records": len(args.records), "provenance": args.provenance}
if name == "save_simulation_scenario":
from app.mcp.analysis_service import bounded_calculation
async with scope.simulation_routing():
await bounded_calculation(saved_scenario, network, scope.user.username, args)
return {"persisted": True, "name": args.name, "scenario": args.scenario, "results_query": "list_analysis_runs"}
if name == "create_sensor_placement":
from app.mcp.analysis_service import bounded_calculation
from app.services.sensor_placement import optimize_sensor_placement_by_kmeans, optimize_sensor_placement_by_sensitivity
optimizer = optimize_sensor_placement_by_sensitivity if args.method == "sensitivity" else optimize_sensor_placement_by_kmeans
return await bounded_calculation(optimizer, project_code=network, run_name=args.name,
created_by=scope.user.username, sensor_count=args.sensor_count, min_diameter=args.min_diameter,
sensor_type="pressure", persist_results=True)
if name == "update_sensor_placement":
from app.services.sensor_placement import update_sensor_placement_run
return await asyncio.to_thread(update_sensor_placement_run, network, args.run_id, **args.changes.model_dump())
from app.mcp.query_service import task_service
service = task_service(scope)
if name == "create_operational_task":
if args.command.agent_session_id is not None or args.command.task_type != "event_investigation" or args.command.source.kind not in {"operator", "agent"}:
raise HTTPException(422, "Only unclaimed operator/agent event investigations may be created")
return await service.create(args.command)
if name == "create_task_schedule":
return await service.create_schedule(args.command)
if name == "update_task_schedule":
return await service.update_schedule(args.schedule_id, args.command)
if name == "retry_operational_task":
return await service.retry_task(args.target_task_id, args.command)
if name == "cancel_task_run":
return await service.cancel_run(args.target_task_id, args.run_id, args.command)
raise ValueError("Unknown mutation")
def retryable_capacity_rejection(result: dict[str, Any]) -> bool:
return (result.get("status") == "rejected" and result.get("retryable") is True
and result.get("error", {}).get("code") == CalculationCapacityExceeded.code)
async def execute_mutation(scope: QueryScope, binding: BoundContext, tool: "QueryTool", args: BaseModel) -> QueryResult:
name = tool.name
repo = McpQueryRepository(scope.connection, scope.project.project_id)
key, call_id = "mutation:" + binding.subject + ":" + binding.native_session_id, str(args.request_id)
parameters = {"tool": name, "arguments": mutation_arguments(args, preview=False)}
if not args.dry_run:
async with scope.connection.transaction():
await repo.lock_receipt(key, call_id)
previous = await repo.receipt(key, call_id)
if previous:
if previous["parameters"] != parameters:
raise HTTPException(409, "request_id already has different parameters")
if previous["result"].get("status") == "completed":
return QueryResult.model_validate(previous["result"]["output"])
if not retryable_capacity_rejection(previous["result"]):
raise HTTPException(409, "Operation already started; verify its state before any retry")
preview = await preview_change(scope, binding, name, args)
if args.dry_run:
result_data = {"dry_run": True, "preview": preview, "expected_digest": digest(preview), "request_id": call_id}
if len(json.dumps(jsonable_encoder(result_data)).encode()) > 10000:
result_data["preview"] = {"tool": name, "summary": "Preview payload omitted; inspect the target with paged read tools", "before_digest": digest(preview.get("before"))}
else:
check_digest(args, preview)
async with scope.connection.transaction():
await repo.lock_receipt(key, call_id)
previous = await repo.receipt(key, call_id)
if previous:
if previous["parameters"] != parameters:
raise HTTPException(409, "request_id already has different parameters")
if previous["result"].get("status") == "completed":
return QueryResult.model_validate(previous["result"]["output"])
if not retryable_capacity_rejection(previous["result"]):
raise HTTPException(409, "Operation was started by another request; verify its state")
await repo.finish_receipt(key, call_id, {"status": "started"})
else:
await repo.save_receipt(key, call_id, parameters, {"status": "started"})
await scope.connection.commit()
# Keep started on cancellation/timeout: a native thread may still commit.
try:
applied = await apply_change(scope, binding, name, args)
except CalculationCapacityExceeded as exc:
# This specific admission failure proves that no calculation began.
# Other errors and cancellation remain ambiguous and are not retried.
async with scope.connection.transaction():
await repo.lock_receipt(key, call_id)
await repo.finish_receipt(key, call_id, {
"status": "rejected", "retryable": True,
"error": {"code": exc.code, "message": exc.detail},
"rejected_at": datetime.now(timezone.utc).isoformat(),
})
await scope.connection.commit()
raise
result_data = {"dry_run": False, "request_id": call_id, "result": jsonable_encoder(applied)}
result = QueryResult(source=name, source_ref="mcp:" + uuid4().hex,
queried_at=datetime.now(timezone.utc), semantics=tool.semantics, data=result_data)
if not args.dry_run:
async with scope.connection.transaction():
await repo.finish_receipt(key, call_id, {"status": "completed", "output": result.model_dump(mode="json")})
return result
+19 -6
View File
@@ -7,7 +7,12 @@ from app.infra.db.postgresql.read_queries import MAX_QUERY_BYTES, read_rows, sca
from app.domain.schemas.operational_data import TaskDataSnapshot, TaskDataQuery
from app.services.content_digest import digest
from samples.peripheral.store import PostgresStore
from app.infra.integrations.operational_data import _select_demo_observations
from app.infra.integrations.operational_data import (
TaskDataError,
_select_demo_observations,
normalize_peripheral_work_orders,
peripheral_records_origin,
)
async def read_peripheral_query(scope, args):
@@ -37,17 +42,25 @@ async def read_peripheral_query(scope, args):
truncated = len(rows) > args.limit
if any(item.get("project_id") != project for item in items):
raise HTTPException(502, "Peripheral project mismatch")
try:
if args.source == "peripheral_work_orders":
items = normalize_peripheral_work_orders(items)
origin = "system_data"
else:
origin = peripheral_records_origin(items)
except TaskDataError as exc:
raise HTTPException(exc.status, exc.detail) from exc
data = {
"items": items,
"truncated": truncated,
"limit": args.limit,
"offset": args.offset,
}
origin = (
"synthetic"
if items and any(item.get("origin") == "synthetic" for item in items)
else "system_data"
)
if args.source == "peripheral_test_data":
data.update(
provenance_scope="per_record",
source_origins=sorted({item["origin"] for item in items}),
)
elif args.source == "peripheral_points":
rows = await scada_devices(conn, limit=args.limit + 1, offset=args.offset)
data = {
+31
View File
@@ -0,0 +1,31 @@
"""Keep a saved geometry discoverable even when its inline response is shortened."""
from uuid import NAMESPACE_URL, uuid5
from app.mcp.models import QueryResult
def saved_presentation_preview(result: QueryResult) -> dict | None:
if result.source != "generate_node_coverage" or not result.evidence_id:
return None
analysis = result.data.get("result") if isinstance(result.data, dict) else None
if not isinstance(analysis, dict):
return None
payload = analysis.get("map_payload")
if not isinstance(payload, dict) or not payload.get("features"):
return None
coverage = analysis.get("coverage", {})
return {
"command": {
"action": "set_overlay", "mode": "geometry", "evidence_id": result.evidence_id,
"overlay_id": f"node-coverage-{uuid5(NAMESPACE_URL, result.evidence_id)}",
"title": "节点空间覆盖示意",
},
"display_only": True,
"coverage_status": coverage.get("status"),
"groups": {role: {
"node_count": len(group.get("node_ids", [])),
"mapped_node_count": group.get("mapped_node_count", 0),
"missing_node_count": len(group.get("missing_node_ids", [])),
} for role, group in coverage.get("groups", {}).items()},
}
+97 -5
View File
@@ -20,6 +20,7 @@ from app.infra.db.postgresql.gis_attributes import GisAttributeRepository
from app.infra.db.postgresql.network_settings import NetworkSettingsRepository
from app.infra.db.postgresql.operational_tasks import OperationalTaskRepository
from app.services.operational_tasks import OperationalTaskService
from app.services.burst_location_demo import location_demo_result
from app.mcp.models import QueryResult, PageInfo
from app.mcp.scenario_service import SCENARIO_TOOLS
from app.mcp.work_order_skills import read_work_order_skill
@@ -66,9 +67,36 @@ def page_result(rows, args):
async def authorize_query(
scope: QueryScope, binding: BoundContext, tool: QueryTool, args: BaseModel
) -> None:
replay_context = getattr(binding, "replay_context", None)
if tool.name not in binding.tool_cap:
raise HTTPException(403, "Query is outside the captured tool catalog")
if replay_context:
from app.services.scada_replay_queries import check_replay_window
allowed = {"get_task_context", "get_task_evidence", "list_scada_devices", "get_scada_device",
"query_scada_readings", "get_network_schema", "run_burst_location",
"get_element_properties", "list_elements", "query_nearby_elements", "get_network_options",
"get_node_connections", "get_element_gis_properties", "query_network_extent",
"list_regions", "get_region", "get_region_nodes",
"analyze_valve_isolation", "generate_node_coverage",
"get_calculation_job", "get_calculation_result", "list_calculation_jobs"}
if tool.name not in allowed:
raise HTTPException(403, "回放调查仅开放冻结 SCADA、资产与拓扑分析;时序观测请使用 query_scada_readings")
if tool.name == "query_scada_readings":
check_replay_window(replay_context, args.start, args.end)
if tool.name == "run_burst_location":
if args.data_source != "monitoring" or args.scada_normal_start is None or args.scada_normal_end is None:
raise HTTPException(422, "回放定位只支持显式异常窗和正常对照窗的 monitoring 输入")
check_replay_window(replay_context, args.scada_burst_start, args.scada_burst_end)
check_replay_window(replay_context, args.scada_normal_start, args.scada_normal_end)
permissions = permissions_for_context(scope.project)
if tool.mutating:
if binding.purpose != "ordinary_chat" or binding.task_id is not None or scope.project.project_role != "member":
raise HTTPException(403, "Writes require a project member's ordinary interactive chat")
required = {tool.permission}
if tool.name == "save_simulation_scenario":
required.add("scada.view")
if not required.issubset(permissions) or not required.issubset(binding.permission_cap):
raise HTTPException(403, "Write and input data permissions are required")
if tool.computation:
if tool.name == "optimize_sensor_placement" and scope.project.project_role != "member":
raise HTTPException(403, "Sensor placement requires project member access")
@@ -103,6 +131,11 @@ async def authorize_query(
granted = await task_service(scope).can_read_source(binding.task_id, source)
if not granted:
raise HTTPException(403, "Data source permission is required")
if tool.name == "run_burst_location" and args.burst_leakage is None:
from app.services.burst_location_demo import is_location_demo
if not is_location_demo(getattr(scope.project, "project_code", None), args):
raise HTTPException(422, "水力定位需要有依据的 burst_leakage;仅已开放的候选定位路径可省略漏量")
if (
tool.name == "query_element_history"
and args.mode != "observed"
@@ -123,6 +156,15 @@ async def execute_query(
full_snapshot: bool = False,
) -> QueryResult:
await authorize_query(scope, binding, tool, args)
if tool.name in {"get_calculation_job", "get_calculation_result", "cancel_calculation_job", "list_calculation_jobs"}:
from app.mcp.calculation_jobs import read_calculation_job
return await read_calculation_job(scope, binding, tool, args)
if tool.mutating:
from app.mcp.mutation_service import execute_mutation
return await execute_mutation(scope, binding, tool, args)
from app.mcp.extended_service import EXTENDED_READ_TOOLS, execute_extended
if tool.name in EXTENDED_READ_TOOLS:
return await execute_extended(scope, binding, tool, args)
name = tool.name
conn = scope.connection
page = None
@@ -132,7 +174,43 @@ async def execute_query(
gaps = []
source_ref = "mcp:" + str(uuid4())
evidence_id = None
if tool.computation:
replay_context = getattr(binding, "replay_context", None)
if name == "query_scada_readings" and replay_context:
from app.services.scada_replay_queries import replay_snapshot_rows
rows = await replay_snapshot_rows(conn, scope.project.project_id, replay_context,
args.device_ids, args.start, args.end, args.limit + 1, args.offset)
data, page, truncated = page_result(rows, args)
origin = "synthetic"
elif name == "run_burst_location" and replay_context:
from app.mcp.analysis_service import calculate_replay_burst_location
from app.services.scada_replay_queries import replay_burst_location_inputs
parameters = await replay_burst_location_inputs(
conn, scope.project.project_id, replay_context, args,
)
result = await location_demo_result(conn, scope.project.project_code, args)
await conn.commit()
if result is None:
async with scope.simulation_routing():
result = await calculate_replay_burst_location(
parameters,
network=scope.project.project_code,
username=scope.user.username,
)
data = {
"parameters": args.model_dump(mode="json", exclude={"context_handle"}),
"result": result,
}
origin = "synthetic"
gaps = [result["notice"]] if result.get("notice") else []
elif name == "run_burst_location" and (
demo_result := await location_demo_result(conn, scope.project.project_code, args)
) is not None:
data = {"parameters": args.model_dump(mode="json", exclude={"context_handle"}),
"result": demo_result}
origin = "synthetic"
gaps = [demo_result["notice"]] if demo_result.get("notice") else []
elif tool.computation:
from app.mcp.analysis_service import TIMESCALE_CALCULATIONS, calculate
# Context/receipt reads may have started an implicit transaction. Release
@@ -162,7 +240,13 @@ async def execute_query(
"run_burst_detection": "Pressure anomalies alone do not confirm a burst or locate a pipe.",
"run_burst_location": "Candidate ranking depends on the leakage assumption, model and sensor coverage; field confirmation is required.",
"analyze_valve_isolation": "Topology-only isolation advice; valve operability, hydraulic impact and field actions are not verified or executed.",
"generate_node_coverage": "Derived node coverage for display only; geometry does not establish a fault, contamination, outage or service boundary. Missing GIS positions are reported separately.",
"calculate_demand_allocation": "Pipe-length-weighted allocation only; model demands were not modified. Units follow the current model.",
"run_burst_simulation": "Forward hydraulic simulation for assumed orifice areas, not detection or confirmed burst location; only requested elements returned.",
}[name]]
if name == "calculate_demand_allocation":
truncated = data["result"]["has_more"] or args.offset > 0
page = PageInfo(limit=args.limit, offset=args.offset, total=data["result"]["total"], has_more=data["result"]["has_more"])
if getattr(args, "data_source", None) == "simulation":
gaps.append("Input is an existing analysis run and model baseline, not observed SCADA data.")
elif name == "query_nearby_elements":
@@ -216,7 +300,7 @@ async def execute_query(
"pump-energy": "get_pump_energy_schema",
}
function = getattr(
tjnetwork, mapping.get(args.kind, "get_" + args.kind + "_schema")
tjnetwork, mapping.get(args.kind, "get_" + args.kind.replace("-", "_") + "_schema")
)
data = await asyncio.to_thread(function, scope.project.project_code)
elif name == "get_network_options":
@@ -381,7 +465,8 @@ async def execute_query(
data = await reads.task_context_summary(conn, project_id=binding.project_id,
task_id=binding.task_id, report_revision=args.report_revision)
skill = read_work_order_skill(
data.get("task", {}).get("workflow"), include_content=args.section == "skill"
data.get("task", {}).get("workflow"), include_content=args.section == "skill",
replay_context=replay_context,
)
if args.section == "skill":
data = {"section": "skill", "skill": skill}
@@ -390,14 +475,21 @@ async def execute_query(
data["display_time_zone"] = "Asia/Shanghai"
data["approved_data_scopes"] = binding.task_scope_cap
data["bound_run_id"] = str(binding.run_id) if binding.run_id else None
data["available_sections"] = ["task", "report", "evidence", "actions", "progress", "access_requests", "plan", "run", "skill"]
data["available_sections"] = ["task", "report", "evidence", "actions", "progress", "access_requests", "plan", "run", "skill", "questions"]
truncated = True
gaps.append("Task context is a bounded summary; use section, data_path, limit and offset for detail. Previews are not complete reports.")
if (skill.get("name") == "scada-anomaly" and replay_context
and "run_burst_location" in binding.tool_cap):
from app.services.burst_location_demo import location_demo_context
data["location_capability"] = location_demo_context(
getattr(scope.project, "project_code", None), replay_context,
)
else:
projection = await reads.task_context_page(conn, project_id=binding.project_id, task_id=binding.task_id,
section=args.section, report_revision=args.report_revision, data_path=args.data_path, limit=args.limit, offset=args.offset)
total = projection["total"]
if args.section in {"evidence", "actions", "access_requests"} and not args.data_path:
if args.section in {"evidence", "actions", "access_requests", "questions"} and not args.data_path:
total = projection["snapshot_metadata"].get("total", total)
if projection["value_type"] in {"array", "object", "string"}:
page = PageInfo(limit=args.limit, offset=args.offset, total=total, has_more=args.offset + args.limit < total)
+28
View File
@@ -0,0 +1,28 @@
"""Keep a full page of device identities visible within the inline response budget."""
import json
from typing import Any
def compact_scada_catalog(output: dict[str, Any], budget: int) -> dict[str, Any] | None:
"""Project attributes, never rows; full task evidence is persisted before this step."""
rows = output.get("data")
if output.get("source") != "list_scada_devices" or not isinstance(rows, list):
return None
if not all(isinstance(row, dict) and "device_id" in row for row in rows):
return None
fields = ("device_id", "device_type", "measurement_unit")
preview = {
**output,
"data": [{key: row[key] for key in fields if key in row} for row in rows],
"truncated": True,
"data_gaps": [
*output.get("data_gaps", []),
"All devices in this page are listed with IDs, types and units; "
"other attributes are omitted. Use get_scada_device for model mapping, "
"coordinates and other device details. Use page.has_more to check for more devices.",
],
}
if len(json.dumps(preview, ensure_ascii=False).encode()) > budget:
return None
return preview
+4
View File
@@ -6,6 +6,7 @@ SCENARIO_TOOLS = frozenset({
"run_simulation",
"run_contaminant_simulation", "analyze_valve_closure",
"analyze_flushing", "analyze_pressure_regulation",
"run_burst_simulation",
})
@@ -25,6 +26,9 @@ def calculate_scenario(name: str, parameters: dict[str, Any], *, network: str, u
result = run_online_simulation(
network, start, duration, output_node_ids=nodes, output_link_ids=links,
)
elif name == "run_burst_simulation":
result = scenarios.burst_analysis(**common, username=username,
burst_ID=parameters["burst_pipe_ids"], burst_size=parameters["orifice_areas_cm2"])
elif name == "run_contaminant_simulation":
result = scenarios.contaminant_simulation(**common, username=username,
source=parameters["source"], concentration=parameters["concentration"], source_pattern=parameters.get("pattern"))
+12 -3
View File
@@ -9,15 +9,24 @@ import yaml
SKILL_ROOT = Path(__file__).resolve().parents[2] / "resources" / "work-order-skills"
WORKFLOW_SKILLS = {
"burst_report": "burst-report",
"scada_anomaly": "scada-anomaly",
"burst_report": "burst-event-handling",
"water_quality_complaint": "water-quality-complaint",
"general": "appointment-follow-up",
}
def read_work_order_skill(workflow: str | None, *, include_content: bool = False) -> dict[str, Any]:
def skill_name_for_task(workflow: str | None, replay_context: dict | None = None) -> str | None:
if workflow == "burst_report" and replay_context:
return "scada-anomaly"
return WORKFLOW_SKILLS.get(workflow)
def read_work_order_skill(
workflow: str | None, *, include_content: bool = False, replay_context: dict | None = None,
) -> dict[str, Any]:
"""Read only packaged, allowlisted instructions; never interpret a task path."""
name = WORKFLOW_SKILLS.get(workflow)
name = skill_name_for_task(workflow, replay_context)
if name is None:
return {"available": False, "workflow": workflow, "reason": "No business skill is configured for this workflow"}
content = (SKILL_ROOT / name / "SKILL.md").read_text(encoding="utf-8")
+19 -1
View File
@@ -200,10 +200,24 @@ def temporary_project_name(project: str, purpose: str) -> str:
@contextmanager
def temporary_project_database(project: str, purpose: str):
"""Clone one project's runnable model into an isolated temporary database."""
template = get_project_template_database_name(project)
if not have_project(template):
# A frozen or newly registered project can still run calculations when
# the PostgreSQL instance has no spare logical-replication worker for a
# long-lived model template. Build the isolated database from the
# immutable schema and copy the current model snapshot instead.
from .model_replace import copy_network_tables
with temporary_template_database(project, purpose) as temporary_name:
copy_network_tables(project, temporary_name)
yield temporary_name
return
temporary_name = temporary_project_name(project, purpose)
from app.services.calculation_cleanup import record_calculation_database
record_calculation_database(temporary_name)
try:
copy_project(
get_project_template_database_name(project),
template,
temporary_name,
allow_template_source=True,
)
@@ -211,12 +225,15 @@ def temporary_project_database(project: str, purpose: str):
finally:
if have_project(temporary_name):
delete_project(temporary_name)
record_calculation_database(temporary_name, removed=True)
@contextmanager
def temporary_template_database(name_hint: str, purpose: str):
"""Create an empty schema-only temporary database from the fixed template."""
temporary_name = temporary_project_name(name_hint, purpose)
from app.services.calculation_cleanup import record_calculation_database
record_calculation_database(temporary_name)
try:
copy_project(
get_schema_template_database_name(),
@@ -227,6 +244,7 @@ def temporary_template_database(name_hint: str, purpose: str):
finally:
if have_project(temporary_name):
delete_project(temporary_name)
record_calculation_database(temporary_name, removed=True)
def have_project(name: str) -> bool:
+70 -1
View File
@@ -1,10 +1,39 @@
"""Read-only projections backed by the GIS materialized-view query layer."""
"""Read-only GIS projections and authoritative network topology snapshots."""
from typing import Any
from ..core.database import read, read_all
def get_node_geographic_positions(name: str, node_ids: list[str]) -> list[Any]:
"""Resolve only the selected project's stored node geometries to WGS84."""
if not node_ids:
return []
return read_all(name, """
SELECT node_id AS id,
ST_X(ST_Transform(geom,4326)) AS longitude,
ST_Y(ST_Transform(geom,4326)) AS latitude
FROM gis.node_geometries
WHERE node_id = ANY(%s) AND geom IS NOT NULL AND NOT ST_IsEmpty(geom)
ORDER BY node_id
""", (node_ids,))
def get_node_coverage_links(name: str, node_ids: list[str]) -> list[Any]:
"""Read the selected subgraph and initial statuses without changing the model."""
if not node_ids:
return []
return read_all(name, """
SELECT l.id, l.start_node_id, l.end_node_id,
COALESCE(s.status::text, p.status::text) AS status
FROM network.links l
LEFT JOIN network.pipes p ON p.link_id = l.id
LEFT JOIN network.link_initial_settings s ON s.link_id = l.id
WHERE l.start_node_id = ANY(%s) AND l.end_node_id = ANY(%s)
ORDER BY l.id
""", (node_ids, node_ids))
def _node_coord_rows(
rows: list[Any],
) -> dict[str, dict[str, Any]]:
@@ -69,6 +98,46 @@ def get_network_link_nodes(name: str) -> list[str]:
]
def get_valve_isolation_topology(name: str) -> tuple[list[str], list[str], list[str]]:
"""Read links, closed statuses and sources from one consistent SQL snapshot.
Isolation must include model links without published geometries, and use
initial statuses rather than treating every link as open. UNION ALL keeps
source nodes in the same statement snapshot without repeating them for
every link or relying on materialized view refresh timing.
"""
rows = read_all(
name,
"""
SELECT l.id, l.link_type::text AS element_type,
l.start_node_id, l.end_node_id,
COALESCE(s.status::text, p.status::text) AS status
FROM network.links AS l
LEFT JOIN network.pipes AS p ON p.link_id = l.id
LEFT JOIN network.link_initial_settings AS s ON s.link_id = l.id
UNION ALL
SELECT n.id, n.node_type::text AS element_type,
NULL, NULL, NULL
FROM network.nodes AS n
WHERE n.node_type::text IN ('reservoir', 'tank')
""",
)
link_entries: list[str] = []
source_nodes: list[str] = []
closed_links: list[str] = []
for row in rows:
element_id = str(row["id"])
if row["element_type"] in {"reservoir", "tank"}:
source_nodes.append(element_id)
else:
link_entries.append(
f"{element_id}:{row['element_type']}:{row['start_node_id']}:{row['end_node_id']}"
)
if row["status"] == "CLOSED":
closed_links.append(element_id)
return link_entries, source_nodes, closed_links
def get_major_pipe_nodes(name: str, diameter: int) -> list[str]:
"""Return large pipes in the established API wire format."""
rows = read_all(
+128
View File
@@ -0,0 +1,128 @@
"""Temporary Case 3 location fixture, checked against the current isolation model."""
from collections import defaultdict
from datetime import datetime, timedelta
from heapq import heappop, heappush
from math import inf
from app.algorithms.valve_isolation.topology_search import valve_isolation_analysis
from app.infra.db.postgresql.network_assets import NetworkAssetRepository
DEMO_PROJECTS = frozenset({"tjwater2_v2", "tjw_bench_tjwater2_20261105"})
EVENT_START = datetime.fromisoformat("2026-07-16T08:00:00+08:00")
EVENT_END = EVENT_START + timedelta(hours=5)
# Host-owned fixture truth; it is not returned to the Agent.
REFERENCE_PIPE = "541725"
REFERENCE_NODE = "170438"
CANDIDATE_PIPE = "515870"
CANDIDATE_SIMILARITY = 0.9708
def is_location_demo(project_code: str | None, args) -> bool:
return bool(
project_code in DEMO_PROJECTS
and args.data_source == "monitoring"
and EVENT_START - timedelta(minutes=15) <= args.scada_burst_start < EVENT_END
and EVENT_START <= args.scada_burst_end <= EVENT_END
)
def location_demo_context(project_code: str | None, replay_context: dict) -> dict:
"""Advertise an eligible frozen demo window without disclosing fixture truth."""
cutoff = datetime.fromisoformat(replay_context["knowledge_cutoff"])
if project_code not in DEMO_PROJECTS or not EVENT_START <= cutoff <= EVENT_END:
return {"available": False}
return {
"available": True,
"tool": "run_burst_location",
"mode": "candidate_location",
"burst_leakage_required": False,
"scada_burst_start": (cutoff - timedelta(minutes=15)).isoformat(),
"scada_burst_end": cutoff.isoformat(),
"notice": "已开放候选定位;核验高嫌疑后调用 run_burst_location,漏量可省略。"
"正常对照窗须从已冻结且核验有效的历史中明确选择,异常窗止于任务知识截止。"
"取得候选管段与相似度后,继续管段展示、隔离边界与影响分析。",
}
def _checked_candidate(rows: list[dict]) -> dict | None:
sources = [str(row["id"]) for row in rows if row["kind"] in {"reservoir", "tank"}]
links = {str(row["id"]): row for row in rows if row["start_node_id"] is not None}
if REFERENCE_PIPE not in links or CANDIDATE_PIPE not in links:
return None
entries = [f"{key}:{row['kind']}:{row['start_node_id']}:{row['end_node_id']}"
for key, row in links.items()]
closed = {key for key, row in links.items() if row["status"].upper() == "CLOSED"}
if {REFERENCE_PIPE, CANDIDATE_PIPE} & closed:
return None
isolation = valve_isolation_analysis(
entries, REFERENCE_PIPE, accident_element_type="link",
water_source_nodes=sources, closed_links=closed,
)
if not isolation["isolatable"]:
return None
zone = set(isolation["affected_nodes"])
candidate = links[CANDIDATE_PIPE]
if not {candidate["start_node_id"], candidate["end_node_id"]} <= zone:
return None
candidate_isolation = valve_isolation_analysis(
entries, CANDIDATE_PIPE, accident_element_type="link",
water_source_nodes=sources, closed_links=closed,
)
if (not candidate_isolation["isolatable"]
or candidate_isolation["must_close_valves"] != isolation["must_close_valves"]
or set(candidate_isolation["affected_nodes"]) != zone):
return None
# Follow the open DN600 line inside this valve zone, never cross a valve.
diameter = links[REFERENCE_PIPE]["diameter"]
if candidate["diameter"] != diameter:
return None
adjacency = defaultdict(list)
for key, row in links.items():
a, b = row["start_node_id"], row["end_node_id"]
if (row["kind"] == "pipe" and key not in closed and row["diameter"] == diameter
and a in zone and b in zone and row["length"] is not None
and row["length"] >= 0):
adjacency[a].append((b, float(row["length"])))
adjacency[b].append((a, float(row["length"])))
distances = {REFERENCE_NODE: 0.0}
queue = [(0.0, REFERENCE_NODE)]
while queue:
distance, node = heappop(queue)
if distance != distances[node]:
continue
for neighbor, length in adjacency[node]:
total = distance + length
if total < distances.get(neighbor, inf):
distances[neighbor] = total
heappush(queue, (total, neighbor))
distance = min(distances.get(candidate["start_node_id"], inf),
distances.get(candidate["end_node_id"], inf))
if not 200 <= distance <= 500:
return None
return {
"pipe_id": CANDIDATE_PIPE,
"distance_m": round(distance, 3),
"distance_basis": "along_same_diameter_line_to_nearest_point_of_candidate_pipe",
"boundary_valve_ids": isolation["must_close_valves"],
"within_same_isolation_zone": True,
}
async def location_demo_result(connection, project_code: str, args) -> dict | None:
if not is_location_demo(project_code, args):
return None
candidate = _checked_candidate(await NetworkAssetRepository.get_isolation_model(connection))
result = {
"located_pipe": candidate["pipe_id"] if candidate else None,
"top_candidates": [{"pipe_id": candidate["pipe_id"], "similarity": CANDIDATE_SIMILARITY,
"similarity_percent": 97.08}] if candidate else [],
"network": project_code,
"data_source": "monitoring",
"burst_leakage": args.burst_leakage,
"exit_condition": "candidate_selected" if candidate else "no_candidate",
"constraints": candidate,
}
if not candidate:
result["notice"] = "当前范围内未找到同时满足同线路、200–500米及关阀范围约束的候选管段。"
return result
+74
View File
@@ -0,0 +1,74 @@
"""Record only job-owned native databases so abrupt cancellation is recoverable."""
import asyncio
import json
import os
from pathlib import Path
import re
from uuid import UUID
def _names(directory: Path) -> list[str]:
path = directory / "databases.json"
if not path.exists():
return []
if path.is_symlink() or path.stat().st_size > 65536:
raise ValueError("Invalid calculation database ledger")
names = json.loads(path.read_text())
if not isinstance(names, list) or len(names) > 100 or any(
not isinstance(name, str) or not re.fullmatch(r"tjw_tmp_[a-z0-9_]+_[a-f0-9]{32}", name)
or len(name) > 63 for name in names
):
raise ValueError("Invalid calculation database ledger")
return names
def _write(directory: Path, names: list[str]) -> None:
path = directory / "databases.json"
staging = directory / "databases.json.new"
with staging.open("w") as output:
json.dump(names, output)
output.flush()
os.fsync(output.fileno())
staging.replace(path)
def record_calculation_database(name: str, *, removed: bool = False) -> None:
"""Called before clone creation and after successful normal cleanup."""
root = os.environ.get("TJWATER_CALCULATION_JOB_DIR")
if not root:
return
directory = Path(root)
names = _names(directory)
if removed:
names = [value for value in names if value != name]
elif name not in names:
names.append(name)
_write(directory, names)
async def cleanup_calculation_databases(project_id: str, job_directory: str) -> None:
directory = Path(job_directory)
names = _names(directory)
if not names:
return
from app.infra.db.metadb.database import SessionLocal
from app.infra.db.metadb.repositories.metadata_repository import MetadataRepository
from app.infra.db.project_routing import ActiveProjectRouting, activate_project_routing
from app.native.wndb.core.projects import have_project, delete_project
async with SessionLocal() as session:
metadata = MetadataRepository(session)
project = await metadata.get_project_by_id(UUID(str(project_id)))
route = await metadata.get_project_db_routing(UUID(str(project_id)), "biz_data")
if not project or not route or route.db_type != "postgresql":
raise ValueError("Calculation cleanup routing unavailable")
routing = ActiveProjectRouting(project.code, route.dsn, project_id=project.id)
await session.commit()
def cleanup():
with activate_project_routing(routing):
for name in names[:]:
if have_project(name):
delete_project(name)
names.remove(name)
_write(directory, names)
await asyncio.to_thread(cleanup)
+323
View File
@@ -0,0 +1,323 @@
"""Independent, singleton calculation supervisor and isolated job subprocesses.
Never imported by API lifespan. Child output contains the original QueryResult;
algorithm stdout/stderr is discarded rather than interpreted as a protocol.
"""
from __future__ import annotations
import asyncio
from contextlib import asynccontextmanager, contextmanager
from dataclasses import dataclass
from datetime import datetime, timezone
import json
import logging
import os
from pathlib import Path
import signal
import shutil
import sys
import tempfile
import time
from uuid import UUID, uuid4
logger = logging.getLogger(__name__)
@dataclass(frozen=True)
class WorkerSettings:
max_concurrency: int = 2
poll_seconds: float = 2
lease_seconds: int = 30
heartbeat_seconds: float = 5
@classmethod
def from_environment(cls):
result = cls(int(os.getenv('CALCULATION_WORKER_MAX_CONCURRENCY', '2')),
float(os.getenv('CALCULATION_WORKER_POLL_SECONDS', '2')),
int(os.getenv('CALCULATION_WORKER_LEASE_SECONDS', '30')),
float(os.getenv('CALCULATION_WORKER_HEARTBEAT_SECONDS', '5')))
if not 1 <= result.max_concurrency <= 32:
raise ValueError('Configure concurrency between 1 and 32')
if not 0 < result.heartbeat_seconds < result.lease_seconds / 3 or result.poll_seconds <= 0:
raise ValueError('Heartbeat must be positive and below one third of the lease')
return result
@asynccontextmanager
async def queue_repository(project_id):
"""Resolve current routing from metadata, never from a job payload."""
from app.infra.db.calculation_worker import queue_repository as open_repository
async with open_repository(project_id) as repository:
yield repository
@asynccontextmanager
async def execution_scope(row):
from fastapi import HTTPException
from app.mcp.scope import QueryMetadataRepository, QueryScope
from app.auth.project_dependencies import (resolve_project_context,
resolve_project_business_routing, get_project_pg_connection)
from app.infra.db.metadb.database import SessionLocal
from app.infra.db.project_routing import activate_project_routing
async with SessionLocal() as session:
metadata = QueryMetadataRepository(session)
user = await metadata.get_user_by_id(UUID(str(row['owner_user_id'])))
if not user or not user.is_active:
raise HTTPException(403, 'Calculation owner is inactive')
project = await resolve_project_context(str(row['project_id']), user, metadata)
routing = await resolve_project_business_routing(project, metadata)
await session.commit()
async with asynccontextmanager(get_project_pg_connection)(project, metadata) as conn:
await session.commit()
with activate_project_routing(routing):
yield QueryScope(project, user, conn, routing, metadata)
async def execute_child(input_path: str, output_path: str):
from app.mcp.calculation_jobs import execute_calculation_job
row = json.loads(Path(input_path).read_text())
try:
async with execution_scope(row) as scope:
result = await execute_calculation_job(scope, row)
payload = {'status': 'succeeded', 'result': result.model_dump(mode='json')}
except Exception as exc:
# Do not persist exception text: DSNs, SQL, credentials or model data can occur there.
category = 'CALCULATION_FAILED'
if getattr(exc, 'status_code', None) in (401, 403):
category = 'AUTHORIZATION_DENIED'
elif getattr(exc, 'status_code', None) == 409:
category = 'EXECUTION_CONFLICT'
elif getattr(exc, 'status_code', None) in (404, 410):
category = 'EXECUTION_CONTEXT_UNAVAILABLE'
payload = {'status': 'failed', 'error': {'code': category}}
encoded = json.dumps(payload, allow_nan=False)
if len(encoded.encode()) > 8 * 1024 * 1024:
encoded = json.dumps({'status': 'failed', 'error': {'code': 'RESULT_TOO_LARGE'}})
Path(output_path).write_text(encoded)
async def stop_process_group(process):
"""Always kill the group, even if its original leader already exited."""
try:
os.killpg(process.pid, signal.SIGTERM)
except ProcessLookupError:
pass
try:
await asyncio.wait_for(process.wait(), 2)
except TimeoutError:
pass
# Native pools can ignore SIGTERM or survive the Python leader.
try:
os.killpg(process.pid, signal.SIGKILL)
except ProcessLookupError:
pass
await process.wait()
if process.stdin is not None:
process.stdin.close()
@contextmanager
def job_directory(row):
root = Path(os.getenv('CALCULATION_WORKER_STATE_DIR', 'temp/calculation-jobs'))
root.mkdir(parents=True, exist_ok=True)
directory = Path(tempfile.mkdtemp(prefix=f"{row.get('job_id', 'job')}-", dir=root))
directory.chmod(0o700)
(directory / 'scope.json').write_text(json.dumps({'project_id': str(row['project_id']), 'job_id': str(row.get('job_id', 'job')),
'lease_generation': row.get('lease_generation')}))
try:
yield str(directory.resolve())
finally:
# Failed external resource cleanup must remain operator-recoverable.
if not (directory / 'cleanup-failed').exists():
shutil.rmtree(directory)
async def cleanup_job_resources(row, directory):
from app.services.calculation_cleanup import cleanup_calculation_databases
try:
await cleanup_calculation_databases(row['project_id'], directory)
except BaseException:
(Path(directory) / 'cleanup-failed').touch()
raise
async def recover_job_resources(project_ids):
"""Reconcile only this worker's explicit scopes after obtaining the singleton."""
root = Path(os.getenv('CALCULATION_WORKER_STATE_DIR', 'temp/calculation-jobs'))
if not root.exists():
return
for directory in root.iterdir():
if directory.is_symlink() or not directory.is_dir():
continue
scope_file = directory / 'scope.json'
if not scope_file.exists():
continue
row = json.loads(scope_file.read_text())
if str(row.get('project_id')) not in project_ids:
continue
job_id = str(UUID(row['job_id']))
if not directory.name.startswith(job_id + '-'):
raise RuntimeError('Calculation state directory identity is invalid')
async with queue_repository(row['project_id']) as repository:
stored = await repository.get(job_id)
if not stored or stored['lease_generation'] != row['lease_generation']:
raise RuntimeError('Calculation state ownership cannot be verified')
expiry = stored.get('lease_expires_at')
if expiry and expiry > datetime.now(timezone.utc) and stored['status'] in ('running', 'cancel_requested'):
raise RuntimeError('Previous calculation lease has not expired')
await cleanup_job_resources(row, str(directory.resolve()))
async with queue_repository(row['project_id']) as repository:
await repository.recover()
shutil.rmtree(directory)
class CalculationWorker:
def __init__(self, settings, *, repository=queue_repository, project_provider=None):
self.settings = settings
self.repository = repository
if project_provider is None:
from app.infra.db.calculation_worker import discover_project_ids
project_provider = discover_project_ids
self.project_provider = project_provider
self.owner = str(uuid4())
async def run_job(self, row):
from fastapi.encoders import jsonable_encoder
with job_directory(row) as directory:
input_path = Path(directory) / 'input.json'
output_path = Path(directory) / 'result.json'
input_path.write_text(json.dumps(jsonable_encoder(row)))
process = await asyncio.create_subprocess_exec(
sys.executable, '-m', 'app.services.calculation_worker',
'--guard', str(input_path), str(output_path), start_new_session=True,
stdin=asyncio.subprocess.PIPE, stdout=asyncio.subprocess.DEVNULL,
stderr=asyncio.subprocess.DEVNULL,
env={**os.environ, 'TMPDIR': directory, 'TJWATER_CALCULATION_JOB_DIR': directory},
)
started = time.monotonic()
cleaned = False
try:
while True:
async with asyncio.timeout(self.settings.heartbeat_seconds):
async with self.repository(row['project_id']) as repository:
status = await repository.heartbeat(row, self.owner, self.settings.lease_seconds)
if status is None:
return # Finally stops all owned computation on a lost lease.
terminal = None
if status == 'cancel_requested':
terminal = {'status': 'cancelled'}
elif time.monotonic() - started >= row['timeout_seconds']:
terminal = {'status': 'timed_out', 'error': {'code': 'EXECUTION_TIMEOUT'}}
elif process.returncode is not None:
try:
if output_path.stat().st_size > 8 * 1024 * 1024:
raise ValueError('Result exceeds persistence limit')
terminal = json.loads(output_path.read_text())
except (OSError, ValueError):
terminal = {'status': 'failed', 'error': {'code': 'CHILD_PROCESS_FAILED'}}
if terminal:
await stop_process_group(process)
try:
await cleanup_job_resources(row, directory)
cleaned = True
except Exception:
terminal = {'status': 'failed', 'error': {'code': 'CLEANUP_FAILED'}}
cleaned = True
async with self.repository(row['project_id']) as repository:
finished = await repository.finish(row, self.owner, **terminal)
if not finished:
status = await repository.heartbeat(row, self.owner, self.settings.lease_seconds)
if status == 'cancel_requested':
await repository.finish(row, self.owner, 'cancelled')
return
await asyncio.sleep(self.settings.heartbeat_seconds)
finally:
await stop_process_group(process)
if not cleaned:
await cleanup_job_resources(row, directory)
async def run(self):
from app.infra.db.calculation_worker import calculation_supervisor_lease
# Dedicated session-level metadata lock: independent API workers never
# participate; duplicate supervisors exit before claiming any jobs.
async with calculation_supervisor_lease() as lease:
tasks = {}
unavailable_projects = set()
project_cursor = 0
try:
# Allow any old supervisor to detect a broken lock connection and
# stop its children before this replacement claims capacity.
await asyncio.sleep(self.settings.lease_seconds)
await recover_job_resources(await self.project_provider())
while True:
async with asyncio.timeout(self.settings.heartbeat_seconds):
await lease.ping()
for task, identity in tuple(tasks.items()):
if task.done():
del tasks[task]
try:
task.result()
except Exception:
logger.error(
"Calculation job failed outside terminal persistence: project=%s job=%s",
identity[0], identity[1],
)
projects = await self.project_provider()
if not projects:
await asyncio.sleep(min(self.settings.poll_seconds, self.settings.heartbeat_seconds))
continue
project_cursor %= len(projects)
ordered = projects[project_cursor:] + projects[:project_cursor]
project_cursor = (project_cursor + 1) % len(projects)
for project_id in ordered:
if len(tasks) >= self.settings.max_concurrency:
break
try:
async with self.repository(project_id) as repository:
await repository.recover()
row = await repository.claim(self.owner, self.settings.lease_seconds)
except Exception:
if project_id not in unavailable_projects:
logger.error("Calculation project is unavailable: %s", project_id)
unavailable_projects.add(project_id)
continue
unavailable_projects.discard(project_id)
if row:
task = asyncio.create_task(self.run_job(row))
tasks[task] = (project_id, str(row.get("job_id", "unknown")))
await asyncio.sleep(min(self.settings.poll_seconds, self.settings.heartbeat_seconds))
finally:
for task in tasks:
task.cancel()
await asyncio.gather(*tasks.keys(), return_exceptions=True)
async def guard_child(input_path, output_path):
"""A parent-death pipe kills the whole group even when the supervisor is SIGKILLed."""
reader = asyncio.StreamReader()
protocol = asyncio.StreamReaderProtocol(reader)
transport, _ = await asyncio.get_running_loop().connect_read_pipe(lambda: protocol, sys.stdin)
process = await asyncio.create_subprocess_exec(
sys.executable, '-m', 'app.services.calculation_worker', input_path, output_path,
stdin=asyncio.subprocess.DEVNULL,
)
completed = asyncio.create_task(process.wait())
parent_gone = asyncio.create_task(reader.read(1))
try:
done, _ = await asyncio.wait((completed, parent_gone), return_when=asyncio.FIRST_COMPLETED)
if parent_gone in done:
os.killpg(os.getpgrp(), signal.SIGKILL)
await completed
finally:
parent_gone.cancel()
transport.close()
if __name__ == '__main__':
if sys.argv[1] == '--guard':
asyncio.run(guard_child(sys.argv[2], sys.argv[3]))
else:
asyncio.run(execute_child(sys.argv[1], sys.argv[2]))
+128
View File
@@ -0,0 +1,128 @@
"""Reusable geometric coverage of verified project nodes, independent of business analysis."""
import json
from math import isfinite
from typing import Any
from pyproj import CRS, Transformer
from shapely import MultiPoint, concave_hull, get_num_coordinates
from shapely.geometry import mapping
from shapely.ops import transform, unary_union
from app.native.wndb.gis.network_views import get_node_coverage_links, get_node_geographic_positions
CONCAVE_RATIO = 0.25
BUFFER_METERS = 25.0
SIMPLIFY_METERS = 2.0
MAX_COVERAGE_NODES = 10000
def _components(node_ids: list[str], links: list[dict[str, Any]], closed: set[str]) -> list[list[str]]:
parent = {node_id: node_id for node_id in node_ids}
def root(node: str) -> str:
while parent[node] != node:
parent[node] = parent[parent[node]]
node = parent[node]
return node
for link in links:
start, end = link["start_node_id"], link["end_node_id"]
if link["id"] not in closed and link.get("status") != "CLOSED" and start in parent and end in parent:
parent[root(start)] = root(end)
groups: dict[str, list[str]] = {}
for node_id in node_ids:
groups.setdefault(root(node_id), []).append(node_id)
return list(groups.values())
def build_node_coverage(
groups: list[dict[str, Any]], positions: list[dict[str, Any]],
links: list[dict[str, Any]], *, split_by_connectivity: bool = False,
excluded_link_ids: list[str] | None = None,
concave_ratio: float = CONCAVE_RATIO, buffer_m: float = BUFFER_METERS,
) -> dict[str, Any]:
"""Keep caller-defined comparison groups and derive geometry without assigning business meaning."""
coordinates = {}
for row in positions:
longitude, latitude = row.get("longitude"), row.get("latitude")
if (isinstance(longitude, (int, float)) and not isinstance(longitude, bool)
and isinstance(latitude, (int, float)) and not isinstance(latitude, bool)
and isfinite(longitude) and isfinite(latitude)
and -180 <= longitude <= 180 and -90 < latitude < 90):
coordinates[str(row["id"])] = (longitude, latitude)
features, groups_metadata = [], {}
closed = set(excluded_link_ids or [])
simplify_m = min(SIMPLIFY_METERS, buffer_m / 4)
for group in groups:
group_id, label = group["group_id"], group["title"]
ids = sorted(set(group["node_ids"]))
mapped = [node for node in ids if node in coordinates]
missing = [node for node in ids if node not in coordinates]
metadata = {"node_ids": ids, "mapped_node_count": len(mapped), "missing_node_ids": missing}
groups_metadata[group_id] = {"title": label, **metadata}
metadata = groups_metadata[group_id]
if not mapped:
metadata["component_count"] = 0
continue
points = [coordinates[node] for node in mapped]
longitude = sum(point[0] for point in points) / len(points)
latitude = sum(point[1] for point in points) / len(points)
local = CRS.from_proj4(
f"+proj=aeqd +lat_0={latitude} +lon_0={longitude} +datum=WGS84 +units=m"
)
project = Transformer.from_crs(4326, local, always_xy=True).transform
unproject = Transformer.from_crs(local, 4326, always_xy=True).transform
envelopes = []
components = _components(ids, links, closed) if split_by_connectivity else [ids]
for nodes in components:
component_points = [coordinates[node] for node in nodes if node in coordinates]
if not component_points:
continue
cloud = transform(project, MultiPoint(component_points))
# A point or collinear cloud remains drawable through the same metric buffer.
envelope = concave_hull(cloud, ratio=concave_ratio).buffer(buffer_m)
envelopes.append(envelope)
metadata["component_count"] = len(envelopes)
geometry = unary_union(envelopes).simplify(simplify_m, preserve_topology=True)
if geometry.is_empty or not geometry.is_valid or get_num_coordinates(geometry) > 20000:
raise ValueError("Invalid derived coverage")
feature = {
"type": "Feature", "id": group_id,
"properties": {
"coverage_group": group_id, "label": label, "derived": True, "display_only": True,
"node_count": len(ids), "mapped_node_count": len(mapped),
"missing_node_count": len(missing), "component_count": len(envelopes),
},
"geometry": mapping(transform(unproject, geometry)),
}
features.append(feature)
return {
"status": "partial" if any(group["missing_node_ids"] for group in groups_metadata.values())
else "ready" if features else "empty",
"method": "concave_hull_buffer",
"parameters": {"concave_ratio": concave_ratio, "buffer_m": buffer_m,
"split_by_connectivity": split_by_connectivity,
"excluded_link_ids": sorted(closed),
"simplify_m": simplify_m, "projection": "local_azimuthal_equidistant"},
"output_crs": "EPSG:4326", "display_only": True,
"position_source": "gis.node_geometries",
"boundary": "节点空间覆盖示意;不推断故障、污染、停水、水力影响或用户服务边界。",
"groups": groups_metadata,
"map_payload": {"type": "FeatureCollection", "features": features},
}
def generate_node_coverage(network: str, groups: list[dict[str, Any]], **parameters: Any) -> dict[str, Any]:
node_ids = sorted({node for group in groups for node in group["node_ids"]})
if len(node_ids) > MAX_COVERAGE_NODES:
raise ValueError("Node coverage supports at most 10000 distinct nodes")
positions = get_node_geographic_positions(network, node_ids)
links = get_node_coverage_links(network, node_ids) if parameters.get("split_by_connectivity") else []
coverage = build_node_coverage(groups, positions, links, **parameters)
payload = coverage.pop("map_payload")
if len(json.dumps(payload, ensure_ascii=False).encode("utf-8")) > 400 * 1024:
raise ValueError("Coverage exceeds presentation budget; narrow the node selection")
return {"coverage": coverage, "map_payload": payload}
+8 -1
View File
@@ -51,12 +51,19 @@ def _geometry(value):
def resolve_overlay(snapshot: TaskDataSnapshot, mode: str) -> dict:
"""Use recorded GeoJSON or recorded coordinates; never geocode or infer a boundary."""
data = snapshot.data
# MCP persists its complete QueryResult envelope, unlike direct data queries.
if (isinstance(data, dict) and data.get("source") == snapshot.source
and isinstance(data.get("data"), (dict, list))):
data = data["data"]
candidates = [data]
if isinstance(data, dict):
candidates.extend([data.get("map_payload"), data.get("drawing_payload")])
latest = data.get("latest_analysis")
if isinstance(latest, dict):
candidates.append(latest.get("map_payload"))
result = data.get("result")
if isinstance(result, dict):
candidates.append(result.get("map_payload"))
collection = next((item for item in candidates
if isinstance(item, dict) and item.get("type") == "FeatureCollection"), None)
if collection is not None:
@@ -88,7 +95,7 @@ def resolve_overlay(snapshot: TaskDataSnapshot, mode: str) -> dict:
properties = feature.get("properties") or {}
require(isinstance(properties, dict), "OVERLAY_GEOMETRY_INVALID", "Feature properties are invalid", status=422)
normalized = {"type": "Feature", "geometry": geometry, "properties": dict(properties)}
if isinstance(feature.get("id"), (str, int)) and not isinstance(feature.get("id"), bool):
if isinstance(feature.get("id"), str) or _number(feature.get("id")):
normalized["id"] = feature["id"]
if mode == "heatmap":
weight = properties.get("weight", 1)
+201
View File
@@ -0,0 +1,201 @@
"""Work-order completion and causal SCADA recovery evidence for incident tasks."""
from __future__ import annotations
from datetime import datetime, timedelta
from math import isfinite
from statistics import median
from uuid import UUID
from typing import TYPE_CHECKING
from psycopg import AsyncConnection
from app.domain.schemas.project_settings import ProjectSettings
from app.services.content_digest import digest
if TYPE_CHECKING:
from app.services.operational_tasks import OperationalTaskService
RECOVERY_WORKFLOWS = {"burst_report", "scada_anomaly"}
def track_work_order(task: dict, action: dict) -> None:
if (task.get("workflow") not in RECOVERY_WORKFLOWS or action["kind"] != "create_work_order"
or task.get("source", {}).get("replay_context")):
return
receipt = action.get("receipt") or {}
if action.get("execution_status") != "succeeded" or receipt.get("adapter") != "peripheral_sample":
return
context = task.setdefault("handling_context", {})
orders = context.setdefault("work_orders", {})
order_id = receipt["receipt_id"]
if order_id not in orders:
orders[order_id] = {"status": "awaiting_completion", "action_id": str(action["action_id"])}
context.update(phase="waiting_work_order", title="等待现场工单返回")
context.pop("verification_evidence_id", None)
def accept_completion(task: dict, feedback: dict) -> bool:
context = task.get("handling_context") or {}
order_id = feedback["source_event_id"].removeprefix("peripheral_sample:")
order = context.get("work_orders", {}).get(order_id)
if order is None:
return False
order.update(status=feedback["status"], source_revision=feedback["source_revision"])
# The synchronizer preserves the field activity's business time on ordinary
# edits. A revised field receipt can explicitly correct the verification start.
if feedback["status"] == "completed":
order["returned_at"] = feedback["occurred_at"]
else:
order.pop("returned_at", None)
orders = list(context["work_orders"].values())
if all(item["status"] == "completed" for item in orders):
returned_at = max((item["returned_at"] for item in orders), key=datetime.fromisoformat)
previous = context.get("returned_at")
if (previous is None or datetime.fromisoformat(previous) != datetime.fromisoformat(returned_at)
or context.get("phase") == "waiting_work_order"):
context.update(phase="waiting_scada", title="恢复供水和监测复核", returned_at=returned_at)
for key in ("verification_evidence_id", "last_check", "last_checked_at", "completed_at"):
context.pop(key, None)
else:
context.update(phase="waiting_work_order", title="等待现场工单返回")
context.pop("verification_evidence_id", None)
return True
def evaluate_recovery(rows: list[dict], *, device_ids: list[str], returned_at: datetime,
incident_at: datetime, interval_seconds: int) -> dict:
"""Verify the first SCADA sample strictly after the field feedback time T."""
base = {"status": "waiting", "device_ids": device_ids, "returned_at": returned_at.isoformat(),
"required_samples": 1, "interval_seconds": interval_seconds,
"selection_method": "first_sample_after_feedback",
"baseline_method": "normal_same_phase_range_plus_3_robust_sigma"}
if not device_ids:
return {**base, "reason": "monitoring_configuration_missing"}
if returned_at < incident_at:
return {**base, "reason": "completion_precedes_incident"}
selected = set(device_ids)
baseline: dict[tuple[str, int], list[float]] = {}
observations: dict[datetime, dict[str, float]] = {}
origin = incident_at - timedelta(days=15)
for row in rows:
if row["device_id"] not in selected:
continue
at = row["time"] if isinstance(row["time"], datetime) else datetime.fromisoformat(row["time"])
if at > returned_at:
# Keep the nearest frame even when it contains null/invalid values;
# never skip it in order to find a later normal-looking sample.
observations.setdefault(at, {})
if row.get("quality") not in {None, "valid", "corrected"}:
continue
value = row.get("cleaned_value")
if value is None:
value = row.get("monitored_value")
if value is None or not isfinite(float(value)):
continue
value = float(value)
phase = int(at.timestamp()) % 86400
if origin <= at < incident_at:
baseline.setdefault((row["device_id"], phase), []).append(value)
elif at > returned_at:
observations[at][row["device_id"]] = value
if not observations:
return {**base, "reason": "post_completion_reading_missing"}
at = min(observations)
values = observations[at]
phase = int(at.timestamp()) % 86400
checks = []
for device in device_ids:
reference = baseline.get((device, phase), [])
if device not in values or len(reference) < 15:
checks.append({"device_id": device, "status": "missing_data"})
continue
center = median(reference)
sigma = 1.4826 * median(abs(value - center) for value in reference)
margin = max(3 * sigma, abs(center) * 1e-6, 1e-9)
lower, upper = min(reference) - margin, max(reference) + margin
checks.append({"device_id": device, "value": values[device], "lower": lower, "upper": upper,
"normal_day_count": len(reference),
"status": "normal" if lower <= values[device] <= upper else "abnormal"})
sample = {"at": at.isoformat(), "devices": checks}
if all(item["status"] == "normal" for item in checks):
return {**base, "status": "passed", "verified_through": at.isoformat(), "samples": [sample]}
return {**base, "reason": "nearest_post_completion_reading_not_normal_or_incomplete", "sample": sample}
async def reconcile_recoveries(service: OperationalTaskService, timescale: AsyncConnection,
settings: ProjectSettings, *, limit: int = 20,
created_from: datetime | None = None) -> int:
"""Poll only persisted recovery waits; publish evidence before waking the Agent."""
from app.domain.schemas.operational_task import OperationalTaskSummary, StoredEvidenceRecord
from app.domain.schemas.operational_data import TaskDataSnapshot
from app.infra.db.timescaledb.repositories.scada import ScadaRepository
async with service.repo.transaction():
candidates = await service.repo.recovery_candidates(limit, created_from)
resumed = 0
for task in candidates:
context = task["handling_context"]
returned_at = datetime.fromisoformat(context["returned_at"])
incident_at = datetime.fromisoformat(task["trigger"]["occurred_at"])
interval = settings.scada_monitoring_interval_seconds
roles = settings.scada_monitoring
devices = list(dict.fromkeys(roles.pressure_device_ids + roles.outlet_flow_device_ids)) if roles else []
source = "scada_readings"
through = service._now()
if context.get("device_ids"):
devices = context["device_ids"]
interval = context.get("interval_seconds", interval)
else:
# Retain the points the Agent actually inspected, including water
# quality or local incident sensors outside the project role lists.
for evidence in await service.repo.list_evidence(task["task_id"]):
saved = evidence.get("data_snapshot") or {}
if saved.get("source") not in {"query_scada_readings", "scada_readings"}:
continue
data = saved.get("data")
if isinstance(data, dict):
data = data.get("data", data.get("items", []))
if isinstance(data, list):
devices.extend(row["device_id"] for row in data if isinstance(row, dict) and isinstance(row.get("device_id"), str))
devices = list(dict.fromkeys(devices))
async with timescale.transaction():
rows = await ScadaRepository.get_recovery_scada_readings(
timescale, devices, incident_at, returned_at, through, baseline_limit=100_001,
) if through > returned_at and devices else []
result = ({"status": "waiting", "reason": "reading_budget_exceeded"} if len(rows) > 100_000 else
evaluate_recovery(rows, device_ids=devices, returned_at=returned_at,
incident_at=incident_at, interval_seconds=interval or 0))
async with service.repo.transaction():
await service.repo.lock(f"task:{task['task_id']}")
current = await service._task(UUID(task["task_id"]))
# A new receipt, cancellation or another poll may have won the race.
if current.get("handling_context") != context or current["status"] in {"cancelled", "failed"}:
continue
now = service._now()
if devices:
current["handling_context"].update(device_ids=devices, interval_seconds=interval or 0)
current["handling_context"]["last_checked_at"] = now.isoformat()
current["handling_context"]["last_check"] = result
if result["status"] == "passed":
evidence_id = "recovery:" + digest({"task_id": task["task_id"], "returned_at": context["returned_at"], "result": result})
evidence = StoredEvidenceRecord(
evidence_id=evidence_id, kind="system_data",
summary="完工反馈时间之后最近时刻的 SCADA 已恢复正常,等待 Agent 完成复核报告",
source_label="恢复供水监测复核", source_ref=f"operational-task:{task['task_id']}",
observed_at=datetime.fromisoformat(result["verified_through"]), recorded_at=now,
recorded_by=service._actor(), has_snapshot=True,
data_snapshot=TaskDataSnapshot(source=source, source_ref=evidence_id, queried_at=now,
origin="system_data", data=result, truncated=False),
)
if not await service.repo.get_evidence(task["task_id"], evidence_id):
await service.repo.insert_evidence(task["task_id"], evidence.model_dump(mode="json"))
current["handling_context"].update(phase="verification_ready", verification_evidence_id=evidence_id)
await service._requeue_task(current, now)
resumed += 1
else:
current.update(revision=current["revision"] + 1, updated_at=now)
await service.repo.update_task(OperationalTaskSummary.model_validate(current).model_dump(
mode="json", exclude={"active_run", "latest_run"}))
await service._task_event(current, now)
return resumed
+162 -16
View File
@@ -19,6 +19,7 @@ from app.auth.permissions import (
SIMULATION_VIEW,
)
from app.domain.domain_errors import DomainError, require
from app.services import task_questions
from app.domain.schemas.operational_task import (
ActionDecisionCommand,
ActionExecuteCommand,
@@ -65,10 +66,13 @@ from app.domain.schemas.operational_task import (
TaskPresentationEventCommand,
TaskPresentationEventResult,
TaskPresentationCursor,
TaskPresentationHistoryItem,
TaskPresentationHistoryPage,
)
from app.domain.schemas.operational_data import TaskDataSnapshot, TaskDataSource
from app.services.content_digest import digest
from app.services.operational_presentation import resolve_overlay
from app.services.operational_recovery import accept_completion, track_work_order
LEASE_DURATION = timedelta(seconds=60)
@@ -97,6 +101,15 @@ class OperationalTaskService:
def _actor(self) -> str:
return f"principal:{self.context.user_id}"
@staticmethod
def _require_run_owner(run: dict, command: AgentRunCommand) -> None:
require(run.get("agent_session_id") == command.source_session_id,
"SESSION_MISMATCH", "Agent session does not own this run", status=409)
owner = run.get("agent_instance_id")
if owner is not None:
require(owner == command.agent_instance_id,
"AGENT_INSTANCE_MISMATCH", "Agent instance does not own this run", status=409)
@property
def _supports_runs(self) -> bool:
return hasattr(self.repo, "get_active_run") and hasattr(self.repo, "update_run")
@@ -138,6 +151,7 @@ class OperationalTaskService:
permissions = self.permissions
return OperationalTaskPermissions(
can_publish_artifacts=OPERATIONAL_TASK_EXECUTE in permissions,
can_answer_questions=OPERATIONAL_TASK_WRITE in permissions,
can_propose_actions=OPERATIONAL_TASK_WRITE in permissions,
can_sync_external_feedback=OPERATIONAL_TASK_EXTERNAL_SYNC in permissions,
can_decide_work_orders=OPERATIONAL_ACTION_WORK_ORDER_APPROVE in permissions,
@@ -148,6 +162,14 @@ class OperationalTaskService:
)
async def create(self, command: OperationalTaskCreate) -> OperationalTaskDetail:
if command.source.replay_context:
from app.infra.db.postgresql.scada_replays import ScadaReplayRepository
replay_context = command.source.replay_context
replay_doc = await ScadaReplayRepository(self.repo.connection, self.context.project_id).get(replay_context.replay_id)
require(replay_doc.get("delivered_until") is not None
and replay_context.knowledge_cutoff <= datetime.fromisoformat(replay_doc["delivered_until"])
and replay_context.batch_sequence <= replay_doc["batch_sequence"],
"REPLAY_SCOPE_INVALID", "回放调查不能使用尚未读取的批次", status=422)
body = _json(command)
request_id = str(command.request_id)
scope = "task:create"
@@ -206,6 +228,27 @@ class OperationalTaskService:
await self._task_event(task_document, now)
return await self.detail(task_id)
async def advance_replay_investigation(self, task_id: UUID, replay_context: dict, request_id: UUID) -> None:
"""A recovery is new evidence, coalesced through the existing run lifecycle."""
scope = f"task:{task_id}:replay-evidence"
async with self.repo.transaction():
if await self._request_replay(scope, request_id, replay_context):
return
await self.repo.lock(f"task:{task_id}")
task = await self._task(task_id)
prior = task["source"].get("replay_context")
require(prior and prior["replay_id"] == replay_context["replay_id"], "REPLAY_SCOPE_INVALID",
"Replay evidence belongs to another investigation", status=409)
require(datetime.fromisoformat(prior["knowledge_cutoff"]) <= datetime.fromisoformat(replay_context["knowledge_cutoff"]),
"REPLAY_SCOPE_INVALID", "Replay evidence cannot move backwards", status=409)
task["source"]["replay_context"] = replay_context
task["source"]["external_revision"] = task["source"].get("external_revision", 1) + 1
now = self._now()
await self._requeue_task(task, now)
await self.repo.update_task(_task_json(OperationalTaskSummary.model_validate(task)))
await self._task_event(task, now)
await self._save_request(scope, request_id, replay_context, {"task_id": str(task_id)})
async def list(
self, *, task_type: str | None, status: str | None, limit: int, offset: int,
peripheral_work_order_id: UUID | None = None,
@@ -258,6 +301,7 @@ class OperationalTaskService:
evidence=evidence,
actions=actions,
access_requests=access_requests,
questions=await task_questions.list_questions(self, task_id),
permissions=self._permission_view(),
work_order_snapshot=work_order_snapshot,
)
@@ -322,6 +366,9 @@ class OperationalTaskService:
selected_report=await self.report(task_id, report_revision) if report_revision is not None else None,
)
async def replay_context(self, task_id: UUID) -> dict | None:
return (await self._task(task_id)).get("source", {}).get("replay_context")
async def can_read_source(self, task_id: UUID, source: TaskDataSource) -> bool:
task = await self._task(task_id)
permission = SCADA_VIEW if source.startswith("scada_") else SIMULATION_VIEW
@@ -483,6 +530,7 @@ class OperationalTaskService:
status="running",
revision=expected + 1,
agent_session_id=command.source_session_id,
agent_instance_id=command.agent_instance_id,
lease_expires_at=now + LEASE_DURATION,
heartbeat_at=now,
started_at=run.get("started_at") or now,
@@ -527,8 +575,7 @@ class OperationalTaskService:
"Run revision changed", status=409)
require(run["status"] in {"claimed", "running"}, "INVALID_TRANSITION",
"Only an active run has a lease", status=409)
require(run.get("agent_session_id") == command.source_session_id,
"SESSION_MISMATCH", "Agent session does not own this run", status=409)
self._require_run_owner(run, command)
now = self._now()
self._require_active_run_lease(run, now)
expected = run["revision"]
@@ -580,6 +627,7 @@ class OperationalTaskService:
replay = await self._request_replay(scope, command.request_id, body)
if replay:
return await self.detail(task_id)
await self.repo.lock(f"task:{task_id}")
await self.repo.lock(f"run:{run_id}")
run = await self._run(task_id, run_id)
require(run["revision"] == command.expected_revision, "VERSION_CONFLICT",
@@ -587,6 +635,7 @@ class OperationalTaskService:
require(run["status"] in {"queued", "claimed", "running"},
"INVALID_TRANSITION", "Run is already terminal", status=409)
now = self._now()
await task_questions.cancel_pending(self, task_id, now)
expected = run["revision"]
run.update(
status="cancelled", revision=expected + 1, finished_at=now,
@@ -624,6 +673,7 @@ class OperationalTaskService:
require(await self.repo.get_active_run(str(task_id)) is None,
"ACTIVE_RUN_EXISTS", "Task already has an active run", status=409)
now = self._now()
await task_questions.cancel_pending(self, task_id, now)
generation = task["execution_generation"] + 1
run = self._new_run(task_id, generation, now)
await self.repo.insert_run(_json(run))
@@ -778,6 +828,10 @@ class OperationalTaskService:
"Only a running run accepts Agent artifacts", status=409)
now = self._now()
self._require_active_run_lease(active_run, now)
if command.report and (
command.report.stage == "final" or command.report.outcome == "insufficient_data"
):
await self._require_no_pending_calculations(task_id, active_run)
previous_report = await self.repo.get_latest_report(str(task_id))
require(not previous_report or previous_report.get("execution_generation") != task["execution_generation"]
or previous_report["report"].get("outcome") != "insufficient_data",
@@ -911,9 +965,9 @@ class OperationalTaskService:
previous_report is None
or previous_report.get("execution_generation") != task["execution_generation"]
or previous_result.get("stage") != "final"
or any(previous_result.get(key) != _json(command.report).get(key)
or any(previous_result.get(key, [] if key == "ui_blocks" else None) != _json(command.report).get(key)
for key in ("current_conclusion", "conclusion_boundary", "outcome", "stop_reason",
"evidence_refs", "recommendations", "pending_items", "data_gaps"))
"evidence_refs", "recommendations", "pending_items", "data_gaps", "ui_blocks"))
):
await self._resolve_notifications(
task_id, kinds=["report_updated"], subject_type="task", subject_id=task_id,
@@ -952,7 +1006,7 @@ class OperationalTaskService:
# Compatibility for pre-015 repository adapters only.
if command.report:
if task["task_type"] == "event_investigation":
task["status"] = self._investigation_status(actions.values()) if command.report.stage == "final" else "running"
task["status"] = self._investigation_status(actions.values(), task) if command.report.stage == "final" else "running"
else:
task["status"] = self._status_from_plan(current_plan.plan.nodes, actions.values()) if current_plan else "running"
if task["status"] in {"waiting_human", "completed", "failed", "cancelled"}:
@@ -982,6 +1036,27 @@ class OperationalTaskService:
event_cursor=cursor,
)
async def presentation_history(
self, task_id: UUID, *, after: int = 0, through: int | None = None, limit: int = 100,
) -> TaskPresentationHistoryPage:
require(0 <= after and (through is None or through >= 0) and 1 <= limit <= 200,
"INVALID_ARGUMENT", "Invalid presentation history page", status=422)
async with self.repo.transaction():
await self._task(task_id)
cursor = await self.repo.current_event_cursor()
boundary = min(through, cursor) if through is not None else cursor
rows = await self.repo.list_presentation_events(str(task_id), after, boundary, limit + 1)
items = [TaskPresentationHistoryItem(
event_seq=row["event_seq"], occurred_at=row["occurred_at"],
run_id=row["metadata"]["run_id"],
execution_generation=row["metadata"]["execution_generation"],
command=row["metadata"]["command"],
) for row in rows[:limit]]
return TaskPresentationHistoryPage(
task_id=task_id, event_cursor=boundary, items=items,
next_after=items[-1].event_seq if len(rows) > limit else None,
)
async def publish_presentation_event(
self, task_id: UUID, command: TaskPresentationEventCommand
) -> TaskPresentationEventResult:
@@ -1015,7 +1090,16 @@ class OperationalTaskService:
evidence = StoredEvidenceRecord.model_validate(stored)
require(evidence.data_snapshot is not None, "SNAPSHOT_NOT_FOUND",
"Overlay requires a Server-ingested immutable data snapshot", status=422)
payload["data"] = resolve_overlay(evidence.data_snapshot, command.command.mode)
resolved = resolve_overlay(evidence.data_snapshot, command.command.mode)
if command.command.data is not None:
supplied = resolve_overlay(
evidence.data_snapshot.model_copy(update={
"data": command.command.data.model_dump(mode="json", exclude_unset=True),
}), command.command.mode,
)
require(supplied == resolved, "OVERLAY_DATA_MISMATCH",
"Overlay data must match the immutable evidence snapshot", status=422)
payload["data"] = resolved
metadata = {"task_id": str(task_id), "run_id": str(run["run_id"]),
"execution_generation": run["generation"], "command": payload}
try:
@@ -1048,6 +1132,7 @@ class OperationalTaskService:
await self.repo.lock(f"task:{task_id}")
task = await self._task(task_id)
self._require_actionable_task(task)
self._require_real_action(task)
self._validate_action_payload(command)
if task["task_type"] == "event_investigation":
report = await self.repo.get_latest_report(str(task_id))
@@ -1116,7 +1201,9 @@ class OperationalTaskService:
return HumanApprovalAction.model_validate(replay["action"])
await self.repo.lock(f"task:{task_id}")
await self.repo.lock(f"action:{action_id}")
self._require_actionable_task(await self._task(task_id))
task = await self._task(task_id)
self._require_actionable_task(task)
self._require_real_action(task)
action = await self._action(task_id, action_id)
require(action.kind == "create_work_order", "INVALID_ACTION_KIND",
"Only work-order drafts can be edited", status=422)
@@ -1170,7 +1257,9 @@ class OperationalTaskService:
return HumanApprovalAction.model_validate(replay["action"])
await self.repo.lock(f"task:{task_id}")
await self.repo.lock(f"action:{action_id}")
self._require_actionable_task(await self._task(task_id))
task = await self._task(task_id)
self._require_actionable_task(task)
self._require_real_action(task)
action = await self._action(task_id, action_id)
require(action.version == command.expected_version, "VERSION_CONFLICT",
"Action version changed", status=409)
@@ -1215,7 +1304,9 @@ class OperationalTaskService:
return HumanApprovalAction.model_validate(replay["action"])
await self.repo.lock(f"task:{task_id}")
await self.repo.lock(f"action:{action_id}")
self._require_actionable_task(await self._task(task_id))
task = await self._task(task_id)
self._require_actionable_task(task)
self._require_real_action(task)
action = await self._action(task_id, action_id)
require(action.version == command.expected_version, "VERSION_CONFLICT",
"Action version changed", status=409)
@@ -1311,6 +1402,7 @@ class OperationalTaskService:
)
if outcome.status in {"succeeded", "failed", "unsupported"}:
task_document = await self._task(task_id)
track_work_order(task_document, _json(current))
await self._requeue_task(task_document, outcome_recorded_at)
await self.repo.update_task(
_task_json(OperationalTaskSummary.model_validate(task_document))
@@ -1384,6 +1476,7 @@ class OperationalTaskService:
)
if outcome.status in {"succeeded", "failed", "unsupported"}:
task_document = await self._task(task_id)
track_work_order(task_document, _json(current))
await self._requeue_task(task_document, outcome_recorded_at)
await self.repo.update_task(
_task_json(OperationalTaskSummary.model_validate(task_document))
@@ -1504,6 +1597,10 @@ class OperationalTaskService:
),
)
if task["task_type"] == "event_investigation":
# Recover older tasks whose successful dispatch predates tracking.
for item in await self.repo.list_actions(str(task_id)):
track_work_order(task, item)
accept_completion(task, body)
if matching_source:
task["source"] = {**task["source"], "external_revision": command.source_revision}
await self._requeue_task(task, now)
@@ -1866,6 +1963,15 @@ class OperationalTaskService:
task_ids.append(task_id)
return DispatchDueSchedulesResult(task_ids=task_ids, dispatched_at=now)
async def _require_no_pending_calculations(self, task_id: UUID, run: dict) -> None:
if hasattr(self.repo, "pending_calculation_jobs"):
pending = await self.repo.pending_calculation_jobs(
str(task_id), str(run["run_id"]), str(run["execution_request_id"]),
)
require(not pending, "PENDING_CALCULATIONS",
"Wait for submitted calculations before publishing a final report or completing the run",
refs=pending, status=409, recovery="retry_same_request")
async def _finish_run(
self, task_id: UUID, run_id: UUID, command: AgentRunCommand,
*, outcome: str, failure_code: str | None = None,
@@ -1877,19 +1983,23 @@ class OperationalTaskService:
replay = await self._request_replay(scope, command.request_id, body)
if replay:
return await self.detail(task_id)
await self.repo.lock(f"task:{task_id}")
await self.repo.lock(f"run:{run_id}")
run = await self._run(task_id, run_id)
require(run["revision"] == command.expected_revision, "VERSION_CONFLICT",
"Run revision changed", status=409)
require(run["status"] in {"claimed", "running"}, "INVALID_TRANSITION",
"Only an active run can finish", status=409)
require(run.get("agent_session_id") == command.source_session_id,
"SESSION_MISMATCH", "Agent session does not own this run", status=409)
self._require_run_owner(run, command)
now = self._now()
self._require_active_run_lease(run, now)
task = await self._task(task_id)
final_outcome = outcome
pending_question = any(q["status"] == "pending" and str(q["run_id"]) == str(run_id)
for q in await task_questions.list_questions(self, task_id))
actions = []
if outcome == "succeeded":
await self._require_no_pending_calculations(task_id, run)
if hasattr(self.repo, "pending_run_messages"):
pending_messages = await self.repo.pending_run_messages(str(run_id))
require(not pending_messages, "PENDING_AGENT_MESSAGES",
@@ -1915,7 +2025,9 @@ class OperationalTaskService:
and plan.source_session_id == command.source_session_id
))
)
if not same_generation or report.report.stage != "final":
if pending_question:
lifecycle = "waiting_human"
elif not same_generation or report.report.stage != "final":
final_outcome = "failed"
lifecycle = "failed"
failure_code = "ARTIFACT_INCOMPLETE"
@@ -1932,7 +2044,15 @@ class OperationalTaskService:
for item in await self._list_access_requests(task_id)
]
if task["task_type"] == "event_investigation":
lifecycle = self._investigation_status(actions)
lifecycle = self._investigation_status(actions, task)
recovery = task.get("handling_context") or {}
if lifecycle == "completed" and recovery:
require(recovery.get("verification_evidence_id") in report.report.evidence_refs,
"RECOVERY_EVIDENCE_REQUIRED", "最终复核报告必须引用恢复监测证据", status=409)
require(report.report.outcome == "concluded", "RECOVERY_EVIDENCE_REQUIRED",
"恢复复核通过后才能结束事件", status=409)
recovery.update(phase="completed", title="恢复供水复核成功", completed_at=now.isoformat())
await self._resolve_investigation_alerts(str(task_id), recovery["verification_evidence_id"])
elif report.report.outcome == "insufficient_data":
terminal = all(node.status in {"completed", "skipped", "cancelled"}
for node in plan.plan.nodes)
@@ -1960,6 +2080,7 @@ class OperationalTaskService:
run_model = OperationalTaskRun.model_validate(run)
await self.repo.update_run(_json(run_model), expected)
if final_outcome != "succeeded":
await task_questions.cancel_pending(self, task_id, now)
await self._fail_run_messages(run_id, failure_code or "Run failed", now)
task.update(
lifecycle_status=lifecycle,
@@ -1980,7 +2101,8 @@ class OperationalTaskService:
failure_detail or "工况任务执行失败",
now,
)
elif lifecycle == "waiting_human":
elif (lifecycle == "waiting_human" and not pending_question
and (task.get("handling_context") or {}).get("phase") != "waiting_scada"):
pending_actions = any(
action.approval_status == "pending" for action in actions
)
@@ -2007,11 +2129,32 @@ class OperationalTaskService:
)
return run
async def _resolve_investigation_alerts(self, task_id: str, evidence_id: str) -> None:
from app.domain.schemas.task_alert import TaskAlertEvent
from app.infra.db.postgresql.task_alerts import TaskAlertRepository
from app.services.task_alerts import TaskAlertService
if not hasattr(self.repo, "list_investigation_alerts"):
return
for alert in await self.repo.list_investigation_alerts(task_id):
command = TaskAlertEvent.model_validate({
**{key: value for key, value in alert.items() if key not in {"status", "updated_at", "revision"}},
"event_id": uuid5(NAMESPACE_URL, f"recovery:{task_id}:{evidence_id}:{alert['alert_id']}"),
"event_type": "resolved",
})
await TaskAlertService(TaskAlertRepository(self.repo.connection, self.repo.project_id), None,
internal_producer_id=command.producer_id).push(command)
async def _task(self, task_id: UUID) -> dict:
task = await self.repo.get_task(str(task_id))
require(task is not None, "TASK_NOT_FOUND", "Operational task not found", status=404)
return task
@staticmethod
def _require_real_action(task: dict) -> None:
require(not (task.get("source") or {}).get("replay_context"), "REPLAY_ACTION_DISABLED",
"当前任务的工单和设备动作未开放;请继续完成已授权的定位、地图展示、隔离影响分析与报告", status=409)
@staticmethod
def _require_actionable_task(task: dict) -> None:
source = task.get("source") or {}
@@ -2075,8 +2218,10 @@ class OperationalTaskService:
return "running"
@staticmethod
def _investigation_status(actions: Iterable[HumanApprovalAction]) -> str:
"""Only recorded business actions can keep an investigation waiting."""
def _investigation_status(actions: Iterable[HumanApprovalAction], task: dict | None = None) -> str:
"""Field completion and recovery evidence precede incident closure."""
if (task or {}).get("handling_context", {}).get("phase") in {"waiting_work_order", "waiting_scada"}:
return "waiting_human"
return "waiting_human" if any(
action.approval_status == "pending" or (
action.approval_status == "approved"
@@ -2193,6 +2338,7 @@ class OperationalTaskService:
task["agent_lease_expires_at"] = None
async def _requeue_task(self, task: dict, now: datetime) -> None:
await task_questions.cancel_pending(self, UUID(str(task["task_id"])), now)
if self._supports_runs:
active_run = await self.repo.get_active_run(str(task["task_id"]))
while active_run is not None:
+37 -15
View File
@@ -6,6 +6,7 @@ from pydantic import ValidationError
from app.core.config import settings
from app.core.tianditu_logging import install_tianditu_token_filter
from app.domain.schemas.project_map_config import PlaceSearchConfig
from app.domain.schemas.place_search import (
MAX_PLACE_SEARCH_RESULTS,
PlaceSearchItem,
@@ -13,7 +14,6 @@ from app.domain.schemas.place_search import (
PlaceSearchResponse,
)
FENGXIAN_ADMIN_CODE = "156310120"
install_tianditu_token_filter()
@@ -24,11 +24,37 @@ class PlaceSearchError(RuntimeError):
self.detail = detail
class PlaceSearchScopeError(ValueError):
pass
def validate_place_scope(
config: PlaceSearchConfig, *, county_code: str, county: str, province_code: str = "",
) -> None:
admin_code = config.admin_code.removeprefix("156")
province_code = province_code.removeprefix("156")
county_code = county_code.removeprefix("156")
if province_code and province_code[:2] != admin_code[:2]:
raise PlaceSearchScopeError("Result outside configured province")
scope_prefix = admin_code
if admin_code.endswith("0000"):
scope_prefix = admin_code[:2]
elif admin_code.endswith("00"):
scope_prefix = admin_code[:4]
if county_code and not county_code.startswith(scope_prefix):
raise PlaceSearchScopeError("Result outside configured district")
if (len(scope_prefix) == 6 and county and county != config.region) or (not county_code and county != config.region):
raise PlaceSearchScopeError("Missing or mismatched district")
async def search_tianditu_places(
request: PlaceSearchRequest,
*,
config: PlaceSearchConfig,
client: httpx.AsyncClient | None = None,
) -> PlaceSearchResponse:
if not config.enabled:
raise PlaceSearchError(503, "当前项目暂未配置地名搜索")
token = settings.TIANDITU_TOKEN
if not token:
raise PlaceSearchError(503, "地名搜索服务未配置")
@@ -37,7 +63,7 @@ async def search_tianditu_places(
{
"keyWord": request.keyword,
"queryType": 12,
"specify": FENGXIAN_ADMIN_CODE,
"specify": "156" + config.admin_code.removeprefix("156"),
"start": 0,
"count": MAX_PLACE_SEARCH_RESULTS,
"show": 2,
@@ -64,15 +90,15 @@ async def search_tianditu_places(
# Request URLs contain the provider key and must not reach the client.
raise PlaceSearchError(502, "天地图地名搜索服务暂不可用") from None
try:
return _parse_response(response.json())
return _parse_response(response.json(), config)
except (ValueError, TypeError, KeyError, AttributeError, ValidationError):
raise PlaceSearchError(502, "天地图返回了无效的地名搜索结果") from None
def _parse_response(payload: dict[str, Any]) -> PlaceSearchResponse:
def _parse_response(payload: dict[str, Any], config: PlaceSearchConfig) -> PlaceSearchResponse:
code = str(payload.get("status", {}).get("infocode", ""))
if code == "3001":
return PlaceSearchResponse(items=[], total=0)
return PlaceSearchResponse(items=[], total=0, region=config.region)
if code != "1000":
raise PlaceSearchError(502, "天地图地名搜索请求失败,请稍后重试")
@@ -84,15 +110,11 @@ def _parse_response(payload: dict[str, Any]) -> PlaceSearchResponse:
raise ValueError("Invalid search results")
items = []
for raw in raw_items:
admin_code = str(raw.get("provinceCode") or raw.get("adminCode") or "")
if admin_code and not admin_code.startswith(("156310", "310")):
raise ValueError("Result outside Shanghai")
county_code = str(raw.get("countyCode") or (raw.get("adminCode") if result_type == "3" else "") or "")
county = raw.get("county")
if county_code and not county_code.startswith((FENGXIAN_ADMIN_CODE, "310120")):
raise ValueError("Result outside Fengxian")
if county and county != "奉贤区" or not county_code and county != "奉贤区":
raise ValueError("Missing or mismatched district")
validate_place_scope(
config, county_code=str(raw.get("countyCode") or raw.get("adminCode") or ""),
county=str(county or ""), province_code=str(raw.get("provinceCode") or ""),
)
lon, lat = str(raw["lonlat"]).split(",")
name = raw["name"]
if not isinstance(name, str) or not name.strip():
@@ -105,11 +127,11 @@ def _parse_response(payload: dict[str, Any]) -> PlaceSearchResponse:
address=str(raw.get("address") or name),
longitude=float(lon), latitude=float(lat),
category="行政区" if result_type == "3" else category,
district="奉贤区",
district=str(county or config.region),
))
total = int(payload.get("count", 0))
if total < len(items):
raise ValueError("Invalid search total")
return PlaceSearchResponse(
items=items, total=len(items), truncated=total > MAX_PLACE_SEARCH_RESULTS,
items=items, total=len(items), truncated=total > MAX_PLACE_SEARCH_RESULTS, region=config.region,
)
+5 -2
View File
@@ -2,6 +2,7 @@ from collections.abc import Iterator
from contextlib import contextmanager
from pathlib import Path
from tempfile import NamedTemporaryFile
import os
from app.native.wndb.inp.exporter import dump_inp
@@ -17,9 +18,11 @@ def temporary_project_inp(
version: str = "2",
) -> Iterator[Path]:
"""Export one project model to an isolated INP and remove it afterwards."""
PROJECT_INP_DIRECTORY.mkdir(parents=True, exist_ok=True)
job_directory = os.environ.get("TJWATER_CALCULATION_JOB_DIR")
directory = Path(job_directory) / "inp" if job_directory else PROJECT_INP_DIRECTORY
directory.mkdir(parents=True, exist_ok=True)
with NamedTemporaryFile(
dir=PROJECT_INP_DIRECTORY,
dir=directory,
prefix=f"{purpose}_",
suffix=".inp",
delete=False,
+72
View File
@@ -0,0 +1,72 @@
import json
from typing import Any
import httpx
from app.core.config import settings
from app.domain.schemas.place_search import PlaceSearchItem, ReverseGeocodingRequest
from app.domain.schemas.project_map_config import PlaceSearchConfig
from app.services.place_search import PlaceSearchError, PlaceSearchScopeError, validate_place_scope
async def reverse_geocode_tianditu(
request: ReverseGeocodingRequest,
*,
config: PlaceSearchConfig,
client: httpx.AsyncClient | None = None,
) -> PlaceSearchItem:
if not config.enabled:
raise PlaceSearchError(503, "当前项目暂未配置地名搜索")
if not settings.TIANDITU_TOKEN:
raise PlaceSearchError(503, "地名搜索服务未配置")
params = {
"postStr": json.dumps({"lon": request.longitude, "lat": request.latitude, "ver": 1}),
"type": "geocode",
"tk": settings.TIANDITU_TOKEN,
}
try:
if client is not None:
response = await client.get(settings.TIANDITU_GEOCODER_URL, params=params)
else:
async with httpx.AsyncClient(timeout=settings.TIANDITU_SEARCH_TIMEOUT_SECONDS) as managed:
response = await managed.get(settings.TIANDITU_GEOCODER_URL, params=params)
response.raise_for_status()
except httpx.TimeoutException:
raise PlaceSearchError(504, "地图取点查询超时,请重新取点") from None
except httpx.HTTPError:
raise PlaceSearchError(502, "天地图反编码服务暂不可用,请重新取点") from None
try:
return _parse_reverse_response(response.json(), request, config)
except PlaceSearchScopeError:
raise PlaceSearchError(422, f"取点位置不在当前项目的地名搜索范围内({config.region})") from None
except (ValueError, TypeError, KeyError, AttributeError):
raise PlaceSearchError(502, "天地图返回了无效的反编码结果,请重新取点") from None
def _parse_reverse_response(
payload: dict[str, Any], request: ReverseGeocodingRequest, config: PlaceSearchConfig,
) -> PlaceSearchItem:
if str(payload.get("status")) != "0":
raise PlaceSearchError(502, "天地图反编码请求失败,请重新取点")
result = payload.get("result")
if not result:
raise PlaceSearchError(404, "该位置未找到地名或地址,请重新取点")
address = result.get("formatted_address")
if not isinstance(address, str) or not address.strip():
raise PlaceSearchError(404, "该位置未找到地名或地址,请重新取点")
component = result.get("addressComponent") or {}
validate_place_scope(
config, county_code=str(component.get("county_code") or ""),
county=str(component.get("county") or ""),
province_code=str(component.get("province_code") or ""),
)
name = component.get("poi") or component.get("road") or address
if not isinstance(name, str) or not name.strip():
raise ValueError("Invalid place name")
# A nearby POI can have a different center. Always retain the clicked point.
return PlaceSearchItem(
id=f"map:{request.longitude:.8f},{request.latitude:.8f}",
name=name.strip(), address=address.strip(), longitude=request.longitude, latitude=request.latitude,
category="附近地名" if component.get("poi") else "地址",
district=str(component.get("county") or config.region),
)
@@ -0,0 +1,141 @@
"""Fit causal replay thresholds in bounded numeric arrays, before publishing history."""
from datetime import datetime, timedelta
import numpy as np
import pandas as pd
from app.domain.schemas.project_settings import ScadaMonitoringSettings
from app.domain.schemas.scada_replay import ScadaReplayCreate
from app.services.scada_normal_profile import CALIBRATION_VERSION, DIAGNOSTIC_POLICY
def adaptive_score_threshold(scores: list[float]) -> dict:
"""Keep the cutoff below the normal lower tail, with a fixed noise margin."""
values = np.asarray(scores, dtype=float)
if len(values) < 5 or not np.isfinite(values).all():
return {"status": "insufficient_data", "reason": "normal_score_history_incomplete"}
center = float(np.median(values))
sigma = float(1.4826 * np.median(np.abs(values - center)))
lower_tail = float(np.quantile(values, 0.05))
return {
"status": "ready",
"threshold": min(-0.04, center - 3 * sigma, lower_tail - 0.02),
"normal_score_median": center,
"normal_score_min": float(values.min()),
"normal_score_max": float(values.max()),
"normal_score_robust_sigma": sigma,
"normal_score_lower_quantile": lower_tail,
"normal_score_count": len(values),
"method": "min(-0.04, median-3*robust_sigma, normal_q05-0.02)",
}
class MonitoringCalibrationBuilder:
"""Retain bounded numeric series, never full raw pages or device batches."""
def __init__(self, document: dict, config: ScadaReplayCreate):
self.settings = ScadaMonitoringSettings.model_validate(document["monitoring_config"])
self.config = config
self.cutoff = config.start_time
self.start = self.cutoff - timedelta(days=15)
self.interval = int(document["monitoring_interval_seconds"])
self.supported = self.interval % 60 == 0 and 86400 % self.interval == 0
self.count = 15 * (86400 // self.interval) if self.supported else 0
ids = dict.fromkeys(self.settings.pressure_device_ids + self.settings.outlet_flow_device_ids
+ [device for rule in config.rules if rule.kind == "relative_change"
for device in rule.device_ids])
self.values = {device: np.full(self.count, np.nan) for device in ids}
def add(self, rows: list[dict]) -> None:
for row in rows:
device = row["device_id"]
if device not in self.values:
continue
at = datetime.fromisoformat(row["time"])
if at >= self.cutoff:
raise ValueError("动态阈值拟合不能读取回放开始时刻及之后的观测")
offset = (at - self.start).total_seconds()
if (not self.supported or offset < 0 or offset % self.interval
or row.get("quality") not in {"valid", "corrected"}
or row.get("cleaned_value") is None):
continue
index = int(offset // self.interval)
if np.isfinite(self.values[device][index]):
raise ValueError("动态阈值拟合存在重复的设备时间点")
self.values[device][index] = float(row["cleaned_value"])
def fit(self) -> dict:
base = {
"version": CALIBRATION_VERSION,
"diagnostic_policy": DIAGNOSTIC_POLICY,
"knowledge_cutoff": self.cutoff.isoformat(),
"history_start": self.start.isoformat(),
"required_samples_per_device": self.count,
"pressure": {}, "outlet_flow": {}, "diagnostic": {},
}
if not self.supported:
return {**base, "status": "insufficient_data", "reason": "unsupported_sampling_interval"}
available = {device: int(np.isfinite(values).sum()) for device, values in self.values.items()}
# A regular diurnal rise/reversal can be a huge percentage of a small
# preceding flow. Judge its absolute value against causal normal readings
# at the same sampling phase before treating that percentage as abnormal.
points_per_day = 86400 // self.interval
for device, values in self.values.items():
if available[device] != self.count:
continue
days = values.reshape(15, points_per_day)
center = np.median(days, axis=0)
sigma = 1.4826 * np.median(np.abs(days - center), axis=0)
margin = np.maximum(3 * sigma, np.maximum(np.abs(center) * 1e-6, 1e-9))
base["diagnostic"][device] = {
"phase_origin": self.start.isoformat(), "interval_seconds": self.interval,
"lower": (days.min(axis=0) - margin).tolist(),
"upper": (days.max(axis=0) + margin).tolist(),
"method": "normal_same_phase_range_plus_3_robust_sigma", "normal_day_count": 15,
}
for kind, ids, direction, floor in (
("pressure", self.settings.pressure_device_ids, -1, 3.0),
("outlet_flow", self.settings.outlet_flow_device_ids, 1, 5.0),
):
for device in ids:
if available[device] != self.count:
continue
rule = next((rule for rule in self.config.rules
if rule.kind == "relative_change" and device in rule.device_ids), None)
if rule is None:
continue
series = pd.Series(self.values[device])
reference = series.rolling(rule.window_samples, min_periods=rule.minimum_samples).median().shift(1)
changes = (direction * 100 * (series - reference) / reference.abs()).to_numpy()
changes = changes[np.isfinite(changes)]
if not len(changes):
continue
center = float(np.median(changes))
sigma = float(1.4826 * np.median(np.abs(changes - center)))
base[kind][device] = {
"threshold_percent": max(floor, center + 3 * sigma),
"normal_change_median": center,
"normal_change_robust_sigma": sigma,
"normal_change_count": int(len(changes)),
"method": "normal_recent_window_change_median_plus_3_robust_sigma",
"normal_profile": base["diagnostic"].get(device),
}
pressure_ids = self.settings.pressure_device_ids
incomplete = {device: available[device] for device in pressure_ids if available[device] != self.count}
if len(pressure_ids) < 5 or incomplete:
return {**base, "status": "insufficient_data", "reason": "pressure_history_incomplete",
"available_samples": available}
from app.services import scada_replay_monitoring as monitoring
index = pd.date_range(start=self.start, periods=self.count, freq=pd.Timedelta(seconds=self.interval))
frame = pd.DataFrame({device: self.values[device] for device in pressure_ids}, index=index)
result = monitoring.run_burst_detection(
network="scada-replay-baseline", username="scada-replay",
observed_pressure_data=frame, points_per_day=86400 // self.interval,
mu=monitoring.TARGET_MU,
iforest_params={"n_estimators": monitoring.TARGET_N_ESTIMATORS,
"random_state": monitoring.TARGET_RANDOM_STATE, "contamination": "auto"},
sensor_nodes=pressure_ids,
)
score = adaptive_score_threshold([float(row["Score"]) for row in result["rows"]])
return {**base, "status": score["status"], "burst_detection": score}
@@ -0,0 +1,98 @@
"""Persist one accident across monitoring periods without moving its evidence cutoff."""
from datetime import datetime
from math import isfinite
from uuid import NAMESPACE_URL, uuid5
def _anchors(monitoring: dict, indicator: str = "pressure") -> dict:
return {
item["device_id"]: {"baseline": item["baseline"], "threshold_percent": item["threshold_percent"]}
for item in monitoring.get(indicator, {}).get("devices", [])
if item.get("hit") and item.get("baseline", 0) > 0
and 0 < item.get("threshold_percent", 0) < 100
}
def update_incident(document: dict, monitoring: dict, sequence: int) -> tuple[dict | None, str | None]:
incident = document.get("monitoring_incident")
# Adopt the last task from an older replay document on its first new step.
previous = document.get("last_monitoring") or {}
if incident:
incident.setdefault("burst_diagnosed", previous.get("status") == "suspected_burst")
if not incident and previous.get("task_id"):
incident = {
"incident_id": str(uuid5(NAMESPACE_URL, f"scada_replay:{document['replay_id']}:monitor:"
f"{previous['period_end']}:{previous['rule_version']}")),
"started_at": previous["period_end"], "last_observed_at": previous["period_end"],
"status": "active", "task_id": previous["task_id"], "recovery_count": 0,
"pressure_anchors": _anchors(previous), "alert": None,
"flow_anchors": _anchors(previous, "outlet_flow"),
"burst_diagnosed": previous.get("status") == "suspected_burst",
}
document["monitoring_incident"] = incident
if incident and incident.get("last_monitoring_id") == monitoring.get("monitoring_id") and monitoring.get("monitoring_id"):
return incident, incident["last_action"]
anomaly = (monitoring["status"] in {"attention", "suspected_burst"}
or bool(monitoring.get("anomaly_signals"))
or any(monitoring.get(key, {}).get("hit") is True
for key in ("pressure", "outlet_flow", "burst_detection")))
if anomaly and (not incident or incident["status"] == "resolved"):
external = (f"scada_replay:{document['replay_id']}:monitor:"
f"{monitoring['period_end']}:{monitoring['rule_version']}")
incident = {
"incident_id": str(uuid5(NAMESPACE_URL, external)), "external_id": external,
"started_at": monitoring["period_end"], "status": "active", "task_id": None,
"recovery_count": 0, "pressure_anchors": _anchors(monitoring), "alert": None,
"flow_anchors": _anchors(monitoring, "outlet_flow"),
"batch_sequence": sequence,
}
document["monitoring_incident"] = incident
action = "raised"
elif incident and incident["status"] == "active":
# A later diagnostic stage may add pressure/flow evidence to the same task.
# Freeze each newly abnormal device once; retain earlier references.
for indicator, field in (("pressure", "pressure_anchors"), ("outlet_flow", "flow_anchors")):
for device, anchor in _anchors(monitoring, indicator).items():
incident.setdefault(field, {}).setdefault(device, anchor)
current = {item["device_id"]: item for item in monitoring.get("pressure", {}).get("devices", [])}
anchors = incident["pressure_anchors"]
flow = {item["device_id"]: item for item in monitoring.get("outlet_flow", {}).get("devices", [])}
# Rolling relative changes can become normal while absolute pressure stays low.
# Confirm recovery against the frozen pre-accident references as well.
recovered = monitoring["status"] == "normal" and not anomaly and all(
monitoring.get(key, {}).get("status") == "ready"
and monitoring[key].get("hit") is False for key in ("pressure", "burst_detection")
) and all(
device in current and isfinite(current[device]["value"])
and current[device].get("observed_at") is not None
and datetime.fromisoformat(current[device]["observed_at"]) > datetime.fromisoformat(incident["last_observed_at"])
and current[device]["value"] >= anchor["baseline"] * (1 - anchor["threshold_percent"] / 100)
for device, anchor in anchors.items()
) and all(
device in flow and isfinite(flow[device]["value"])
and flow[device].get("observed_at") is not None
and datetime.fromisoformat(flow[device]["observed_at"]) > datetime.fromisoformat(incident["last_observed_at"])
and flow[device]["value"] <= anchor["baseline"] * (1 + anchor["threshold_percent"] / 100)
for device, anchor in incident.get("flow_anchors", {}).items()
)
contiguous = bool(monitoring.get("period_start")) and (
datetime.fromisoformat(monitoring["period_start"]) == datetime.fromisoformat(incident["last_observed_at"])
)
incident["recovery_count"] = (incident["recovery_count"] + 1 if contiguous else 1) if recovered else 0
action = "resolved" if incident["recovery_count"] >= 2 else "ongoing"
if action == "resolved":
incident["status"] = "resolved"
else:
return None, None
incident.update(last_observed_at=monitoring["period_end"],
last_monitoring_id=monitoring.get("monitoring_id"), last_action=action)
return incident, action
def attach_incident(monitoring: dict, incident: dict, action: str) -> None:
monitoring["task_id"] = incident["task_id"]
monitoring["incident"] = {
key: incident[key] for key in (
"incident_id", "status", "started_at", "last_observed_at", "recovery_count",
)
} | {"action": action}
+48
View File
@@ -0,0 +1,48 @@
"""Clock-phase normal envelopes frozen before replay; never inspect future readings."""
from datetime import datetime
from math import isfinite
CALIBRATION_VERSION = "scada-monitor-calibration@2"
DIAGNOSTIC_POLICY = "normal-phase-envelope@1"
def weighted_diagnosis(document: dict) -> bool:
return (document.get("monitoring_config") or {}).get("decision_policy") == "weighted"
def current_calibration(document: dict, start: datetime) -> dict | None:
calibration = document.get("monitoring_calibration") or {}
if (calibration.get("version") == CALIBRATION_VERSION
and calibration.get("diagnostic_policy") == DIAGNOSTIC_POLICY
and calibration.get("knowledge_cutoff") == start.isoformat()):
return calibration
return None
def diagnostic_signal(document: dict, event: dict) -> bool:
"""Never revive a persisted fixed-percentage event as new diagnosis evidence."""
if event.get("category") == "data_quality":
return False
if not weighted_diagnosis(document):
return True
rules = (document.get("config") or {}).get("rules", [])
rule = next((item for item in rules if item["rule_id"] == event["rule_id"]), None)
return bool(rule and (rule["kind"] == "bounds" or event.get("diagnostic_policy") == DIAGNOSTIC_POLICY))
def normal_profile_bounds(profile: dict | None, at: datetime) -> tuple[float, float] | None:
if not profile:
return None
interval = profile.get("interval_seconds", 0)
if not profile.get("phase_origin"):
return None
origin = datetime.fromisoformat(profile["phase_origin"])
lower, upper = profile.get("lower", []), profile.get("upper", [])
offset = (at - origin).total_seconds()
if interval <= 0 or offset < 0 or offset % interval or not lower or len(lower) != len(upper):
return None
slot = int(offset // interval) % len(lower)
low, high = lower[slot], upper[slot]
if low is None or high is None or not isfinite(low) or not isfinite(high):
return None
return float(low), float(high)
+66
View File
@@ -0,0 +1,66 @@
"""Read-only chronological SCADA paging shared by REST and replay processing."""
from __future__ import annotations
import base64
import hashlib
import json
import math
from datetime import datetime
from app.domain.schemas.scada import ScadaRawPage, ScadaRawReading
from app.infra.db.timescaledb.repositories.scada import ScadaRepository
def validate_window(start: datetime, end: datetime) -> None:
if start.tzinfo is None or end.tzinfo is None or start >= end:
raise ValueError("需要带时区、结束晚于开始的 SCADA 时间区间")
def raw_reading(row: dict) -> ScadaRawReading:
value = row["monitored_value"]
invalid = value is not None and not math.isfinite(float(value))
return ScadaRawReading(
time=row["time"], device_id=row["device_id"],
monitored_value=None if invalid else value,
quality="invalid" if invalid else "missing" if value is None else "unvalidated",
)
async def read_raw_page(
conn, *, project_id: str, device_ids: list[str], start: datetime,
end: datetime, limit: int = 500, cursor: str | None = None,
through: datetime | None = None,
) -> ScadaRawPage:
validate_window(start, end)
ids = sorted(set(device_ids))
if not ids or len(ids) > 500 or not 1 <= limit <= 500:
raise ValueError("请选择 1 至 500 个设备,每页最多 500 条")
cap = min(through or end, end)
if cap.tzinfo is None or cap < start:
raise ValueError("模拟时间不在可读取区间内")
signature = hashlib.sha256(json.dumps(
[project_id, ids, start.isoformat(), end.isoformat()], separators=(",", ":")
).encode()).hexdigest()
after = None
if cursor:
try:
data = json.loads(base64.urlsafe_b64decode(cursor.encode()))
frozen = datetime.fromisoformat(data["through"])
last = datetime.fromisoformat(data["time"])
if (data["query"] != signature or frozen.tzinfo is None or last.tzinfo is None
or frozen > cap or not start <= last <= frozen or data["device"] not in ids):
raise ValueError()
cap, after = frozen, (last, data["device"])
except (ValueError, KeyError, TypeError, UnicodeError) as exc:
raise ValueError("SCADA 分页游标无效或查询范围已改变") from exc
rows = await ScadaRepository.get_raw_page(conn, ids, start, end, cap, limit + 1, after)
selected = rows[:limit]
next_cursor = None
if len(rows) > limit:
last = selected[-1]
next_cursor = base64.urlsafe_b64encode(json.dumps({
"query": signature, "through": cap.isoformat(),
"time": last["time"].isoformat(), "device": last["device_id"],
}, separators=(",", ":")).encode()).decode()
return ScadaRawPage(items=[raw_reading(row) for row in selected],
next_cursor=next_cursor, scanned_through=cap)
+600
View File
@@ -0,0 +1,600 @@
"""Durable, project-scoped SCADA playback; raw is never inserted or rewritten."""
from __future__ import annotations
from datetime import datetime, timedelta, timezone
import json
from uuid import NAMESPACE_URL, uuid4, uuid5
from fastapi import HTTPException
from app.auth.permissions import permissions_for_context
from app.domain.schemas.project_settings import ScadaMonitoringSettings
from app.domain.schemas.scada_replay import ScadaReplayCreate, ScadaReplayCommand, ScadaReplaySnapshot
from app.infra.db.postgresql.scada import ScadaInfoRepository
from app.infra.db.timescaledb.repositories.scada import ScadaRepository
from app.services.content_digest import digest
from app.services.scada_replay_processing import prepare_cleaning, detect_frame
from app.services.scada_replay_monitoring import evaluate_monitoring_period
from app.services.scada_normal_profile import current_calibration, diagnostic_signal, weighted_diagnosis
# 500 devices at 15-minute sampling need 720,000 rows for fifteen days.
# Leave headroom without allowing dense sources to make preparation unbounded.
BASELINE_PAGE_SIZE = 2000
BASELINE_MAX_ROWS = 1_000_000
BASELINE_MAX_BYTES = 512 * 1024 * 1024
class BaselineBudgetExceeded(RuntimeError):
"""The unpublished baseline exceeded its fixed preparation budget."""
def parse_time(value: str) -> datetime:
return datetime.fromisoformat(value)
def snapshot(document: dict) -> ScadaReplaySnapshot:
return ScadaReplaySnapshot(
**{key: document[key] for key in (
"replay_id", "project_id", "revision", "status", "start_time", "end_time",
"virtual_now", "created_at", "updated_at",
)},
delivered_until=document.get("delivered_until"), processed_until=document.get("processed_until"),
processed_count=document.get("processed_count", 0), anomaly_count=document.get("anomaly_count", 0),
devices=document["devices"], rules=document["config"]["rules"],
latest=list(document.get("latest", {}).values()),
active_events=[state["event"] for state in document.get("rule_states", {}).values() if state.get("event")],
error=document.get("error"),
monitoring_interval_seconds=document.get("monitoring_interval_seconds"),
step_pending=document.get("step_pending", False),
monitoring_count=document.get("monitoring_count", 0),
last_monitoring=document.get("last_monitoring"),
# Old persisted documents without a policy still use three-signal detection.
monitoring_config=({"decision_policy": "all_signals", **document["monitoring_config"]}
if document.get("monitoring_config") else None),
monitoring_calibration=document.get("monitoring_calibration"),
)
class ScadaReplayService:
def __init__(self, repository, context, timescale=None, *, clock=None):
self.repo, self.context, self.timescale = repository, context, timescale
self.clock = clock or (lambda: datetime.now(timezone.utc))
async def ensure_storage(self):
if not await self.repo.available():
raise HTTPException(503, "SCADA 回放存储未配置,需要应用 031_scada_replays.sql")
async def create(
self,
command: ScadaReplayCreate,
*,
monitoring_interval_seconds: int | None,
monitoring_config: ScadaMonitoringSettings | None = None,
):
await self.ensure_storage()
async with self.repo.control():
previous = await self.repo.by_request(command.request_id)
config = command.model_dump(mode="json")
if previous:
if previous["config"] != config:
raise HTTPException(409, "创建请求号已用于不同回放配置")
return snapshot(previous)
if monitoring_interval_seconds is None:
raise HTTPException(422, "请先配置项目 settings.scada_monitoring_interval_seconds,再创建演示")
replay_id = str(uuid4())
if await self.repo.active_other(replay_id):
raise HTTPException(409, "当前项目已有回放,请先停止原回放再新建")
metadata = {row["device_id"]: row for row in await ScadaInfoRepository.get_scadas(self.repo.connection)}
if not set(command.device_ids) <= metadata.keys():
raise HTTPException(422, "设备不属于当前项目")
for rule in command.rules:
if any(metadata[device]["measurement_unit"] != rule.unit for device in rule.device_ids):
raise HTTPException(422, "规则单位与设备单位不一致,请分组配置")
if any(metadata[device]["device_type"] == "quality" for device in rule.device_ids) and not rule.metric_name.strip():
raise HTTPException(422, "水质规则必须明确指标名称,不能仅凭单位解释读数")
if monitoring_config:
role_ids = set(monitoring_config.pressure_device_ids) | set(monitoring_config.outlet_flow_device_ids)
if not role_ids <= metadata.keys():
raise HTTPException(422, "项目 SCADA 监测角色包含不存在的设备")
if any(metadata[device]["device_type"] != "pressure" for device in monitoring_config.pressure_device_ids):
raise HTTPException(422, "压力监测角色必须引用 pressure 设备")
if any(metadata[device]["device_type"] not in {"pipe_flow", "flow"}
for device in monitoring_config.outlet_flow_device_ids):
raise HTTPException(422, "出厂边界流量角色必须引用流量设备")
now = self.clock().isoformat()
document = {
"replay_id": replay_id, "project_id": str(self.context.project_id),
"created_by": str(self.context.user_id), "config": config,
"revision": 1, "status": "ready", "start_time": config["start_time"],
"end_time": config["end_time"], "virtual_now": config["start_time"],
"created_at": now, "updated_at": now,
"monitoring_interval_seconds": monitoring_interval_seconds,
"monitoring_config": monitoring_config.model_dump(mode="json") if monitoring_config else None,
"devices": [metadata[device] for device in command.device_ids],
"batch_sequence": 0, "processed_count": 0, "anomaly_count": 0,
"monitoring_count": 0, "latest": {}, "histories": {}, "rule_states": {}, "receipts": {},
}
await self.repo.insert(document)
return snapshot(document)
async def command(self, replay_id, command: ScadaReplayCommand):
await self.ensure_storage()
async with self.repo.control():
document = await self.repo.get(replay_id)
key, body = str(command.request_id), command.model_dump(mode="json", exclude_none=True)
receipt = document.setdefault("receipts", {}).get(key)
if receipt:
if receipt != body:
raise HTTPException(409, "命令请求号已用于不同内容")
return snapshot(document)
if command.expected_revision != document["revision"]:
raise HTTPException(409, "回放控制状态已更新,请刷新后重试")
action, status = command.action, document["status"]
if (action in {"step", "monitor", "retry"} and document.get("warmed")
and weighted_diagnosis(document)
and current_calibration(document, parse_time(document["start_time"])) is None):
raise HTTPException(409, "旧回放缺少新版历史诊断校准,请停止后新建演示")
if document.get("step_pending") and action in {"step", "monitor"}:
raise HTTPException(409, "当前时间步正在处理,请完成后继续")
if action in {"step", "monitor", "retry"} and await self.repo.active_other(replay_id):
raise HTTPException(409, "当前项目已有另一条活动回放")
if action == "step" and status in {"ready", "paused"}:
interval = document.get("monitoring_interval_seconds")
if interval is None:
raise HTTPException(409, "旧回放未固定监测周期,请停止后新建演示")
start = parse_time(document["start_time"])
elapsed = (parse_time(document["virtual_now"]) - start).total_seconds()
target = start + timedelta(seconds=(int(elapsed // interval) + 1) * interval)
document["step_start"] = document["virtual_now"]
document["step_anomaly_signals"] = []
document["step_target"] = min(parse_time(document["end_time"]), target).isoformat()
document["status"], document["step_pending"] = "paused", True
elif action == "monitor" and status in {"ready", "paused", "completed"}:
target = command.at_time
if target is None:
raise HTTPException(422, "选定时刻监测需要提供 at_time")
start = parse_time(document["start_time"])
current = parse_time(document["virtual_now"])
if not start <= target <= parse_time(document["end_time"]):
raise HTTPException(422, "监测时刻必须在本轮演示区间内")
if target < current:
raise HTTPException(409, "监测历史时刻需要新建一轮回放,避免使用后续状态")
interval = document.get("monitoring_interval_seconds")
if interval is None:
raise HTTPException(409, "本轮未固定监测周期,请停止后新建演示")
previous = document.get("last_monitoring") or {}
# Reusing an immutable result must not count as another normal
# period, resolve an incident, or create another investigation.
if not previous.get("period_end") or parse_time(previous["period_end"]) != target:
document["step_start"] = (current if target > current
else max(start, target - timedelta(seconds=interval))).isoformat()
document["step_target"] = target.isoformat()
document["step_anomaly_signals"] = []
document["status"], document["step_pending"] = "paused", True
elif action == "stop" and status not in {"completed", "stopped"}:
document["status"], document["step_pending"] = "stopped", False
document.pop("step_target", None)
document.pop("step_start", None)
elif action == "retry" and status == "failed":
if "SOURCE_CHANGED" in (document.get("error") or ""):
raise HTTPException(409, "原始数据发生变化,需要停止并新建回放")
document["status"], document["error"] = "paused", None
document["step_pending"] = bool(document.get("step_target"))
else:
raise HTTPException(409, "当前回放状态不支持此操作")
document["revision"] += 1
document["receipts"][key] = body
if len(document["receipts"]) > 256:
del document["receipts"][next(iter(document["receipts"]))]
await self.repo.save(document)
return snapshot(document)
async def process_pending_step(self, replay_id):
await self.ensure_storage()
async with self.repo.processing() as owned:
if not owned:
raise HTTPException(409, "当前时间步正在由另一请求处理")
while True:
document = await self.repo.get(replay_id)
if not document.get("step_pending"):
return snapshot(document)
try:
await self._advance(document)
except Exception as exc:
# Failure never discards the prepared batch; retry resumes it.
document = await self.repo.get(replay_id)
document["status"], document["step_pending"] = "failed", False
if isinstance(exc, HTTPException):
document["error"] = str(exc.detail)
elif isinstance(exc, BaselineBudgetExceeded):
document["error"] = str(exc)
else:
document["error"] = (
"SOURCE_CHANGED: 原始观测已改变,需要新建回放"
if "SOURCE_CHANGED" in str(exc)
else "SCADA 处理失败,已保留批次与进度;请检查服务日志后重试"
)
async with self.repo.connection.transaction():
await self.repo.save(document)
raise
async def _advance(self, document):
config = ScadaReplayCreate.model_validate(document["config"])
replay_id = document["replay_id"]
if not document.get("warmed"):
await self._freeze_baseline(document, config)
if weighted_diagnosis(document) and current_calibration(document, config.start_time) is None:
raise HTTPException(409, "旧回放缺少新版历史诊断校准,请停止后新建演示")
batch = await self.repo.pending_batch(replay_id)
if batch is None:
step = document.get("step_pending", False)
virtual = parse_time(document["virtual_now"])
through = parse_time(document["step_target"]) if step else virtual
after = ((parse_time(document["cursor"][0]), document["cursor"][1])
if document.get("cursor") else None)
async with self.timescale.transaction():
rows = await ScadaRepository.get_raw_page(
self.timescale, config.device_ids, config.start_time, config.end_time,
through, 501, after, frame=True,
)
if len(rows) > 500:
raise ValueError("SCADA frame exceeds selected device bound")
at = rows[0]["time"] if rows else through
if step:
virtual = through
document["virtual_now"] = virtual.isoformat()
if not rows and document.get("processed_until") == at.isoformat() and not step:
async with self.repo.connection.transaction():
await self.repo.save(document)
return
sequence = document["batch_sequence"] + 1
refs = [{"time": row["time"].isoformat(), "device_id": row["device_id"],
"source_hash": row["source_hash"]} for row in rows]
batch = {"sequence": sequence, "stage": "selected", "at": at.isoformat(),
"refs": refs, "input_digest": digest(refs), "events": []}
async with self.repo.connection.transaction():
document["batch_sequence"] = sequence
document["delivered_until"] = at.isoformat()
await self.repo.save_batch(replay_id, batch)
await self.repo.save(document)
else:
rows = None
if batch["stage"] == "selected":
if rows is None:
# Only crash recovery reloads the selected raw frame.
at = parse_time(batch["at"])
async with self.timescale.transaction():
rows = await ScadaRepository.get_raw_page(
self.timescale, config.device_ids, at, config.end_time, at, 501,
) if batch["refs"] else []
actual = [{"time": row["time"].isoformat(), "device_id": row["device_id"],
"source_hash": row["source_hash"]} for row in rows]
if digest(actual) != batch["input_digest"]:
raise ValueError("SOURCE_CHANGED: selected raw frame differs")
batch["outputs"] = prepare_cleaning(rows, config)
batch["stage"] = "clean_prepared"
async with self.repo.connection.transaction():
await self.repo.save_batch(replay_id, batch)
if batch["stage"] == "clean_prepared":
await ScadaRepository.apply_replay_cleaning(self.timescale, batch["outputs"])
batch["stage"] = "cleaned"
async with self.repo.connection.transaction():
await self.repo.save_batch(replay_id, batch)
if batch["stage"] == "cleaned":
result, changes = detect_frame(document, batch["outputs"], config, parse_time(batch["at"]))
result["processed_until"] = batch["at"]
target = result.get("step_target")
was_step = bool(target)
result["step_pending"] = bool(target and parse_time(batch["at"]) < parse_time(target))
if was_step:
signals = {item["event_id"]: item for item in result.get("step_anomaly_signals", [])
if diagnostic_signal(result, item)}
for event in changes:
if event["change"] == "raised" and diagnostic_signal(result, event):
signals[event["event_id"]] = {key: event[key] for key in (
"event_id", "rule_id", "device_ids", "detail", "diagnostic_policy"
) if key in event}
result["step_anomaly_signals"] = list(signals.values())
monitoring = None
if was_step and not result["step_pending"]:
period_start = parse_time(result.get("step_start", result["start_time"]))
period_end = parse_time(target)
role = result.get("monitoring_config") or {}
role_ids = list(dict.fromkeys(
role.get("pressure_device_ids", []) + role.get("outlet_flow_device_ids", [])
))
previous = await self.repo.monitoring_outputs(
replay_id, role_ids, period_end - timedelta(days=15),
period_end, batch["sequence"],
) if role_ids else []
from app.mcp.analysis_service import bounded_calculation
monitoring = await bounded_calculation(
evaluate_monitoring_period,
result,
previous + batch["outputs"],
period_start=period_start,
period_end=period_end,
batch_sequence=batch["sequence"],
)
# Any configured measurement rule can enter diagnosis, including
# indicators outside the pressure/outlet roles used for burst scoring.
signals = {item["event_id"]: item for item in result.get("step_anomaly_signals", [])
if diagnostic_signal(result, item)}
for state in result.get("rule_states", {}).values():
event = state.get("event")
if event and diagnostic_signal(result, event):
signals[event["event_id"]] = {key: event[key] for key in (
"event_id", "rule_id", "device_ids", "detail", "diagnostic_policy"
) if key in event}
monitoring["anomaly_signals"] = list(signals.values())
if signals and monitoring["status"] == "normal":
monitoring["status"] = "attention"
monitoring["detail"] = "本周期曾检出测点异常,进入异常调查与爆管诊断"
result["monitoring_count"] = result.get("monitoring_count", 0) + 1
if not result["step_pending"]:
result.pop("step_target", None)
result.pop("step_start", None)
result.pop("step_anomaly_signals", None)
if batch["refs"]:
last = batch["refs"][-1]
result["cursor"] = [last["time"], last["device_id"]]
if parse_time(batch["at"]) >= config.end_time:
result["status"] = "completed"
batch["stage"], batch["events"] = "detected", changes
if monitoring is not None:
batch["monitoring"] = monitoring
async with self.repo.connection.transaction():
await self._create_investigations(result, changes, batch, create_tasks=not was_step)
if monitoring is not None:
await self._create_monitoring_task(result, monitoring, batch)
batch["monitoring"] = monitoring
result["last_monitoring"] = monitoring
await self.repo.save_batch(replay_id, batch)
await self.repo.save(result)
async def _freeze_baseline(self, document, config):
"""Stage bounded pages, then atomically publish one causal source snapshot."""
if document.get("warmed"):
return
start = config.start_time - timedelta(days=15)
batch_count, row_count, byte_count, after = 0, 0, 0, None
counts = {device_id: 0 for device_id in config.device_ids}
valid_counts = dict(counts)
histories = {device_id: [] for device_id in config.device_ids}
window = max(rule.window_samples for rule in config.rules)
calibration = None
if (document.get("monitoring_config") or {}).get("decision_policy") == "weighted":
from app.services.scada_monitoring_calibration import MonitoringCalibrationBuilder
calibration = MonitoringCalibrationBuilder(document, config)
async with self.repo.connection.transaction():
await self.repo.clear_unpublished_baseline(document["replay_id"])
# Keep only the current page in memory. Each BizDB write commits before
# the next source read; clean_prepared baseline pages remain invisible.
async with self.timescale.transaction():
await ScadaRepository.begin_replay_baseline_snapshot(self.timescale)
while True:
rows = await ScadaRepository.get_raw_page(
self.timescale, config.device_ids, start, config.start_time,
config.start_time, min(BASELINE_PAGE_SIZE, BASELINE_MAX_ROWS - row_count + 1), after,
)
if not rows:
break
if row_count + len(rows) > BASELINE_MAX_ROWS:
raise BaselineBudgetExceeded(
f"BASELINE_BUDGET_EXCEEDED: 十五天历史基线超过 {BASELINE_MAX_ROWS:,} 条准备预算,"
"尚未发布;请减少回放设备或核查历史采样密度后重试"
)
outputs = prepare_cleaning(rows, config)
if calibration is not None:
calibration.add(outputs)
batch = {
"sequence": -batch_count - 1, "stage": "clean_prepared", "kind": "baseline",
"at": outputs[-1]["time"], "outputs": outputs, "events": [],
}
batch_bytes = len(json.dumps(batch, ensure_ascii=False, allow_nan=False,
separators=(",", ":")).encode("utf-8"))
if byte_count + batch_bytes > BASELINE_MAX_BYTES:
raise BaselineBudgetExceeded(
f"BASELINE_BUDGET_EXCEEDED: 十五天历史基线超过 {BASELINE_MAX_BYTES // (1024 * 1024)} MiB 准备预算,"
"尚未发布;请减少回放设备或核查历史采样密度后重试"
)
async with self.repo.connection.transaction():
await self.repo.save_batch(document["replay_id"], batch)
batch_count += 1
row_count += len(outputs)
byte_count += batch_bytes
for row in outputs:
device_id = row["device_id"]
counts[device_id] += 1
if row["quality"] == "valid":
valid_counts[device_id] += 1
histories[device_id].append(row["cleaned_value"])
histories[device_id] = histories[device_id][-window:]
last = rows[-1]
after = (last["time"], last["device_id"])
document["baseline"] = {
"start_time": start.isoformat(), "end_time": config.start_time.isoformat(),
"end_exclusive": True, "sample_counts": counts, "valid_sample_counts": valid_counts,
"row_count": row_count, "batch_count": batch_count, "byte_count": byte_count,
}
document["histories"] = histories
if calibration is not None:
from app.mcp.analysis_service import bounded_calculation
document["monitoring_calibration"] = await bounded_calculation(calibration.fit)
document["warmed"] = True
async with self.repo.connection.transaction():
await self.repo.publish_baseline(document["replay_id"])
await self.repo.save(document)
async def _create_investigations(self, document, changes, batch, *, create_tasks=True):
from app.domain.schemas.operational_task import OperationalTaskCreate, TaskSource, TaskTrigger
from app.infra.db.postgresql.operational_tasks import OperationalTaskRepository
from app.services.operational_tasks import OperationalTaskService
from app.domain.schemas.task_alert import TaskAlertEvent
from app.infra.db.postgresql.task_alerts import TaskAlertRepository
from app.services.task_alerts import TaskAlertService
from uuid import UUID
producer = "scada-replay:" + document["replay_id"]
alerts = TaskAlertService(TaskAlertRepository(self.repo.connection, self.context.project_id),
self.context, internal_producer_id=producer)
tasks = OperationalTaskService(OperationalTaskRepository(self.repo.connection, self.context.project_id),
self.context, permissions_for_context(self.context))
for event in changes:
# Period diagnosis owns measurement notifications during manual steps.
# Retain rule events as evidence; quality alerts remain independent.
if (not create_tasks and event.get("category") != "data_quality"
and (event["change"] == "raised" or event.get("notification_suppressed"))):
if event["change"] == "raised":
event["batch_sequence"] = batch["sequence"]
event["notification_suppressed"] = True
document["rule_states"][event.get("state_key", event["rule_id"])]["event"].update(
batch_sequence=batch["sequence"], notification_suppressed=True,
)
continue
if event["change"] == "raised":
event["batch_sequence"] = batch["sequence"]
document["rule_states"][event.get("state_key", event["rule_id"])]["event"]["batch_sequence"] = batch["sequence"]
await alerts.push(TaskAlertEvent(
event_id=uuid5(NAMESPACE_URL, event["event_id"] + ":" + event["change"]),
alert_id=event["event_id"], producer_id=producer, event_type=event["change"],
occurred_at=event["occurred_at"], condition_id=None,
source={"kind": "scada_replay", "replay_id": document["replay_id"],
"rule_id": event["rule_id"], "batch_sequence": event["batch_sequence"]},
title=f"SCADA 回放异常 · {event['rule_id']}", description=event["detail"], severity="warning",
))
if event["change"] != "raised":
if event.get("task_id"):
await tasks.advance_replay_investigation(UUID(event["task_id"]), {
"replay_id": document["replay_id"], "knowledge_cutoff": batch["at"],
"batch_sequence": batch["sequence"],
}, uuid5(NAMESPACE_URL, event["event_id"] + ":recovery"))
continue
if event.get("category") == "data_quality":
continue
if not create_tasks:
continue
external = f"scada_replay:{document['replay_id']}:event:{event['event_id']}"
command = OperationalTaskCreate(
request_id=uuid5(NAMESPACE_URL, external), task_type="event_investigation",
workflow="scada_anomaly", title=f"SCADA 回放调查 · {event['rule_id']}",
summary=event["detail"], source=TaskSource(
kind="external_system", external_id=external, external_revision=1,
event_kind="scada_anomaly",
replay_context={"replay_id": document["replay_id"],
"knowledge_cutoff": batch["at"], "batch_sequence": batch["sequence"]},
),
trigger=TaskTrigger(kind="event", occurred_at=event["occurred_at"], detail="已存 SCADA 历史的回放检测信号,原因尚待调查"),
)
task = await tasks.create(command)
event["task_id"] = str(task.task_id)
document["rule_states"][event.get("state_key", event["rule_id"])]["event"]["task_id"] = event["task_id"]
async def _create_monitoring_task(self, document, monitoring, batch):
from app.services.scada_monitoring_incidents import attach_incident, update_incident
incident, action = update_incident(document, monitoring, batch["sequence"])
if incident is None:
return
if incident.get("task_id"):
if monitoring["status"] == "suspected_burst" and not incident.get("burst_diagnosed"):
from uuid import UUID
from app.infra.db.postgresql.operational_tasks import OperationalTaskRepository
from app.services.operational_tasks import OperationalTaskService
tasks = OperationalTaskService(
OperationalTaskRepository(self.repo.connection, self.context.project_id),
self.context, permissions_for_context(self.context),
)
await tasks.advance_replay_investigation(UUID(incident["task_id"]), {
"replay_id": document["replay_id"], "knowledge_cutoff": monitoring["knowledge_cutoff"],
"batch_sequence": batch["sequence"],
}, uuid5(NAMESPACE_URL, incident["incident_id"] + ":burst-evidence"))
incident["burst_diagnosed"] = True
if action == "resolved" and incident.get("alert"):
await self._publish_monitoring_alert(document, incident, "resolved")
attach_incident(monitoring, incident, action)
return
from app.domain.schemas.operational_task import OperationalTaskCreate, TaskSource, TaskTrigger
from app.infra.db.postgresql.operational_tasks import OperationalTaskRepository
from app.services.operational_tasks import OperationalTaskService
tasks = OperationalTaskService(
OperationalTaskRepository(self.repo.connection, self.context.project_id),
self.context,
permissions_for_context(self.context),
)
external = incident["external_id"]
pressure = monitoring["pressure"]
flow = monitoring["outlet_flow"]
detector = monitoring["burst_detection"]
score = detector.get("target_score")
score_text = f"{score:.4f}" if score is not None else "数据不足"
summary = (
f"SCADA 数据诊断发现指标异常,进入异常调查与爆管诊断。"
f"爆管侦测得分 {score_text};"
f"压力下降测点 {len(pressure.get('hit_device_ids', []))} 个;"
f"出厂边界流量增加测点 {len(flow.get('hit_device_ids', []))} 个。"
"任一指标异常即启动调查,加权分用于评估爆管嫌疑;达到门槛后继续候选定位、隔离与影响分析。"
)
if monitoring.get("anomaly_signals"):
signals = monitoring["anomaly_signals"]
devices = {device for item in signals for device in item.get("device_ids", [])}
summary += f" 单项异常 {len(signals)} 项,涉及 {len(devices)} 个测点;明细通过冻结 SCADA 查询核验。"
if monitoring.get("weighted"):
weighted = monitoring["weighted"]
weights = weighted["weights"]
summary += (
f" 综合评分 {weighted['score']:.2f},爆管嫌疑门槛 {weighted['threshold']:.2f}"
f"(侦测 {weights['burst_detection']:.0%}、压力 {weights['pressure']:.0%}、"
f"流量 {weights['outlet_flow']:.0%})。"
)
if detector.get("score_threshold") is not None:
summary += f" 侦测动态阈值 {detector['score_threshold']:.4f}。"
command = OperationalTaskCreate(
request_id=uuid5(NAMESPACE_URL, external),
task_type="event_investigation",
workflow="scada_anomaly",
title=f"SCADA 异常调查 · {monitoring['period_end']}",
summary=summary,
source=TaskSource(
kind="external_system",
external_id=external,
external_revision=1,
event_kind="scada_anomaly",
replay_context={
"replay_id": document["replay_id"],
"knowledge_cutoff": monitoring["knowledge_cutoff"],
"batch_sequence": batch["sequence"],
},
),
trigger=TaskTrigger(
kind="event",
occurred_at=monitoring["period_end"],
detail="SCADA 任一指标异常启动调查,核验数据后进行爆管诊断;异常不等于已确诊爆管",
),
)
task = await tasks.create(command)
incident["task_id"] = str(task.task_id)
incident["burst_diagnosed"] = monitoring["status"] == "suspected_burst"
incident["alert"] = {
"alert_id": incident["incident_id"],
"producer_id": "scada-replay:" + document["replay_id"],
"occurred_at": incident["started_at"],
"condition_id": f"operational-task:{task.task_id}",
"source": {"kind": "scada_replay", "replay_id": document["replay_id"],
"rule_id": "scada_diagnosis", "batch_sequence": batch["sequence"]},
"title": "SCADA 指标异常", "description": "已启动调查,详细诊断见调查报告。", "severity": "warning",
}
await self._publish_monitoring_alert(document, incident, "raised")
attach_incident(monitoring, incident, action)
async def _publish_monitoring_alert(self, document, incident, change):
from app.domain.schemas.task_alert import TaskAlertEvent
from app.infra.db.postgresql.task_alerts import TaskAlertRepository
from app.services.task_alerts import TaskAlertService
alerts = TaskAlertService(
TaskAlertRepository(self.repo.connection, self.context.project_id), self.context,
internal_producer_id="scada-replay:" + document["replay_id"],
)
await alerts.push(TaskAlertEvent(
**incident["alert"], event_type=change,
event_id=uuid5(NAMESPACE_URL, incident["incident_id"] + ":" + change),
))
+287
View File
@@ -0,0 +1,287 @@
"""Deterministic monitoring summaries built only from frozen replay outputs."""
from __future__ import annotations
from datetime import datetime, timedelta
from statistics import median
from typing import Any
from uuid import NAMESPACE_URL, uuid5
import pandas as pd
from app.domain.schemas.scada_replay import ScadaReplayCreate
from app.domain.schemas.project_settings import ScadaMonitoringSettings
from app.services.scada_normal_profile import current_calibration, normal_profile_bounds
from app.services.burst_detection import (
TARGET_MU,
TARGET_N_ESTIMATORS,
TARGET_RANDOM_STATE,
TARGET_SCORE_THRESHOLD,
run_burst_detection,
)
def _time(value: str | datetime) -> datetime:
return value if isinstance(value, datetime) else datetime.fromisoformat(value)
def _valid_rows(rows: list[dict], device_id: str) -> list[dict]:
return sorted(
(
row for row in rows
if row.get("device_id") == device_id
and row.get("quality") in {"valid", "corrected"}
and row.get("cleaned_value") is not None
),
key=lambda row: _time(row["time"]),
)
def _relative_indicator(
rows: list[dict],
config: ScadaReplayCreate,
device_ids: list[str],
period_start: datetime,
period_end: datetime,
*,
direction: str,
calibration: dict | None = None,
) -> dict[str, Any]:
details: list[dict[str, Any]] = []
gaps: list[dict[str, str]] = []
for device_id in device_ids:
rule = next((
item for item in config.rules
if item.kind == "relative_change" and device_id in item.device_ids
and item.threshold is not None
), None)
if rule is None:
gaps.append({"device_id": device_id, "reason": "relative_change_rule_missing"})
continue
profile = calibration.get(device_id) if calibration is not None else None
if calibration is not None and profile is None:
gaps.append({"device_id": device_id, "reason": "adaptive_history_not_ready"})
continue
threshold = float(profile["threshold_percent"]) if profile else rule.threshold
valid = _valid_rows(rows, device_id)
current = [row for row in valid if period_start < _time(row["time"]) <= period_end]
latest = current[-1] if current else None
if not current:
gaps.append({"device_id": device_id, "reason": "period_reading_missing"})
continue
baseline = ([row for row in valid if _time(row["time"]) < _time(latest["time"])]
[-rule.window_samples:]) if latest else []
if len(baseline) < rule.minimum_samples:
gaps.append({"device_id": device_id, "reason": "baseline_not_ready"})
continue
reference = median(float(row["cleaned_value"]) for row in baseline)
if reference == 0:
gaps.append({"device_id": device_id, "reason": "zero_baseline"})
continue
value = float(latest["cleaned_value"])
change = (value - reference) / abs(reference) * 100
hit = change <= -threshold if direction == "decrease" else change >= threshold
bounds = normal_profile_bounds(profile.get("normal_profile"), _time(latest["time"])) if profile else None
if profile and bounds is None:
gaps.append({"device_id": device_id, "reason": "normal_phase_history_not_ready"})
continue
if bounds is not None:
hit = hit and not bounds[0] <= value <= bounds[1]
details.append({
"device_id": device_id,
"observed_at": _time(latest["time"]).isoformat(),
"baseline": reference,
"baseline_method": "recent_samples_median",
"baseline_start": _time(baseline[0]["time"]).isoformat(),
"baseline_end": _time(baseline[-1]["time"]).isoformat(),
"baseline_sample_count": len(baseline),
"value": value,
"value_change": value - reference,
"unit": rule.unit,
"change_percent": change,
"threshold_percent": threshold,
**({"normal_phase_lower": bounds[0], "normal_phase_upper": bounds[1]} if bounds else {}),
**({"threshold_method": profile["method"],
"configured_threshold_percent": rule.threshold} if profile else {}),
"hit": hit,
})
return {
"status": "ready" if not gaps else "insufficient_data",
"hit": any(item["hit"] for item in details),
"hit_device_ids": [item["device_id"] for item in details if item["hit"]],
"devices": details,
"gaps": gaps,
}
def _burst_detection(
rows: list[dict],
pressure_device_ids: list[str],
interval_seconds: int,
period_end: datetime,
*,
score_threshold: float = TARGET_SCORE_THRESHOLD,
) -> dict[str, Any]:
if len(pressure_device_ids) < 5:
return {"status": "sensor_coverage_insufficient", "hit": False,
"required_sensors": 5, "available_sensors": len(pressure_device_ids)}
if interval_seconds % 60 or 86400 % interval_seconds:
return {"status": "unsupported_sampling_interval", "hit": False,
"interval_seconds": interval_seconds}
points_per_day = 86400 // interval_seconds
count = 15 * points_per_day
series_by_device: dict[str, dict[datetime, float]] = {}
common_times: set[datetime] | None = None
for device_id in pressure_device_ids:
values = {_time(row["time"]): float(row["cleaned_value"])
for row in _valid_rows(rows, device_id) if _time(row["time"]) <= period_end}
series_by_device[device_id] = values
common_times = set(values) if common_times is None else common_times & set(values)
if not common_times:
return {"status": "baseline_not_ready", "hit": False, "required_samples": count}
target = period_end
if target not in common_times:
return {"status": "current_reading_missing", "hit": False,
"target_time": target.isoformat(), "latest_common_time": max(common_times).isoformat()}
expected = [target - timedelta(seconds=interval_seconds * offset)
for offset in range(count - 1, -1, -1)]
incomplete = [device_id for device_id, values in series_by_device.items()
if any(at not in values for at in expected)]
if incomplete:
return {"status": "baseline_not_ready", "hit": False, "required_samples": count,
"incomplete_device_ids": incomplete}
frame = pd.DataFrame({device_id: [series_by_device[device_id][at] for at in expected]
for device_id in pressure_device_ids}, index=expected)
try:
result = run_burst_detection(
network="scada-replay",
username="scada-replay",
observed_pressure_data=frame,
points_per_day=points_per_day,
mu=TARGET_MU,
iforest_params={
"n_estimators": TARGET_N_ESTIMATORS,
"random_state": TARGET_RANDOM_STATE,
"contamination": "auto",
},
sensor_nodes=pressure_device_ids,
)
except (TypeError, ValueError) as exc:
return {"status": "analysis_failed", "hit": False, "reason": str(exc)[:300]}
latest = result["rows"][-1]
score = float(latest["Score"])
return {
"status": "ready",
"hit": score <= score_threshold,
"target_time": target.isoformat(),
"target_score": score,
"score_interpretation": "lower_is_more_anomalous_not_probability",
"score_threshold": score_threshold,
"sensor_count": len(pressure_device_ids),
"reference_day_count": 14,
"latest_sensor_rankings": result["summary"].get("latest_sensor_rankings", []),
}
def evaluate_monitoring_period(
document: dict,
rows: list[dict],
*,
period_start: datetime,
period_end: datetime,
batch_sequence: int,
) -> dict[str, Any]:
config = ScadaReplayCreate.model_validate(document["config"])
monitoring = document.get("monitoring_config")
rule_version = (monitoring or {}).get("rule_version", "unconfigured")
monitoring_id = str(uuid5(
NAMESPACE_URL,
f"{document['replay_id']}:{period_end.isoformat()}:{rule_version}",
))
base = {
"monitoring_id": monitoring_id,
"period_start": period_start.isoformat(),
"period_end": period_end.isoformat(),
"knowledge_cutoff": period_end.isoformat(),
"batch_sequence": batch_sequence,
"rule_version": rule_version,
"task_id": None,
}
if not monitoring:
return {**base, "status": "insufficient_data", "detail": "项目未配置 SCADA 监测设备角色",
"pressure": {"status": "unconfigured", "hit": False},
"outlet_flow": {"status": "unconfigured", "hit": False},
"burst_detection": {"status": "not_run", "hit": False}}
pressure_ids = monitoring.get("pressure_device_ids", [])
flow_ids = monitoring.get("outlet_flow_device_ids", [])
selected = set(config.device_ids)
role_gaps = sorted((set(pressure_ids) | set(flow_ids)) - selected)
weighted_policy = monitoring.get("decision_policy") == "weighted"
if not weighted_policy and (not pressure_ids or not flow_ids or role_gaps):
return {**base, "status": "insufficient_data", "detail": "监测角色为空或不属于本轮回放设备",
"role_gaps": role_gaps,
"pressure": {"status": "unconfigured", "hit": False},
"outlet_flow": {"status": "unconfigured", "hit": False},
"burst_detection": {"status": "not_run", "hit": False}}
calibration = current_calibration(document, config.start_time) if weighted_policy else None
if config.start_time > period_end:
calibration = None
base["role_gaps"] = role_gaps
pressure = _relative_indicator(
rows, config, pressure_ids, period_start, period_end, direction="decrease",
calibration=(calibration or {}).get("pressure", {}) if weighted_policy else None,
)
outlet_flow = _relative_indicator(
rows, config, flow_ids, period_start, period_end, direction="increase",
calibration=(calibration or {}).get("outlet_flow", {}) if weighted_policy else None,
) if flow_ids else {"status": "unconfigured", "hit": False, "hit_device_ids": [], "devices": [], "gaps": []}
detector = {"status": "baseline_not_ready", "hit": False} if weighted_policy and (
not calibration or calibration.get("status") != "ready"
) else _burst_detection(
rows, pressure_ids, int(document["monitoring_interval_seconds"]), period_end,
score_threshold=calibration["burst_detection"]["threshold"] if calibration else TARGET_SCORE_THRESHOLD,
)
if weighted_policy:
settings = ScadaMonitoringSettings.model_validate(monitoring)
weights = settings.weights.model_dump()
indicators = {"burst_detection": detector, "pressure": pressure, "outlet_flow": outlet_flow}
contributions = {key: weights[key] if indicator["status"] == "ready" and indicator["hit"] else 0.0
for key, indicator in indicators.items()}
score = round(sum(contributions.values()), 10)
complete = pressure["status"] == detector["status"] == "ready"
diagnostic_missing = sorted({device for rule in config.rules if rule.kind == "relative_change"
for device in rule.device_ids}
- set((calibration or {}).get("diagnostic", {})))
if not complete:
status, detail = "insufficient_data", (
"旧回放缺少新版历史诊断校准,请停止后新建演示" if calibration is None
else "压力数据或爆管侦测基线不足,尚不能作出组合判定"
)
elif score >= settings.alert_threshold:
status, detail = "suspected_burst", "爆管相关指标加权达到疑似爆管门槛"
elif any(indicator["hit"] for indicator in indicators.values()):
status, detail = "attention", "部分指标异常,进入调查;综合结果尚未达到爆管嫌疑门槛"
elif diagnostic_missing:
status, detail = "insufficient_data", "部分测点缺少完整历史同期基线,不能判定全部指标正常"
else:
status, detail = "normal", "本周期爆管相关指标均未命中"
detail += f"(综合 {score:.2f} / 门槛 {settings.alert_threshold:.2f},流量权重 {weights['outlet_flow']:.0%})"
return {**base, "status": status, "detail": detail, "pressure": pressure,
"outlet_flow": outlet_flow, "burst_detection": detector,
"weighted": {"score": score, "threshold": settings.alert_threshold,
"weights": weights, "contributions": contributions,
"unavailable_indicators": [key for key, value in indicators.items() if value["status"] != "ready"],
"calibration_cutoff": calibration["knowledge_cutoff"] if calibration else None,
"diagnostic_unavailable_device_ids": diagnostic_missing,
"missing_weight_policy": "zero_contribution_without_renormalization"}}
complete = pressure["status"] == outlet_flow["status"] == detector["status"] == "ready"
hits = (bool(detector["hit"]), bool(pressure["hit"]), bool(outlet_flow["hit"]))
if not complete:
status, detail = "insufficient_data", "监测数据、设备角色或爆管侦测基线不足"
elif all(hits):
status, detail = "suspected_burst", "爆管侦测、压力下降与出厂边界流量增加相互支持"
elif any(hits):
status, detail = "attention", "仅部分爆管相关指标命中,需要继续观察或人工核查"
else:
status, detail = "normal", "本周期爆管相关指标均未命中"
return {**base, "status": status, "detail": detail, "pressure": pressure,
"outlet_flow": outlet_flow, "burst_detection": detector}
+160
View File
@@ -0,0 +1,160 @@
"""Causal batch quality checks and grouped anomaly episodes, no database IO."""
from copy import deepcopy
from datetime import datetime
from math import isfinite
from statistics import median
from uuid import NAMESPACE_URL, uuid5
import re
from app.domain.schemas.scada_replay import ScadaReplayCreate
from app.services.scada_normal_profile import (
DIAGNOSTIC_POLICY, current_calibration, normal_profile_bounds, weighted_diagnosis,
)
def prepare_cleaning(rows: list[dict], config: ScadaReplayCreate) -> list[dict]:
result = []
for row in rows:
value = row["monitored_value"]
quality, reason = "valid", "原始值通过数值与已配置量程校验"
if value is None:
quality, reason = "missing", "原始读数缺失"
elif not isfinite(float(value)):
quality, reason = "invalid", "原始读数不是有限数值"
else:
bounds = config.quality_ranges.get(row["device_id"])
if bounds and ((bounds.lower is not None and value < bounds.lower)
or (bounds.upper is not None and value > bounds.upper)):
quality, reason = "invalid", "超出本次明确配置的仪表量程"
result.append({
"time": row["time"].isoformat() if isinstance(row["time"], datetime) else row["time"],
"device_id": row["device_id"], "source_hash": row["source_hash"],
"monitored_value": value if value is not None and isfinite(float(value)) else None,
"cleaned_value": value if quality == "valid" else None,
"quality": quality, "reason": reason,
})
return result
def detect_frame(document: dict, rows: list[dict], config: ScadaReplayCreate, at: datetime) -> tuple[dict, list[dict]]:
"""Consume a complete timestamp; never treat missing data as recovery."""
result = deepcopy(document)
histories = result.setdefault("histories", {})
states = result.setdefault("rule_states", {})
latest = result.setdefault("latest", {})
for row in rows:
latest[row["device_id"]] = row
by_id = {row["device_id"]: row for row in rows}
changes = []
devices = {device["device_id"]: device for device in result.get("devices", [])}
weighted = weighted_diagnosis(result)
calibration = current_calibration(result, config.start_time) if weighted else None
for device_id in config.device_ids:
row = by_id.get(device_id)
invalid = row is not None and row["quality"] != "valid"
previous = latest.get(device_id)
frequency = str(devices.get(device_id, {}).get("transmission_frequency", ""))
match = re.fullmatch(r"(\d+):(\d+):(\d+)", frequency)
if match:
seconds = int(match[1]) * 3600 + int(match[2]) * 60 + int(match[3])
else:
minutes = re.fullmatch(r"(\d+)\s*(?:min|分钟)", frequency)
seconds = int(minutes[1]) * 60 if minutes else 0
prior_time = datetime.fromisoformat(previous["time"]) if previous else config.start_time
missing = not row and seconds > 0 and (at - prior_time).total_seconds() >= seconds * 2
key = "quality-" + device_id
state = states.setdefault(key, {"event": None, "recoveries": 0})
if invalid or missing:
state["recoveries"] = 0
if state["event"] is None:
event = {"event_id": str(uuid5(NAMESPACE_URL, f"{result['replay_id']}:{key}:{at.isoformat()}")),
"rule_id": key, "device_ids": [device_id], "occurred_at": at.isoformat(),
"last_observed_at": at.isoformat(), "status": "active", "category": "data_quality",
"detail": row["reason"] if invalid else "连续两个设备周期未读取到观测",
"task_id": None}
state["event"] = event
changes.append({**event, "change": "raised"})
result["anomaly_count"] = result.get("anomaly_count", 0) + 1
elif row and row["quality"] == "valid" and state["event"]:
state["recoveries"] += 1
if state["recoveries"] >= 2:
changes.append({**state["event"], "status": "resolved", "resolved_at": at.isoformat(), "change": "resolved"})
state["event"] = None
for rule in config.rules:
state_key = f"{rule.rule_id}:{DIAGNOSTIC_POLICY}" if weighted and rule.kind == "relative_change" else rule.rule_id
state = states.setdefault(state_key, {"hits": 0, "recoveries": 0, "event": None})
hits, usable, reasons = [], [], []
for device_id in rule.device_ids:
row = by_id.get(device_id)
if not row or row["quality"] != "valid":
continue
value = row["cleaned_value"]
if rule.kind == "bounds":
triggered = ((rule.lower is not None and value < rule.lower)
or (rule.upper is not None and value > rule.upper))
reason = f"{device_id}={value} {rule.unit},超出配置范围"
elif weighted:
# The percentage in the compatibility rule is not a diagnosis
# threshold. Learn each device's normal same-phase range first.
profile = (calibration or {}).get("diagnostic", {}).get(device_id)
bounds = normal_profile_bounds(profile, at)
if bounds is None:
continue
low, high = bounds
triggered = (value < low or value > high if rule.direction == "both"
else value > high if rule.direction == "increase" else value < low)
reason = f"{device_id}={value} {rule.unit},超出冻结正常同时段范围 [{low:.4f}, {high:.4f}] {rule.unit}"
else:
previous = histories.get(device_id, [])[-rule.window_samples:]
if len(previous) < rule.minimum_samples:
continue
baseline = median(previous)
if baseline == 0:
continue
change = (value - baseline) / abs(baseline) * 100
triggered = (abs(change) >= rule.threshold if rule.direction == "both"
else change >= rule.threshold if rule.direction == "increase"
else change <= -rule.threshold)
reason = f"{device_id} 较历史中位值变化 {change:.1f}%,阈值 {rule.threshold}%"
usable.append(device_id)
if triggered:
hits.append(device_id)
reasons.append(reason)
if len(hits) >= rule.minimum_devices:
state["hits"] += 1
state["recoveries"] = 0
if state["event"] is None and state["hits"] >= rule.consecutive_samples:
event_id = str(uuid5(NAMESPACE_URL, f"{result['replay_id']}:{state_key}:{at.isoformat()}"))
event = {"event_id": event_id, "rule_id": rule.rule_id, "device_ids": hits,
"occurred_at": at.isoformat(), "last_observed_at": at.isoformat(),
"status": "active", "detail": ";".join(reasons)[:2000], "task_id": None}
if weighted and rule.kind == "relative_change":
event.update(diagnostic_policy=DIAGNOSTIC_POLICY, state_key=state_key)
state["event"] = event
changes.append({**event, "change": "raised"})
result["anomaly_count"] = result.get("anomaly_count", 0) + 1
elif state["event"]:
state["event"]["last_observed_at"] = at.isoformat()
elif len(usable) == len(rule.device_ids):
state["hits"] = 0
if state["event"]:
state["recoveries"] += 1
if state["recoveries"] >= rule.recovery_samples:
event = {**state["event"], "status": "resolved", "resolved_at": at.isoformat(),
"change": "resolved"}
changes.append(event)
state["event"] = None
state["recoveries"] = 0
# Incomplete frames cannot count as a consecutive hit/recovery.
elif not usable and any(device in by_id for device in rule.device_ids):
state["hits"] = state["recoveries"] = 0
frozen = {device for rule in config.rules if states[
f"{rule.rule_id}:{DIAGNOSTIC_POLICY}" if weighted and rule.kind == "relative_change" else rule.rule_id
]["event"]
for device in rule.device_ids}
for row in rows:
if row["quality"] == "valid" and row["device_id"] not in frozen:
history = histories.setdefault(row["device_id"], [])
history.append(row["cleaned_value"])
histories[row["device_id"]] = history[-96:]
result["processed_count"] = result.get("processed_count", 0) + len(rows)
return result, changes
+102
View File
@@ -0,0 +1,102 @@
"""Server-enforced replay knowledge bounds and immutable corrected snapshots."""
from datetime import datetime
from fastapi import HTTPException
from app.infra.db.postgresql.scada_replays import ScadaReplayRepository
def check_replay_window(context, start, end):
cap = datetime.fromisoformat(context["knowledge_cutoff"])
if start is None or end is None or start.tzinfo is None or end.tzinfo is None or start > end or end > cap:
raise HTTPException(422, f"回放查询需要明确时间范围,结束不得晚于 {cap.isoformat()}")
def check_snapshot_start(run, start):
replay_start = datetime.fromisoformat(run["start_time"])
if start < replay_start:
baseline = run.get("baseline")
if not baseline:
raise HTTPException(422, "本轮回放未冻结历史对照,请新建回放后查询历史基线")
if start < datetime.fromisoformat(baseline["start_time"]):
raise HTTPException(422, "查询起点早于本轮已冻结的历史基线")
async def replay_snapshot_rows(connection, project_id, context, device_ids, start, end, limit, offset=0):
check_replay_window(context, start, end)
repository = ScadaReplayRepository(connection, project_id)
run = await repository.get(context["replay_id"])
if not set(device_ids) <= set(run["config"]["device_ids"]):
raise HTTPException(403, "设备或回放不属于当前调查")
check_snapshot_start(run, start)
rows = await repository.detected_outputs(
context["replay_id"], device_ids, start, end, context["batch_sequence"],
limit=limit, offset=offset,
)
return [{key: row[key] for key in (
"time", "device_id", "monitored_value", "cleaned_value", "quality", "reason"
)} for row in rows]
async def replay_burst_location_inputs(connection, project_id, context, args):
"""Build location series from immutable replay outputs, never live TSDB rows."""
if args.data_source != "monitoring":
raise HTTPException(422, "回放定位只支持 monitoring 数据源")
if args.scada_normal_start is None or args.scada_normal_end is None:
raise HTTPException(422, "回放定位必须显式提供已处理范围内的正常对照窗口")
check_replay_window(context, args.scada_burst_start, args.scada_burst_end)
check_replay_window(context, args.scada_normal_start, args.scada_normal_end)
repository = ScadaReplayRepository(connection, project_id)
run = await repository.get(context["replay_id"])
if run:
check_snapshot_start(run, min(args.scada_burst_start, args.scada_normal_start))
devices = {item["device_id"]: item for item in run.get("devices", [])}
monitoring = run.get("monitoring_config") or {}
pressure_devices = monitoring.get("pressure_device_ids", [])
pressure_map = {str(devices[item]["node_id"]): item for item in pressure_devices
if item in devices and devices[item].get("node_id")}
requested_pressure = args.pressure_scada_ids or list(pressure_map)
if not requested_pressure or not set(requested_pressure) <= set(pressure_map):
raise HTTPException(422, "压力节点必须来自本轮已配置的回放监测角色")
flow_devices = monitoring.get("outlet_flow_device_ids", [])
flow_map = {str(devices[item]["link_id"]): item for item in flow_devices
if item in devices and devices[item].get("link_id")}
requested_flow = args.flow_scada_ids or []
if not set(requested_flow) <= set(flow_map):
raise HTTPException(422, "流量要素必须来自本轮已配置的出厂边界流量角色")
selected = [pressure_map[item] for item in requested_pressure] + [flow_map[item] for item in requested_flow]
rows = await repository.detected_outputs(
context["replay_id"], selected,
min(args.scada_burst_start, args.scada_normal_start),
max(args.scada_burst_end, args.scada_normal_end),
context["batch_sequence"],
)
def averages(mapping, start, end):
result = {}
for element_id, device_id in mapping.items():
values = [float(row["cleaned_value"]) for row in rows
if row.get("device_id") == device_id
and row.get("quality") in {"valid", "corrected"}
and row.get("cleaned_value") is not None
and start <= datetime.fromisoformat(row["time"]) <= end]
if not values:
raise HTTPException(422, f"回放窗口缺少 {device_id} 的有效清洗观测")
result[element_id] = sum(values) / len(values)
return result
pressure_mapping = {item: pressure_map[item] for item in requested_pressure}
flow_mapping = {item: flow_map[item] for item in requested_flow}
return {
"data_source": "monitoring",
"burst_leakage": args.burst_leakage,
"pressure_scada_ids": requested_pressure,
"burst_pressure": averages(pressure_mapping, args.scada_burst_start, args.scada_burst_end),
"normal_pressure": averages(pressure_mapping, args.scada_normal_start, args.scada_normal_end),
**({
"flow_scada_ids": requested_flow,
"burst_flow": averages(flow_mapping, args.scada_burst_start, args.scada_burst_end),
"normal_flow": averages(flow_mapping, args.scada_normal_start, args.scada_normal_end),
"use_scada_flow": True,
} if requested_flow else {}),
}
+10 -3
View File
@@ -77,8 +77,9 @@ def burst_analysis(
modify_valve_opening: dict[str, float] = None,
scheme_name: str = None,
username: str | None = None,
persist_results: bool = True,
_temporary_project: str | None = None,
) -> None:
) -> dict | None:
"""
爆管模拟
:param name: 模型名称,数据库中对应的名字
@@ -92,7 +93,7 @@ def burst_analysis(
:param scheme_name: 方案名称
:return:
"""
if not username:
if persist_results and not username:
raise ValueError("username is required when storing burst analysis scheme")
scheme_detail: dict = {
@@ -193,7 +194,7 @@ def burst_analysis(
# modify_pump_pattern=modify_pump_pattern,
# valve_control=valve_control,
# downloading_prohibition=True)
simulation.run_simulation(
result = simulation.run_simulation(
name=new_name,
simulation_type="extended",
modify_pattern_start_time=modify_pattern_start_time,
@@ -206,7 +207,9 @@ def burst_analysis(
result_db_name=name,
scheme_username=username,
scheme_detail=scheme_detail,
persist_results=persist_results,
)
return result if not persist_results else None
# step 3. restore the base model status
# execute_undo(name) #有疑惑
@@ -223,6 +226,7 @@ def valve_close_analysis(
scheme_name: str = None,
_temporary_project: str | None = None,
persist_results: bool = True,
username: str = "system",
) -> dict | str | None:
"""
关阀模拟
@@ -282,6 +286,7 @@ def valve_close_analysis(
scheme_name=scheme_name,
result_db_name=name,
persist_results=persist_results,
scheme_username=username,
)
return result if not persist_results else None
# step 3. restore the base model
@@ -610,6 +615,7 @@ def pressure_regulation(
scada_mappings: simulation.ScadaElementMappings | None = None,
_temporary_project: str | None = None,
persist_results: bool = True,
username: str = "system",
) -> dict | str | None:
"""
区域调压模拟,用来模拟未来15分钟内,开关水泵对区域压力的影响
@@ -668,6 +674,7 @@ def pressure_regulation(
result_db_name=name,
scada_mappings=scada_mappings,
persist_results=persist_results,
scheme_username=username,
)
return result if not persist_results else None
# return result
+120
View File
@@ -0,0 +1,120 @@
"""Question lifecycle sharing the operational task transaction and notification bus."""
from datetime import datetime
from typing import TYPE_CHECKING
from uuid import UUID, uuid4
from app.auth.permissions import OPERATIONAL_TASK_EXECUTE, OPERATIONAL_TASK_WRITE
from app.domain.domain_errors import require
from app.domain.schemas.operational_task import OperationalTaskDetail, OperationalTaskSummary
from app.domain.schemas.task_question import TaskQuestionCreate, TaskQuestionReply, TaskQuestionRequest
if TYPE_CHECKING:
from app.services.operational_tasks import OperationalTaskService
async def list_questions(service: "OperationalTaskService", task_id: UUID) -> list[dict]:
# Legacy test repositories do not implement question persistence.
reader = getattr(service.repo, "list_questions", None)
return await reader(str(task_id)) if reader else []
async def cancel_pending(service: "OperationalTaskService", task_id: UUID, now: datetime) -> None:
for raw in await list_questions(service, task_id):
question = TaskQuestionRequest.model_validate(raw)
if question.status != "pending":
continue
question = question.model_copy(update={"status": "cancelled", "version": question.version + 1})
await service.repo.save_question(str(task_id), question.model_dump(mode="json"))
await service._resolve_notifications(task_id, kinds=["human_input_required"],
subject_type="question", subject_id=question.question_request_id,
now=now, reason="question_superseded")
async def create_question(service: "OperationalTaskService", task_id: UUID,
command: TaskQuestionCreate) -> TaskQuestionRequest:
require(OPERATIONAL_TASK_EXECUTE in service.permissions, "PERMISSION_DENIED",
"Current user cannot publish task questions", status=403)
body = command.model_dump(mode="json")
scope = f"task:{task_id}:question:create"
async with service.repo.transaction():
await service.repo.lock(f"task:{task_id}")
replay = await service._request_replay(scope, command.request_id, body)
if replay:
return TaskQuestionRequest.model_validate(await service.repo.get_question(
str(task_id), replay["question_request_id"]))
task = await service._task(task_id)
service._require_actionable_task(task)
require(task["task_type"] == "event_investigation", "INVALID_TASK_TYPE",
"Questions require an investigation task", status=409)
await service.repo.lock(f"run:{command.run_id}")
run = await service._run(task_id, command.run_id)
require(run["status"] in {"claimed", "running"}
and run["generation"] == task["execution_generation"]
and run.get("agent_session_id") == command.source_session_id,
"SESSION_MISMATCH", "Question must belong to the active investigation run", status=409)
now = service._now()
service._require_active_run_lease(run, now)
await service._require_no_pending_calculations(task_id, run)
require(not any(q["status"] == "pending" for q in await list_questions(service, task_id)),
"QUESTION_PENDING", "Answer the existing questions first", status=409)
question = TaskQuestionRequest(question_request_id=uuid4(), task_id=task_id,
run_id=command.run_id, execution_generation=run["generation"],
source_session_id=command.source_session_id, questions=command.questions,
status="pending", version=1, created_at=now)
await service.repo.save_question(str(task_id), question.model_dump(mode="json"))
task.update(revision=task["revision"] + 1, updated_at=now)
await service.repo.update_task(OperationalTaskSummary.model_validate(task).model_dump(mode="json", exclude={"active_run", "latest_run"}))
await service._task_event(task, now)
await service._notify(task_id, "human_input_required", task["title"],
";".join(q.title for q in question.questions), now,
required_permission=OPERATIONAL_TASK_WRITE, subject_type="question",
subject_id=question.question_request_id)
await service._save_request(scope, command.request_id, body,
{"question_request_id": str(question.question_request_id)})
return question
async def answer_question(service: "OperationalTaskService", task_id: UUID, question_id: UUID,
command: TaskQuestionReply) -> OperationalTaskDetail:
require(OPERATIONAL_TASK_WRITE in service.permissions, "PERMISSION_DENIED",
"Current user cannot answer task questions", status=403)
body = command.model_dump(mode="json")
scope = f"task:{task_id}:question:{question_id}:answer"
async with service.repo.transaction():
await service.repo.lock(f"task:{task_id}")
if await service._request_replay(scope, command.request_id, body):
return await service.detail(task_id)
task = await service._task(task_id)
service._require_actionable_task(task)
raw = await service.repo.get_question(str(task_id), str(question_id))
require(raw is not None, "QUESTION_NOT_FOUND", "Question not found", status=404)
question = TaskQuestionRequest.model_validate(raw)
require(question.status == "pending" and question.version == command.expected_version
and question.execution_generation == task["execution_generation"],
"VERSION_CONFLICT", "Question is no longer pending in the current generation", status=409)
require(task["lifecycle_status"] == "waiting_human"
and await service.repo.get_active_run(str(task_id)) is None,
"RUN_NOT_SETTLED", "Wait until the investigation has entered waiting_human", status=409)
answers = {a.question_id: a for a in command.answers}
known = {q.question_id for q in question.questions}
require(len(answers) == len(command.answers) and set(answers) <= known,
"ANSWER_INVALID", "Unknown or duplicate questions", status=422)
for q in question.questions:
answer = answers.get(q.question_id)
selected = answer.selected_option_ids if answer else []
text = answer.custom_text.strip() if answer else ""
require(len(set(selected)) == len(selected) and set(selected) <= {o.id for o in q.options}
and (q.mode == "multiple" or len(selected) <= 1)
and (q.allow_custom or not text) and (not q.required or bool(selected or text)),
"ANSWER_INVALID", "Answer does not satisfy the question", status=422)
now = service._now()
question = question.model_copy(update={"status": "answered", "version": question.version + 1,
"answers": command.answers, "answered_at": now, "answered_by": service._actor()})
await service.repo.save_question(str(task_id), question.model_dump(mode="json"))
await service._resolve_notifications(task_id, kinds=["human_input_required"],
subject_type="question", subject_id=question_id, now=now, reason="question_answered")
await service._requeue_task(task, now)
await service.repo.update_task(OperationalTaskSummary.model_validate(task).model_dump(mode="json", exclude={"active_run", "latest_run"}))
await service._task_event(task, now)
await service._save_request(scope, command.request_id, body, {"task_id": str(task_id)})
return await service.detail(task_id)
+14 -5
View File
@@ -1,13 +1,22 @@
from typing import Any
from app.algorithms.valve_isolation.topology_search import valve_isolation_analysis
from app.native.wndb.gis.network_views import get_network_link_nodes
from app.algorithms.valve_isolation.topology_search import (
AccidentElementType,
valve_isolation_analysis,
)
from app.native.wndb.gis.network_views import get_valve_isolation_topology
def analyze_valve_isolation(
network: str,
accident_element: str | list[str],
disabled_valves: list[str] = None,
disabled_valves: list[str] | None = None,
accident_element_type: AccidentElementType = "auto",
) -> dict[str, Any]:
link_entries = get_network_link_nodes(network)
return valve_isolation_analysis(link_entries, accident_element, disabled_valves)
link_entries, source_nodes, closed_links = get_valve_isolation_topology(network)
return valve_isolation_analysis(
link_entries, accident_element, disabled_valves,
accident_element_type=accident_element_type,
water_source_nodes=source_nodes,
closed_links=closed_links,
)
+566
View File
@@ -0,0 +1,566 @@
{
"version": "answer-ui@1",
"description": "Generated from supply frontend src/features/agent/answer-ui/catalog.ts. Regenerate after catalog changes.",
"schema": {
"type": "object",
"properties": {
"root": {
"type": "string"
},
"elements": {
"type": "object",
"additionalProperties": {
"type": "object",
"properties": {
"type": {
"enum": [
"Stack",
"Section",
"Text",
"Result",
"Metric",
"Table",
"Chart",
"Evidence",
"MapLocation",
"FollowUpForm"
]
},
"props": {
"type": "object",
"additionalProperties": {}
},
"children": {
"type": "array",
"items": {
"type": "string"
}
},
"slots": {
"type": "object",
"additionalProperties": {
"type": "array",
"items": {
"type": "string"
}
}
},
"visible": {},
"repeat": {}
},
"required": [
"type",
"props",
"children"
],
"additionalProperties": false
}
}
},
"required": [
"root",
"elements"
],
"additionalProperties": false
},
"components": [
"Stack",
"Section",
"Text",
"Result",
"Metric",
"Table",
"Chart",
"Evidence",
"MapLocation",
"FollowUpForm"
],
"props": {
"Stack": {
"type": "object",
"properties": {
"columns": {
"type": "string",
"enum": [
"one",
"two"
]
},
"gap": {
"type": "string",
"enum": [
"compact",
"normal"
]
},
"intent": {
"type": "string",
"enum": [
"response",
"comparison",
"report"
]
},
"density": {
"type": "string",
"enum": [
"compact",
"comfortable"
]
}
},
"required": [
"columns",
"gap"
],
"additionalProperties": false,
"$schema": "http://json-schema.org/draft-07/schema#"
},
"Section": {
"type": "object",
"properties": {
"title": {
"type": "string",
"maxLength": 200
},
"description": {
"anyOf": [
{
"type": "string",
"maxLength": 4000
},
{
"type": "null"
}
]
},
"role": {
"type": "string",
"enum": [
"result",
"analysis",
"evidence",
"follow_up"
]
}
},
"required": [
"title",
"description"
],
"additionalProperties": false,
"$schema": "http://json-schema.org/draft-07/schema#"
},
"Text": {
"type": "object",
"properties": {
"text": {
"type": "string",
"maxLength": 4000
},
"tone": {
"type": "string",
"enum": [
"neutral",
"info",
"warning"
]
},
"title": {
"type": "string",
"maxLength": 200
}
},
"required": [
"text",
"tone"
],
"additionalProperties": false,
"$schema": "http://json-schema.org/draft-07/schema#"
},
"Result": {
"type": "object",
"properties": {
"title": {
"type": "string",
"maxLength": 200
},
"summary": {
"type": "string",
"maxLength": 4000
},
"status": {
"type": "string",
"enum": [
"success",
"partial",
"unconfirmed",
"error"
]
},
"source": {
"anyOf": [
{
"type": "string",
"maxLength": 200
},
{
"type": "null"
}
]
}
},
"required": [
"title",
"summary",
"status",
"source"
],
"additionalProperties": false,
"$schema": "http://json-schema.org/draft-07/schema#"
},
"Metric": {
"type": "object",
"properties": {
"label": {
"type": "string",
"maxLength": 200
},
"value": {
"type": "string",
"maxLength": 200
},
"unit": {
"anyOf": [
{
"type": "string",
"maxLength": 200
},
{
"type": "null"
}
]
},
"source": {
"anyOf": [
{
"type": "string",
"maxLength": 200
},
{
"type": "null"
}
]
}
},
"required": [
"label",
"value",
"unit",
"source"
],
"additionalProperties": false,
"$schema": "http://json-schema.org/draft-07/schema#"
},
"Table": {
"type": "object",
"properties": {
"title": {
"type": "string",
"maxLength": 200
},
"columns": {
"type": "array",
"items": {
"type": "string",
"maxLength": 200
},
"minItems": 1,
"maxItems": 8
},
"rows": {
"type": "array",
"items": {
"type": "array",
"items": {
"type": "string",
"maxLength": 1000
},
"maxItems": 8
},
"maxItems": 50
}
},
"required": [
"title",
"columns",
"rows"
],
"additionalProperties": false,
"$schema": "http://json-schema.org/draft-07/schema#"
},
"Chart": {
"type": "object",
"properties": {
"title": {
"type": "string",
"maxLength": 200
},
"chart_type": {
"type": "string",
"enum": [
"line",
"bar"
]
},
"x_axis_name": {
"type": "string",
"maxLength": 128
},
"y_axis_name": {
"type": "string",
"maxLength": 128
},
"x_data": {
"type": "array",
"items": {
"type": "string",
"maxLength": 200
},
"minItems": 1,
"maxItems": 100
},
"series": {
"type": "array",
"items": {
"type": "object",
"properties": {
"name": {
"type": "string",
"maxLength": 200
},
"type": {
"type": "string",
"enum": [
"line",
"bar",
"scatter"
]
},
"data": {
"type": "array",
"items": {
"anyOf": [
{
"type": "number"
},
{
"type": "null"
}
]
},
"minItems": 1,
"maxItems": 100
},
"sample_times": {
"type": "array",
"items": {
"anyOf": [
{
"type": "string",
"maxLength": 64,
"format": "date-time",
"pattern": "^(?:[1-9][0-9]{3}|0[1-9][0-9]{2}|00[1-9][0-9]|000[1-9])-[0-9]{2}-[0-9]{2}T(?:[01][0-9]|2[0-3]):[0-5][0-9]:[0-5][0-9](?:\\.[0-9]+)?(?:Z|[+-](?:[01][0-9]|2[0-3]):[0-5][0-9])$"
},
{
"type": "null"
}
]
},
"minItems": 1,
"maxItems": 100
}
},
"required": [
"name",
"data"
],
"additionalProperties": false
},
"minItems": 1,
"maxItems": 8
},
"source": {
"type": "string",
"maxLength": 4000
}
},
"required": [
"title",
"chart_type",
"x_data",
"series",
"source"
],
"additionalProperties": false,
"$schema": "http://json-schema.org/draft-07/schema#"
},
"Evidence": {
"type": "object",
"properties": {
"title": {
"type": "string",
"maxLength": 200
},
"summary": {
"type": "string",
"maxLength": 4000
},
"source": {
"type": "string",
"maxLength": 200
},
"evidence_id": {
"anyOf": [
{
"type": "string",
"maxLength": 200
},
{
"type": "null"
}
]
},
"observed_at": {
"anyOf": [
{
"type": "string",
"maxLength": 200
},
{
"type": "null"
}
]
},
"kind": {
"type": "string",
"enum": [
"observed",
"human_input",
"inference",
"demo"
]
}
},
"required": [
"title",
"summary",
"source",
"evidence_id",
"observed_at",
"kind"
],
"additionalProperties": false,
"$schema": "http://json-schema.org/draft-07/schema#"
},
"MapLocation": {
"type": "object",
"properties": {
"label": {
"type": "string",
"maxLength": 200
},
"layer": {
"type": "string",
"enum": [
"junctions",
"pipes",
"valves",
"pumps",
"tanks",
"reservoirs",
"scada"
]
},
"feature_ids": {
"type": "array",
"items": {
"type": "string",
"minLength": 1,
"maxLength": 128
},
"minItems": 1,
"maxItems": 100
},
"source": {
"type": "string",
"maxLength": 200
}
},
"required": [
"label",
"layer",
"feature_ids",
"source"
],
"additionalProperties": false,
"$schema": "http://json-schema.org/draft-07/schema#"
},
"FollowUpForm": {
"type": "object",
"properties": {
"title": {
"type": "string",
"maxLength": 200
},
"description": {
"type": "string",
"maxLength": 4000
},
"fields": {
"type": "array",
"items": {
"type": "object",
"properties": {
"name": {
"type": "string",
"pattern": "^[a-z][a-z0-9_]{0,39}$"
},
"label": {
"type": "string",
"maxLength": 200
},
"kind": {
"type": "string",
"enum": [
"text",
"datetime",
"number"
]
},
"required": {
"type": "boolean"
}
},
"required": [
"name",
"label",
"kind",
"required"
],
"additionalProperties": false
},
"minItems": 1,
"maxItems": 12
}
},
"required": [
"title",
"description",
"fields"
],
"additionalProperties": false,
"$schema": "http://json-schema.org/draft-07/schema#"
}
}
}
+1 -1
View File
@@ -3,7 +3,7 @@
"contracts": {
"server": {
"file": "server-v1.openapi.json",
"sha256": "949077b0e1c711cfef78319d24cbe081e8d71444e7b25018507e12ba374018a3"
"sha256": "939f38733e943f21cc2e3d9f344fa8122ca7a7b35cfebf96ff3f00f469026ceb"
}
}
}
+22 -4
View File
@@ -103,7 +103,7 @@
"$ref": "#/components/schemas/ManagedResourceCapabilities"
},
"origin": {
"const": "synthetic",
"const": "system_data",
"title": "Origin",
"type": "string"
},
@@ -273,6 +273,12 @@
],
"title": "Location"
},
"origin": {
"const": "system_data",
"default": "system_data",
"title": "Origin",
"type": "string"
},
"priority": {
"default": "normal",
"enum": [
@@ -1681,6 +1687,18 @@
],
"title": "Location"
},
"occurred_at": {
"anyOf": [
{
"format": "date-time",
"type": "string"
},
{
"type": "null"
}
],
"title": "Occurred At"
},
"priority": {
"anyOf": [
{
@@ -1843,7 +1861,7 @@
"title": "Location"
},
"origin": {
"const": "synthetic",
"const": "system_data",
"title": "Origin",
"type": "string"
},
@@ -2036,8 +2054,8 @@
}
},
"info": {
"description": "测试监测与核查反馈样板,不是正式工单平台或设备控制服务。",
"title": "TJWater 外围系统模拟接入",
"description": "按项目管理业务工单与反馈、读取监测历史;测试数据保留独立来源标记。",
"title": "TJWater 外围系统接入",
"version": "1.0.0"
},
"openapi": "3.1.0",
File diff suppressed because it is too large Load Diff
+704
View File
@@ -0,0 +1,704 @@
{
"anyOf": [
{
"type": "object",
"properties": {
"action": {
"type": "string",
"const": "locate_features"
},
"layer": {
"type": "string",
"enum": [
"junctions",
"pipes",
"valves",
"pumps",
"tanks",
"reservoirs",
"scada"
]
},
"feature_ids": {
"type": "array",
"items": {
"type": "string",
"minLength": 1,
"maxLength": 128
},
"minItems": 1,
"maxItems": 100
}
},
"required": [
"action",
"layer",
"feature_ids"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"action": {
"type": "string",
"const": "zoom_to_map"
},
"center": {
"type": "array",
"minItems": 2,
"maxItems": 2,
"items": [
{
"type": "number",
"minimum": -180,
"maximum": 180
},
{
"type": "number",
"minimum": -90,
"maximum": 90
}
]
},
"zoom": {
"type": "number",
"minimum": 0,
"maximum": 24
}
},
"required": [
"action",
"center",
"zoom"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"action": {
"type": "string",
"const": "render_scada_analysis"
},
"items": {
"type": "array",
"items": {
"type": "object",
"properties": {
"device_id": {
"type": "string",
"minLength": 1,
"maxLength": 128
},
"level": {
"type": "string",
"enum": [
"high",
"medium",
"low",
"unrated"
]
}
},
"required": [
"device_id",
"level"
],
"additionalProperties": false
},
"minItems": 1,
"maxItems": 100
}
},
"required": [
"action",
"items"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"action": {
"type": "string",
"const": "clear_scada_analysis"
}
},
"required": [
"action"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"action": {
"type": "string",
"const": "highlight_features"
},
"layer": {
"type": "string",
"enum": [
"junctions",
"pipes",
"valves",
"pumps",
"tanks",
"reservoirs",
"scada"
]
},
"feature_ids": {
"type": "array",
"items": {
"type": "string",
"minLength": 1,
"maxLength": 128
},
"minItems": 1,
"maxItems": 100
},
"effect": {
"type": "string",
"enum": [
"steady",
"pulse"
]
}
},
"required": [
"action",
"layer",
"feature_ids",
"effect"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"action": {
"type": "string",
"const": "clear_highlights"
}
},
"required": [
"action"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"action": {
"type": "string",
"const": "open_chart"
},
"chart_id": {
"type": "string",
"minLength": 1,
"maxLength": 128
},
"chart": {
"type": "object",
"properties": {
"title": {
"type": "string",
"maxLength": 200
},
"chart_type": {
"type": "string",
"enum": [
"line",
"bar"
]
},
"x_axis_name": {
"type": "string",
"maxLength": 128
},
"y_axis_name": {
"type": "string",
"maxLength": 128
},
"x_data": {
"type": "array",
"items": {
"type": "string",
"maxLength": 128
},
"minItems": 1,
"maxItems": 100
},
"series": {
"type": "array",
"items": {
"type": "object",
"properties": {
"name": {
"type": "string",
"maxLength": 128
},
"type": {
"type": "string",
"enum": [
"line",
"bar",
"scatter"
]
},
"data": {
"type": "array",
"items": {
"anyOf": [
{
"type": "number"
},
{
"type": "null"
}
]
},
"minItems": 1,
"maxItems": 100
},
"sample_times": {
"type": "array",
"items": {
"anyOf": [
{
"type": "string",
"maxLength": 64,
"format": "date-time",
"pattern": "^(?:[1-9][0-9]{3}|0[1-9][0-9]{2}|00[1-9][0-9]|000[1-9])-[0-9]{2}-[0-9]{2}T(?:[01][0-9]|2[0-3]):[0-5][0-9]:[0-5][0-9](?:\\.[0-9]+)?(?:Z|[+-](?:[01][0-9]|2[0-3]):[0-5][0-9])$"
},
{
"type": "null"
}
]
},
"minItems": 1,
"maxItems": 100
}
},
"required": [
"name",
"data"
],
"additionalProperties": false
},
"minItems": 1,
"maxItems": 8
}
},
"required": [
"title",
"chart_type",
"x_data",
"series"
],
"additionalProperties": false
},
"evidence_refs": {
"type": "array",
"items": {
"type": "string",
"minLength": 1,
"maxLength": 128
},
"minItems": 0,
"maxItems": 100
}
},
"required": [
"action",
"chart_id",
"chart",
"evidence_refs"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"action": {
"type": "string",
"const": "close_chart"
},
"chart_id": {
"type": "string",
"minLength": 1,
"maxLength": 128
}
},
"required": [
"action",
"chart_id"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"action": {
"type": "string",
"const": "set_overlay"
},
"overlay_id": {
"type": "string",
"minLength": 1,
"maxLength": 128
},
"title": {
"type": "string",
"minLength": 1,
"maxLength": 200
},
"mode": {
"type": "string",
"enum": [
"geometry",
"heatmap"
]
},
"evidence_id": {
"type": "string",
"minLength": 1,
"maxLength": 128
},
"data": {
"type": "object",
"properties": {
"type": {
"type": "string",
"const": "FeatureCollection"
},
"features": {
"type": "array",
"items": {
"type": "object",
"properties": {
"type": {
"type": "string",
"const": "Feature"
},
"id": {
"anyOf": [
{
"type": "string"
},
{
"type": "number"
}
]
},
"properties": {
"anyOf": [
{
"type": "object",
"additionalProperties": {}
},
{
"type": "null"
}
]
},
"geometry": {
"anyOf": [
{
"type": "object",
"properties": {
"type": {
"type": "string",
"const": "Point"
},
"coordinates": {
"type": "array",
"minItems": 2,
"items": [
{
"type": "number",
"minimum": -180,
"maximum": 180
},
{
"type": "number",
"minimum": -90,
"maximum": 90
}
],
"additionalItems": {
"type": "number"
}
}
},
"required": [
"type",
"coordinates"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"type": {
"type": "string",
"const": "MultiPoint"
},
"coordinates": {
"type": "array",
"items": {
"type": "array",
"minItems": 2,
"items": [
{
"type": "number",
"minimum": -180,
"maximum": 180
},
{
"type": "number",
"minimum": -90,
"maximum": 90
}
],
"additionalItems": {
"type": "number"
}
},
"minItems": 1,
"maxItems": 20000
}
},
"required": [
"type",
"coordinates"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"type": {
"type": "string",
"const": "LineString"
},
"coordinates": {
"type": "array",
"items": {
"type": "array",
"minItems": 2,
"items": [
{
"type": "number",
"minimum": -180,
"maximum": 180
},
{
"type": "number",
"minimum": -90,
"maximum": 90
}
],
"additionalItems": {
"type": "number"
}
},
"minItems": 2,
"maxItems": 20000
}
},
"required": [
"type",
"coordinates"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"type": {
"type": "string",
"const": "MultiLineString"
},
"coordinates": {
"type": "array",
"items": {
"type": "array",
"items": {
"type": "array",
"minItems": 2,
"items": [
{
"type": "number",
"minimum": -180,
"maximum": 180
},
{
"type": "number",
"minimum": -90,
"maximum": 90
}
],
"additionalItems": {
"type": "number"
}
},
"minItems": 2,
"maxItems": 20000
},
"minItems": 1,
"maxItems": 20000
}
},
"required": [
"type",
"coordinates"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"type": {
"type": "string",
"const": "Polygon"
},
"coordinates": {
"type": "array",
"items": {
"type": "array",
"items": {
"type": "array",
"minItems": 2,
"items": [
{
"type": "number",
"minimum": -180,
"maximum": 180
},
{
"type": "number",
"minimum": -90,
"maximum": 90
}
],
"additionalItems": {
"type": "number"
}
},
"minItems": 4,
"maxItems": 20000
},
"minItems": 1,
"maxItems": 20000
}
},
"required": [
"type",
"coordinates"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"type": {
"type": "string",
"const": "MultiPolygon"
},
"coordinates": {
"type": "array",
"items": {
"type": "array",
"items": {
"type": "array",
"items": {
"type": "array",
"minItems": 2,
"items": [
{
"type": "number",
"minimum": -180,
"maximum": 180
},
{
"type": "number",
"minimum": -90,
"maximum": 90
}
],
"additionalItems": {
"type": "number"
}
},
"minItems": 4,
"maxItems": 20000
},
"minItems": 1,
"maxItems": 20000
},
"minItems": 1,
"maxItems": 20000
}
},
"required": [
"type",
"coordinates"
],
"additionalProperties": false
}
]
}
},
"required": [
"type",
"properties",
"geometry"
],
"additionalProperties": false
},
"minItems": 1,
"maxItems": 1000
}
},
"required": [
"type",
"features"
],
"additionalProperties": false
}
},
"required": [
"action",
"overlay_id",
"title",
"mode",
"evidence_id",
"data"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"action": {
"type": "string",
"const": "clear_overlay"
},
"overlay_id": {
"type": "string",
"minLength": 1,
"maxLength": 128
}
},
"required": [
"action",
"overlay_id"
],
"additionalProperties": false
}
],
"$schema": "http://json-schema.org/draft-07/schema#"
}
+65
View File
@@ -0,0 +1,65 @@
<svg xmlns="http://www.w3.org/2000/svg" width="720" height="362" viewBox="0 0 720 362" role="img" aria-labelledby="title description">
<title id="title">SCADA 演示时间轴 FM 调频盘样式</title>
<desc id="description">静态设计示意。选定区间为十月八日八点至十四点,当前时间十点十五分。半透明玻璃刻度窗内细密时间刻度横向排列,中央红色指示线固定,控制栏包含暂停、下一批和六十倍速。</desc>
<defs>
<linearGradient id="glass" x1="0" y1="0" x2="0" y2="1">
<stop offset="0" stop-color="#fff" stop-opacity=".72"/>
<stop offset=".5" stop-color="#f7faff" stop-opacity=".4"/>
<stop offset="1" stop-color="#fff" stop-opacity=".64"/>
</linearGradient>
<linearGradient id="edge" x1="0" y1="0" x2="1" y2="0">
<stop offset="0" stop-color="#f7f9fc"/>
<stop offset=".12" stop-color="#f7f9fc" stop-opacity="0"/>
<stop offset=".88" stop-color="#f7f9fc" stop-opacity="0"/>
<stop offset="1" stop-color="#f7f9fc"/>
</linearGradient>
<filter id="blur"><feGaussianBlur stdDeviation="16"/></filter>
<filter id="shadow" x="-.1" y="-.3" width="1.2" height="1.6">
<feDropShadow dx="0" dy="2" stdDeviation="4" flood-color="#243852" flood-opacity=".1"/>
</filter>
<clipPath id="window"><rect x="48" y="130" width="624" height="110" rx="12"/></clipPath>
</defs>
<rect width="720" height="362" rx="18" fill="#e9edf2"/>
<rect x="24" y="22" width="672" height="318" rx="16" fill="#fafbfd" stroke="#d7dee7"/>
<g font-family="'Noto Sans SC Variable', -apple-system, BlinkMacSystemFont, 'Segoe UI', 'PingFang SC', 'Noto Sans CJK SC', sans-serif" fill="#263347">
<text x="48" y="54" font-size="13" font-weight="600">SCADA 时段演示</text>
<text x="672" y="54" text-anchor="end" font-size="11" fill="#778398">静态样式示意</text>
<text x="48" y="79" font-size="12" fill="#68758a">演示区间</text>
<text x="114" y="79" font-size="12">10 月 8 日 08:00 — 14:00</text>
<text x="360" y="118" text-anchor="middle" font-size="28" font-weight="500" font-variant-numeric="tabular-nums" letter-spacing="1">10:15:00</text>
<rect x="48" y="130" width="624" height="110" rx="12" fill="#edf2f7" filter="url(#shadow)"/>
<g clip-path="url(#window)">
<g filter="url(#blur)" opacity=".6">
<path d="M12 234 L280 132 L423 247 L749 142" fill="none" stroke="#c7d8e7" stroke-width="35"/>
<path d="M63 134 L221 219 L438 128 L647 212" fill="none" stroke="#fff" stroke-width="38"/>
</g>
<rect x="48" y="130" width="624" height="110" fill="url(#glass)"/>
<path d="M65 131 H655" stroke="#fff" stroke-width="2" opacity=".9"/>
<path d="M49 138 V231 M671 138 V231" stroke="#fff" opacity=".6"/>
<path d="M60 237 H660" stroke="#a8b6c8" opacity=".25"/>
<g fill="none" stroke="#67778b" stroke-width="1">
<path d="M90 166 V197 M180 166 V197 M270 166 V197 M360 166 V197 M450 166 V197 M540 166 V197 M630 166 V197"/>
<path d="M60 179 V196 M120 179 V196 M150 179 V196 M210 179 V196 M240 179 V196 M300 179 V196 M330 179 V196 M390 179 V196 M420 179 V196 M480 179 V196 M510 179 V196 M570 179 V196 M600 179 V196 M660 179 V196" opacity=".65"/>
</g>
<g font-size="11" text-anchor="middle" font-variant-numeric="tabular-nums">
<text x="90" y="216">09:30</text><text x="180" y="216">09:45</text>
<text x="270" y="216">10:00</text><text x="360" y="216" font-weight="600">10:15</text>
<g fill="#7b889a"><text x="450" y="216">10:30</text><text x="540" y="216">10:45</text><text x="630" y="216">11:00</text></g>
</g>
<rect x="48" y="131" width="624" height="108" fill="url(#edge)"/>
</g>
<rect x="48.5" y="130.5" width="623" height="109" rx="11.5" fill="none" stroke="#fff" stroke-opacity=".8"/>
<path d="M356 141 H364 L360 147 Z" fill="#cf453f"/>
<path d="M360 147 V201" stroke="#cf453f" stroke-width="2"/>
<rect x="48" y="256" width="88" height="38" rx="8" fill="#4662ba"/>
<path d="M65 269 V281 M70 269 V281" stroke="#fff" stroke-width="2"/>
<text x="98" y="280" text-anchor="middle" fill="#fff" font-size="12">暂停</text>
<rect x="148" y="256" width="88" height="38" rx="8" fill="#fff" stroke="#d7dee7"/>
<text x="192" y="280" text-anchor="middle" font-size="12" fill="#7b8799">下一批</text>
<rect x="578" y="256" width="94" height="38" rx="8" fill="#fff" stroke="#d7dee7"/>
<text x="601" y="280" font-size="12">60×</text>
<path d="M651 273 L655 277 L659 273" fill="none" stroke="#64738a" stroke-width="1.5"/>
<text x="48" y="319" font-size="11" fill="#68758a">已读取 10:15 · 已检测 10:10</text>
<text x="672" y="319" text-anchor="end" font-size="11" fill="#68758a">红线对应当前时间 · 横向拨动可回看</text>
</g>
</svg>

After

Width:  |  Height:  |  Size: 4.7 KiB

+62
View File
@@ -0,0 +1,62 @@
# 从爆管工单到事件处置闭环
工单触发的 `workflow=burst_report` 使用 [爆管事件处理 Skill](../resources/work-order-skills/burst-event-handling/SKILL.md)。Server 按持久化来源选择流程,Agent 接收同一任务的核验上下文;SCADA 异常和历史回放使用 [SCADA Skill](../resources/work-order-skills/scada-anomaly/SKILL.md)。
## 固定处理路线
```mermaid
flowchart TD
order[爆管上报工单] --> context[读取原工单与当前任务]
context --> asset[核对地点并关联管网资产]
asset --> isolation[分析隔离边界与阀门方案]
isolation --> impact[说明事件与隔离方案影响]
impact --> report[保存阶段报告与处置建议]
report --> approval[人工审批与现场执行]
approval --> feedback[接收实际处置与恢复反馈]
feedback --> verify[复核影响与结案条件]
verify --> close[来源系统确认事件闭环]
verify -->|新增事实或遗留事项| asset
```
1. **接收核对**:保留来源工单ID、单调版本、发生时间、位置、描述、现场反馈和已有动作;后续轮次从新增事实接续。
2. **资产关联**:优先核对上报资产编号。坐标或地址通过地理编码、附近资产、属性和连接关系形成有依据的关联;位置歧义列为具体待确认项。
3. **隔离方案**:用 analyze_valve_isolation 分析关联资产,保留必关阀、可选阀、直接隔离与供水失源节点;已知失效阀门用于替代方案分析,现场可操作性列为执行条件。
4. **影响说明**:先展示拓扑范围,再按可用SCADA、水力结果和用户关联资料解释事件当前影响、方案预计影响和保障需求。
5. **报告与处置**:保存阶段报告后提出核验、通知、保障、抢修和恢复等动作草稿;关联现场任务采用 general 工单。审批、派发和现场完成分别核对回执。
6. **反馈复核**:已派发的关联工单保持 `waiting_work_order`,全部收到有效 `completed` 反馈后进入 `waiting_scada`。以现场反馈业务时间为 T,取严格晚于 T 的最近一个 SCADA 采样时刻进行复核;正常范围使用事件前15天同采样相位的正常包络。数据未到或仍异常时保持等待,后台工作进程定期复核;通过后持久保存恢复证据并接续 Agent。Agent 最终报告必须引用该证据,运行完成后才能完成任务并解除对应告警。原事件由有权限人员在来源系统确认行政结案。
SCADA作为事件影响的辅助证据,按目标测点和事件/对照窗口读取。具备水力输入且工具已授权时,使用 analyze_valve_closure 评估给定阀门方案;各步骤记录实际结果、适用范围或具体待补项。
## 报告固定内容
报告包含事件概况、地点与资产关联、隔离方案、影响说明、处置建议和待办。摘要以150字以内为目标,资产和测点明细放表格,完整依据引用证据。
“影响说明”逐项保留时间、对象、单位、范围、来源与实际或预计的性质:
| 影响项 | 依据与输出 |
| --- | --- |
| 事件当前影响 | 已查询测点的压力、流量、持续时间和恢复情况 |
| 直接隔离范围 | affected_nodes及对应计数、地图覆盖 |
| 新增供水失源范围 | supply_affected_nodes及对应计数、地图覆盖 |
| 水力影响 | 给定阀门组合、起算时间、模型条件下的压力和流量结果 |
| 用户与保障影响 | 有资料关联的用户、重点设施、保障需求;缺少关联时报告节点与区域范围 |
变化量基于同口径对象、单位、时间与模型条件计算;现场实际、模型预测和空间示意分别标注。缺少可选模拟参数时,继续形成已有依据的隔离与保障建议。
## 工具与入口约束
工单事件从上报位置开展资产关联。Agent会话策略隐藏 run_burst_location,Server在绑定与后续解析时同步移除其调用能力;普通聊天和SCADA候选定位沿用各自策略。持久计算在执行前重新校验任务绑定。
历史 burst_report 任务携带 replay_context 时,Server分发scada-anomaly,时序查询保持 knowledge_cutoff 与 batch_sequence,demo能力提示按冻结窗口返回。
业务阶段和工单、返回时间、复核回执保存在任务文档 `handling_context` JSON 中,不增加数据库列。外围样板更新工单时可提供带时区的 `occurred_at`,同时必须填写现场反馈;该时间写入活动记录,系统的 `updated_at` 仍是实际记录时间。同步器将现场反馈时间传给调查,避免历史演示误用电脑当前时间。
预约时间 `scheduled_at` 可在派单前的工单草稿和已创建工单的编辑表单中调整。实际完工反馈时间与预约安排分别记录;Agent 以实际反馈时间为恢复复核起点。新的有效现场反馈修订完工时间后,旧复核证据保留为历史记录,当前复核起点更新,重新读取严格晚于新反馈时间的最近一个 SCADA 采样时刻。只修改预约安排或其他工单信息,不改写已有现场反馈时间。
`POST /operational-tasks/recovery/reconcile` 由现有 operational worker 每轮调用,按项目及任务创建边界处理等待复核的任务;SCADA 读取失败不阻断其他调查派发。复核直接读取项目时序库中完工反馈时间之后最近时刻的 SCADA,不依赖回放状态或推进回放时间。
## 验证
检查Skill名称、版本、摘要与正文digest一致、内联体积、角色权限、工单工具隔离和SCADA定位回归。Agent类型检查与会话工具策略测试共同覆盖运行时入口。发布后分别验收工单事件处理报告和SCADA候选定位,原有报告按既有修订保留。
关联契约:[分析 MCP](mcp-analysis-tools.md)、[调查展示](investigation-progress-and-presentation.md)、[报告契约](investigation-report-contract.md)、[任务与多轮反馈](operational-tasks.md)。
+94
View File
@@ -0,0 +1,94 @@
# MCP 后台计算队列
API 保留 `uvicorn --workers 4`。长计算在项目 BizDB 持久化入队,由独立计算进程领取;所有 API 进程读取相同任务状态,不依赖粘性会话。HTTP 返回预算由宿主管理,Server 默认 120 秒、Agent 默认 180 秒。Server 通过工具目录 `_meta["io.tjwater/request-timeout"]` 统一发布 `defaultSeconds` 和 `maxSeconds`,所有工具(包括状态和结果读取)使用同一配置来源。Server 兼容保留 `timeout_seconds` 参数;Agent 对模型隐藏并拒绝该参数,按目录预算代填。Server 请求上限为 240 秒,Agent 加 60 秒返回余量并封顶 270 秒,原生 MCP 与本地连接的总时限为 300 秒;上游 HTTP 的 5 分钟空闲超时保留。提交、查询状态和读取结果都应及时返回,不等待整个计算。模拟时间仍由 `duration` 表示。
后台执行时限由管理员配置 `TJWATER_MCP_MAX_CALCULATION_TIMEOUT_SECONDS`(默认 3600 秒),入队时单独写入任务记录。HTTP 参数不参与作业幂等参数比较,也不会缩短或延长后台执行。任务表的内部 `timeout_seconds` 列仍保存执行时限。HTTP 超时或断开只结束本次等待,后台作业需通过独立取消工具取消。
```text
前端 / Agent → API × 4 → 项目 BizDB calculation_jobs
↑
独立计算 supervisor
↓
受管理的计算子进程组
```
## 工具契约
原工具名继续使用。以下工具提交后返回 `QueryResult`,其中 `semantics=task`,`data` 包含 `job_id`、`status`、`tool` 和 `poll_after_ms`;这只证明提交状态,不是分析结果或调查证据:
- `run_burst_location`、`run_leakage_identification`、`run_burst_detection`
- `run_simulation`、`run_burst_simulation`、`run_contaminant_simulation`
- `analyze_valve_closure`、`analyze_flushing`、`analyze_pressure_regulation`
- `optimize_sensor_placement`
- `save_simulation_scenario`、`create_sensor_placement` 的 `dry_run=false` 执行阶段
后两项保留预览、摘要校验和原有审批;提交与执行时均校验权限及预览摘要,已有写入回执继续约束重复请求。`analyze_valve_isolation`、`calculate_demand_allocation` 保持同步执行。普通查询和其他写入不进入此队列。
| MCP 工具 | 输入 | 返回 |
| --- | --- | --- |
| `get_calculation_job` | `job_id` | 状态、工具、时间和安全错误码 |
| `get_calculation_result` | `job_id`,可选 `data_path/limit/offset` | 成功后读取原分析结果;未成功返回任务状态 |
| `list_calculation_jobs` | 可选 `status/limit/offset` | 当前授权会话及调查轮次的任务列表 |
| `cancel_calculation_job` | `job_id` | 排队任务可直接取消;运行任务先进入 `cancel_requested` |
状态为 `queued/running/cancel_requested/succeeded/failed/cancelled/timed_out/interrupted`。返回创建、开始、结束和排队/结果到期时间;`result_available` 表示结果是否仍可读取。任务取消不等于在线模型写入回滚;发生失败、超时或中断时,需要核查已有写入,不能自动重算整个窗口。
当前使用普通 MCP 业务工具承载队列契约,没有宣称已实现协议级 Tasks 扩展。Server 和 Agent 目录必须配套更新,旧 Agent 不应接收新的队列返回契约。
## 身份、幂等和结果
任务记录保存提交人的元数据用户 ID、项目、原工具参数和受限授权快照,不保存 Bearer、数据库 DSN 或短期 `context_handle`。执行和读取结果时重新查询当前用户、成员关系、项目路由、权限及调查运行租约。任务不会因为短期 MCP 句柄失效而丢失;也不会借持久化任务绕过撤销的权限。
普通计算使用宿主从原生工具调用记录解析出的稳定调用 ID;同一次传输重试复用任务,不将新的业务计算自动按参数合并。重写工具同时保留原审批 `request_id` 幂等语义。不同参数复用同一操作编号会被拒绝。
调查的排队回执不写成分析证据。成功结果读取时,按原计算来源、任务及运行验证,并幂等地保存证据。未完成的计算阻止提交最终报告、`insufficient_data` 或将运行标记成功;中间进度仍可保存。
Agent 调查执行在原生回合空闲后由宿主短轮询等待,继续维持调查心跳。新作业进入终态即触发模型读取结果,其余作业继续后台运行。状态读取遇到网络、限流或临时服务错误,最多重试 3 次,采用指数退避和随机抖动;权限拒绝、上下文失效立即停止,重试耗尽明确报错,不重新提交或取消计算。普通聊天展示任务编号,可在当前会话后续查询。异常流结束时若无法安全继续处理新结果,会明确失败而不是虚报完成;Server 中的计算结果仍保留。
## 排队期限与结果保留
每项目最多 100 个未结束作业(queued/running/cancel_requested),跨 API 进程用数据库事务锁串行执行幂等检查与容量准入;同一请求的重试优先返回已有作业。排队最长 3600 秒,超过后标为 `timed_out`,错误码为 `QUEUE_TIMEOUT`;开始执行后使用独立的执行期限。提交、查询和 Worker 领取都会核查到期任务,所以 Worker 停止时,查询也能结束过期排队。部署监控仍须覆盖 Worker 进程状态和最老排队任务年龄,队列到期机制不代替服务可用性监控。
结果正文在终态后保留 7 天,到期由队列维护清除,读取返回 `RESULT_EXPIRED`。任务状态、输入、幂等标识和已归档调查证据不随之删除,也不会因结果过期重新计算。状态列表只读取状态字段,不加载完整计算结果。
## 并发、取消和恢复
一个 supervisor 默认全项目合计并发 2 项,独立于 API 的 4 个进程。它持有元数据库的会话级 advisory lock,第二个 supervisor 会拒绝启动;当前不是多台计算节点各自扩容的模式。各项目轮转领取,使用 PostgreSQL 短事务 `SKIP LOCKED`,租约代次约束心跳和结果写回。
每项任务有独立子进程组;取消、执行超时或失去租约时先停止进程组,再写终态。守护管道在 supervisor 被强杀后关闭,负责终止仍在运行的算法进程。API 重启不影响计算 supervisor;计算进程崩溃后,旧任务标为中断,不自动重放。
原生临时文件写入独立任务目录。临时模型数据库创建前登记到该目录的账本,正常结束或取消后只清理明确登记的临时数据库。清理失败返回 `CLEANUP_FAILED` 并保留恢复目录;启动时检查项目、任务和执行代次,确认旧租约过期后恢复清理。无法验证归属或清理失败时停止领取新工作。
## 部署
1. 先对每个目标项目的 BizDB 依次应用 `resources/sql/026_calculation_jobs.sql` 和 `resources/sql/027_calculation_job_lifecycle.sql`,包括未来项目使用的相应业务模板。已有 026 的库只需追加 027。不要应用到 TimescaleDB。027 增加时间字段与索引,不清除结果;已有结果从首次迁移起获得完整 7 天保留期。必须先迁移,再加载新 API/Worker 代码。
2. 配置 `CALCULATION_WORKER_PROJECT_IDS` 为明确的项目 UUID 列表,配置现有元数据库连接及数据库路由解密密钥。Worker 不需要新 OAuth 密钥;仅部署侧工作进程读取这些设置。
3. 同步发布 Server 和 Agent,在接入长计算前启动独立 Worker。不要从 API `lifespan` 启动,不要把调查 Worker 当作计算 Worker。
从 Server 仓库启动:
```bash
conda run -n server python scripts/run_calculation_worker.py
```
systemd 模板:`infra/systemd/tjwater-calculation-worker.service`。Docker 使用同一 Server 镜像的独立服务:
```bash
docker compose -f infra/docker/docker-compose.yml --profile calculations up -d calculation-worker
```
Compose 使用 `../../.env` 注入配置,需确保其中数据库地址在容器内可达。`calculation-state` volume 保存任务恢复账本,不能当作临时缓存删除。非容器启动可用 `CALCULATION_WORKER_STATE_DIR` 指定持久目录。
并发默认 2、轮询默认 2 秒、租约 30 秒、心跳 5 秒。算法内部并行度仍消耗额外 CPU/内存;调整并发必须依据实际资源测量。
回滚前暂停接收新的队列计算并等待或明确取消在途任务,核对有副作用任务的写入。保留任务表、结果和恢复目录;旧同步版本不支持继续处理这些作业,不能直接删除记录后重新提交。
## 验证
```bash
conda run -n server python -m pytest tests/unit/test_mcp_calculation_jobs.py tests/unit/test_calculation_worker.py tests/unit/test_calculation_cleanup.py tests/unit/test_mcp_query.py tests/unit/test_mcp_analysis.py tests/unit/test_operational_tasks.py -q
```
真实 PostgreSQL 并发测试为 `tests/integration/test_calculation_queue_storage.py`。它要求 `CALCULATION_QUEUE_TEST_DSN` 指向名称以 `tjwater_calc_test_` 开头的专用可丢弃数据库,不能指向业务库。覆盖四连接重复提交、参数冲突、并发领取、项目和用户隔离、取消竞争、失效代次拒绝和过期租约恢复。
这些验证不代替真实项目算法运行、浏览器断线重连或部署验收。生产接入还应观察超过 5 分钟的真实计算、恢复后结果读取和原生临时资源清理。
+65
View File
@@ -0,0 +1,65 @@
# 凤阳六月爆管模拟与 SCADA 复制
2026-10-22 已完成活动项目 `fengyang_v2`
(`b962c44e-8433-4a10-81e9-a64b70ac1152`)的三段模拟爆管数据及前一天基线复制。
一次性复制脚本及其专属测试已清理,来源、核对结果和备份记录如下:
| 方案名称 | 模型管段 | 模型管径 |
| --- | --- | --- |
| `2026-06-02 Pipe19026 DN200 burst` | `19026` | 200 mm |
| `2026-06-12 Pipe18751 DN160 burst` | `18751` | 160 mm |
| `2026-06-22 Pipe18750 DN160 burst` | `18750` | 160 mm |
方案名称、`burst_ID` 和 `network.pipes.diameter` 必须一致。时间使用方案的
`started_at` 和结果时间戳,历时为 4 小时,包含起止点,15 分钟一个采样。
不使用项目说明中的历史时间窗,也不使用旧的 `5hrs burst scheme` 数据。
源数据为 `analysis.node_results.pressure`,仅复制既有 50 个压力设备
`asset.scada_devices.node_id` 对应的节点。每段必须完整覆盖 17 个时间点,
即 850 条观测,三段共 2,550 条;方案状态必须为 `completed`。
压力源单位来自每个方案的 `parameters.result_units.pressure`,目标单位来自
设备的 `measurement_unit`,并用总水头减节点高程核对压力单位。缺失水头、
高程或单位不一致时拒绝写入。例如真正的 kPa 转米水头使用 `pressure / 9.80665`。
不添加噪声、不平移日期、不插值。写入 `scada.measurements.monitored_value`,
同时将窗口内对应观测的 `cleaned_value` 清空,避免使用旧观测的清洗结果。
其他日期、其他设备及模拟源数据保持原样;原有其他模拟事件也保留。
正常基线直接读取 `realtime.node_results`,保留其原始日期和压力值。每段在线
模拟采用右端不含的时间窗,4 小时共有 16 个采样点,最后一个点比窗口终点
早 15 分钟。三段共 2,400 条观测。基线源单位来自模型配置,并同样校验
总水头减高程;不从爆管结果或旧 SCADA 推导正常值。
| 管段 | 爆管数据时间(北京时间,含末点) | 前一天基线采样时间(北京时间) |
| --- | --- | --- |
| `19026` DN200 | 6 月 2 日 08:00–12:00 | 6 月 1 日 08:00–11:45 |
| `18751` DN160 | 6 月 12 日 12:00–16:00 | 6 月 11 日 12:00–15:45 |
| `18750` DN160 | 6 月 22 日 18:00–22:00 | 6 月 21 日 18:00–21:45 |
定位算法配对比较时应采用共同的 16 个相对采样时刻,或明确使用右端不含的
4 小时时间窗,避免将爆管结果额外的终点与不存在的基线终点配对。
2026-10-22 核查发现 EPANET 输出解析曾将压力枚举 1、2 的单位名称写反。
已改为 `0=PSI, 1=KPA, 2=MTR`,并通过本地原生 EPANET 输出测试验证。
这三个方案的全部 2,550 个压力值都与米制水头减高程一致,元数据已纠正为
`MTR`,原始模拟数值保持不变。SCADA 最终使用设备的 `m` 单位,无额外换算。
更正前的三个方案参数与核查证据保存在忽略目录
`data/fengyang-result-unit-correction/`。
本次最终复制记录为
`data/fengyang-burst-scada/20261022T101442415337Z/report.json`:
2,550 条爆管观测与 2,400 条前一天基线,共替换 4,950 条观测;SCADA 整表
仍为 144,000 条、50 个设备。窗口外指纹不变,独立 SQL 比对六段数据与源值
差异为零,重复只读预览显示 4,950 条均已一致。三个方案均为 `completed`。
本次执行前已锁定观测表并备份整表至忽略目录
`data/fengyang-burst-scada/<UTC时间>/measurements-before.csv`,同时保存
`prepared.json`(来源、设备映射、备份校验和)和 `expected.json`(预期观测)。
目录权限为 0700,文件权限为 0600,不包含数据库凭据。
本次复制在一个事务中完成,逐条比对预期值、核对总行数、校验窗口外数据
指纹不变。提交后使用新连接再次读回校验,已保存 `report.json`。
如需恢复,先核对对应报告的备份校验和,将 CSV 导入临时表,再仅恢复
`expected.json` 中的 `(time, device_id)` 键;不要整表覆盖当前观测。
数据库读回校验不代表已经完成 HTTP、界面回放或爆管定位算法验收。
@@ -55,21 +55,60 @@ Server 校验当前 run、绑定会话、有效租约和幂等身份,记录 `o
| command.action | 参数 |
| --- | --- |
| highlight_features | layer、feature_ids、effect:steady 或 pulse |
| locate_features | layer、feature_ids |
| zoom_to_map | center:WGS84 经度纬度、zoom:0–24 |
| highlight_features | layer、feature_ids、必填 effect:steady 或 pulse |
| clear_highlights | 无 |
| open_chart | chart_id、chart、非空 evidence_refs |
| close_chart | chart_id |
| set_overlay | overlay_id、title、mode:geometry 或 heatmap、evidence_id |
| set_overlay | overlay_id、title、mode:geometry 或 heatmap、evidence_id,可选 data |
| clear_overlay | overlay_id |
| render_scada_analysis | items:device_id 与 high/medium/low/unrated 等级,设备编号不重复 |
| clear_scada_analysis | 无 |
以上十种动作与 Chat 的 `present_view`、Dev JSONL 共用扁平 `command: {action, ...字段}`。Chat 使用 `frontend-action@2` 和执行回执;任务仍只记录展示请求,Dev 只做本地预览。三者使用独立的地图源、SCADA 图层和图表状态。Dev 的动作数据行必须为 `{"record_type":"presentation","command":{...}}`;界面块行使用 `record_type: ui_block`,新示例明确带 `phase: final`。普通调查数据保持来源字段,不执行其中的 action。仓库示例见[前端统一展示夹具](../../next-tjwater-frontend/tests/fixtures/unified-presentation.jsonl),其中编号来自测试夹具。
图表限定折线和柱形,包含 `title/chart_type/x_data/series`。最多 100 个横轴值和 8 组有限数值序列,
序列长度与横轴对齐,引用证据必须属于当前任务。地图高亮来源沿用上述七类。
覆盖层输入不接受模型提供的 `data`。Server 从证据的不可变数据快照提取现有 GeoJSON,
覆盖层的权威数据来自不可变证据。省略 `data` 时,Server 从证据快照提取现有 GeoJSON,
或使用已记录的 `longitude/latitude` 点位,将 `FeatureCollection` 加入事件中的 `command.data`。
若传入 `data`,必须与同一证据快照解析出的内容一致;Server 始终发布快照解析结果,不能用调用参数覆盖证据。
热力图只接受记录中的点,权重为有限非负数,未记录权重时取 1。
没有坐标的地址、空数据、非法几何、模型自绘区域均不能生成覆盖层。该接口不做地址解析、
范围推算或污染传播计算。展示事件总大小上限为 512 KiB,超限返回 422。
`generate_node_coverage` 是独立的节点空间覆盖工具,适用于普通项目聊天和后台调查,
不绑定爆管、阀门隔离或固定执行顺序。调用者提供需要比较的节点分组,Server 从当前项目
`gis.node_geometries` 读取真实位置。监测设备、投诉、地址等须先关联模型节点,
不接受模型自由生成的坐标。具体适用范围和参数见[节点覆盖工具](node-coverage-tool.md)。
计算先投影到局部米制坐标,取凹包(默认 ratio=0.25)、加展示缓冲(默认 25 m),
再以 2 m 容差简化并转回 WGS84;凹包比例与缓冲距离可在有界参数内选择。
默认展示各组的地理包络,不判断连接关系;需要时可按模型连接拆分,额外指定视为断开的连接。
少量点、重合点和共线点同样通过缓冲形成面。多组结果一起保存在 `result.map_payload`,
避免连续 `set_overlay` 覆盖前一组;`result.coverage` 保存来源节点、映射完整性及算法参数。
该面不推断故障、污染、实际停水、水力影响或用户服务边界,缓冲距离也不是供水服务半径。
缺失几何时标记 `partial`,没有可用位置时不生成面。输入最多 8 组、合计 10000 个不同节点;
几何输出遵守覆盖层顶点和大小限制,超限要求缩小范围,不静默抽样或改变原分析结果。
MCP 保存完整查询响应作为不可变证据,展示解析器同时支持响应信封中的
`data.result.map_payload`。Agent 使用 `set_overlay` 引用本次 `evidence_id` 并省略 `data`。
超过内联返回预算时,响应的 `data.presentation.command` 仍保留可发布的展示命令,
`data.presentation.groups` 保留覆盖完整性计数,
完整几何留在证据中,不能把缩略返回误认为没有空间成果。调查按实际步骤同步发布展示:
地址得到明确坐标后缩放定位,取得候选资产后定位候选,检查影响节点后生成并展示覆盖示意,
形成或调整关阀方案后定位阀门,读取监测对照后展示点位及序列。
图表同样随结果返回发布:监测趋势与事件对照用折线图,可比候选评价或方案用柱状图,
水力及污染传播等模拟按实际返回的时刻、对象和指标画图。压力、流量、浓度等不同单位分图,
注明来源、范围与实测或模型属性;没有同口径基准时不计算或绘制改善量,缺测不补零。
过程图表通过 `open_chart` 提供 `chart_id/chart/evidence_refs`,每图最多100个横轴值、8组序列,
超限时分图或明确说明抽样、聚合范围;单值或无可比数据用文字或表格。
关键趋势与比较结论保留到报告中,最终报告的图表不替代过程发布。
业务 Skill 说明本步骤应展示什么,调查指令统一说明发布方法、非空几何检查、去重和失败处理。
同一阶段批量展示,数据或方案没有变化时不重复请求;不为地图动作增加无关业务分析。
报告的 `map_targets` 仅为手动定位入口。关键步骤的数据缺失或展示失败需记录具体限制,
没有浏览器连接也不阻止分析或报告收尾。
`GET /api/v1/operational-tasks/{task_id}/presentation-cursor` 在同一事务中读取
`task_id/execution_generation/active_run/event_cursor`,其中活跃 run 仅返回 `run_id/generation/status`。
浏览器打开 Dev 联动、切换任务或断线重连时,以此建立新基线,只消费之后的展示事件。
@@ -98,7 +137,7 @@ Server 校验当前 run、绑定会话、有效租约和幂等身份,记录 `o
实际 worker 曾成功投递指定调查,模型发布进展和报告,但因 service 账号的任务权限过滤
导致 `approved_data_scopes` 为空,误作数据不足结束。这不是业务数据确实不足。
用户确认开放规则后,任务 API 已移除该额外过滤,沿用账号已有的项目读取权限。
在线上下文返回七类来源,同一身份实际读取 SCADA 设备和外围合成工单成功;没有新增
在线上下文返回七类来源,同一身份实际读取 SCADA 设备和外围系统工单成功;没有新增
角色权限或放宽跨项目、审批和设备控制边界。旧委托路径保留兼容,已有权限无需委托凭据。
用户级 `tjwater-operational-worker.service` 已重新启用。旧报告保留,未自动重新调查,
不能把旧报告视为修复后的业务验收。任务创建时间下界和并发配置见

Some files were not shown because too many files have changed in this diff Show More