Project Atlas AetherDev project hub
Atlas project production

ASCIP Sync Engine

Provide a deterministic, auditable way to keep district staff data aligned with ASCIP LMS by turning CSV extracts into a repeatable diff-and-apply workflow. Load config and mapping files, validate file paths, and establish API connectivity. Pull ASCIP reference data (groups, buckets, roles, timezones) to build local lookup maps. Read the district CSV, apply matching heuristics, and compute add/update/remove actions by group. Optionally run a dry run to generate reports without writes, then apply changes and emit audit/email reports.

Playbook available

Operational guidance is available for this system.

Open playbook
Type
System
Lifecycle
Active
Last touched
2025-11-17
Visibility
Public
Why it's showcased

Showcases a full diff-and-apply sync engine with mapping, dry-run, and audit reporting against a real-world LMS API.

Purpose

Provide a deterministic, auditable way to keep district staff data aligned with ASCIP LMS by turning CSV extracts into a repeatable diff-and-apply workflow.

Current state

Last touched: 2025-11-17. Functionality and completeness: Core sync, validation, and reporting flows are implemented and documented; additional tests are still desired.

Next step

Add baseline automated tests to cover critical flows; Add CI pipeline for build/test/lint; Document deployment/runtime environment (or add Dockerfile); Document interfaces (CLI flags, API endpoints, file formats); Add structured logging and basic health checks.

Interfaces

Inputs
  • District staff CSV, ASCIP LMS API, bucket/group mapping JSON
  • CSV (district staff extract)
  • mapping files (bucket/group/value IDs)
Outputs
  • ASCIP user/group updates, CSV reference exports, diff/audit reports
  • ASCIP state changes
  • CSV exports
  • diff reports
  • audit logs

Reality to Action trace

Reality Ingestion

Contributes in this stage.

Canonical Storage

Contributes in this stage.

Automation Engines

Contributes in this stage.

Human Interfaces

Not in scope.

Operational Adoption

Contributes in this stage.

Core workflow

  1. Load config, mappings, and validate file paths and credentials.
  2. Fetch ASCIP reference data (groups, buckets, roles, timezones) and build lookup maps.
  3. Load the district CSV and normalize identities (email/username matching).
  4. Fetch ASCIP people list plus per-person detail for bucket fields.
  5. Compute diff by group for adds, updates, and removals.
  6. Optionally run dry-run to produce diff reports without writes.
  7. Apply changes via API and enforce group memberships.
  8. Emit audit reports, exports, and optional email notifications.
  9. Record run artifacts and skip future runs when the input CSV is unchanged.

Data integrity and contracts

Canonical schema definitions

  • CSV staff extract schema with required identity and attribute columns.
  • Mapping JSON schema for bucket/group/value ID translation.
  • TOML config schema for API keys, base URL, file paths, and logging.
  • ASCIP API payload shapes for People, Groups, and Buckets.

Source of truth rules

  • District CSV is canonical for staff identity and attribute values.
  • Mapping files define authoritative translations to ASCIP IDs.
  • ASCIP is treated as target state and overwritten to match the CSV.
  • Dry-run outputs are used for approval before writes.

Data quality checks

  • Validate required CSV headers and non-empty identity fields.
  • Validate mapping IDs exist in ASCIP reference data.
  • Detect duplicate or conflicting email identities.
  • Flag records missing group or bucket mappings.
  • Skip sync when the CSV hash matches the last successful run.

Safe handling

  • Keep config.local.toml, CSVs, and mapping files out of git.
  • Restrict file permissions on shared servers and rotate API keys.
  • Avoid logging raw PII; redact sensitive fields in logs and reports.
  • Store audit artifacts in protected directories.

Downstream integration map

  • ASCIP People records.
  • ASCIP Groups and Buckets.
  • Audit reports for IT and data services.
  • Mapping maintenance workflows.

Artifacts

  • Sample CSV input
  • Mapping file sample
  • Diff report example
  • Log excerpts showing add/update/remove
  • Data flow diagram

Operational notes

Constraints and scars

  • ASCIP list endpoints omit some bucket fields, requiring per-person detail fetches.
  • HTTP 411 content-length behavior required a query-parameter refactor.
  • Group membership must be enforced per group call, adding extra API round trips.
  • List endpoint missing bucket fields forced per-person detail fetch.
  • HTTP 411 content-length requirements surfaced in query-param refactor.
  • Group membership must be enforced per group call.

Reliability posture

Failure modes and safe behavior: missing files or auth failures abort early; dry-run mode prevents writes. Idempotency / retries / batching behavior: diff-first design; detail fetches are retried once with backoff; run is skipped when input CSV is unchanged.

Observability

  • Logs: structured console logs; optional rotating log file via config
  • Metrics/health checks: validate-config emits a readiness report (tenant profile + connectivity checks)
  • optional log file
  • special-case exception logs
  • config validation report (optional)

Security and privacy

Handles staff PII and ASCIP credentials; keep config files and CSVs out of version control. Use local config overrides (e.g., config.local.toml) and restrict file permissions on shared servers. SMTP credentials and API keys should be rotated if exposure is suspected.

Dependencies

Upstream
  • ASCIP LMS API, optional SMTP relay for report delivery
  • Aeries/HR exports (source of truth)
Downstream
  • ASCIP People
  • ASCIP Groups (locations)
  • ASCIP Buckets (attributes)

Ownership

Owners

Josh Barton

Users

IT automation, data services, systems owner (you), Josh Barton (owner)

ASCIP-API

Architecture & Major Components

  • High-level diagram (text):

    • Config + mappings -> reference data fetch -> CSV diff engine -> apply actions -> reports/logs

  • Entry points: src/main.rs

  • Top-level folders: docs, example, src, output, logs, config templates

  • Key abstractions: config loader, HMAC auth signer, API clients, diff engine, report/email builder, validation health checks

Setup / Build / Run

  • Build system(s): Cargo.
  • Requires a TOML config file plus CSV and mapping JSON inputs; run in dry-run mode before enabling writes.
  • Detailed environment preparation and scheduling guidance live in docs/SETUP.md and docs/operations.md.

Related playbooks