Project Atlas AetherDev project hub
Atlas project production

BOUSD-AeriesDataExportFormatter-Extract

Shell-orchestrated Aeries extract workflow that runs configured SQL to CSV and synchronizes results to a Google Sheet range used for downstream formatting/reporting.

Internal-only entry. Do not publish externally without review.
Type
Field Tool
Lifecycle
Active
Last touched
2026-03-26
Visibility
Internal

Purpose

Maintain a focused extract-and-sheet-sync workflow for Aeries data that downstream users consume in spreadsheet form.

Current state

Implementation details are concrete: `run.sh` executes `fetch_and_upload "staff"`, reads `conf.d/staff.toml`, writes `data/staff.csv`, and overwrites `SourceData-staff!A1:ZZ10000` in the configured sheet. This repo is retained as adjacent development/production-adjacent evidence; current scheduler and escalation ownership are still unresolved in Atlas.

Next step

Confirm whether runtime is manual-only or scheduled, identify host/owner/escalation contacts, and reconcile whether this workflow should stay adjacent or join the broader production extract-family inventory.

Interfaces

Inputs
  • SQL queries in sql/
  • Extract config in conf.d/*.toml
  • Aeries SQL source data
Outputs
  • `data/staff.csv`
  • `temp/staff.csv`
  • `SourceData-staff!A1:ZZ10000` sheet range

Reality to Action trace

Reality Ingestion

Contributes in this stage.

Canonical Storage

Not in scope.

Automation Engines

Not in scope.

Human Interfaces

Contributes in this stage.

Operational Adoption

Not in scope.

Core workflow

TBD. Document the 5-10 steps that define the core workflow.

Artifacts

  • data/*.csv
  • temp/*.csv
  • sql/staff.sql
  • conf.d/staff.toml

Operational notes

Constraints and scars

  • Sheet tab/range contracts must stay aligned with query output shape.
  • ODBC and credential setup remain environment-sensitive.

Reliability posture

Single-extract deterministic pipeline through `run.sh` and helper binaries. Failures surface through script exit/log output; explicit retry/alert orchestration is not documented in-repo.

Observability

  • stdout/stderr from run.sh
  • Helper tool output
  • Generated `data/staff.csv` and `temp/staff.csv` artifacts

Security and privacy

Handles staff-related operational data and credentialed integrations. Keep DB/sheet credentials out of source and treat generated files as restricted.

Dependencies

Upstream
  • Aeries SQL Server via ODBC
  • Google Sheets API
Downstream
  • Spreadsheet-based formatting and reporting workflows

Ownership

Owners

Josh Barton

Users

Technology Services

BOUSD-AeriesDataExportFormatter-Extract

Implementation Notes

  • run.sh is the workflow boundary.
  • fetch_and_upload composes extract-specific config, SQL output paths, and destination sheet ranges.
  • Current configured extract is staff only (conf.d/staff.toml + sql/staff.sql).
  • Destination contract in code is SourceData-staff!A1:ZZ10000.
  • Bundled helpers in bin/ perform SQL export and sheet synchronization.

Validation Needed

  • Confirm active runtime trigger: scheduled host run versus operator-invoked/manual run
  • Confirm named owner and escalation route for failed syncs