Data Management

COASTSim provides comprehensive data management capabilities for simulating onboard data storage, generation, and downlink operations. This system allows mission planners to evaluate:

  • Data recorder sizing requirements

  • Downlink pass frequency and duration needs

  • Data generation rates from science instruments

  • Storage capacity utilization over mission timelines

  • Alert thresholds for recorder fullness

Overview

The data management system consists of three main components:

  1. OnboardRecorder: Simulates the spacecraft’s solid-state recorder (SSR) or data storage device

  2. DataGeneration: Models how instruments produce data during observations

  3. Downlink Operations: Simulates data transfer during ground station passes

These components work together seamlessly in DITL simulations to track data flow throughout the spacecraft’s operations.

OnboardRecorder

The OnboardRecorder class simulates an onboard data storage device with configurable capacity and alert thresholds.

Configuration

from conops.config import OnboardRecorder

# Create a recorder with 128 Gigabit capacity
recorder = OnboardRecorder(
    name="Solid State Recorder",
    capacity_gb=128.0,          # Maximum capacity in Gigabits
    current_volume_gb=0.0,      # Starting empty
    yellow_threshold=0.7,       # Warning at 70% full
    red_threshold=0.9           # Critical at 90% full
)

Key Features

Capacity Management

The recorder automatically caps data storage at the configured capacity. When the recorder is full, additional data generation is lost (simulating data that cannot be stored).

# Add data to recorder
data_stored = recorder.add_data(10.5)  # Returns amount actually stored

# Check current state
print(f"Current volume: {recorder.current_volume_gb} Gb")
print(f"Fill fraction: {recorder.get_fill_fraction()}")
print(f"Available space: {recorder.available_capacity()} Gb")

Alert System

The recorder provides three alert levels based on fill fraction:

  • 0 (Green): Below yellow threshold - normal operations

  • 1 (Yellow): At or above yellow threshold - warning state

  • 2 (Red): At or above red threshold - critical state

alert_level = recorder.get_alert_level()
if alert_level == 2:
    print("CRITICAL: Recorder nearly full!")
elif alert_level == 1:
    print("WARNING: Recorder filling up")

Data Removal

During ground station passes, data is removed from the recorder to simulate downlink:

# Remove data during downlink
data_downlinked = recorder.remove_data(25.0)  # Returns amount actually removed
print(f"Downlinked {data_downlinked} Gb")

DataGeneration

The DataGeneration class models how instruments produce data. It supports two generation modes:

Rate-Based Generation

Instruments generate data continuously at a specified rate in Gigabits per second (Gbps). This is appropriate for instruments like cameras or spectrometers that produce continuous data streams.

from conops.config import DataGeneration, Instrument

# Camera generating 0.5 Gbps continuously
camera_data = DataGeneration(rate_gbps=0.5)

camera = Instrument(
    name="High-Resolution Camera",
    data_generation=camera_data
)

# Calculate data generated over 60 seconds
data = camera.data_generation.data_generated(60)  # Returns 30.0 Gb

Per-Observation Generation

Instruments generate a fixed amount of data per observation, regardless of duration. This is appropriate for instruments that capture discrete snapshots or measurements.

# Spectrometer generating 5 Gb per observation
spec_data = DataGeneration(per_observation_gb=5.0)

spectrometer = Instrument(
    name="X-ray Spectrometer",
    data_generation=spec_data
)

# Returns 5.0 Gb regardless of duration
data = spectrometer.data_generation.data_generated(60)

Payload-Level Data Rates

The Payload class aggregates data generation across all instruments:

from conops.config import Payload

payload = Payload(instruments=[camera, spectrometer])

# Get total data rate across all instruments
total_rate = payload.total_data_rate_gbps()  # 0.5 Gbps from camera

# Calculate total data generated over a period
total_data = payload.data_generated(60)  # Camera: 30 Gb + Spec: 5 Gb = 35 Gb

Integration with DITL

The data management system is automatically integrated into DITL simulations.

Configuration

Add the recorder to your spacecraft configuration:

from conops.config import MissionConfig

config = MissionConfig(
    name="Science Mission",
    spacecraft_bus=spacecraft_bus,
    solar_panel=solar_panel,
    payload=payload,
    battery=battery,
    constraint=constraint,
    ground_stations=ground_stations,
    recorder=recorder  # Add recorder here
)

Telemetry Output

During simulation, DITL automatically records data management telemetry in structured format:

from conops.ditl import QueueDITL
from conops import MissionConfig

config = MissionConfig()
ditl = QueueDITL(config=config)
ditl.calc()

# Access data management telemetry through structured telemetry system
telemetry = ditl.telemetry

# Housekeeping data (recorded at each timestep)
recorder_volume = telemetry.housekeeping.recorder_volume_gb      # Current volume at each timestep
fill_fraction = telemetry.housekeeping.recorder_fill_fraction    # Fill level (0-1) at each timestep
alert_level = telemetry.housekeeping.recorder_alert              # Alert level (0/1/2) at each timestep

# Payload data (recorded when data is generated)
data_events = telemetry.data  # List of PayloadData records
data_sizes = [pd.data_size_gb for pd in data_events]

For backward compatibility, the legacy array access is still available:

# Legacy access (still supported)
recorder_volume = ditl.recorder_volume_gb
fill_fraction = ditl.recorder_fill_fraction
alert_level = ditl.recorder_alert
data_generated = ditl.data_generated_gb
data_downlinked = ditl.data_downlinked_gb

Visualization Example

import matplotlib.pyplot as plt
from conops.visualization import plot_data_management_telemetry

# Use the built-in plotting function
fig, axes = plot_data_management_telemetry(ditl)
plt.show()

# Or create custom plots using telemetry data
fig, axes = plt.subplots(4, 1, figsize=(12, 10), sharex=True)

# Get timestamps for plotting
times = telemetry.housekeeping.timestamp

# Recorder volume
axes[0].plot(times, telemetry.housekeeping.recorder_volume_gb, 'b-')
axes[0].set_ylabel('Volume (Gb)')
axes[0].set_title('Onboard Recorder Status')
axes[0].grid(True)

# Fill fraction with thresholds
axes[1].plot(times, telemetry.housekeeping.recorder_fill_fraction, 'g-', label='Fill Level')
axes[1].axhline(y=0.7, color='orange', linestyle='--', label='Yellow Threshold')
axes[1].axhline(y=0.9, color='red', linestyle='--', label='Red Threshold')
axes[1].set_ylabel('Fill Fraction')
axes[1].legend()
axes[1].grid(True)

# Data generation events
if telemetry.data:
    data_times = [pd.timestamp for pd in telemetry.data]
    data_sizes = [pd.data_size_gb for pd in telemetry.data]
    axes[2].bar(data_times, data_sizes, width=0.01, alpha=0.7, color='purple')
    axes[2].set_ylabel('Generated (Gb)')
    axes[2].set_title('Data Generation Events')
    axes[2].grid(True)

# Alert levels
colors = ['green', 'orange', 'red']  # 0=normal, 1=warning, 2=critical
for i, alert in enumerate(telemetry.housekeeping.recorder_alert):
    if alert > 0:  # Only plot non-zero alerts
        axes[3].scatter(times[i], alert, c=colors[alert], s=20, alpha=0.8)
axes[3].set_ylabel('Alert Level')
axes[3].set_title('Recorder Alerts')
axes[3].set_yticks([0, 1, 2])
axes[3].set_yticklabels(['Normal', 'Warning', 'Critical'])
axes[3].grid(True)

plt.tight_layout()
plt.show()

Fault Management Integration

The recorder’s fill fraction is automatically monitored by the fault management system when configured.

Configuration

from conops.config import FaultManagement

fault_mgmt = FaultManagement()

config = MissionConfig(
    # ... other config ...
    recorder=recorder,
    fault_management=fault_mgmt
)

# Initialize fault management with default thresholds
config.init_fault_management_defaults()

This automatically adds a recorder_fill_fraction threshold using the recorder’s yellow and red threshold values. The fault management system will:

  • Monitor the fill fraction at each timestep

  • Track state transitions (GREEN → YELLOW → RED)

  • Log threshold violations

  • Optionally trigger safe mode on RED alerts (if configured)

Manual Threshold Configuration

You can also manually configure recorder thresholds:

fault_mgmt.add_threshold(
    name="recorder_fill_fraction",
    yellow=0.75,      # Custom yellow threshold
    red=0.95,         # Custom red threshold
    direction="above"  # Alert when value goes above thresholds
)

Use Cases and Examples

Sizing the Onboard Recorder

Determine the minimum recorder capacity needed for your mission:

# Run simulation with different recorder sizes
capacities = [32, 64, 128, 256]  # Gigabits

for capacity in capacities:
    recorder = OnboardRecorder(capacity_gb=capacity)
    config.recorder = recorder

    ditl = DITL(config=config)
    ditl.calc()

    max_fill = max(ditl.recorder_fill_fraction)
    print(f"Capacity: {capacity} Gb, Max Fill: {max_fill:.1%}")

    if max_fill < 0.9:
        print(f"✓ {capacity} Gb is sufficient")
        break

Optimizing Observation Schedules

Balance science observations with data downlink availability:

# Analyze periods when recorder is full
full_periods = []
for i, fill in enumerate(ditl.recorder_fill_fraction):
    if fill >= 1.0:
        full_periods.append(ditl.utime[i])

if full_periods:
    print(f"WARNING: Recorder full for {len(full_periods)} timesteps")
    print("Consider:")
    print("  - Adding more ground station passes")
    print("  - Increasing downlink rate")
    print("  - Reducing instrument data rates")
    print("  - Increasing recorder capacity")

Best Practices

  1. Start with Conservative Estimates

    Begin with generous recorder capacity and downlink capabilities, then optimize based on simulation results.

  2. Monitor Alert Levels

    Pay attention to yellow and red alerts during simulation. Persistent red alerts indicate insufficient downlink capability or excessive data generation.

  3. Account for Growth

    Size your recorder with margin for:

    • Mission lifetime degradation

    • Potential instrument mode additions

    • Missed passes due to weather or station outages

  4. Balance Generation and Downlink

    The ideal scenario maintains recorder fill between 30-70%, providing buffer for both data accumulation and unexpected gaps in downlink opportunities.

  5. Validate with Different Scenarios

    Test your configuration with:

    • Best case: All passes succeed

    • Worst case: 20-30% of passes fail

    • Realistic case: Mix of successful and failed passes

Troubleshooting

Recorder Frequently Full

Symptoms: Red alerts, data loss, fill fraction at 100%

Solutions:

  • Increase recorder capacity

  • Add more ground station passes

  • Increase antenna downlink rate

  • Reduce instrument data generation rates

  • Adjust observation scheduling to prioritize downlink windows

Recorder Always Empty

Symptoms: Fill fraction near 0%, unused capacity

Solutions:

  • Reduce recorder capacity (weight/cost savings)

  • Reduce number of ground station contacts

  • Consider lower downlink rates

  • Optimize pass scheduling

API Reference

For detailed API documentation, see:

  • OnboardRecorder

  • DataGeneration

  • Instrument

  • Payload

  • Antenna

  • DITL

  • QueueDITL