Writing Data

Learn how to write data to channels in Synnax, including real-time streaming and historical writes.

This guide covers writing data to Synnax channels. For live data acquisition, using a Writer is the recommended approach. Historical writes are useful for backfilling data or ingesting data from files.

If you’d like a conceptual overview of how writes work in Synnax, check out the writes concepts guide. The rules of writes are especially important to understand.

Writing with Writers

Writers are designed for streaming data as it’s acquired. This is the recommended approach for live data acquisition, control sequences, and real-time data processing. Writers maintain a file-like interface governed by transactions. To learn more about transactions and how writes work in Synnax, see the concepts page.

Opening a Writer

To open a writer, use the open_writer method with a starting timestamp and a list of channels to write to.

These examples write 100 samples to the temperature channel, each spaced roughly 100ms apart, and commit all writes when finished. It’s typical to write and commit millions of samples over the course of hours or days, intermittently calling commit to persist data to the cluster.

Persistence/Streaming Mode

By default, writers are opened in stream + persist mode. To change the mode of a writer, specify the mode argument when opening the writer. The available modes are:

• persist - Only persist data to the database (no streaming to subscribers)
• stream - Only stream data to subscribers (no persistence)
• persist_stream - Both persist and stream (default)

For example, to open a writer that only persists data:

Writing Data

The write method accepts several argument formats. Use the one that best fits your use case.

Writing Variable-Length Data

Strings, JSON, and bytes are written through the same writer API as fixed-width samples. The channel’s data type determines how the writer encodes each value.

JSON channels accept and return native objects. Bytes channels accept any Python bytes value or TypeScript Uint8Array. Variable-length channels can share a writer with fixed-width channels and a common index, so a single write call can commit timestamps, numeric samples, and string or JSON annotations together.

Closing a Writer

After you’re done writing, it’s essential to close the writer to release network connections and other resources. If a writer is not closed, other writers may not be able to write to the same channels.

Using structured cleanup patterns ensures the writer is always closed, even if an exception is thrown.

Historical Writes

Historical writes are useful for backfilling data, ingesting data from files, or writing data at specific timestamps that have already passed.

Writing to a Channel

Index and Data Alignment

Notice how the two arrays are aligned using the common start timestamp. This tells Synnax that the first sample in the temperatures array is associated with the first timestamp in the timestamps array.

Synnax will raise a ValidationError if the index channel does not contain a corresponding timestamp for every sample in the data channel. After all, it wouldn’t make sense to have a temperature reading without an associated timestamp.

Writing Multiple Data Channels

It’s common to have multiple data channels that share the same index. For example, a weather station might record temperature, humidity, and pressure all at the same timestamps.

Single Time Index

When writing a single sample to multiple channels at the same instant, use a dictionary with scalar values.

Multiple Time Indices

When writing multiple samples over time, use arrays for both the timestamps and data values.

All data channels use the same start timestamp for alignment, which tells Synnax they share the same index. The arrays must all have the same length as the timestamps array.

Common Pitfalls

There are several common pitfalls to avoid when writing data to Synnax. These can lead to performance degradation and/or control issues.

Using Many Individual Write Calls Instead of a Writer

When writing large volumes of data in a streaming fashion (or in batches), use a writer instead of making individual write calls to a channel. Calls to write on a channel use an entirely new transaction for each call - constantly creating, committing, and closing transactions has a dramatic impact on performance.

Avoid this pattern - Writing directly to channels in a loop:

Also avoid this pattern - Opening and closing a writer for every write:

Recommended approach - Repeatedly call write on a single writer: