rdsp-s3¶
rdsp-s3 is the official reference provider for the dataLOFT platform. It is written in Rust and implements the run-dsrpc gRPC provider interface, serving file data from an S3-compatible object store.
Purpose¶
rdsp-s3 demonstrates how to build a provider backend that integrates with RUN-DSP. It handles the three core provider responsibilities:
- Catalogue: Returns metadata about available files so that consumers can discover them.
- Dataset information: Returns detailed information about a specific dataset on request.
- Download instructions: Returns a signed URL or equivalent instruction that allows the consumer to retrieve the file.
Architecture¶
rdsp-s3 runs as two cooperating services:
Provider service: Implements the run-dsrpc gRPC interface. Responds to RUN-DSP's catalogue and dataset queries by reading from the metadata index.
Publisher service: Handles the actual file transfer. Generates download URLs for consumer pull requests and accepts upload URLs for push transfers.
Metadata¶
rdsp-s3 indexes files from the configured S3 bucket on a configurable refresh interval. For each file, it reads a JSON sidecar metadata file (.metadata) stored alongside the data file. The metadata file contains:
- File UUID
- Name and description
- Creation and modification timestamps
- File size and MIME type
- Owner information
- S3 path
Files without a sidecar are not indexed and are not visible in the catalogue.
Configuration¶
rdsp-s3 is configured via a TOML file or environment variables. Key settings include the S3 endpoint, credentials, bucket name, and the metadata refresh interval.