Skip to main content
CSV AWS S3 refers to CSV (Comma-Separated Values) files stored in Amazon Simple Storage Service (S3), which is a scalable cloud storage service. This allows you to store and access CSV data files in the cloud, providing reliable and secure data storage with high availability.

​
1. Add your CSV AWS S3 access

  1. In the Destinations tab, click on the “Add destination” button located on the top right of your screen. Then, select the CSV (AWS S3) option from the list of connectors.
  2. Click Next and you’ll be prompted to add your access:

​
Required Configuration

  • Bucket name: The AWS S3 bucket name where you want to store the CSV files.
    • If using a Nekt-managed bucket (pattern: nekt-lakehouse-{company-name}), no additional authentication is required.
    • If using your own AWS S3 bucket, you must provide AWS credentials (see below).

​
AWS Authentication (Required for Custom Buckets)

If you’re using your own AWS S3 bucket (not a Nekt-managed bucket), you need to provide AWS credentials to authenticate.
  • Access Key ID: Your AWS access key ID for authentication. You can create this in the AWS IAM console.
  • Secret Access Key: Your AWS secret access key for authentication. Keep this secure and never share it publicly.
To create AWS credentials for accessing your S3 bucket:
  1. Log in to your AWS IAM Console
  2. Navigate to Users and select your user (or create a new one)
  3. Go to the Security credentials tab
  4. Click Create access key
  5. Select Application running outside AWS or Other as the use case
  6. Copy the Access Key ID and Secret Access Key (you won’t be able to see the secret again)
Required IAM Permissions:Your AWS user/role needs the following S3 permissions:
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::your-bucket-name",
        "arn:aws:s3:::your-bucket-name/*"
      ]
    }
  ]
}

​
File Configuration

  • Output folder path: Folder path where to store output files inside the S3 bucket. Leave empty if the file should be stored in the bucket’s root folder.
    • Example: data/exports or reports/daily
  • Stream name: A custom name for the stream that can be used to compose the file name.
  • File naming format: The format to be used to generate the output file names. Available variables:
    • {stream_name} - The custom stream name you defined
    • {datestamp} - Date in YYYY-MM-DD format
    • {timestamp} - Timestamp in YYYY-MM-DDTHHMMSS format
    • {batch_id} - Unique identifier for the batch
    • {batch_index} - Sequential index for the batch
    • Example: {stream_name}_{timestamp}_{batch_index}.csv
  • Maximum CSV file size (MB): (Optional, default: 1000 MB) Define a maximum size for each CSV file. If your input data exceeds this size, multiple files will be generated respecting the maximum size defined.
  1. Click Next.

​
2. Select your catalog data to send

  1. The next step is letting us know which data you want to export to S3. Select the layer and then the desired table.
    Tip: The table can be found more easily by typing its name.
  2. Click Next.

​
3. Configure your CSV (AWS S3) data destination

  1. Describe your destination for easy identification within your organization. You can inform things like what data it exports, to which team it belongs, etc.
  2. To define your Trigger, consider how frequently you need to have the data updated in S3 (every day, once a week, only at specific times, etc.).
  3. Click Done.

​
4. Check your new destination

  1. Once completed, you’ll receive confirmation that your new destination is set up!
  2. You can view your new destination on the Destinations page. The pipeline will run according to your configured trigger schedule. You can monitor it on the Destinations page to see its execution and completion.
  3. If needed, manually trigger the pipeline by clicking on the refresh icon. Once executed, your CSV files will be available in your S3 bucket at the path you specified.
You can access your exported CSV files directly in your S3 bucket using the AWS S3 Console or any S3-compatible tool.
Make sure your S3 bucket has the appropriate permissions and policies configured to prevent unauthorized access to your data.
If you encounter any issues, reach out to us via Slack, and we’ll gladly assist you!