Single file output from multiple source files with fluent-bit - amazon-s3

Question

We are using fluent-bit to capture multiple logs within a directory, do some basic parsing and filtering, and sending output to s3. Each source file seems to correspond to a separate output file in the bucket rather than a combined output.
Is it possible to send multiple input files to a single output file in fluent-bit, or is this simply how the buffer flush behavior works?
Here is our config for reference:
[SERVICE]
Daemon Off
Flush 1
Log_Level warn
Parsers_File parsers.conf
Parsers_File custom_parsers.conf
Health_Check Off
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_Port 2020
storage.path /tmp/fluentbit/
storage.max_chunks_up 128
[INPUT]
Name tail
Path /var/log/containers/*.log
multiline.parser docker, cri
Tag kube.*
storage.type filesystem
Mem_Buf_Limit 10MB
Buffer_Chunk_Size 2M
Buffer_Max_size 256M
Skip_Long_Lines On
Skip_Empty_Lines On
[FILTER]
Name kubernetes
Match kube.*
Merge_Log On
Keep_Log Off
Merge_Log_Key msg-json
K8S-Logging.Parser On
K8S-Logging.Exclude On
Cache_Use_Docker_Id On
[FILTER]
Name nest
Match kube.*
Operation lift
Nested_under kubernetes
Add_prefix kubernetes_
[FILTER]
Name nest
Match kube.*
Operation lift
Nested_under kubernetes_labels
Add_prefix kubernetes_labels_
[FILTER]
Name aws
Match *
imds_version v1
az true
ec2_instance_id true
ec2_instance_type true
private_ip true
account_id true
hostname true
vpc_id true
[OUTPUT]
Name s3
Match *
bucket <bucket name redacted>
region us-east-1
total_file_size 100M
upload_timeout 60s
use_put_object true
compression gzip
store_dir_limit_size 500m
s3_key_format /fluentbit/team/%Y.%m.%d.%H_%M_%S.$UUID.gz
static_file_path On

Answer 1

It is possible to send multiple input files to single output file.
The issue here might be with your use of the s3_key_format.
Your current file name format is '/fluentbit/team/%Y.%m.%d.%H_%M_%S.$UUID.gz' and this has a UUID which causes each input file being written to a separate output file in S3.
To combine and send to single output file, just modify it to '/fluentbit/team/%Y.%m.%d.gz'