Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

out_azure_blob: add support for Azure Blob SAS authentication #8243

Merged
merged 1 commit into from
Jan 14, 2024

Conversation

RobinLin666
Copy link
Contributor

@RobinLin666 RobinLin666 commented Dec 3, 2023

Sometimes user cannot use shared key authentication. So, this patch support authenticate Azure Blob Storage with shared access signatures.

Here is the document about shared access signatures: https://learn.microsoft.com/en-us/azure/storage/common/storage-sas-overview

This patch add two configuration parameters for out_azure_blob plugin: auth_type and sas_token.

- auth_type, default value is "key", it can be "key" or "sas".
- sas_token, default value is NULL, it is the SAS token, it is required when auth_type is "sas".

Fixes #8230


Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

  • Example configuration file for the change
[INPUT]
    name tail
    path /workspaces/fluent-bit/build/test.log
    tag test.txt

[OUTPUT]
    name                  azure_blob
    match                 *
    account_name         my_account
    auth_type             sas
    sas_token             sv=2022-11-02&ss=bfqt&srt=sco&sp=rwdlacupyx&se=2023-12-04T15:08:48Z&st=2023-12-03T07:08:48Z&spr=https&sig=xxxxxx
    path                  test_fs/test_fluent_bit
    container_name        my_container
    auto_create_container on
    tls                   on
  • Debug log output from testing the change
[2023/12/03 13:02:01] [debug] [input:tail:tail.0] inode=15033, /workspaces/fluent-bit/build/test.log, events: IN_ATTRIB
[2023/12/03 13:02:01] [debug] [input chunk] update output instances with new chunk size diff=108, records=3, input=tail.0
[2023/12/03 13:02:01] [debug] [input:tail:tail.0] inode=15033, /workspaces/fluent-bit/build/test.log, events: IN_MODIFY
[2023/12/03 13:02:01] [debug] [task] created task=0xffff7c051b30 id=0 OK
[2023/12/03 13:02:02] [debug] [upstream] KA connection #40 to my_account.blob.core.windows.net:443 is connected
[2023/12/03 13:02:02] [debug] [http_client] not using http_proxy for header
[2023/12/03 13:02:03] [debug] [upstream] KA connection #40 to my_account.blob.core.windows.net:443 is now available
[2023/12/03 13:02:03] [debug] [upstream] KA connection #40 to my_account.blob.core.windows.net:443 has been assigned (recycled)
[2023/12/03 13:02:03] [debug] [http_client] not using http_proxy for header
[2023/12/03 13:02:03] [debug] [upstream] KA connection #40 to my_account.blob.core.windows.net:443 is now available
[2023/12/03 13:02:03] [ info] [output:azure_blob:azure_blob.0] blob not found: /my_container/test_fs/test_fluent_bit/test.txt?comp=appendblock&sv=2022-11-02&ss=bfqt&srt=sco&sp=rwdlacupyx&se=2023-12-13T15:08:48Z&st=2023-12-03T07:08:48Z&spr=https
[2023/12/03 13:02:03] [debug] [upstream] KA connection #40 to my_account.blob.core.windows.net:443 has been assigned (recycled)
[2023/12/03 13:02:03] [debug] [http_client] not using http_proxy for header
[2023/12/03 13:02:03] [ info] [output:azure_blob:azure_blob.0] blob created successfully: /my_container/test_fs/test_fluent_bit/test.txt?sv=2022-11-02&ss=bfqt&srt=sco&sp=rwdlacupyx&se=2023-12-13T15:08:48Z&st=2023-12-03T07:08:48Z&spr=https
[2023/12/03 13:02:03] [debug] [upstream] KA connection #40 to my_account.blob.core.windows.net:443 is now available
[2023/12/03 13:02:03] [debug] [upstream] KA connection #40 to my_account.blob.core.windows.net:443 has been assigned (recycled)
[2023/12/03 13:02:03] [debug] [http_client] not using http_proxy for header
[2023/12/03 13:02:03] [debug] [upstream] KA connection #40 to my_account.blob.core.windows.net:443 is now available
[2023/12/03 13:02:03] [ info] [output:azure_blob:azure_blob.0] content appended to blob successfully
[2023/12/03 13:02:03] [debug] [out flush] cb_destroy coro_id=0
[2023/12/03 13:02:03] [debug] [task] destroy task=0xffff7c051b30 (task_id=0)
  • Attached Valgrind output that shows no leaks or memory corruption was found
fluent-bit@008906c25bf5:/workspaces/fluent-bit/build$ valgrind ./bin/fluent-bit -c fluent-bit.conf
==50903== Memcheck, a memory error detector
==50903== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==50903== Using Valgrind-3.16.1 and LibVEX; rerun with -h for copyright info
==50903== Command: ./bin/fluent-bit -c fluent-bit.conf
==50903==
Fluent Bit v2.2.1
* Copyright (C) 2015-2023 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/12/03 13:33:34] [ info] Configuration:
[2023/12/03 13:33:34] [ info]  flush time     | 1.000000 seconds
[2023/12/03 13:33:34] [ info]  grace          | 5 seconds
[2023/12/03 13:33:34] [ info]  daemon         | 0
[2023/12/03 13:33:34] [ info] ___________
[2023/12/03 13:33:34] [ info]  inputs:
[2023/12/03 13:33:34] [ info]      tail
[2023/12/03 13:33:34] [ info] ___________
[2023/12/03 13:33:34] [ info]  filters:
[2023/12/03 13:33:34] [ info] ___________
[2023/12/03 13:33:34] [ info]  outputs:
[2023/12/03 13:33:34] [ info]      azure_blob.0
[2023/12/03 13:33:34] [ info] ___________
[2023/12/03 13:33:34] [ info]  collectors:
[2023/12/03 13:33:34] [ info] [fluent bit] version=2.2.1, commit=fad58c93b9, pid=50903
[2023/12/03 13:33:34] [debug] [engine] coroutine stack size: 196608 bytes (192.0K)
[2023/12/03 13:33:34] [ info] [storage] ver=1.5.1, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/12/03 13:33:34] [ info] [cmetrics] version=0.6.5
[2023/12/03 13:33:34] [ info] [ctraces ] version=0.3.1
[2023/12/03 13:33:34] [ info] [input:tail:tail.0] initializing
[2023/12/03 13:33:34] [ info] [input:tail:tail.0] storage_strategy='memory' (memory only)
[2023/12/03 13:33:34] [debug] [tail:tail.0] created event channels: read=21 write=22
[2023/12/03 13:33:34] [debug] [input:tail:tail.0] flb_tail_fs_inotify_init() initializing inotify tail input
[2023/12/03 13:33:34] [debug] [input:tail:tail.0] inotify watch fd=27
[2023/12/03 13:33:34] [debug] [input:tail:tail.0] scanning path /workspaces/fluent-bit/build/test.log
[2023/12/03 13:33:34] [debug] [input:tail:tail.0] inode=15033 with offset=815 appended as /workspaces/fluent-bit/build/test.log
[2023/12/03 13:33:34] [debug] [input:tail:tail.0] scan_glob add(): /workspaces/fluent-bit/build/test.log, inode 15033
[2023/12/03 13:33:34] [debug] [input:tail:tail.0] 1 new files found on path '/workspaces/fluent-bit/build/test.log'
[2023/12/03 13:33:34] [debug] [azure_blob:azure_blob.0] created event channels: read=29 write=30
[2023/12/03 13:33:34] [ info] [output:azure_blob:azure_blob.0] account_name=my_account, container_name=my_container, blob_type=blockblob, emulator_mode=no, endpoint=my_account.blob.core.windows.net, auth_type=sas
[2023/12/03 13:33:34] [ info] [sp] stream processor started
[2023/12/03 13:33:34] [debug] [input:tail:tail.0] inode=15033 file=/workspaces/fluent-bit/build/test.log promote to TAIL_EVENT
[2023/12/03 13:33:34] [ info] [input:tail:tail.0] inotify_fs_add(): inode=15033 watch_fd=1 name=/workspaces/fluent-bit/build/test.log
[2023/12/03 13:33:34] [debug] [input:tail:tail.0] [static files] processed 0b, done
[2023/12/03 13:33:42] [debug] [input:tail:tail.0] inode=15033, /workspaces/fluent-bit/build/test.log, events: IN_ATTRIB
[2023/12/03 13:33:42] [debug] [input chunk] update output instances with new chunk size diff=72, records=2, input=tail.0
[2023/12/03 13:33:42] [debug] [input:tail:tail.0] inode=15033, /workspaces/fluent-bit/build/test.log, events: IN_MODIFY
[2023/12/03 13:33:42] [debug] [task] created task=0x5363fe0 id=0 OK
[2023/12/03 13:33:44] [debug] [upstream] KA connection #40 to my_account.blob.core.windows.net:443 is connected
[2023/12/03 13:33:44] [debug] [http_client] not using http_proxy for header
[2023/12/03 13:33:44] [debug] [upstream] KA connection #40 to my_account.blob.core.windows.net:443 is now available
[2023/12/03 13:33:44] [debug] [upstream] KA connection #40 to my_account.blob.core.windows.net:443 has been assigned (recycled)
[2023/12/03 13:33:44] [debug] [http_client] not using http_proxy for header
[2023/12/03 13:33:44] [debug] [upstream] KA connection #40 to my_account.blob.core.windows.net:443 is now available
[2023/12/03 13:33:44] [ info] [output:azure_blob:azure_blob.0] content appended to blob successfully
[2023/12/03 13:33:44] [debug] [upstream] KA connection #40 to my_account.blob.core.windows.net:443 has been assigned (recycled)
[2023/12/03 13:33:44] [debug] [http_client] not using http_proxy for header
[2023/12/03 13:33:45] [ info] [output:azure_blob:azure_blob.0] blob id ZmxiLTE3MDE2MTA0MjQuNTI3My5pZA== committed successfully
[2023/12/03 13:33:45] [debug] [upstream] KA connection #40 to my_account.blob.core.windows.net:443 is now available
[2023/12/03 13:33:45] [debug] [out flush] cb_destroy coro_id=0
[2023/12/03 13:33:45] [debug] [task] destroy task=0x5363fe0 (task_id=0)

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

  • [N/A] Run local packaging test showing all targets (including any new ones) build.
  • [N/A ] Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

  • Documentation required for this feature

fluent/fluent-bit-docs#1265

Backporting

  • [N/A] Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Sometimes user cannot use shared key authentication. So, this patch support
authenticate Azure Blob Storage with shared access signatures.
Here is the document about shared access signatures:
https://learn.microsoft.com/en-us/azure/storage/common/storage-sas-overview

This patch add two configuration parameters for out_azure_blob plugin:
auth_type and sas_token.
    - auth_type, default value is "key", it can be "key" or "sas".
    - sas_token, default value is NULL, it is the SAS token,
    it is required when auth_type is "sas".

Signed-off-by: Yi Lin <[email protected]>
@Lyndon1994
Copy link

kindly ping

@RobinLin666
Copy link
Contributor Author

Hi Happy New Year! Can anyone help to review the PR? Thanks !

@edsiper edsiper merged commit 3cf8cba into fluent:master Jan 14, 2024
50 checks passed
@edsiper
Copy link
Member

edsiper commented Jan 14, 2024

thank you !

shaerpour pushed a commit to shaerpour/fluent-bit that referenced this pull request Jan 16, 2024
…#8243)

Sometimes user cannot use shared key authentication. So, this patch support
authenticate Azure Blob Storage with shared access signatures.
Here is the document about shared access signatures:
https://learn.microsoft.com/en-us/azure/storage/common/storage-sas-overview

This patch add two configuration parameters for out_azure_blob plugin:
auth_type and sas_token.
    - auth_type, default value is "key", it can be "key" or "sas".
    - sas_token, default value is NULL, it is the SAS token,
    it is required when auth_type is "sas".

Signed-off-by: Yi Lin <[email protected]>
shaerpour pushed a commit to shaerpour/fluent-bit that referenced this pull request Jan 16, 2024
…#8243)

Sometimes user cannot use shared key authentication. So, this patch support
authenticate Azure Blob Storage with shared access signatures.
Here is the document about shared access signatures:
https://learn.microsoft.com/en-us/azure/storage/common/storage-sas-overview

This patch add two configuration parameters for out_azure_blob plugin:
auth_type and sas_token.
    - auth_type, default value is "key", it can be "key" or "sas".
    - sas_token, default value is NULL, it is the SAS token,
    it is required when auth_type is "sas".

Signed-off-by: Yi Lin <[email protected]>
Signed-off-by: ahspw <[email protected]>
pwhelan pushed a commit that referenced this pull request Jan 16, 2024
Sometimes user cannot use shared key authentication. So, this patch support
authenticate Azure Blob Storage with shared access signatures.
Here is the document about shared access signatures:
https://learn.microsoft.com/en-us/azure/storage/common/storage-sas-overview

This patch add two configuration parameters for out_azure_blob plugin:
auth_type and sas_token.
    - auth_type, default value is "key", it can be "key" or "sas".
    - sas_token, default value is NULL, it is the SAS token,
    it is required when auth_type is "sas".

Signed-off-by: Yi Lin <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Azure Blob output should support shared access signatures
3 participants