Ceph RADOS Gateway Charm

centralized logging is missing due to rgw_enable_ops_log

Bug #2022065 reported by Nobuto Murata
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ceph RADOS Gateway Charm
New
Undecided
Unassigned

Bug Description

Ceph radosgw has a mechanism to log operations into Ceph itself. The handy feature is not enabled by default so it would be nice if the charm enables it for auditing purposes.

Currently GET operation for example is logged at each radosgw unit.
e.g. /var/log/ceph/ceph-client.rgw.juju-d642a4-0-lxd-1.log

However, the equivalent log cannot be retrieved via the Ceph embedded feature.

# radosgw-admin log list | head
[
 "obj_delete_at_hint.0000000078",
 "obj_delete_at_hint.0000000070",
 "obj_delete_at_hint.0000000104",
 "obj_delete_at_hint.0000000026",
 "obj_delete_at_hint.0000000028",
 "obj_delete_at_hint.0000000040",
 "obj_delete_at_hint.0000000015",
 "obj_delete_at_hint.0000000069",
 "obj_delete_at_hint.0000000095",

# radosgw-admin log show --bucket simplestreams \
    --bucket-id 52b9a1c1-1c52-42b8-85a8-89df0fbcc471.6774.1 \
    --date 2023-06-01-12
error reading log 2023-06-01-12-52b9a1c1-1c52-42b8-85a8-89df0fbcc471.6774.1-simplestreams: (2) No such file or directory

By enabling the feature by hand, it allows to see the log through the radsgw-admin interface.
https://docs.ceph.com/en/quincy/radosgw/config-ref/#confval-rgw_enable_ops_log

$ juju config ceph-radosgw config-flags='{"global":{"rgw enable ops log": true}}'

$ curl -v https://$HOSTNAME/simplestreams/foo

# radosgw-admin log show --bucket simplestreams \
    --bucket-id 52b9a1c1-1c52-42b8-85a8-89df0fbcc471.6774.1 \
    --date 2023-06-01-12 | head
{
 "bucket_id": "52b9a1c1-1c52-42b8-85a8-89df0fbcc471.6774.1",
 "bucket_owner": "1515718f5892428fa97d4641243848ed",
 "bucket": "simplestreams",
 "log_entries": [
     {
         "bucket": "simplestreams",
         "time": "2023-06-01T12:21:17.098687Z",
         "time_local": "2023-06-01T12:21:17.098687+0000",
         "remote_addr": "127.0.0.1",

# radosgw-admin log show --bucket simplestreams \
    --bucket-id 52b9a1c1-1c52-42b8-85a8-89df0fbcc471.6774.1 \
    --date 2023-06-01-12 | grep foo
         "uri": "GET /simplestreams/foo HTTP/1.1",

Revision history for this message
Nobuto Murata (nobuto) wrote :

We should additionally set either "rgw remote addr param" or "rgw log http headers" with "HTTP_X_FORWARDED_FOR" otherwise the remote ip is always 127.0.0.1.

$ juju config ceph-radosgw \
    config-flags='{"global":{"rgw enable ops log": true, "rgw remote addr param": "HTTP_X_FORWARDED_FOR"}}'

        {
            "bucket": "simplestreams",
            "time": "2023-06-01T13:10:00.060561Z",
            "time_local": "2023-06-01T13:10:00.060561+0000",
            "remote_addr": "192.168.151.107",
            "object_owner": "1515718f5892428fa97d4641243848ed",
            "user": "anonymous",
            "operation": "get_obj",
            "uri": "GET /simplestreams/foo HTTP/1.1",
            "http_status": "200",
            "error_code": "",
            "bytes_sent": 0,
            "bytes_received": 0,
            "object_size": 0,
            "total_time": 4,
            "user_agent": "curl/7.81.0",
            "referrer": "",
            "trans_id": "tx00000ff5efd6ded2863e0-00647898a8-9207-default",
            "authentication_type": "Local",
            "temp_url": false
        },

$ juju config ceph-radosgw \
    config-flags='{"global":{"rgw enable ops log": true, "rgw log http headers": "http_x_forwarded_for"}}'

        {
            "bucket": "simplestreams",
            "time": "2023-06-01T12:44:27.316514Z",
            "time_local": "2023-06-01T12:44:27.316514+0000",
            "remote_addr": "127.0.0.1",
            "object_owner": "1515718f5892428fa97d4641243848ed",
            "user": "anonymous",
            "operation": "get_obj",
            "uri": "GET /simplestreams/foo HTTP/1.1",
            "http_status": "200",
            "error_code": "",
            "bytes_sent": 0,
            "bytes_received": 0,
            "object_size": 0,
            "total_time": 8,
            "user_agent": "curl/7.81.0",
            "referrer": "",
            "http_x_headers": [
                {
                    "HTTP_X_FORWARDED_FOR": "192.168.151.107"
                }
            ],
            "trans_id": "tx000002dcd7463dff3832f-00647892ab-8c1f-default",
            "authentication_type": "Local",
            "temp_url": false
        }

Revision history for this message
Nobuto Murata (nobuto) wrote :

"good-first-bug" to have a config option as "enable-ops-log" or something like that.

tags: added: good-first-bug
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.