openstack rocky 版本 多后端存储支持 保存镜像失败

[firejoke]

官方文档Multi Store Support

官方说在 rocky 版本是作为实验功能引入,

参考后的配置(/etc/glance/glance-api.conf)

[DEFAULT]
...
enabled_backends = reliable:file, fast:rbd
...

[glance_store]
default_backend = reliable

[reliable]
filesystem_store_datadir = /opt/stack/data/glance/images/
store_description = "Reliable filesystem backend"

[fast]
store_description = "Fast rbd backend"
rbd_store_pool = images
rbd_store_user = glance
rbd_store_ceph_conf = /etc/ceph/ceph.conf
rbd_store_chunk_size = 8
rados_connect_timeout = 0

[os_staging]
filesystem_store_datadir = /opt/stack/data/glance/os_staging/
store_description = "Filesystem store for staging purpose"

源码内判断后端存储位置的代码(glance/glance/api/v2/image_data.py class ImageDataController)

...
...
    @utils.mutating
    def upload(self, req, image_id, data, size):
        backend = None
        if CONF.enabled_backends:
            backend = req.headers.get('x-image-meta-store',
                                      CONF.glance_store.default_backend)

            try:
                LOG.debug(" = = = = = = backend: %s = = = = = = " % backend)
                glance_store.get_store_from_store_identifier(backend)
                LOG.debug("glance_store.backend %s" % glance_store.backend)
            except glance_store.UnknownScheme as exc:
                raise webob.exc.HTTPBadRequest(explanation=exc.msg,
                                               request=req,
                                               content_type='text/plain')

        image_repo = self.gateway.get_repo(req.context)
        image = None
        refresher = None
        cxt = req.context
        try:
            self.policy.enforce(cxt, 'upload_image', {})
            image = image_repo.get(image_id)
            image.status = 'saving'
            try:
                if CONF.data_api == 'glance.db.registry.api':
                    # create a trust if backend is registry
                    try:
                        # request user plugin for current token
                        user_plugin = req.environ.get('keystone.token_auth')
                        roles = []
                        # use roles from request environment because they
                        # are not transformed to lower-case unlike cxt.roles
                        for role_info in req.environ.get(
                                'keystone.token_info')['token']['roles']:
                            roles.append(role_info['name'])
                        refresher = trust_auth.TokenRefresher(user_plugin,
                                                              cxt.tenant,
                                                              roles)
                    except Exception as e:
                        LOG.info(_LI("Unable to create trust: %s "
                                     "Use the existing user token."),
                                 encodeutils.exception_to_unicode(e))

                image_repo.save(image, from_state='queued')
                image.set_data(data, size, backend=backend)

                try:
                    image_repo.save(image, from_state='saving')
                except exception.NotAuthenticated:
                    if refresher is not None:
                        # request a new token to update an image in database
                        cxt.auth_token = refresher.refresh_token()
                        image_repo = self.gateway.get_repo(req.context)
                        image_repo.save(image, from_state='saving')
                    else:
                        raise

                try:
                    # release resources required for re-auth
                    if refresher is not None:
                        refresher.release_resources()
                except Exception as e:
                    LOG.info(_LI("Unable to delete trust %(trust)s: %(msg)s"),
                             {"trust": refresher.trust_id,
                              "msg": encodeutils.exception_to_unicode(e)})
...
...

目前请求头并没有 x-image-meta-store 属性,直接按默认的 reliable 后端配置来

相关日志:

12 月 24 15:59:25 controller [email protected][6524]: DEBUG glance.api.v2.image_data [None req-081110c4-8d29-4567-a4fb-bbbc7313efbf alt_demo admin]  = = = = = = backend: reliable = = = = = =  {{(pid=6527) upload /opt/stack/glance/glance/api/v2/image_data.py:127}}
12 月 24 15:59:25 controller [email protected][6524]: DEBUG glance.api.v2.image_data [None req-081110c4-8d29-4567-a4fb-bbbc7313efbf alt_demo admin] glance_store.backend <module 'glance_store.backend' from '/usr/lib/python2.7/site-packages/glance_store/backend.pyc'> {{(pid=6527) upload /opt/stack/glance/glance/api/v2/image_data.py:129}}
......
12 月 24 15:59:26 controller [email protected][6524]: DEBUG glance_store._drivers.filesystem [None req-081110c4-8d29-4567-a4fb-bbbc7313efbf alt_demo admin] Wrote 0 bytes to /opt/stack/data/glance/images/9da337be-bb04-43af-ac1d-b9e5a15a0c70 with checksum d41d8cd98f00b204e9800998ecf8427e and multihash cf83e1357eefb8bdf1542850d66d8007d620e4050b5715dc83f4a921d36ce9ce47d0d13c5d85f2b0ff8318d2877eec2f63b931bd47417a81a538327af927da3e {{(pid=6527) add /usr/lib/python2.7/site-packages/glance_store/_drivers/filesystem.py:728}}
12 月 24 15:59:27 controller [email protected][6524]: DEBUG glance_store.multi_backend [None req-081110c4-8d29-4567-a4fb-bbbc7313efbf alt_demo admin] Skipping store.set_acls... not implemented {{(pid=6527) set_acls_for_multi_store /usr/lib/python2.7/site-packages/glance_store/multi_backend.py:439}}
......

在 http 上传缓存目录能找到上传成功的文件,但在从缓存目录写入到 image 目录时,但写入数据就变成了 Wrote 0 bytes

可能是最近比较浮躁

一直跑流程也还是找不到为什么

请求能有大大指点一二

[firejoke]

回滚后测试,已经确定不是代码层的问题

[firejoke]

定位到了最终问题,是在
def upload(self, req, image_id, data, size):
backend = None
if CONF.enabled_backends:
下面打了个日志记录了整个 req,结果就导致写入数据 0
具体原因不明