[rhos-list] Gluster UFO 3.4 swift Multi tenant question

Tue Sep 17 15:13:09 UTC 2013

Luis
well thats intresting because it was my impression that Gluster UFO
3.4 was based on the Grizzly version of Swift.
Also I was previously unaware of this new rpm which doesnt seem to be
in a repo any where.

also there is a line in this new howto that is extreamly unclear

"
/usr/bin/gluster-swift-gen-builders test
"
in place of "test" what should go there is it the tenant ID string,
the tenant name, or just a generic volume you can name whatever you
want?
in other words how should the Gluster volumes be named?

On Tue, Sep 17, 2013 at 10:10 AM, Luis Pabon <lpabon at redhat.com> wrote:
> First thing I can see is that you have Essex based gluster-ufo-* which has
> been replaced by the gluster-swift project.  We are currently in progress of
> replacing the gluster-ufo-* with RPMs from the gluster-swift project in
> Fedora.
>
> Please checkout the following quickstart guide which show how to download
> the Grizzly version of gluster-swift:
> https://github.com/gluster/gluster-swift/blob/master/doc/markdown/quick_start_guide.md
> .
>
> For more information please visit: https://launchpad.net/gluster-swift
>
> - Luis
>
>
> On 09/16/2013 05:02 PM, Paul Robert Marino wrote:
>
> Sorry for the delay on reporting the details. I got temporarily pulled
> off the project and dedicated to a different project which was
> considered higher priority by my employer. I'm just getting back to
> doing my normal work today.
>
> first here are the rpms I have installed
> "
>  rpm -qa |grep -P -i '(gluster|swift)'
> glusterfs-libs-3.4.0-8.el6.x86_64
> glusterfs-server-3.4.0-8.el6.x86_64
> openstack-swift-plugin-swift3-1.0.0-0.20120711git.el6.noarch
> openstack-swift-proxy-1.8.0-2.el6.noarch
> glusterfs-3.4.0-8.el6.x86_64
> glusterfs-cli-3.4.0-8.el6.x86_64
> glusterfs-geo-replication-3.4.0-8.el6.x86_64
> glusterfs-api-3.4.0-8.el6.x86_64
> openstack-swift-1.8.0-2.el6.noarch
> openstack-swift-container-1.8.0-2.el6.noarch
> openstack-swift-object-1.8.0-2.el6.noarch
> glusterfs-fuse-3.4.0-8.el6.x86_64
> glusterfs-rdma-3.4.0-8.el6.x86_64
> openstack-swift-account-1.8.0-2.el6.noarch
> glusterfs-ufo-3.4.0-8.el6.noarch
> glusterfs-vim-3.2.7-1.el6.x86_64
> python-swiftclient-1.4.0-1.el6.noarch
>
> here are some key config files note I've changed the passwords I'm
> using and hostnames
> "
>  cat /etc/swift/account-server.conf
> [DEFAULT]
> mount_check = true
> bind_port = 6012
> user = root
> log_facility = LOG_LOCAL2
> devices = /swift/tenants/
>
> [pipeline:main]
> pipeline = account-server
>
> [app:account-server]
> use = egg:gluster_swift_ufo#account
> log_name = account-server
> log_level = DEBUG
> log_requests = true
>
> [account-replicator]
> vm_test_mode = yes
>
> [account-auditor]
>
> [account-reaper]
>
> "
>
> "
>  cat /etc/swift/container-server.conf
> [DEFAULT]
> devices = /swift/tenants/
> mount_check = true
> bind_port = 6011
> user = root
> log_facility = LOG_LOCAL2
>
> [pipeline:main]
> pipeline = container-server
>
> [app:container-server]
> use = egg:gluster_swift_ufo#container
>
> [container-replicator]
> vm_test_mode = yes
>
> [container-updater]
>
> [container-auditor]
>
> [container-sync]
> "
>
> "
>  cat /etc/swift/object-server.conf
> [DEFAULT]
> mount_check = true
> bind_port = 6010
> user = root
> log_facility = LOG_LOCAL2
> devices = /swift/tenants/
>
> [pipeline:main]
> pipeline = object-server
>
> [app:object-server]
> use = egg:gluster_swift_ufo#object
>
> [object-replicator]
> vm_test_mode = yes
>
> [object-updater]
>
> [object-auditor]
> "
>
> "
> cat /etc/swift/proxy-server.conf
> [DEFAULT]
> bind_port = 8080
> user = root
> log_facility = LOG_LOCAL1
> log_name = swift
> log_level = DEBUG
> log_headers = True
>
> [pipeline:main]
> pipeline = healthcheck cache authtoken keystone proxy-server
>
> [app:proxy-server]
> use = egg:gluster_swift_ufo#proxy
> allow_account_management = true
> account_autocreate = true
>
> [filter:tempauth]
> use = egg:swift#tempauth
> # Here you need to add users explicitly. See the OpenStack Swift Deployment
> # Guide for more information. The user and user64 directives take the
> # following form:
> #     user_<account>_<username> = <key> [group] [group] [...] [storage_url]
> #     user64_<account_b64>_<username_b64> = <key> [group] [group]
> [...] [storage_url]
> # Where you use user64 for accounts and/or usernames that include
> underscores.
> #
> # NOTE (and WARNING): The account name must match the device name specified
> # when generating the account, container, and object build rings.
> #
> # E.g.
> #     user_ufo0_admin = abc123 .admin
>
> [filter:healthcheck]
> use = egg:swift#healthcheck
>
> [filter:cache]
> use = egg:swift#memcache
>
>
> [filter:keystone]
> use = egg:swift#keystoneauth
> #paste.filter_factory = keystone.middleware.swift_auth:filter_factory
> operator_roles = Member,admin,swiftoperator
>
>
> [filter:authtoken]
> paste.filter_factory = keystone.middleware.auth_token:filter_factory
> auth_host = keystone01.vip.my.net
> auth_port = 35357
> auth_protocol = http
> admin_user = swift
> admin_password = PASSWORD
> admin_tenant_name = service
> signing_dir = /var/cache/swift
> service_port = 5000
> service_host = keystone01.vip.my.net
>
> [filter:swiftauth]
> use = egg:keystone#swiftauth
> auth_host = keystone01.vip.my.net
> auth_port = 35357
> auth_protocol = http
> keystone_url = https://keystone01.vip.my.net:5000/v2.0
> admin_user = swift
> admin_password = PASSWORD
> admin_tenant_name = service
> signing_dir = /var/cache/swift
> keystone_swift_operator_roles = Member,admin,swiftoperator
> keystone_tenant_user_admin = true
>
> [filter:catch_errors]
> use = egg:swift#catch_errors
> "
>
> "
> cat /etc/swift/swift.conf
> [DEFAULT]
>
>
> [swift-hash]
> # random unique string that can never change (DO NOT LOSE)
> swift_hash_path_suffix = gluster
> #3d60c9458bb77abe
>
>
> # The swift-constraints section sets the basic constraints on data
> # saved in the swift cluster.
>
> [swift-constraints]
>
> # max_file_size is the largest "normal" object that can be saved in
> # the cluster. This is also the limit on the size of each segment of
> # a "large" object when using the large object manifest support.
> # This value is set in bytes. Setting it to lower than 1MiB will cause
> # some tests to fail. It is STRONGLY recommended to leave this value at
> # the default (5 * 2**30 + 2).
>
> # FIXME: Really? Gluster can handle a 2^64 sized file? And can the fronting
> # web service handle such a size? I think with UFO, we need to keep with the
> # default size from Swift and encourage users to research what size their
> web
> # services infrastructure can handle.
>
> max_file_size = 18446744073709551616
>
>
> # max_meta_name_length is the max number of bytes in the utf8 encoding
> # of the name portion of a metadata header.
>
> #max_meta_name_length = 128
>
>
> # max_meta_value_length is the max number of bytes in the utf8 encoding
> # of a metadata value
>
> #max_meta_value_length = 256
>
>
> # max_meta_count is the max number of metadata keys that can be stored
> # on a single account, container, or object
>
> #max_meta_count = 90
>
>
> # max_meta_overall_size is the max number of bytes in the utf8 encoding
> # of the metadata (keys + values)
>
> #max_meta_overall_size = 4096
>
>
> # max_object_name_length is the max number of bytes in the utf8 encoding of
> an
> # object name: Gluster FS can handle much longer file names, but the length
> # between the slashes of the URL is handled below. Remember that most web
> # clients can't handle anything greater than 2048, and those that do are
> # rather clumsy.
>
> max_object_name_length = 2048
>
> # max_object_name_component_length (GlusterFS) is the max number of bytes in
> # the utf8 encoding of an object name component (the part between the
> # slashes); this is a limit imposed by the underlying file system (for XFS
> it
> # is 255 bytes).
>
> max_object_name_component_length = 255
>
> # container_listing_limit is the default (and max) number of items
> # returned for a container listing request
>
> #container_listing_limit = 10000
>
>
> # account_listing_limit is the default (and max) number of items returned
> # for an account listing request
>
> #account_listing_limit = 10000
>
>
> # max_account_name_length is the max number of bytes in the utf8 encoding of
> # an account name: Gluster FS Filename limit (XFS limit?), must be the same
> # size as max_object_name_component_length above.
>
> max_account_name_length = 255
>
>
> # max_container_name_length is the max number of bytes in the utf8 encoding
> # of a container name: Gluster FS Filename limit (XFS limit?), must be the
> same
> # size as max_object_name_component_length above.
>
> max_container_name_length = 255
>
> "
>
>
> The volumes
> "
>  gluster volume list
> cindervol
> unified-storage-vol
> a07d2f39117c4e5abdeba722cf245828
> bd74a005f08541b9989e392a689be2fc
> f6da0a8151ff43b7be10d961a20c94d6
> "
>
> if I run the command
> "
>  gluster-swift-gen-builders unified-storage-vol
> a07d2f39117c4e5abdeba722cf245828 bd74a005f08541b9989e392a689be2fc
> f6da0a8151ff43b7be10d961a20c94d6
> "
>
> because of a change in the script in this version as compaired to the
> version I got from
> http://repos.fedorapeople.org/repos/kkeithle/glusterfs/ the
> gluster-swift-gen-builders script only takes the first option and
> ignores the rest.
>
> other than the location of the config files none of the changes Ive
> made are functionally different than the ones mentioned in
> http://www.gluster.org/2012/09/howto-using-ufo-swift-a-quick-and-dirty-setup-guide/
>
> The result is that the first volume named "unified-storage-vol" winds
> up being used for every thing regardless of the tenant, and users and
> see and manage each others objects regardless of what tenant they are
> members of.
> through the swift command or via horizon.
>
> In a way this is a good thing for me it simplifies thing significantly
> and would be fine if it just created a directory for each tenant and
> only allow the user to access the individual directories, not the
> whole gluster volume.
> by the way seeing every thing includes the service tenants data so
> unprivileged users can delete glance images without being a member of
> the service group.
>
>
>
>
> On Mon, Sep 2, 2013 at 9:58 PM, Paul Robert Marino <prmarino1 at gmail.com>
> wrote:
>
> Well I'll give you the full details in the morning but simply I used the
> stock cluster ring builder script that came with the 3.4 rpms and the old
> version from 3.3 took the list of volumes and would add all of them the
> version with 3.4 only takes the first one.
>
> Well I ran the script expecting the same behavior but instead they all used
> the first volume in the list.
>
> Now I knew from the docs I read that the per tenant directories in a single
> volume were one possible plan for 3.4 to deal with the scalding issue with a
> large number of tenants, so when I saw the difference in the script and that
> it worked I just assumed that this was done and I missed something.
>
>
>
> -- Sent from my HP Pre3
>
> ________________________________
> On Sep 2, 2013 20:55, Ramana Raja <rraja at redhat.com> wrote:
>
> Hi Paul,
>
> Currently, gluster-swift doesn't support the feature of multiple
> accounts/tenants accessing the same volume. Each tenant still needs his own
> gluster volume. So I'm wondering how you were able to observe the reported
> behaviour.
>
> How did you prepare the ringfiles for the different tenants, which use the
> same gluster volume? Did you change the configuration of the servers? Also,
> how did you access the files that you mention? It'd be helpful if you could
> share the commands you used to perform these actions.
>
> Thanks,
>
> Ram
>
>
> ----- Original Message -----
> From: "Vijay Bellur" <vbellur at redhat.com>
> To: "Paul Robert Marino" <prmarino1 at gmail.com>
> Cc: rhos-list at redhat.com, "Luis Pabon" <lpabon at redhat.com>, "Ramana Raja"
> <rraja at redhat.com>, "Chetan Risbud" <crisbud at redhat.com>
> Sent: Monday, September 2, 2013 4:17:51 PM
> Subject: Re: [rhos-list] Gluster UFO 3.4 swift Multi tenant question
>
> On 09/02/2013 01:39 AM, Paul Robert Marino wrote:
>
> I have Gluster UFO installed as a back end for swift from here
> http://download.gluster.org/pub/gluster/glusterfs/3.4/3.4.0/RHEL/epel-6/
> with RDO 3
>
> Its working well except for one thing. All of the tenants are seeing
> one Gluster volume which is some what nice, especially when compared
> to the old 3.3 behavior of creating one volume per tenant named after
> the tenant ID number.
>
> The problem is I expected to see is sub directory created under the
> volume root for each tenant but instead what in seeing is that all of
> the tenants can see the root of the Gluster volume. The result is that
> all of the tenants can access each others files and even delete them.
> even scarier is that the tennants can see and delete each others
> glance images and snapshots.
>
> Can any one suggest options to look at or documents to read to try to
> figure out how to modify the behavior?
>
> Adding gluster swift developers who might be able to help.
>
> -Vijay
>
>