[rhos-list] Gluster UFO 3.4 swift Multi tenant question

[Luis Pabon]

First thing I can see is that you have Essex based gluster-ufo-* which 
has been replaced by the gluster-swift project.  We are currently in 
progress of replacing the gluster-ufo-* with RPMs from the gluster-swift 
project in Fedora.

Please checkout the following quickstart guide which show how to 
download the Grizzly version of gluster-swift:
https://github.com/gluster/gluster-swift/blob/master/doc/markdown/quick_start_guide.md 
.

For more information please visit: https://launchpad.net/gluster-swift

- Luis

On 09/16/2013 05:02 PM, Paul Robert Marino wrote:
> Sorry for the delay on reporting the details. I got temporarily pulled
> off the project and dedicated to a different project which was
> considered higher priority by my employer. I'm just getting back to
> doing my normal work today.
>
> first here are the rpms I have installed
> "
>   rpm -qa |grep -P -i '(gluster|swift)'
> glusterfs-libs-3.4.0-8.el6.x86_64
> glusterfs-server-3.4.0-8.el6.x86_64
> openstack-swift-plugin-swift3-1.0.0-0.20120711git.el6.noarch
> openstack-swift-proxy-1.8.0-2.el6.noarch
> glusterfs-3.4.0-8.el6.x86_64
> glusterfs-cli-3.4.0-8.el6.x86_64
> glusterfs-geo-replication-3.4.0-8.el6.x86_64
> glusterfs-api-3.4.0-8.el6.x86_64
> openstack-swift-1.8.0-2.el6.noarch
> openstack-swift-container-1.8.0-2.el6.noarch
> openstack-swift-object-1.8.0-2.el6.noarch
> glusterfs-fuse-3.4.0-8.el6.x86_64
> glusterfs-rdma-3.4.0-8.el6.x86_64
> openstack-swift-account-1.8.0-2.el6.noarch
> glusterfs-ufo-3.4.0-8.el6.noarch
> glusterfs-vim-3.2.7-1.el6.x86_64
> python-swiftclient-1.4.0-1.el6.noarch
>
> here are some key config files note I've changed the passwords I'm
> using and hostnames
> "
>   cat /etc/swift/account-server.conf
> [DEFAULT]
> mount_check = true
> bind_port = 6012
> user = root
> log_facility = LOG_LOCAL2
> devices = /swift/tenants/
>
> [pipeline:main]
> pipeline = account-server
>
> [app:account-server]
> use = egg:gluster_swift_ufo#account
> log_name = account-server
> log_level = DEBUG
> log_requests = true
>
> [account-replicator]
> vm_test_mode = yes
>
> [account-auditor]
>
> [account-reaper]
>
> "
>
> "
>   cat /etc/swift/container-server.conf
> [DEFAULT]
> devices = /swift/tenants/
> mount_check = true
> bind_port = 6011
> user = root
> log_facility = LOG_LOCAL2
>
> [pipeline:main]
> pipeline = container-server
>
> [app:container-server]
> use = egg:gluster_swift_ufo#container
>
> [container-replicator]
> vm_test_mode = yes
>
> [container-updater]
>
> [container-auditor]
>
> [container-sync]
> "
>
> "
>   cat /etc/swift/object-server.conf
> [DEFAULT]
> mount_check = true
> bind_port = 6010
> user = root
> log_facility = LOG_LOCAL2
> devices = /swift/tenants/
>
> [pipeline:main]
> pipeline = object-server
>
> [app:object-server]
> use = egg:gluster_swift_ufo#object
>
> [object-replicator]
> vm_test_mode = yes
>
> [object-updater]
>
> [object-auditor]
> "
>
> "
> cat /etc/swift/proxy-server.conf
> [DEFAULT]
> bind_port = 8080
> user = root
> log_facility = LOG_LOCAL1
> log_name = swift
> log_level = DEBUG
> log_headers = True
>
> [pipeline:main]
> pipeline = healthcheck cache authtoken keystone proxy-server
>
> [app:proxy-server]
> use = egg:gluster_swift_ufo#proxy
> allow_account_management = true
> account_autocreate = true
>
> [filter:tempauth]
> use = egg:swift#tempauth
> # Here you need to add users explicitly. See the OpenStack Swift Deployment
> # Guide for more information. The user and user64 directives take the
> # following form:
> #     user_<account>_<username> = <key> [group] [group] [...] [storage_url]
> #     user64_<account_b64>_<username_b64> = <key> [group] [group]
> [...] [storage_url]
> # Where you use user64 for accounts and/or usernames that include underscores.
> #
> # NOTE (and WARNING): The account name must match the device name specified
> # when generating the account, container, and object build rings.
> #
> # E.g.
> #     user_ufo0_admin = abc123 .admin
>
> [filter:healthcheck]
> use = egg:swift#healthcheck
>
> [filter:cache]
> use = egg:swift#memcache
>
>
> [filter:keystone]
> use = egg:swift#keystoneauth
> #paste.filter_factory = keystone.middleware.swift_auth:filter_factory
> operator_roles = Member,admin,swiftoperator
>
>
> [filter:authtoken]
> paste.filter_factory = keystone.middleware.auth_token:filter_factory
> auth_host = keystone01.vip.my.net
> auth_port = 35357
> auth_protocol = http
> admin_user = swift
> admin_password = PASSWORD
> admin_tenant_name = service
> signing_dir = /var/cache/swift
> service_port = 5000
> service_host = keystone01.vip.my.net
>
> [filter:swiftauth]
> use = egg:keystone#swiftauth
> auth_host = keystone01.vip.my.net
> auth_port = 35357
> auth_protocol = http
> keystone_url = https://keystone01.vip.my.net:5000/v2.0
> admin_user = swift
> admin_password = PASSWORD
> admin_tenant_name = service
> signing_dir = /var/cache/swift
> keystone_swift_operator_roles = Member,admin,swiftoperator
> keystone_tenant_user_admin = true
>
> [filter:catch_errors]
> use = egg:swift#catch_errors
> "
>
> "
> cat /etc/swift/swift.conf
> [DEFAULT]
>
>
> [swift-hash]
> # random unique string that can never change (DO NOT LOSE)
> swift_hash_path_suffix = gluster
> #3d60c9458bb77abe
>
>
> # The swift-constraints section sets the basic constraints on data
> # saved in the swift cluster.
>
> [swift-constraints]
>
> # max_file_size is the largest "normal" object that can be saved in
> # the cluster. This is also the limit on the size of each segment of
> # a "large" object when using the large object manifest support.
> # This value is set in bytes. Setting it to lower than 1MiB will cause
> # some tests to fail. It is STRONGLY recommended to leave this value at
> # the default (5 * 2**30 + 2).
>
> # FIXME: Really? Gluster can handle a 2^64 sized file? And can the fronting
> # web service handle such a size? I think with UFO, we need to keep with the
> # default size from Swift and encourage users to research what size their web
> # services infrastructure can handle.
>
> max_file_size = 18446744073709551616
>
>
> # max_meta_name_length is the max number of bytes in the utf8 encoding
> # of the name portion of a metadata header.
>
> #max_meta_name_length = 128
>
>
> # max_meta_value_length is the max number of bytes in the utf8 encoding
> # of a metadata value
>
> #max_meta_value_length = 256
>
>
> # max_meta_count is the max number of metadata keys that can be stored
> # on a single account, container, or object
>
> #max_meta_count = 90
>
>
> # max_meta_overall_size is the max number of bytes in the utf8 encoding
> # of the metadata (keys + values)
>
> #max_meta_overall_size = 4096
>
>
> # max_object_name_length is the max number of bytes in the utf8 encoding of an
> # object name: Gluster FS can handle much longer file names, but the length
> # between the slashes of the URL is handled below. Remember that most web
> # clients can't handle anything greater than 2048, and those that do are
> # rather clumsy.
>
> max_object_name_length = 2048
>
> # max_object_name_component_length (GlusterFS) is the max number of bytes in
> # the utf8 encoding of an object name component (the part between the
> # slashes); this is a limit imposed by the underlying file system (for XFS it
> # is 255 bytes).
>
> max_object_name_component_length = 255
>
> # container_listing_limit is the default (and max) number of items
> # returned for a container listing request
>
> #container_listing_limit = 10000
>
>
> # account_listing_limit is the default (and max) number of items returned
> # for an account listing request
>
> #account_listing_limit = 10000
>
>
> # max_account_name_length is the max number of bytes in the utf8 encoding of
> # an account name: Gluster FS Filename limit (XFS limit?), must be the same
> # size as max_object_name_component_length above.
>
> max_account_name_length = 255
>
>
> # max_container_name_length is the max number of bytes in the utf8 encoding
> # of a container name: Gluster FS Filename limit (XFS limit?), must be the same
> # size as max_object_name_component_length above.
>
> max_container_name_length = 255
>
> "
>
>
> The volumes
> "
>   gluster volume list
> cindervol
> unified-storage-vol
> a07d2f39117c4e5abdeba722cf245828
> bd74a005f08541b9989e392a689be2fc
> f6da0a8151ff43b7be10d961a20c94d6
> "
>
> if I run the command
> "
>   gluster-swift-gen-builders unified-storage-vol
> a07d2f39117c4e5abdeba722cf245828 bd74a005f08541b9989e392a689be2fc
> f6da0a8151ff43b7be10d961a20c94d6
> "
>
> because of a change in the script in this version as compaired to the
> version I got from
> http://repos.fedorapeople.org/repos/kkeithle/glusterfs/ the
> gluster-swift-gen-builders script only takes the first option and
> ignores the rest.
>
> other than the location of the config files none of the changes Ive
> made are functionally different than the ones mentioned in
> http://www.gluster.org/2012/09/howto-using-ufo-swift-a-quick-and-dirty-setup-guide/
>
> The result is that the first volume named "unified-storage-vol" winds
> up being used for every thing regardless of the tenant, and users and
> see and manage each others objects regardless of what tenant they are
> members of.
> through the swift command or via horizon.
>
> In a way this is a good thing for me it simplifies thing significantly
> and would be fine if it just created a directory for each tenant and
> only allow the user to access the individual directories, not the
> whole gluster volume.
> by the way seeing every thing includes the service tenants data so
> unprivileged users can delete glance images without being a member of
> the service group.
>
>
>
>
> On Mon, Sep 2, 2013 at 9:58 PM, Paul Robert Marino <prmarino1 at gmail.com> wrote:
>> Well I'll give you the full details in the morning but simply I used the
>> stock cluster ring builder script that came with the 3.4 rpms and the old
>> version from 3.3 took the list of volumes and would add all of them the
>> version with 3.4 only takes the first one.
>>
>> Well I ran the script expecting the same behavior but instead they all used
>> the first volume in the list.
>>
>> Now I knew from the docs I read that the per tenant directories in a single
>> volume were one possible plan for 3.4 to deal with the scalding issue with a
>> large number of tenants, so when I saw the difference in the script and that
>> it worked I just assumed that this was done and I missed something.
>>
>>
>>
>> -- Sent from my HP Pre3
>>
>> ________________________________
>> On Sep 2, 2013 20:55, Ramana Raja <rraja at redhat.com> wrote:
>>
>> Hi Paul,
>>
>> Currently, gluster-swift doesn't support the feature of multiple
>> accounts/tenants accessing the same volume. Each tenant still needs his own
>> gluster volume. So I'm wondering how you were able to observe the reported
>> behaviour.
>>
>> How did you prepare the ringfiles for the different tenants, which use the
>> same gluster volume? Did you change the configuration of the servers? Also,
>> how did you access the files that you mention? It'd be helpful if you could
>> share the commands you used to perform these actions.
>>
>> Thanks,
>>
>> Ram
>>
>>
>> ----- Original Message -----
>> From: "Vijay Bellur" <vbellur at redhat.com>
>> To: "Paul Robert Marino" <prmarino1 at gmail.com>
>> Cc: rhos-list at redhat.com, "Luis Pabon" <lpabon at redhat.com>, "Ramana Raja"
>> <rraja at redhat.com>, "Chetan Risbud" <crisbud at redhat.com>
>> Sent: Monday, September 2, 2013 4:17:51 PM
>> Subject: Re: [rhos-list] Gluster UFO 3.4 swift Multi tenant question
>>
>> On 09/02/2013 01:39 AM, Paul Robert Marino wrote:
>>> I have Gluster UFO installed as a back end for swift from here
>>> http://download.gluster.org/pub/gluster/glusterfs/3.4/3.4.0/RHEL/epel-6/
>>> with RDO 3
>>>
>>> Its working well except for one thing. All of the tenants are seeing
>>> one Gluster volume which is some what nice, especially when compared
>>> to the old 3.3 behavior of creating one volume per tenant named after
>>> the tenant ID number.
>>>
>>> The problem is I expected to see is sub directory created under the
>>> volume root for each tenant but instead what in seeing is that all of
>>> the tenants can see the root of the Gluster volume. The result is that
>>> all of the tenants can access each others files and even delete them.
>>> even scarier is that the tennants can see and delete each others
>>> glance images and snapshots.
>>>
>>> Can any one suggest options to look at or documents to read to try to
>>> figure out how to modify the behavior?
>>>
>> Adding gluster swift developers who might be able to help.
>>
>> -Vijay

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/rhos-list/attachments/20130917/27d8299f/attachment.htm>