Hi folks, We ran a test of 4 experienced system administrators (current RHEL users) in the Westford Red Hat office last week. Attached is my write-up of how the tests went along with a list of all of the issues encountered across users. It's in both markdown and HTML format. The markdown is pretty readable as plain text. ~m
Background: We had 4 experienced system adminstrators run through the Fedora 19 TC 1 Anaconda UI. They were given sheets outlining a specific LVM or RAID disk layout they were supposed to configure, and were otherwise asked to explore the user interface and let us know what they thought. They used a VM with two vda disks connected; one was 8 GB and one was 6 GB. The storage specifications we gave them required the usage of both disks.
Only have a couple of RHEL 6 systems.
Has trouble unlocking gnome-shell screensaver.
(because it crashed)
anaconda usability test notes ============================= MÃ¡irÃn Duffy <duffy redhat com> **Background:** We had 4 experienced system adminstrators run through the Fedora 19 TC 1 Anaconda UI. They were given sheets outlining a specific LVM or RAID disk layout they were supposed to configure, and were otherwise asked to explore the user interface and let us know what they thought. They used a VM with two vda disks connected; one was 8 GB and one was 6 GB. The storage specifications we gave them required the usage of both disks. Participant #1 -------------- * Her company runs RHEL 5, so she doesn't normally read the latest docs. * RHEL 5.5=>5.6 - see if it works. * Only have a couple of RHEL 6 systems. * Has trouble unlocking gnome-shell screensaver. ### Screen 1 - Welcome/Language ### * Checkbox for default layout * Beta nag ### Hub #1 ### * Hovers over 'installation source', clicks on it ### Installation Source Spoke ### * Looks at 'update' checkbox... "I don't want updates" * Plays with 'install media' checkboxes, "Typically we use kickstart." Checks off auto-detected media option. * Clicks on 'done' in the upper left. ### Hub #1 ### * Clicks on 'Software Selection' ### Software Selection ### * Looks through the list on the left * "I want RHEL 7 OS only. I don't want a webserver or anything. What do I choose?" Looks through the list. * Clicks on infrastructure server - "I thought that one would be the best" * She didn't scroll the list and didn't seem to realize the list scrolled... so she never saw the minimal installation. * "I don't want a desktop." ### Hub #1 ### * Clicks on 'Install Destination' ### Install Destination ### * Picks first vda disk, does not select second vda disk. * Clicks on 'specialized' * "What's that for?" * Leaves specialized section * Hovers over the 'disk summary' link in the lower left. * Hits 'done' on the main install destination screen. * Reads 'install options' lightbox dialog * "It gives me an option to modify partitions" * "Doing automatically - we wouldn't do automatic. We'd use kickstart. I don't use automatic - I do my own partitioning for my own machines." ### Hub #1 ### * "I want to proceed further." * "The keyboard looks fine already, the network looks like it's auto-configured so I'm fine." * "The data/time looks fine." * Do you see anything that says what they are set to? "No, but I'm fine with the defaults." * "How will I understand what I've already visited on this screen? If I came back later and forgot - which things did I do?" * "Show me which ones I visited before." ### Date/Time Screen ### * What is NY time? "In RHEL 5 it asks for New York time, this is fine." * Would you change the settings? "The default timezone is fine." ### Keyboard ### * "I'm fine not using this screen." ### Network Cofiguration ### * "If I have two NIC cards, is it possible?" * "I only see one NIC - I want to add another one." * Clicks on bond pop-up box... dialog confused her. * "Generally a bond is 2 NICs, how is this possible?" Crashes. ### Anaconda Restarted ### (because it crashed) ### Hub #1 ### * What do you think about the orange triangles on the screen? * "Is it an error? Warning..." * Reads error message. * "These are the required items to fill out." ### Software Selection ### * Selects infrastructure server * "What if I want both a webserver and an infrastructure server?" * Doesn't notice the 'web server' option in the add-ons column on the right side. * How would you interpret the word "environment"? * Left-hand side won't allow multiple selections * Seems very confused about only being able to choose one option on the left. ### Disk Selection ### * Noted orange bar on the bottom - said she sees it fine. ### Custom Partitioning ### * clicks on autopart link in the left hand side to create auto part scheme * doesn't think she can change / boot partition options * Clicks on the '+' icon on the left hand side * Click on 'Root' - tries to change its capacity, "it's maxed out" * "I think it's ext4, but it isn't showing anything" * Hits 'apply changes' button - confused - doesn't see anything change or move so doesn't know what happened when she pressed the button. * Looks at sswap - changes it to 1024 MB and hits 'apply changes' - change is applied but she's not confident it worked * clicks on 'file system options' disclosure triangle, "Swap partition... not an LVM. But it's letting me pick LVM device type." * hits 'apply changes' and seems confused * "Now I'll add home" - clicks on the '+' button in the left hand side, selects '/home' from the dropdown rather than typing out '/home'.... * In the home right hand side, types 3096 MB, and it automatically corrects to 2.97 GB. She doesn't understand why the number changed. * triggers the GNOME messaging bar, seems confused * "I want to add another disk." She hits the rescan button and pops up the rescan dialog, then hits cancel. * "If I knew you gave this spec for laying out the discs in the first place, I would have selected more disks" * "From here, I don't know how to add more disks." ### Overall Feedback ### * "The easiest thing - everything is given, I have to only click." Participant #2 -------------- ### Welcome Screen ### * "I usually use the keyboard, and not the mouse." ### Hub #1 ### * "The date and time looks fine... the install source looks fine." * Sees "software selection" and "install desitnation" - "I think the /!\ means I need to set those up." ### Software Selection ### * Worried about the network on software selection - if the network isn't connected will my install work? * "Don't normally use GNOME. Normally don't use desktop" * "Don't normally use web server" * "Like to start with a base system and then build on top of that post install." * Do these give enough info? "Sure, if you've never used this OS you might not know what GNOME is but it's fine if you know what you're doing." ### Storage ### * Chooses 8 GB vda - not sure if he chose that disk correctly - "Can I select both?" * Clicks on the second 6 GB vda, "Oh, okay, it's that simple." ### Storage Dialog: Options ### * selects custom part option ### Custom Partitioning ### * doesn't click on auto-part link in left hand side of screen * clicks on "+", creates "/", types 5 GB.... * clicks on FS/device options... reviews the information, seems okay, hits 'apply changes' * "Typically we use KS so we don't go through GUIs. VM, ECE, UCS - we build templates and deploy from that." * Adds another mount point, /home, types '5 GB', hits 'add' * Hits 'apply changes' * Adds another mount point, /srv, types '2.5 GB' * On the right-hand side of /srv, changes the filesystem to vfat, hits 'apply changes' * Adds another mount point, swap... uses drop down in add a mount point dialog to select 'swap', types '1 GB', hits add button * Hits apply changes button * Adds /boot, 500 GB - it gets automatically sized down to 243 GB * Participant doesn't know why it sizes down. * "There's not enough space?" * Counts up... not sure * "It *should* have 500 GB left." * Sizes /srv down to 2 GB * Resizes boot to 500 GB. * Clicks done - sees summary of changes. * Reads through summary carefully * Doesn't see any errors, accepts changes. ### Hub #1 ### * Clicks on storage - crash * Complains about done being in the upper left corner, but in practice didn't seem to have much difficulty using it. * When asked about which screens he felt he should visit, he said "If it has a !, it's required." * "I know I can't ocntinue because the continue button is greyed out. It's pretty intuitive what I need to do." ### Software Selection ### * "I didn't see minimal install before - I like it. I guess I didn't scroll down - I didn't notice the scroll bar." ### Hub #1 ### * Do you expect to set the time & date and keyboard manually? "No, I usually use kickstart. If I do a manual install though, sure, I might. I saw it was already set, otherwise I'd expect it to prompt me to set something." * When asked about the orange bar at the bottom of hub #1 - "I saw it, but I didn't read it." ### Network Configuration ### * clicks 'configure' * Goes through tabs - "static would be manual" * looks through IPV6 * sets up static ip in IPV4 tab * "These make sense." * "For someone who knows what they're doing [with network configuration], this is all user-friendly." * Didn't seem to notice hostname field. * "If I disconnect the network on this screen after filling out the custom [static ip] settings, then reconnect, will come back with my custom settings?" * "If I disconnect and go to the hub, can I install? I guess it doesn't have to be on the network to install." ### Install Source ### * clicks verify - "Will it work without network?" It did. ### Date / Time ### * "That's pretty slick" * If network is turned off, you can't turn NTP on and there is no error message. ### Probing Questions ### * How do you learn about new features, etc.? "Usually I read the release notes and announcements, but I'm too busy to sit around and read everything." * When asked about confidence install would proceed correctly, "I feel fairly confident I got it done correctly." * When asked what the hardest part of the process was, * "I wasn't sure why I couldn't turn NTP on, I forgot that the network was disconnected." * "I'm not sure how verify was able to work without a network connection." * "I didn't have space left for the custom partition scheme, and I don't know why." * When asked what the easiest part of the process was, * "I liked the /!\ on the hub - and the hover tooltips gave good information." Participant #3 -------------- ### Welcome Screen ### * When asked about checkbox: * "I don't know how it was different in the old UI... usually we use kickstart so I don't know what the old GUI looked like." * "I wanted to make sure the default keyboard was set, so I checked it off." * "All of our deployments are kickstart-based from a Satellite Server." ### Hub #1 ### * Hovers over date/time spoke.... hovers over keyboard... hovers over network... hovers over source... examines each but doesn't click on any of them. * Clicks on software selection. ### Software Selection ### * Scrolls left pane * chooses 'infrastructure server' * "For general purpose we'd use an infrastructure server. For DMZ systems, we'd use a minimal install." * When asked about add-ons, "In our kickstart profiles, we add them based on the specific needs of the server being installed.... we do builds for install time." * In the add-ons pane on the right, he checks off devel tools ("I want gcc") * "When a new release comes out, we have a team that sets a standard build doc based on package group memberships." * "They go through Red Hat and third party security documentation and come up with environment standards." * He scrolls through more add-ons, doesn't check any more off. * When asked if the add-on descriptions were sufficient, he responds "These descriptions are enough information for me." * Had some trouble finding "done" button ### Network Configuration ### * Looks through the screen * Looks at the hostname field and changes the hostname * hits 'configure' * pokes around the various tabs * disables IPv6 "We usually disable IPv6." ### Install Destination ### * Looks at chart he was handed with rthe equested storage configuration. * Clicks on the first disk on disk selection screen. * Clicks on network disk / multipath add disk button. * "I don't understand this screen. How do I go back?" * See link in lower left corner, "1 device selected" * clicks 'Done' after being advised to * Back on disk selection screen... * Clicks on 'devices selected' link in lower left corner * "This is where the bootloader will be located" * "I was looking for custom partitioning." * Clicks on done, gets install options, clicks on custom radio button. Looks at partition type dropdown. Hits ok. ### Custom Partitioning ### * Reads through the left-hand side pane. * Clicks on the "+" button on the bottom of the left-hand side pane. * Adds '/'... types 5 then seems confused. Hovers over field and gets a tooltip which seems to help, then types 'GB.' * Presses '+', adds /home, reads tooltip again. * "I need more space." * "Only 2.99 GB is left... I need more than that." * "How do I go back?" * Hits "Done" * Gets summary, hits cancel. * Hits disk summary / disk 'shopping cart' link. Exits it. * Hits done again. * Hits 'accept changes' * dumped out to Hub #1 * Clicks on install destination again. * Selects second disk, hits done, goes to custom again. * Starts over! * adds '/' (using dropdown) * adds '/home' (using dropdown) * adds '/srv' (types out after finding it not in dropdown) * adds swap (dropdown) * adds /boot - 487 MB (that's what anaconda says is left in lower left) * Tries putting in 500MB.... * gets autochanged to 243 MB. * "243 - is this a default requirement, how far will it go?" * Presses '+' on spinner for disk size... it goes up, even though it then gets auto-corrected to 243 when hitting apply changes." * deletes /boot * creates new /boot - "I don't know why it's making it 243 MB, there's 487 MB left." * Checks filesystem settings on other parititons... * changes /boot to vfat. Doesn't hit 'apply changes' - when asked he says he's not sure his changes are sticking. * makes /boot ext4 and hits apply changes. * checks /srv, /home, /root, and swap fs and device types in that order. * Clicks "Done", gets to summary, reads through * "It saved everything I did." * clicks on apply changes ### Hub #1 ### * "I haven't done installation source." * "Any screens I didn't do before had red marks, so I knew I had to go to them." ### Date/Time ### * Mouses over map * Selects US > Eastern.... map pin and highlight on map does not update! ### Keyboard ### * "We always use US English." ### Installation Source ### * Likes the network install src option. "These dropdowns make it easy." * Hovers over the 'don't install latest updates' checkbox. "We usually install to 5.3, 6.1, or 6.2. If you want the latest updates from the channel, this option is good." * Hovers over additional repos section, "This is a good feature. Today we handle this with post-install scripts." ### Probing Questions ### * "The only thing I'm not sure about is why I couldn't have a 500 MB /boot. Otherwise I'm confident I did things right." * What was the hardest thing? "The screens are new. The partition layout and disk selection [in particular.] It would be nice to add more disks on the fly. I didn't know if it queued things up or took instant action at first. If I did this a few times, I'd get used to it. It's very user-friendly other than that." * Was there anything in particular you liked? "Not really... overall more user-friendly than in the past. Seems more informative, particularly in the partition layouts." Participant #4 -------------- ### Welcome Screen ### * Clicks 'Continue' ### Hub #1 ### * "I'm used to the older, linear method." * "Date & Time, Keyboard, and Network look fine." * Clicks on network configuration spoke. ### Network Configuration ### * Reviews the screen * "This all looks fine: I have an IP, default route, DNS..." * When prompted to "add a NIC", clicks the '+' button in the lower left. * "Does this system have more than one NIC? Why is it offering to create a bond if I only have one NIC - that makes no sense." * Hits 'configure' button to change settings and looks through it. * "This seems clearer than the older anaconda, look & feel wise." * Asked whether or not he ever sets up a sandbox and uses the installer GUI for it. "I used to. Now it's harder to find spare hardware to do that." * Seems to fine done without issue and returns to the hub. ### Hub #1 ### * "I think I should configure storage before software selection..." He reads the orange alert bar on the bottom. * Clicks on install destination spoke. ### Install Destination ### * Clicks on the 1st disk... double-clicks it. * Why did you double-click it? "I thought it would take me to the next screen if I double-clicked it." * Clicks on the 'disk shopping cart' link in the lower left. * Hovers over network disks area but doesn't click. * Deselects, then re-selects the disk. * Keeps looking around. Asked what he's looking for. "I'm trying to find where it will let me set up partitions. I'm looking for 'advanced' as in the old installer." * "Is it part of specialized & network disks?" * Clicks on specialized & network. * "This looks like it's all for *remote* storage." * Clicks on disk shopping cart in lower left of specialized & network disks screen. "This is just showing me the one disk I selected. I was wondering if I could get to partitioning from here." * Advised to click done, so he clicks done. "Ah, okay. I was looking for this!" * Clicks on the custom partitioning radio button. * Looks at the autopart drpodown... doesn't see RAID. Leaves it on LVM, and hits continue. ### Custom Partitioning ### * Not sure if the boxes in the lower left were clickable or not. * "7.99 instead of 8.00 GB listed here, I'm guessing the .1 is for the MBR." * Clicks "+"... adds /home (via dropdown), types in 4 for capacity, then noticed tooltip ("oh!") and then types "GB." Says that the tooltip was helpful. * Clicks on 'customize' * Device type is LVM... selects standard partition, then BTRFS. * "Filesystem is ext?" * Why did you pick BTRFS? "The others wouldn't let me choose a RAID level." * Hits apply changes. Hits RAID level error - requires more disks than selected. * Is able to click done anyway, reads summary of changes, hits accept, goes back to storage. * Crash! ### Hub #1 (after restart) ### * "Do I have to wait for all these things to load before I can enter any of them? * Clicks on install destination. ### Install destination ### * Selects 2 disks this time. * Hits done. Selects custom partitioning. ### Custom Partitioning ### * Adds /home - 4 GB. * Not sure if his changes worked. * Adds / - checks tooltip. * Hits customize disclosure triangle for / * Would like confirmation when a change takes place.... thoughts a pop up or something would show up when he hit apply changes to let him know what it was doing. * Adds /boot * boot defaults to ext4 instead of LVM. "Is that intentional? I didn't expect that." * Adds swap - 2 GB. * Goes to /boot, chooses standard partition, swap. For some reason, when he created swap, it didn't default to swap. * Hits done, reads through summary, applies changes. ### Hub #1 ### * "Install source looks set." Prompted to click on it anyway and take a look. ### Installation Source ### * "I won't click verify, since I'm guessing it'll take a while." * "I heard about adding additional repos - sounds good. We have weird one-offs where we need bleeding edge PHP packages, so this would be a good way to add them." ### Software Selection ### * "I'm glad to see I don't have to choose individual packages anymore." * "It takes forever to load - I just don't want a desktop." * "This is more descriptive than the old way." * "Infratructure server - what does this actually mean?" * "Normally I'd Google it or check the Red Hat portal to get more information about what it means." * "Will it install mail and DNS? Or NIS & LDAP, and everything? I'm not sure." * Only the radio button in the software selection list is clickable - the whole row isn't. He seemed to have trouble targeting the small radio button icon. * Confused that no add-ons are selected by default with infrastructure server. * "Typically I'd pick desktop, maybe add NFS. Then I'd install the add-ons, but post-install." * "It is nice that this gives you add-ons without having to go through package-by-package." * Likes the curated list. * Really likes the lightweight desktop options - XFCE for remote work display * Software dep error * gvfs broken * There's a quit button. Sigh. * When he hits done from the software spoke, no issues or errors appear. ### Date & Time ### * Got network time greyed out * Figured out how to add NTP server no issue. * Liked the simplicity of the NTP configuration options. ### Probing Questions ### * Aside from no warnings, he picked software, defined partitions, ethernet works - confident this install would succeed if he proceeded. * "I assumed there was no real order required." * What was the easiest part? "Software selection. The storage issues were partly my fault - I didn't know I could pick 2 disks at first." * Did anything about the UI stick out to you in particular? "Storage allocation is much clearer. Before, I'd have to wait for the GUI to refresh and it would take forever - it didn't bring the screen back. Now you're not stuck waiting on it." Issues ------ 1. Multiple users didn't seem to realize the base platform / left-hand side list on the software selection spoke scrolled. Minimal install seemed like it would be a desirable option for them but they never saw it. 1. On the network screen, Anaconda offers you to create a network bond even though there is only one NIC on the machine. You need a minimum of 2 NICs to create a bond. This seems like a bug. Two participants hit it. 1. On the network screen, if you click to create a bond when there is one NIC, Anaconda crashes. 1. On the software selection screen, participant #1 seemed confused about the relationship between the left hand side of the screen and the right hand side of the screen. For example, she wanted a server that had a webserver and infrastructure server on it - but she didn't realize she could get both by picking infrastructure server on the left and web on the right, or a minimal system and picking infrastructure and web on the right. Might make sense to change "Environment" as the label to "Base System" and to tune down the number of options offered as "base system" 1. When you try to change the capacity of a partition and there's no space left, you don't get an error message - the size just goes back to what it was. It would be good to have some messaging around what is going on there. 1. By default, the filesystem and device type options should be displayed rather than hidden behind a disclosure triangle 1. The 'apply changes' button shouldn't be lit unless you've made changes. Maybe highlight the fields that have pending changes? 1. Users who ran out of disk space in custom partitioning were very confused, didn't understand how to go back, add another disk, and continue with custom partitioning. Happened to multiple participants - the loop that participant #3 got into is especially illustrative of the problem here. 1. The hostname field is hard to see. Even a recent blog post on Planet Fedora gave a 'workaround' for this 'missing' field - we should make it more visible: http://feedproxy.google.com/~r/xenodeblogfedora/~3/vbLmaM4rMdY/cambiar-el-nombre-de-la-pc-en-fedora-18.html 1. Not sure of answer to question from participant #2 - "If I disconnect the network on this screen after filling out the custom [static ip] settings, then reconnect, will it come back with my custom settings?" 1. Participant #2 confused about whether or not network changes / disconnects would make install fail. Might be worth having a note on the network screen to say a network connection isn't required for install. 1. Participant #2 confused about how verify button on install source spoke was able to work without network... thoughts? 1. If network is turned off, you can't turn NTP on and there is no error message as to why you can't turn it on. We should have a reasonable error message for this scenario. 1. Participant #3 didn't understand the network storage filter screen and didn't know how to get back to the main disk selection screen. 1. Participant #3 couldn't find custom partitioning, wasn't intuitive to him to leave disk selection to get to it. 1. The '+' on the disk capacity spinner in custom partitioning does not limit the number you can enter, even though when you hit apply changes the number gets cut down. 1. Certain timezones, when selected from the dropdown, don't result in the map and pin on the map getting updated. Participant #3 ran into this. Seems like an outright bug. 1. Participant #4 had to pick BTRFS to pick a RAID level, the other types (LVM, Standard partitions) greyed out the raid selector.... RAID should be possible with standard partition. But he only had one disk selected, that's why it didn't show up. 1. When participant #4 selected a RAID level he didn't have enough disks for and hit done anyway, anaconda crashed. 1. Participant #4 wasn't sure if he could enter any of the spokes when some where still loading. 1. In custom partitioning, creating a swap partition doesn't default to a swap fs type. It seems like it really should. 1. Participant 4 wasn't sure about what some of the base system selections under software selcetion meant. Eg. what is an infrastructure server? Might be worth having some kind of dialog pop up for each one with a sampling of the packages or something. Or just have better descriptions :) 1. Only the radio button in the software selection list is clickable - the whole row isn't. Participant #4 seemed to have trouble targeting the small radio button icon. If we can make the whole row clickable, it would make it easier on users I think. Participant #4 was confused that no add-ons are selected by default with infrastructure server. 1. The software dependency error dialog should not have a quit button on it. There's no need to quit the whole installer because it's a fixable issue (just unselect the thing you picked.) The user almost clicked on quit thinking it was the same as cancel to get out of the dialog. 1. Participant #4 (and I think other users) didn't realize they could pick more than one disk on the disk selection screen. Maybe a checkbox graphic in the upper left of those icons would make it more clear?