[rhos-list] EXTERNAL: Re: Problems after Update.
Minton, Rich
rich.minton at lmco.com
Thu Mar 21 17:16:52 UTC 2013
Found it! The NFS mount to my instances directory was missing. I remounted the directory and I can see all of my instances now.
I have one other issue with metadata and ssh keys but I'll create a new thread for that.
Thanks for the help.
-----Original Message-----
From: rhos-list-bounces at redhat.com [mailto:rhos-list-bounces at redhat.com] On Behalf Of Minton, Rich
Sent: Thursday, March 21, 2013 1:03 PM
To: Russell Bryant
Cc: rhos-list at redhat.com
Subject: Re: [rhos-list] EXTERNAL: Re: Problems after Update.
Here is some of what I found in "compute.log" and it doesn't look good.
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
2013-03-21 12:15:21 42682 WARNING nova.compute.manager [-] Found 5 in the database and 0 on the hypervisor.
2013-03-21 12:15:21 42682 WARNING nova.compute.manager [-] [instance: 2d87a89a-fcbb-4581-a7c3-0e2b2c30b568] Instance shutdown by itself. Calling the stop API.
2013-03-21 12:15:21 42682 WARNING nova.compute.manager [-] [instance: b7e64392-1f28-446f-8fb3-2b75dc895951] Instance shutdown by itself. Calling the stop API.
2013-03-21 12:15:21 42682 INFO nova.virt.libvirt.driver [-] [instance: 2d87a89a-fcbb-4581-a7c3-0e2b2c30b568] Instance destroyed successfully.
2013-03-21 12:15:21 42682 WARNING nova.compute.manager [-] [instance: 40153f6e-e963-4a30-bb0a-3f7e612d5638] Instance shutdown by itself. Calling the stop API.
2013-03-21 12:15:21 42682 INFO nova.virt.libvirt.driver [-] [instance: b7e64392-1f28-446f-8fb3-2b75dc895951] Instance destroyed successfully.
2013-03-21 12:15:22 42682 WARNING nova.compute.manager [-] [instance: a39787b8-2c66-4aaa-b154-42ec05c0e3c8] Instance shutdown by itself. Calling the stop API.
2013-03-21 12:15:22 42682 INFO nova.virt.libvirt.driver [-] [instance: 40153f6e-e963-4a30-bb0a-3f7e612d5638] Instance destroyed successfully.
2013-03-21 12:15:22 42682 WARNING nova.compute.manager [-] [instance: be67a3ef-036b-4df8-90dd-04cbd0ffafdb] Instance shutdown by itself. Calling the stop API.
2013-03-21 12:15:22 42682 INFO nova.virt.libvirt.driver [-] [instance: a39787b8-2c66-4aaa-b154-42ec05c0e3c8] Instance destroyed successfully.
2013-03-21 12:15:23 42682 INFO nova.virt.libvirt.driver [-] [instance: be67a3ef-036b-4df8-90dd-04cbd0ffafdb] Instance destroyed successfully.
2013-03-21 12:15:23 42682 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 237303
2013-03-21 12:15:23 42682 AUDIT nova.compute.resource_tracker [-] Free disk (GB): -14
2013-03-21 12:15:23 42682 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 38
2013-03-21 12:15:23 42682 INFO nova.compute.resource_tracker [-] Compute_service record updated for uvslp-diu01os.lmdit.us.lmco.com
2013-03-21 12:16:02 AUDIT nova.compute.manager [req-a00e05db-32f4-4be0-8885-e789e8474b4f f015ea92bf4848a891b73ae2fbf6c75b e4ba09fbf76c41a29218dc68cacf78d5] [instance: be67a3ef-036b-4df8-90dd-04cbd0ffafdb] Get console output
2013-03-21 12:16:02 42682 ERROR nova.openstack.common.rpc.amqp [-] Exception during message handling
2013-03-21 12:16:02 42682 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2013-03-21 12:16:02 42682 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 276, in _process_data
2013-03-21 12:16:02 42682 TRACE nova.openstack.common.rpc.amqp rval = self.proxy.dispatch(ctxt, version, method, **args)
2013-03-21 12:16:02 42682 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/dispatcher.py", line 145, in dispatch
2013-03-21 12:16:02 42682 TRACE nova.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs)
2013-03-21 12:16:02 42682 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/exception.py", line 117, in wrapped
2013-03-21 12:16:02 42682 TRACE nova.openstack.common.rpc.amqp temp_level, payload)
2013-03-21 12:16:02 42682 TRACE nova.openstack.common.rpc.amqp File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2013-03-21 12:16:02 42682 TRACE nova.openstack.common.rpc.amqp self.gen.next()
2013-03-21 12:16:02 42682 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/exception.py", line 92, in wrapped
2013-03-21 12:16:02 42682 TRACE nova.openstack.common.rpc.amqp return f(*args, **kw)
2013-03-21 12:16:02 42682 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 196, in decorated_function
2013-03-21 12:16:02 42682 TRACE nova.openstack.common.rpc.amqp kwargs['instance']['uuid'], e, sys.exc_info())
2013-03-21 12:16:02 42682 TRACE nova.openstack.common.rpc.amqp File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2013-03-21 12:16:02 42682 TRACE nova.openstack.common.rpc.amqp self.gen.next()
2013-03-21 12:16:02 42682 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 190, in decorated_function
2013-03-21 12:16:02 42682 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2013-03-21 12:16:02 42682 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1931, in get_console_output
2013-03-21 12:16:02 42682 TRACE nova.openstack.common.rpc.amqp output = self.driver.get_console_output(instance)
2013-03-21 12:16:02 42682 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/exception.py", line 117, in wrapped
2013-03-21 12:16:02 42682 TRACE nova.openstack.common.rpc.amqp temp_level, payload)
2013-03-21 12:16:02 42682 TRACE nova.openstack.common.rpc.amqp File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2013-03-21 12:16:02 42682 TRACE nova.openstack.common.rpc.amqp self.gen.next()
2013-03-21 12:16:02 42682 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/exception.py", line 92, in wrapped
2013-03-21 12:16:02 42682 TRACE nova.openstack.common.rpc.amqp return f(*args, **kw)
2013-03-21 12:16:02 42682 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1150, in get_console_output
2013-03-21 12:16:02 42682 TRACE nova.openstack.common.rpc.amqp libvirt_utils.chown(path, os.getuid())
2013-03-21 12:16:02 42682 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/utils.py", line 332, in chown
2013-03-21 12:16:02 42682 TRACE nova.openstack.common.rpc.amqp execute('chown', owner, path, run_as_root=True)
2013-03-21 12:16:02 42682 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/utils.py", line 53, in execute
2013-03-21 12:16:02 42682 TRACE nova.openstack.common.rpc.amqp return utils.execute(*args, **kwargs)
2013-03-21 12:16:02 42682 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/utils.py", line 206, in execute
2013-03-21 12:16:02 42682 TRACE nova.openstack.common.rpc.amqp cmd=' '.join(cmd))
2013-03-21 12:16:02 42682 TRACE nova.openstack.common.rpc.amqp ProcessExecutionError: Unexpected error while running command.
2013-03-21 12:16:02 42682 TRACE nova.openstack.common.rpc.amqp Command: sudo nova-rootwrap /etc/nova/rootwrap.conf chown 162 /var/lib/nova/instances/instance-000000f2/console.log
2013-03-21 12:16:02 42682 TRACE nova.openstack.common.rpc.amqp Exit code: 1
2013-03-21 12:16:02 42682 TRACE nova.openstack.common.rpc.amqp Stdout: ''
2013-03-21 12:16:02 42682 TRACE nova.openstack.common.rpc.amqp Stderr: "/bin/chown: cannot access `/var/lib/nova/instances/instance-000000f2/console.log': No such file or directory\n"
2013-03-21 12:16:02 42682 TRACE nova.openstack.common.rpc.amqp
2013-03-21 12:16:02 42682 ERROR nova.openstack.common.rpc.common [-] Returning exception Unexpected error while running command.
Command: sudo nova-rootwrap /etc/nova/rootwrap.conf chown 162 /var/lib/nova/instances/instance-000000f2/console.log
Exit code: 1
Stdout: ''
Stderr: "/bin/chown: cannot access `/var/lib/nova/instances/instance-000000f2/console.log': No such file or directory\n" to caller
2013-03-21 12:16:02 42682 ERROR nova.openstack.common.rpc.common [-] ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 276, in _process_data\n rval
= self.proxy.dispatch(ctxt, version, method, **args)\n', ' File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/dispatcher.py", line 145, in dispatch\n return getattr(proxyobj, method)(ctxt, **kwargs)\n', ' F
ile "/usr/lib/python2.6/site-packages/nova/exception.py", line 117, in wrapped\n temp_level, payload)\n', ' File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__\n self.gen.next()\n', ' File "/usr/lib/python
2.6/site-packages/nova/exception.py", line 92, in wrapped\n return f(*args, **kw)\n', ' File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 196, in decorated_function\n kwargs[\'instance\'][\'uuid\'],
e, sys.exc_info())\n', ' File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__\n self.gen.next()\n', ' File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 190, in decorated_function\n retu
rn function(self, context, *args, **kwargs)\n', ' File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1931, in get_console_output\n output = self.driver.get_console_output(instance)\n', ' File "/usr/lib
/python2.6/site-packages/nova/exception.py", line 117, in wrapped\n temp_level, payload)\n', ' File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__\n self.gen.next()\n', ' File "/usr/lib/python2.6/site-pack
ages/nova/exception.py", line 92, in wrapped\n return f(*args, **kw)\n', ' File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1150, in get_console_output\n libvirt_utils.chown(path, os.getuid())\
n', ' File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/utils.py", line 332, in chown\n execute(\'chown\', owner, path, run_as_root=True)\n', ' File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/utils.py", l
ine 53, in execute\n return utils.execute(*args, **kwargs)\n', ' File "/usr/lib/python2.6/site-packages/nova/utils.py", line 206, in execute\n cmd=\' \'.join(cmd))\n', 'ProcessExecutionError: Unexpected error while run
ning command.\nCommand: sudo nova-rootwrap /etc/nova/rootwrap.conf chown 162 /var/lib/nova/instances/instance-000000f2/console.log\nExit code: 1\nStdout: \'\'\nStderr: "/bin/chown: cannot access `/var/lib/nova/instances/insta
nce-000000f2/console.log\': No such file or directory\\n"\n']
-----Original Message-----
From: Russell Bryant [mailto:rbryant at redhat.com]
Sent: Thursday, March 21, 2013 12:42 PM
To: Minton, Rich
Cc: rhos-list at redhat.com
Subject: EXTERNAL: Re: Problems after Update.
Based on the instance data, there should be a Traceback in the log somewhere. Check nova-compute.log.
--
Russell Bryant
On 03/21/2013 12:23 PM, Minton, Rich wrote:
> I found the source of a couple of my problems...
>
>
>
> The update wiped out my glance-api-paste.ini and
> glance-registry-paste.ini. Thank fully I had backups.
>
>
>
> I still have instances that are in shutoff/shutdown mode. I was able
> to get them to back to "active" but then they went to "shutoff" soon
> after. Here is the "nova show" for one of the instances.
>
>
>
> +-------------------------------------+------------------------------------------------------------------------------------------+
>
> | Property |
> Value
> |
>
> +-------------------------------------+------------------------------------------------------------------------------------------+
>
> | OS-DCF:diskConfig |
> MANUAL
> |
>
> | OS-EXT-SRV-ATTR:host |
> uvslp-diu01os.lmdit.us.lmco.com
> |
>
> | OS-EXT-SRV-ATTR:hypervisor_hostname |
> uvslp-diu01os.lmdit.us.lmco.com
> |
>
> | OS-EXT-SRV-ATTR:instance_name |
> instance-000000ea
> |
>
> | OS-EXT-STS:power_state |
> 4
> |
>
> | OS-EXT-STS:task_state |
> None
> |
>
> | OS-EXT-STS:vm_state |
> error
> |
>
> | accessIPv4
> |
> |
>
> | accessIPv6
> |
> |
>
> | config_drive
> |
> |
>
> | created |
> 2013-03-18T15:39:29Z
> |
>
> | fault | {u'message': u'IOError',
> u'code': 500, u'created': u'2013-03-21T16:17:12Z'} |
>
> | flavor | m1.medium
> (3)
> |
>
> | hostId |
> e55588f13411a788271d5273f0dabfa2eb755e0323dab51039dab866
> |
>
> | id |
> a39787b8-2c66-4aaa-b154-42ec05c0e3c8
> |
>
> | image | Red Hat Enterprise Linux 6.4,
> x86_64, Base Server (71c092f5-55de-4b3b-a9d9-4fcb25e15a89) |
>
> | key_name |
> TestKeys
> |
>
> | lmicc-access-nic network |
> 10.10.16.11
> |
>
> | metadata |
> {}
> |
>
> | name |
> demo
> |
>
> | security_groups | [{u'name':
> u'default'}]
> |
>
> | status |
> ERROR
> |
>
> | tenant_id |
> e4ba09fbf76c41a29218dc68cacf78d5
> |
>
> | updated |
> 2013-03-21T16:17:12Z
> |
>
> | user_id |
> f015ea92bf4848a891b73ae2fbf6c75b
> |
>
> +-------------------------------------+------------------------------------------------------------------------------------------+
>
>
>
> *From:*rhos-list-bounces at redhat.com
> [mailto:rhos-list-bounces at redhat.com] *On Behalf Of *Minton, Rich
> *Sent:* Thursday, March 21, 2013 11:45 AM
> *To:* rhos-list at redhat.com
> *Subject:* EXTERNAL: [rhos-list] Problems after Update.
>
>
>
> Where to begin... Basically, I'm hosed.
>
>
>
> I performed an update to the latest versions of the Openstack distro
> and rebooted my hosts. Then problems started...
>
> * Can't launch new instances-the in progress star comes up and
> stays there almost forever.
>
> * Several of my instances went to an error state. When I tried
> to reset their state to "active" they went into shutdown/shutoff state.
> These are the instances on my controller/compute node. I typed "virsh
> list" and the list was empty.
>
> * I tried to bring up a VNC Console on one of the running
> instances and after several minutes received this error in the browser:
>
> ----------------------------------------------------------------------
> ---------------------------------------------------------------------
>
> CommunicationError at
> /nova/instances/1f6cb3f6-c4d4-45ac-890b-bf2770a90c82/
>
>
>
> _Error communicating with http://10.10.12.245:9292 timed out_
>
>
>
> Request Method: GET
>
> Request URL:
> http://10.10.12.245/dashboard/nova/instances/1f6cb3f6-c4d4-45ac-890b-b
> f2770a90c82/?tab=instance_details__vnc
>
> Django Version: 1.4.2
>
> Exception Type: CommunicationError
>
> Exception Value:
>
>
>
> Error communicating with http://10.10.12.245:9292 timed out
>
>
>
> Exception Location:
> /usr/lib/python2.6/site-packages/glanceclient/common/http.py in
> _http_request, line 145
>
> Python Executable: /usr/bin/python
>
> Python Version: 2.6.6
>
> Python Path:
>
>
>
> ['/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../..',
>
> '/usr/lib64/python26.zip',
>
> '/usr/lib64/python2.6',
>
> '/usr/lib64/python2.6/plat-linux2',
>
> '/usr/lib64/python2.6/lib-tk',
>
> '/usr/lib64/python2.6/lib-old',
>
> '/usr/lib64/python2.6/lib-dynload',
>
> '/usr/lib64/python2.6/site-packages',
>
> '/usr/lib64/python2.6/site-packages/gtk-2.0',
>
> '/usr/lib/python2.6/site-packages/WebOb-1.0.8-py2.6.egg',
>
> '/usr/lib/python2.6/site-packages/PasteDeploy-1.5.0-py2.6.egg',
>
> '/usr/lib/python2.6/site-packages',
>
> '/usr/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg-info',
>
> '/usr/share/openstack-dashboard/openstack_dashboard']
>
>
>
> Server time: Thu, 21 Mar 2013 15:26:38 +0000
>
> ----------------------------------------------------------------------
> --------------------------------------------------------------
>
> * I tried deleting iptables and rebooting but did not solve
> anything.
>
> * I'm not seeing anything out of the ordinary in the nova logs.
>
>
>
> Any ideas on where I should start looking?
>
>
>
> Thanks for any help.
>
> Rick
>
>
>
> _Richard Minton_
>
> LMICC Systems Administrator
>
> 4000 Geerdes Blvd, 13D31
>
> King of Prussia, PA 19406
>
> Phone: 610-354-5482
>
>
>
_______________________________________________
rhos-list mailing list
rhos-list at redhat.com
https://www.redhat.com/mailman/listinfo/rhos-list
More information about the rhos-list
mailing list