[Spacewalk-list] IOError: [Errno 28] No space left on device

Fri Jul 17 16:16:57 UTC 2015

Do you have any suggestions outside of disk space and inode utilization?

On 7/15/15, 11:07 AM, "Robert Paschedag" <robert.paschedag at web.de> wrote:

>Can you give output of "df -h". I assume, spacewalk temporarily writes to
>/var or /tmp.
>
>Am 15.07.2015 3:50 nachm. schrieb "Camp, Neil (NIH/NCI) [C]"
><neil.camp at nih.gov>:
>>
>> Hello,
>>
>> I am trying to upload a deb file that is 1019MB. It keeps timing out:
>>
>> INFO: 1/4301 : ns3-doc_3.17+dfsg-1build1_all.deb
>> Error pushing /var/tmp/ns3-doc_3.17+dfsg-1build1_all.deb:  (500)
>> Waiting 1 seconds and trying again...
>> Error pushing /var/tmp/ns3-doc_3.17+dfsg-1build1_all.deb:  (500)
>> Waiting 4 seconds and trying again...
>> Error pushing /var/tmp/ns3-doc_3.17+dfsg-1build1_all.deb:  (500)
>> Waiting 4 seconds and trying again...
>> Giving up after 3 attempts
>> Internal server error 500 Internal Server Error
>> Internal server error 500 Internal Server Error
>> Internal server error 500 Internal Server Error
>> ERROR: rhnpush failed
>>
>> And the apache error_log shows the following:
>>
>> [Wed Jul 15 09:37:27 2015] [error] Spacewalk 3027 2015/07/15 09:37:27
>>-04:00: ('Unhandled exception',)
>> [Wed Jul 15 09:38:14 2015] [error] Exception reported from ---
>> [Wed Jul 15 09:38:14 2015] [error] Time: Wed Jul 15 09:38:14 2015
>> [Wed Jul 15 09:38:14 2015] [error] Exception type <type
>>'exceptions.IOError'>
>> [Wed Jul 15 09:38:14 2015] [error] Exception while handling function
>>server.apacheUploadServer._wrapper
>> [Wed Jul 15 09:38:14 2015] [error] Request object information:
>> [Wed Jul 15 09:38:14 2015] [error] URI: /PACKAGE-PUSH
>> [Wed Jul 15 09:38:14 2015] [error] Remote Host: localhost
>> [Wed Jul 15 09:38:14 2015] [error] Server Name: localhost:443
>> [Wed Jul 15 09:38:14 2015] [error] Headers passed in:
>> [Wed Jul 15 09:38:14 2015] [error] \tAccept-Encoding: identity
>> [Wed Jul 15 09:38:14 2015] [error] \tCONTENT_LENGTH: 1067972176
>> [Wed Jul 15 09:38:14 2015] [error] \tCONTENT_TYPE: application/x-rpm
>> [Wed Jul 15 09:38:14 2015] [error] \tDOCUMENT_ROOT: /var/www/html
>> [Wed Jul 15 09:38:14 2015] [error] \tGATEWAY_INTERFACE: CGI/1.1
>> [Wed Jul 15 09:38:14 2015] [error] \tHTTPS: 1
>> [Wed Jul 15 09:38:14 2015] [error] \tHTTP_ACCEPT_ENCODING: identity
>> [Wed Jul 15 09:38:14 2015] [error] \tHTTP_HOST: localhost
>> [Wed Jul 15 09:38:14 2015] [error] \tHTTP_USER_AGENT: rhnpush
>> [Wed Jul 15 09:38:14 2015] [error] \tHTTP_X_RHN_UPLOAD_AUTH_SESSION:
>>51612x7d6f79c28c87cebea3a6fa646c188c57778bff4d466dd03af8f45a70fafca30f
>> [Wed Jul 15 09:38:14 2015] [error] \tHTTP_X_RHN_UPLOAD_FILE_MD5SUM:
>>8bbfcd73ef4814742f5c7a0a6a900427
>> [Wed Jul 15 09:38:14 2015] [error] \tHTTP_X_RHN_UPLOAD_FORCE: 0
>> [Wed Jul 15 09:38:14 2015] [error] \tHTTP_X_RHN_UPLOAD_PACKAGE_ARCH:
>>all-deb
>> [Wed Jul 15 09:38:14 2015] [error] \tHTTP_X_RHN_UPLOAD_PACKAGE_NAME:
>>ns3-doc
>> [Wed Jul 15 09:38:14 2015] [error] \tHTTP_X_RHN_UPLOAD_PACKAGE_RELEASE:
>>1build1
>> [Wed Jul 15 09:38:14 2015] [error] \tHTTP_X_RHN_UPLOAD_PACKAGE_VERSION:
>>3.17+dfsg
>> [Wed Jul 15 09:38:14 2015] [error] \tHTTP_X_RHN_UPLOAD_PACKAGING: deb
>> [Wed Jul 15 09:38:14 2015] [error] \tHost: localhost
>> [Wed Jul 15 09:38:14 2015] [error] \tPATH_INFO:
>> [Wed Jul 15 09:38:14 2015] [error] \tQUERY_STRING:
>> [Wed Jul 15 09:38:14 2015] [error] \tREMOTE_ADDR: ::1
>> [Wed Jul 15 09:38:14 2015] [error] \tREMOTE_PORT: 51536
>> [Wed Jul 15 09:38:14 2015] [error] \tREQUEST_METHOD: POST
>> [Wed Jul 15 09:38:14 2015] [error] \tREQUEST_URI: /PACKAGE-PUSH
>> [Wed Jul 15 09:38:14 2015] [error] \tSCRIPT_FILENAME:
>>/usr/share/rhn/wsgi/package_push.py
>> [Wed Jul 15 09:38:14 2015] [error] \tSCRIPT_NAME: /PACKAGE-PUSH
>> [Wed Jul 15 09:38:14 2015] [error] \tSCRIPT_URI:
>>https://localhost/PACKAGE-PUSH
>> [Wed Jul 15 09:38:14 2015] [error] \tSCRIPT_URL: /PACKAGE-PUSH
>> [Wed Jul 15 09:38:14 2015] [error] \tSERVER_ADDR: ::1
>> [Wed Jul 15 09:38:14 2015] [error] \tSERVER_ADMIN: root at localhost
>> [Wed Jul 15 09:38:14 2015] [error] \tSERVER_NAME: localhost
>> [Wed Jul 15 09:38:14 2015] [error] \tSERVER_PORT: 443
>> [Wed Jul 15 09:38:14 2015] [error] \tSERVER_PROTOCOL: HTTP/1.1
>> [Wed Jul 15 09:38:14 2015] [error] \tSERVER_SIGNATURE: <address>Apache
>>Server at localhost Port 443</address>
>> [Wed Jul 15 09:38:14 2015] [error]
>> [Wed Jul 15 09:38:14 2015] [error] \tSERVER_SOFTWARE: Apache
>> [Wed Jul 15 09:38:14 2015] [error] \tUser-Agent: rhnpush
>> [Wed Jul 15 09:38:14 2015] [error] \tX-RHN-Upload-Auth-Session:
>>51612x7d6f79c28c87cebea3a6fa646c188c57778bff4d466dd03af8f45a70fafca30f
>> [Wed Jul 15 09:38:14 2015] [error] \tX-RHN-Upload-File-Checksum:
>>8bbfcd73ef4814742f5c7a0a6a900427
>> [Wed Jul 15 09:38:14 2015] [error] \tX-RHN-Upload-File-Checksum-Type:
>>md5
>> [Wed Jul 15 09:38:14 2015] [error] \tX-RHN-Upload-File-MD5sum:
>>8bbfcd73ef4814742f5c7a0a6a900427
>> [Wed Jul 15 09:38:14 2015] [error] \tX-RHN-Upload-Force: 0
>> [Wed Jul 15 09:38:14 2015] [error] \tX-RHN-Upload-Package-Arch: all-deb
>> [Wed Jul 15 09:38:14 2015] [error] \tX-RHN-Upload-Package-Name: ns3-doc
>> [Wed Jul 15 09:38:14 2015] [error] \tX-RHN-Upload-Package-Release:
>>1build1
>> [Wed Jul 15 09:38:14 2015] [error] \tX-RHN-Upload-Package-Version:
>>3.17+dfsg
>> [Wed Jul 15 09:38:14 2015] [error] \tX-RHN-Upload-Packaging: deb
>> [Wed Jul 15 09:38:14 2015] [error] \tmod_ssl.is_https: <built-in method
>>ssl_is_https of mod_wsgi.Adapter object at 0x7f4f2f38ab70>
>> [Wed Jul 15 09:38:14 2015] [error] \tmod_ssl.var_lookup: <built-in
>>method ssl_var_lookup of mod_wsgi.Adapter object at 0x7f4f2f38ab70>
>> [Wed Jul 15 09:38:14 2015] [error] \tmod_wsgi.application_group:
>>ncias-d1480-v.nci.nih.gov|/package-push
>> [Wed Jul 15 09:38:14 2015] [error] \tmod_wsgi.callable_object:
>>application
>> [Wed Jul 15 09:38:14 2015] [error] \tmod_wsgi.handler_script:
>> [Wed Jul 15 09:38:14 2015] [error] \tmod_wsgi.input_chunked: 0
>> [Wed Jul 15 09:38:14 2015] [error] \tmod_wsgi.listener_host:
>> [Wed Jul 15 09:38:14 2015] [error] \tmod_wsgi.listener_port: 443
>> [Wed Jul 15 09:38:14 2015] [error] \tmod_wsgi.process_group:
>> [Wed Jul 15 09:38:14 2015] [error] \tmod_wsgi.request_handler:
>>wsgi-script
>> [Wed Jul 15 09:38:14 2015] [error] \tmod_wsgi.script_reloading: 1
>> [Wed Jul 15 09:38:14 2015] [error] \tmod_wsgi.version: (3, 2)
>> [Wed Jul 15 09:38:14 2015] [error] \twsgi.errors: <mod_wsgi.Log object
>>at 0x7f4f2f3bfbb0>
>> [Wed Jul 15 09:38:14 2015] [error] \twsgi.file_wrapper: <built-in
>>method file_wrapper of mod_wsgi.Adapter object at 0x7f4f2f38ab70>
>> [Wed Jul 15 09:38:14 2015] [error] \twsgi.input: <mod_wsgi.Input object
>>at 0x7f4f2f3bfb70>
>> [Wed Jul 15 09:38:14 2015] [error] \twsgi.multiprocess: True
>> [Wed Jul 15 09:38:14 2015] [error] \twsgi.multithread: False
>> [Wed Jul 15 09:38:14 2015] [error] \twsgi.run_once: False
>> [Wed Jul 15 09:38:14 2015] [error] \twsgi.url_scheme: https
>> [Wed Jul 15 09:38:14 2015] [error] \twsgi.version: (1, 1)
>> [Wed Jul 15 09:38:14 2015] [error]
>> [Wed Jul 15 09:38:14 2015] [error] Exception Handler Information
>> [Wed Jul 15 09:38:14 2015] [error] Traceback (most recent call last):
>> [Wed Jul 15 09:38:14 2015] [error]   File
>>"/usr/lib/python2.6/site-packages/spacewalk/server/apacheUploadServer.py"
>>, line 102, in _wrapper
>> [Wed Jul 15 09:38:14 2015] [error]     log_debug(5, "Calling", function)
>> [Wed Jul 15 09:38:14 2015] [error]   File
>>"/usr/share/rhn/upload_server/handlers/package_push/package_push.py",
>>line 115, in handler
>> [Wed Jul 15 09:38:14 2015] [error]     self.file_checksum_type,
>>self.file_checksum)
>> [Wed Jul 15 09:38:14 2015] [error]   File
>>"/usr/lib/python2.6/site-packages/spacewalk/server/rhnPackageUpload.py",
>>line 272, in save_uploaded_package
>> [Wed Jul 15 09:38:14 2015] [error]     a_pkg.read_header()
>> [Wed Jul 15 09:38:14 2015] [error]   File
>>"/usr/lib/python2.6/site-packages/spacewalk/common/rhn_deb.py", line
>>118, in read_header
>> [Wed Jul 15 09:38:14 2015] [error]
>>self._stream_copy(self.input_stream, self.header_data)
>> [Wed Jul 15 09:38:14 2015] [error]   File
>>"/usr/lib/python2.6/site-packages/spacewalk/common/rhn_pkg.py", line
>>119, in _stream_copy
>> [Wed Jul 15 09:38:14 2015] [error]     dest.write(buf)
>> [Wed Jul 15 09:38:14 2015] [error] IOError: [Errno 28] No space left on
>>device
>> [Wed Jul 15 09:38:14 2015] [error]
>> [Wed Jul 15 09:38:14 2015] [error] Spacewalk 3023 2015/07/15 09:38:14
>>-04:00: ('Unhandled exception¹,)
>>
>> I hace seen similar issues but none of them explain this problem. I
>>have plenty of space on the file system, 28GB free so the file system is
>>not the problem. Is it the file size or some other issue? I appreciate
>>any help, thank you.