[Rdo-list] Quick rabbitmq question....
Erich Weiler
weiler at soe.ucsc.edu
Fri Nov 20 05:32:49 UTC 2015
Totally fixed it. Thanks man. I learn something new about systemd
every day....
On 11/19/15 7:55 PM, Jeff Weber wrote:
> I struggled with this as well until I found out the limits.conf entries
> don't apply to systemd managed services.
>
> If you create a /etc/systemd/system/rabbitmq-server.service.d directory
> and place a limits.conf file in there with contents similar to
>
> [Service]
> LimitNOFILE=4096
>
> Then reload + restart
>
> systemctl daemon-reload
> systemctl restart rabbitmq-server
>
> https://fedoraproject.org/wiki/Systemd#How_do_I_customize_a_unit_file.2F_add_a_custom_unit_file.3F
>
>
>
> On Thu, Nov 19, 2015 at 10:43 PM, Erich Weiler <weiler at soe.ucsc.edu
> <mailto:weiler at soe.ucsc.edu>> wrote:
>
> Hi Y'all,
>
> I'm sure someone has encountered this issue... basically my
> rabbitmq instance on my controller node is running out of file
> descriptors, this is on RHEL 7. I've upped the max file descriptors
> (nofile) to 1000000 in /etc/security/limits.conf, and my sysctl
> limit for file descriptors is equally huge. Yet, I can't get my
> rabbitmq process to get it's limit's past 1000 or so:
>
> [root at os-con-01 ~]# ps -afe | grep rabbit
> rabbitmq 4989 1 4 16:42 ? 00:07:10
> /usr/lib64/erlang/erts-5.10.4/bin/beam.smp -W w -K true -A30 -P
> 1048576 -- -root /usr/lib64/erlang -progname erl -- -home
> /var/lib/rabbitmq -- -pa
> /usr/lib/rabbitmq/lib/rabbitmq_server-3.3.5/sbin/../ebin -noshell
> -noinput -s rabbit boot -sname rabbit at os-con-01 -boot start_sasl
> -kernel inet_default_connect_options [{nodelay,true}] -sasl
> errlog_type error -sasl sasl_error_logger false -rabbit error_logger
> {file,"/var/log/rabbitmq/rabbit at os-con-01.log"} -rabbit
> sasl_error_logger
> {file,"/var/log/rabbitmq/rabbit at os-con-01-sasl.log"} -rabbit
> enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit
> plugins_dir
> "/usr/lib/rabbitmq/lib/rabbitmq_server-3.3.5/sbin/../plugins"
> -rabbit plugins_expand_dir
> "/var/lib/rabbitmq/mnesia/rabbit at os-con-01-plugins-expand" -os_mon
> start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup
> false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit at os-con-01"
> -kernel inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672
> rabbitmq 5004 1 0 16:42 ? 00:00:00
> /usr/lib64/erlang/erts-5.10.4/bin/epmd -daemon
> rabbitmq 5129 4989 0 16:42 ? 00:00:00 inet_gethost 4
> rabbitmq 5130 5129 0 16:42 ? 00:00:00 inet_gethost 4
> root 17470 17403 0 19:34 pts/0 00:00:00 grep --color=auto rabbit
>
> [root at os-con-01 ~]# cat /proc/4989/limits
> Limit Soft Limit Hard Limit
> Units
> Max cpu time unlimited unlimited
> seconds
> Max file size unlimited unlimited
> bytes
> Max data size unlimited unlimited
> bytes
> Max stack size 8388608 unlimited
> bytes
> Max core file size 0 unlimited
> bytes
> Max resident set unlimited unlimited
> bytes
> Max processes 127788 127788 processes
> Max open files 1024 4096
> files
> Max locked memory 65536 65536
> bytes
> Max address space unlimited unlimited
> bytes
> Max file locks unlimited unlimited
> locks
> Max pending signals 127788 127788
> signals
> Max msgqueue size 819200 819200
> bytes
> Max nice priority 0 0
> Max realtime priority 0 0
> Max realtime timeout unlimited unlimited us
> [root at os-con-01 ~]#
>
> This is causing huge problems in my OpenStack cluster (Kilo
> Release). I've read that you can set this limit in
> /etc/rabbitmq/rabbitmq-env.conf or /etc/rabbitmq/rabbitmq.config but
> no matter what I do there I get nothing, after restarting rabbitmq
> many times. Does this have something to do with systemd?
>
> [root at os-con-01 ~]# rabbitmqctl status
> Status of node 'rabbit at os-con-01' ...
> [{pid,4989},
> {running_applications,[{rabbit,"RabbitMQ","3.3.5"},
> {os_mon,"CPO CXC 138 46","2.2.14"},
> {mnesia,"MNESIA CXC 138 12","4.11"},
> {xmerl,"XML parser","1.3.6"},
> {sasl,"SASL CXC 138 11","2.3.4"},
> {stdlib,"ERTS CXC 138 10","1.19.4"},
> {kernel,"ERTS CXC 138 10","2.16.4"}]},
> {os,{unix,linux}},
> {erlang_version,"Erlang R16B03-1 (erts-5.10.4) [source] [64-bit]
> [smp:32:32] [async-threads:30] [hipe] [kernel-poll:true]\n"},
> {memory,[{total,645523200},
> {connection_procs,32257624},
> {queue_procs,48513416},
> {plugins,0},
> {other_proc,15448376},
> {mnesia,1209984},
> {mgmt_db,0},
> {msg_index,292800},
> {other_ets,1991744},
> {binary,517865992},
> {code,16698259},
> {atom,602729},
> {other_system,10642276}]},
> {alarms,[]},
> {listeners,[{clustering,25672,"::"},{amqp,5672,"::"}]},
> {vm_memory_high_watermark,0.4},
> {vm_memory_limit,13406973132},
> {disk_free_limit,50000000},
> {disk_free,37354610688},
> {file_descriptors,[{total_limit,924}, <----- ??????
> {total_used,831},
> {sockets_limit,829},
> {sockets_used,829}]},
> {processes,[{limit,1048576},{used,8121}]},
> {run_queue,0},
> {uptime,10537}]
> ...done.
>
> Anyone know how to get the file descriptor limits up for rabbitmq?
> I've only got like 40 nodes in my OpenStack cluster, and it's
> choking, and I need to add several hundred more nodes...
>
> Any help much appreciated!!! I looked around the list and couldn't
> find anything on this, and I've RTFM'd as much as I could...
>
> cheers,
> erich
>
> _______________________________________________
> Rdo-list mailing list
> Rdo-list at redhat.com <mailto:Rdo-list at redhat.com>
> https://www.redhat.com/mailman/listinfo/rdo-list
>
> To unsubscribe: rdo-list-unsubscribe at redhat.com
> <mailto:rdo-list-unsubscribe at redhat.com>
>
>
More information about the rdo-list
mailing list