Sunday, October 08, 2017

OpenStack Neutron Server configuration parameters and their values

During troubleshooting I was having difficulty tracing a specific Neutron configuration parameter, specifically I needed to determine what order the default router types are applied. In the case of VMware Integrated OpenStack (aka VIO), Neutron Server uses NSX as the SDN controller, so two configuration files are in play:

  • /etc/neutron/neutron.conf
  • /etc/neutron/plugins/vmware/nsxv.ini

I was looking for the tenant_router_types value, the neutron-server log states it was set to shared, distributed, exclusive and I want to set it to distributed, shared, exclusive. I know this is an easy fix, just update the configuration files and restart neutron-server but from some reason my brain wasn’t working well this day and I wasn’t able to find that parameter in either nsxv.ini.

I ultimately did some grepping in the neutron-server log to find the Python method that is used to retrieve the configuration parameters from the configuration files. The specific method, oslo_service.service, can be searched for in any of the OpenStack service logs and will give you not only the list of parameters used by that service but also all of the values that are assigned. I like to do this when I’m troubleshooting, this way I can keep the configuration version controlled while I change things.

Hopefully you are running a centralized log aggregation solution (e.g. vRealize Log Insight, Splunk, ELK), you should be able to search for that string and retrieve the output. If you aren’t and you unfamiliar with the OpenStack logs here are the exact steps to retrieve the parameters straight from the logs, note that if the logs that contain the Neutron Server start up sequence have rolled you will have to restart the service.

  1. SSH to the server running the Neutron-Server service. In the case of VIO you must SSH to the OMS first, then SSH to either loadbalancer01 if running in Compact Mode or controller01 in HA mode.

  2. Next, run the following command to grep the logs for the specific entry.

    grep -R 'oslo_service.service' /var/log/neutron
  3. This should return a bunch of lines that look like this:

    /var/log/neutron/neutron-server.log.1:2017-10-07 16:51:54.408 27313 DEBUG oslo_service.service [-] agent_down_time                = 75 log_opt_values /usr/lib/python2.7/dist-packages/oslo_config/cfg.py:2738
  4. You can also filter the log metadata out, this sed command isn’t perfect but it works for me.

    grep -R 'DEBUG oslo_service.service \[\-\]' /var/log/neutron/ | sed 's/.* \[\-\]//;s/ log_opt_values .*//;s/^[ \t]*//' | sed '/.*[\*]/,$!d' | sed -n 'G; s/\n/&&/; /^\([ -~]*\n\).*\n\1/d; s/\n//; h; P'

    This should result in a bunch of lines that now look like this:

    agent_down_time                = 75