2026-04-20 05:21:28.243 [info] <0.8.0> Log file opened with Lager
2026-04-20 05:21:31.668 [info] <0.8.0> Feature flags: list of feature flags found:
2026-04-20 05:21:31.668 [info] <0.8.0> Feature flags: feature flag states written to disk: yes
2026-04-20 05:21:31.694 [info] <0.240.0> 
 Starting RabbitMQ 3.7.28 on Erlang 21.3.8.21
 Copyright (c) 2007-2020 Pivotal Software, Inc.
 Licensed under the MPL.  See https://www.rabbitmq.com/
2026-04-20 05:21:31.694 [info] <0.240.0> 
 node           : contrail@cn-jenkins-deploy-platform-ansible-os-5649-3
 home dir       : /var/lib/rabbitmq
 config file(s) : /etc/rabbitmq/rabbitmq.config
 cookie hash    : esZsDxSN6VGbi9JkMSxNZA==
 log(s)         : /var/log/contrail/config-database-rabbitmq/rabbitmq.log
                : /var/log/rabbitmq/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3_upgrade.log
 database dir   : /var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3
2026-04-20 05:21:31.706 [info] <0.240.0> Running boot step pre_boot defined by app rabbit
2026-04-20 05:21:31.707 [info] <0.240.0> Running boot step rabbit_core_metrics defined by app rabbit
2026-04-20 05:21:31.707 [info] <0.240.0> Running boot step rabbit_alarm defined by app rabbit
2026-04-20 05:21:31.712 [info] <0.246.0> Memory high watermark set to 12637 MiB (13251081011 bytes) of 15796 MiB (16563851264 bytes) total
2026-04-20 05:21:31.717 [info] <0.248.0> Enabling free disk space monitoring
2026-04-20 05:21:31.717 [info] <0.248.0> Disk free limit set to 50MB
2026-04-20 05:21:31.728 [info] <0.240.0> Running boot step code_server_cache defined by app rabbit
2026-04-20 05:21:31.728 [info] <0.240.0> Running boot step file_handle_cache defined by app rabbit
2026-04-20 05:21:31.728 [info] <0.251.0> Limiting to approx 65436 file handles (58890 sockets)
2026-04-20 05:21:31.728 [info] <0.252.0> FHC read buffering:  OFF
2026-04-20 05:21:31.728 [info] <0.252.0> FHC write buffering: ON
2026-04-20 05:21:31.729 [info] <0.240.0> Running boot step worker_pool defined by app rabbit
2026-04-20 05:21:31.729 [info] <0.241.0> Will use 4 processes for default worker pool
2026-04-20 05:21:31.729 [info] <0.241.0> Starting worker pool 'worker_pool' with 4 processes in it
2026-04-20 05:21:31.729 [info] <0.240.0> Running boot step database defined by app rabbit
2026-04-20 05:21:31.731 [info] <0.240.0> Node database directory at /var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3 is empty. Assuming we need to join an existing cluster or initialise from scratch...
2026-04-20 05:21:31.731 [info] <0.240.0> Configured peer discovery backend: rabbit_peer_discovery_classic_config
2026-04-20 05:21:31.731 [info] <0.240.0> Will try to lock with peer discovery backend rabbit_peer_discovery_classic_config
2026-04-20 05:21:31.731 [info] <0.240.0> Peer discovery backend does not support locking, falling back to randomized delay
2026-04-20 05:21:31.731 [info] <0.240.0> Peer discovery backend rabbit_peer_discovery_classic_config supports registration.
2026-04-20 05:21:31.731 [info] <0.240.0> Will wait for 35753 milliseconds before proceeding with registration...
2026-04-20 05:22:07.485 [info] <0.240.0> All discovered existing cluster peers: contrail@cn-jenkins-deploy-platform-ansible-os-5649-1, contrail@cn-jenkins-deploy-platform-ansible-os-5649-2, contrail@cn-jenkins-deploy-platform-ansible-os-5649-3
2026-04-20 05:22:07.485 [info] <0.240.0> Peer nodes we can cluster with: contrail@cn-jenkins-deploy-platform-ansible-os-5649-1, contrail@cn-jenkins-deploy-platform-ansible-os-5649-2
2026-04-20 05:22:07.489 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,tables_not_present}
2026-04-20 05:22:07.493 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:22:07.493 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 9 retries left...
2026-04-20 05:22:07.995 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,tables_not_present}
2026-04-20 05:22:07.996 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:22:07.996 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 8 retries left...
2026-04-20 05:22:08.498 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,tables_not_present}
2026-04-20 05:22:08.499 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:22:08.499 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 7 retries left...
2026-04-20 05:22:09.004 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,tables_not_present}
2026-04-20 05:22:09.008 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:22:09.009 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 6 retries left...
2026-04-20 05:22:09.375 [info] <0.300.0> RabbitMQ hasn't finished starting yet. Waiting for startup to finish before stopping...
2026-04-20 05:22:09.509 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,tables_not_present}
2026-04-20 05:22:09.510 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:22:09.510 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 5 retries left...
2026-04-20 05:22:10.012 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,tables_not_present}
2026-04-20 05:22:10.013 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:22:10.013 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 4 retries left...
2026-04-20 05:22:10.514 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,tables_not_present}
2026-04-20 05:22:10.515 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:22:10.515 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 3 retries left...
2026-04-20 05:22:11.017 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,tables_not_present}
2026-04-20 05:22:11.017 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:22:11.018 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 2 retries left...
2026-04-20 05:22:11.518 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,tables_not_present}
2026-04-20 05:22:11.520 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:22:11.520 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 1 retries left...
2026-04-20 05:22:12.021 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,tables_not_present}
2026-04-20 05:22:12.022 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:22:12.023 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 0 retries left...
2026-04-20 05:22:12.524 [warning] <0.240.0> Could not successfully contact any node of: contrail@cn-jenkins-deploy-platform-ansible-os-5649-1,contrail@cn-jenkins-deploy-platform-ansible-os-5649-2 (as in Erlang distribution). Starting as a blank standalone node...
2026-04-20 05:22:12.526 [info] <0.43.0> Application mnesia exited with reason: stopped
2026-04-20 05:22:12.659 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2026-04-20 05:22:12.659 [info] <0.240.0> Successfully synced tables from a peer
2026-04-20 05:22:12.691 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2026-04-20 05:22:12.692 [info] <0.240.0> Successfully synced tables from a peer
2026-04-20 05:22:12.723 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2026-04-20 05:22:12.723 [info] <0.240.0> Successfully synced tables from a peer
2026-04-20 05:22:12.724 [info] <0.240.0> Will register with peer discovery backend rabbit_peer_discovery_classic_config
2026-04-20 05:22:12.724 [info] <0.240.0> Running boot step database_sync defined by app rabbit
2026-04-20 05:22:12.724 [info] <0.240.0> Running boot step feature_flags defined by app rabbit
2026-04-20 05:22:12.726 [info] <0.240.0> Running boot step codec_correctness_check defined by app rabbit
2026-04-20 05:22:12.726 [info] <0.240.0> Running boot step external_infrastructure defined by app rabbit
2026-04-20 05:22:12.727 [info] <0.240.0> Running boot step rabbit_registry defined by app rabbit
2026-04-20 05:22:12.727 [info] <0.240.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit
2026-04-20 05:22:12.727 [info] <0.240.0> Running boot step rabbit_queue_location_random defined by app rabbit
2026-04-20 05:22:12.728 [info] <0.240.0> Running boot step rabbit_event defined by app rabbit
2026-04-20 05:22:12.728 [info] <0.240.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit
2026-04-20 05:22:12.729 [info] <0.240.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit
2026-04-20 05:22:12.729 [info] <0.240.0> Running boot step rabbit_exchange_type_direct defined by app rabbit
2026-04-20 05:22:12.729 [info] <0.240.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit
2026-04-20 05:22:12.729 [info] <0.240.0> Running boot step rabbit_exchange_type_headers defined by app rabbit
2026-04-20 05:22:12.729 [info] <0.240.0> Running boot step rabbit_exchange_type_topic defined by app rabbit
2026-04-20 05:22:12.729 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit
2026-04-20 05:22:12.729 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit
2026-04-20 05:22:12.729 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit
2026-04-20 05:22:12.729 [info] <0.240.0> Running boot step rabbit_priority_queue defined by app rabbit
2026-04-20 05:22:12.729 [info] <0.240.0> Priority queues enabled, real BQ is rabbit_variable_queue
2026-04-20 05:22:12.730 [info] <0.240.0> Running boot step rabbit_queue_location_client_local defined by app rabbit
2026-04-20 05:22:12.730 [info] <0.240.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit
2026-04-20 05:22:12.730 [info] <0.240.0> Running boot step kernel_ready defined by app rabbit
2026-04-20 05:22:12.730 [info] <0.240.0> Running boot step rabbit_sysmon_minder defined by app rabbit
2026-04-20 05:22:12.730 [info] <0.240.0> Running boot step rabbit_epmd_monitor defined by app rabbit
2026-04-20 05:22:12.730 [info] <0.462.0> epmd monitor knows us, inter-node communication (distribution) port: 25673
2026-04-20 05:22:12.731 [info] <0.240.0> Running boot step guid_generator defined by app rabbit
2026-04-20 05:22:12.734 [info] <0.240.0> Running boot step rabbit_node_monitor defined by app rabbit
2026-04-20 05:22:12.734 [info] <0.466.0> Starting rabbit_node_monitor
2026-04-20 05:22:12.734 [info] <0.240.0> Running boot step delegate_sup defined by app rabbit
2026-04-20 05:22:12.735 [info] <0.240.0> Running boot step rabbit_memory_monitor defined by app rabbit
2026-04-20 05:22:12.735 [info] <0.240.0> Running boot step core_initialized defined by app rabbit
2026-04-20 05:22:12.735 [info] <0.240.0> Running boot step upgrade_queues defined by app rabbit
2026-04-20 05:22:12.757 [info] <0.240.0> message_store upgrades: 1 to apply
2026-04-20 05:22:12.757 [info] <0.240.0> message_store upgrades: Applying rabbit_variable_queue:move_messages_to_vhost_store
2026-04-20 05:22:12.757 [info] <0.240.0> message_store upgrades: No durable queues found. Skipping message store migration
2026-04-20 05:22:12.757 [info] <0.240.0> message_store upgrades: Removing the old message store data
2026-04-20 05:22:12.757 [info] <0.240.0> message_store upgrades: All upgrades applied successfully
2026-04-20 05:22:12.784 [info] <0.240.0> Running boot step rabbit_connection_tracking defined by app rabbit
2026-04-20 05:22:12.784 [info] <0.240.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit
2026-04-20 05:22:12.785 [info] <0.240.0> Running boot step rabbit_exchange_parameters defined by app rabbit
2026-04-20 05:22:12.785 [info] <0.240.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit
2026-04-20 05:22:12.786 [info] <0.240.0> Running boot step rabbit_policies defined by app rabbit
2026-04-20 05:22:12.787 [info] <0.240.0> Running boot step rabbit_policy defined by app rabbit
2026-04-20 05:22:12.787 [info] <0.240.0> Running boot step rabbit_queue_location_validator defined by app rabbit
2026-04-20 05:22:12.787 [info] <0.240.0> Running boot step rabbit_vhost_limit defined by app rabbit
2026-04-20 05:22:12.788 [info] <0.240.0> Running boot step rabbit_mgmt_reset_handler defined by app rabbitmq_management
2026-04-20 05:22:12.788 [info] <0.240.0> Running boot step rabbit_mgmt_db_handler defined by app rabbitmq_management_agent
2026-04-20 05:22:12.788 [warning] <0.240.0> force_fine_statistics set to true; ignored.
Replaced by {rates_mode, none} in the rabbitmq_management application.
2026-04-20 05:22:12.788 [info] <0.240.0> Management plugin: using rates mode 'basic'
2026-04-20 05:22:12.789 [info] <0.240.0> Running boot step recovery defined by app rabbit
2026-04-20 05:22:12.790 [info] <0.240.0> Running boot step load_definitions defined by app rabbitmq_management
2026-04-20 05:22:12.791 [info] <0.240.0> Applying definitions from /etc/rabbitmq/definitions.json
2026-04-20 05:22:12.791 [info] <0.240.0> Asked to import definitions. Acting user: rmq-internal
2026-04-20 05:22:12.792 [info] <0.240.0> Importing users...
2026-04-20 05:22:12.792 [info] <0.240.0> Creating user 'guest'
2026-04-20 05:22:12.798 [info] <0.240.0> Setting user tags for user 'guest' to [administrator]
2026-04-20 05:22:12.802 [info] <0.240.0> Importing vhosts...
2026-04-20 05:22:12.802 [info] <0.240.0> Adding vhost '/'
2026-04-20 05:22:12.824 [info] <0.514.0> Making sure data directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists
2026-04-20 05:22:12.829 [info] <0.514.0> Starting message stores for vhost '/'
2026-04-20 05:22:12.829 [info] <0.518.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index
2026-04-20 05:22:12.829 [info] <0.514.0> Started message store of type transient for vhost '/'
2026-04-20 05:22:12.830 [info] <0.521.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
2026-04-20 05:22:12.830 [warning] <0.521.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": rebuilding indices from scratch
2026-04-20 05:22:12.830 [info] <0.514.0> Started message store of type persistent for vhost '/'
2026-04-20 05:22:12.831 [info] <0.240.0> Importing user permissions...
2026-04-20 05:22:12.831 [info] <0.240.0> Setting permissions for 'guest' in '/' to '.*', '.*', '.*'
2026-04-20 05:22:12.835 [info] <0.240.0> Importing topic permissions...
2026-04-20 05:22:12.835 [info] <0.240.0> Importing parameters...
2026-04-20 05:22:12.835 [info] <0.240.0> Importing global parameters...
2026-04-20 05:22:12.835 [info] <0.240.0> Importing policies...
2026-04-20 05:22:12.843 [info] <0.240.0> Importing queues...
2026-04-20 05:22:12.843 [info] <0.240.0> Importing exchanges...
2026-04-20 05:22:12.843 [info] <0.240.0> Importing bindings...
2026-04-20 05:22:12.843 [info] <0.240.0> Running boot step empty_db_check defined by app rabbit
2026-04-20 05:22:12.843 [info] <0.240.0> Running boot step rabbit_looking_glass defined by app rabbit
2026-04-20 05:22:12.843 [info] <0.240.0> Running boot step rabbit_core_metrics_gc defined by app rabbit
2026-04-20 05:22:12.843 [info] <0.240.0> Running boot step background_gc defined by app rabbit
2026-04-20 05:22:12.843 [info] <0.240.0> Running boot step connection_tracking defined by app rabbit
2026-04-20 05:22:12.848 [info] <0.240.0> Setting up a table for connection tracking on this node: 'tracked_connection_on_node_contrail@cn-jenkins-deploy-platform-ansible-os-5649-3'
2026-04-20 05:22:12.854 [info] <0.240.0> Setting up a table for per-vhost connection counting on this node: 'tracked_connection_per_vhost_on_node_contrail@cn-jenkins-deploy-platform-ansible-os-5649-3'
2026-04-20 05:22:12.854 [info] <0.240.0> Running boot step routing_ready defined by app rabbit
2026-04-20 05:22:12.855 [info] <0.240.0> Running boot step pre_flight defined by app rabbit
2026-04-20 05:22:12.855 [info] <0.240.0> Running boot step notify_cluster defined by app rabbit
2026-04-20 05:22:12.855 [info] <0.240.0> Running boot step networking defined by app rabbit
2026-04-20 05:22:12.856 [info] <0.567.0> started TCP listener on [::]:5673
2026-04-20 05:22:12.857 [info] <0.240.0> Running boot step cluster_name defined by app rabbit
2026-04-20 05:22:12.857 [info] <0.240.0> Running boot step direct_client defined by app rabbit
2026-04-20 05:22:12.857 [info] <0.240.0> Running boot step os_signal_handler defined by app rabbit
2026-04-20 05:22:12.857 [info] <0.569.0> Swapping OS signal event handler (erl_signal_server) for our own
2026-04-20 05:22:12.884 [info] <0.619.0> Management plugin: HTTP (non-TLS) listener started on port 15673
2026-04-20 05:22:12.885 [info] <0.725.0> Statistics database started.
2026-04-20 05:22:12.885 [info] <0.724.0> Starting worker pool 'management_worker_pool' with 3 processes in it
2026-04-20 05:22:12.885 [notice] <0.106.0> Changed loghwm of /var/log/contrail/config-database-rabbitmq/rabbitmq.log to 50
2026-04-20 05:22:12.963 [info] <0.8.0> Server startup complete; 3 plugins started.
 * rabbitmq_management
 * rabbitmq_management_agent
 * rabbitmq_web_dispatch
2026-04-20 05:22:13.025 [info] <0.300.0> RabbitMQ is asked to stop...
2026-04-20 05:22:13.065 [info] <0.300.0> Stopping RabbitMQ applications and their dependencies in the following order:
    rabbitmq_management
    amqp_client
    rabbitmq_web_dispatch
    cowboy
    cowlib
    rabbitmq_management_agent
    rabbit
    mnesia
    rabbit_common
    sysmon_handler
    os_mon
2026-04-20 05:22:13.065 [info] <0.300.0> Stopping application 'rabbitmq_management'
2026-04-20 05:22:13.067 [warning] <0.575.0> RabbitMQ HTTP listener registry could not find context rabbitmq_management_tls
2026-04-20 05:22:13.068 [info] <0.43.0> Application rabbitmq_management exited with reason: stopped
2026-04-20 05:22:13.068 [info] <0.300.0> Stopping application 'amqp_client'
2026-04-20 05:22:13.070 [info] <0.43.0> Application amqp_client exited with reason: stopped
2026-04-20 05:22:13.070 [info] <0.300.0> Stopping application 'rabbitmq_web_dispatch'
2026-04-20 05:22:13.073 [info] <0.300.0> Stopping application 'cowboy'
2026-04-20 05:22:13.073 [info] <0.43.0> Application rabbitmq_web_dispatch exited with reason: stopped
2026-04-20 05:22:13.074 [info] <0.43.0> Application cowboy exited with reason: stopped
2026-04-20 05:22:13.074 [info] <0.300.0> Stopping application 'cowlib'
2026-04-20 05:22:13.075 [info] <0.43.0> Application cowlib exited with reason: stopped
2026-04-20 05:22:13.075 [info] <0.300.0> Stopping application 'rabbitmq_management_agent'
2026-04-20 05:22:13.076 [info] <0.43.0> Application rabbitmq_management_agent exited with reason: stopped
2026-04-20 05:22:13.076 [info] <0.300.0> Stopping application 'rabbit'
2026-04-20 05:22:13.076 [info] <0.240.0> Will unregister with peer discovery backend rabbit_peer_discovery_classic_config
2026-04-20 05:22:13.077 [info] <0.567.0> stopped TCP listener on [::]:5673
2026-04-20 05:22:13.077 [info] <0.491.0> Closing all connections in vhost '/' on node 'contrail@cn-jenkins-deploy-platform-ansible-os-5649-3' because the vhost is stopping
2026-04-20 05:22:13.077 [info] <0.521.0> Stopping message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent'
2026-04-20 05:22:13.082 [info] <0.521.0> Message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent' is stopped
2026-04-20 05:22:13.082 [info] <0.518.0> Stopping message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient'
2026-04-20 05:22:13.086 [info] <0.518.0> Message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient' is stopped
2026-04-20 05:22:13.089 [info] <0.43.0> Application rabbit exited with reason: stopped
2026-04-20 05:22:13.089 [info] <0.300.0> Stopping application 'mnesia'
2026-04-20 05:22:13.093 [info] <0.43.0> Application mnesia exited with reason: stopped
2026-04-20 05:22:13.093 [info] <0.300.0> Stopping application 'rabbit_common'
2026-04-20 05:22:13.093 [info] <0.43.0> Application rabbit_common exited with reason: stopped
2026-04-20 05:22:13.093 [info] <0.300.0> Stopping application 'sysmon_handler'
2026-04-20 05:22:13.094 [info] <0.43.0> Application sysmon_handler exited with reason: stopped
2026-04-20 05:22:13.094 [info] <0.300.0> Stopping application 'os_mon'
2026-04-20 05:22:13.095 [info] <0.43.0> Application os_mon exited with reason: stopped
2026-04-20 05:22:13.096 [info] <0.300.0> Successfully stopped RabbitMQ and its dependencies
2026-04-20 05:22:19.503 [info] <0.747.0> Resetting Rabbit forcefully
2026-04-20 05:22:20.406 [info] <0.759.0> RabbitMQ is asked to stop...
2026-04-20 05:22:20.483 [info] <0.759.0> Stopping RabbitMQ applications and their dependencies in the following order:
    rabbitmq_management
    amqp_client
    rabbitmq_management_agent
    rabbitmq_web_dispatch
    rabbit
    mnesia
    rabbit_common
    sysmon_handler
    os_mon
2026-04-20 05:22:20.484 [info] <0.759.0> Stopping application 'rabbitmq_management'
2026-04-20 05:22:20.484 [info] <0.759.0> Stopping application 'amqp_client'
2026-04-20 05:22:20.484 [info] <0.759.0> Stopping application 'rabbitmq_management_agent'
2026-04-20 05:22:20.484 [info] <0.759.0> Stopping application 'rabbitmq_web_dispatch'
2026-04-20 05:22:20.484 [info] <0.759.0> Stopping application 'rabbit'
2026-04-20 05:22:20.484 [info] <0.759.0> Stopping application 'mnesia'
2026-04-20 05:22:20.484 [info] <0.759.0> Stopping application 'rabbit_common'
2026-04-20 05:22:20.484 [info] <0.759.0> Stopping application 'sysmon_handler'
2026-04-20 05:22:20.484 [info] <0.759.0> Stopping application 'os_mon'
2026-04-20 05:22:20.484 [info] <0.759.0> Successfully stopped RabbitMQ and its dependencies
2026-04-20 05:22:20.484 [info] <0.759.0> Halting Erlang VM with the following applications:
    ranch
    ssl
    credentials_obfuscation
    public_key
    asn1
    observer_cli
    jsx
    tools
    crypto
    stdout_formatter
    recon
    inets
    xmerl
    lager
    goldrush
    compiler
    syntax_tools
    sasl
    stdlib
    kernel
2026-04-20 05:22:26.556 [info] <0.8.0> Log file opened with Lager
2026-04-20 05:22:29.440 [info] <0.8.0> Feature flags: list of feature flags found:
2026-04-20 05:22:29.440 [info] <0.8.0> Feature flags: feature flag states written to disk: yes
2026-04-20 05:22:29.462 [info] <0.252.0> 
 Starting RabbitMQ 3.7.28 on Erlang 21.3.8.21
 Copyright (c) 2007-2020 Pivotal Software, Inc.
 Licensed under the MPL.  See https://www.rabbitmq.com/
2026-04-20 05:22:29.462 [info] <0.252.0> 
 node           : contrail@cn-jenkins-deploy-platform-ansible-os-5649-3
 home dir       : /var/lib/rabbitmq
 config file(s) : /etc/rabbitmq/rabbitmq.config
 cookie hash    : esZsDxSN6VGbi9JkMSxNZA==
 log(s)         : /var/log/contrail/config-database-rabbitmq/rabbitmq.log
                : /var/log/rabbitmq/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3_upgrade.log
 database dir   : /var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3
2026-04-20 05:22:29.475 [info] <0.252.0> Running boot step pre_boot defined by app rabbit
2026-04-20 05:22:29.475 [info] <0.252.0> Running boot step rabbit_core_metrics defined by app rabbit
2026-04-20 05:22:29.476 [info] <0.252.0> Running boot step rabbit_alarm defined by app rabbit
2026-04-20 05:22:29.480 [info] <0.258.0> Memory high watermark set to 12637 MiB (13251081011 bytes) of 15796 MiB (16563851264 bytes) total
2026-04-20 05:22:29.485 [info] <0.260.0> Enabling free disk space monitoring
2026-04-20 05:22:29.485 [info] <0.260.0> Disk free limit set to 50MB
2026-04-20 05:22:29.489 [info] <0.252.0> Running boot step code_server_cache defined by app rabbit
2026-04-20 05:22:29.489 [info] <0.252.0> Running boot step file_handle_cache defined by app rabbit
2026-04-20 05:22:29.489 [info] <0.263.0> Limiting to approx 65436 file handles (58890 sockets)
2026-04-20 05:22:29.489 [info] <0.264.0> FHC read buffering:  OFF
2026-04-20 05:22:29.489 [info] <0.264.0> FHC write buffering: ON
2026-04-20 05:22:29.489 [info] <0.252.0> Running boot step worker_pool defined by app rabbit
2026-04-20 05:22:29.489 [info] <0.253.0> Will use 4 processes for default worker pool
2026-04-20 05:22:29.489 [info] <0.253.0> Starting worker pool 'worker_pool' with 4 processes in it
2026-04-20 05:22:29.490 [info] <0.252.0> Running boot step database defined by app rabbit
2026-04-20 05:22:29.490 [info] <0.252.0> Node database directory at /var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3 is empty. Assuming we need to join an existing cluster or initialise from scratch...
2026-04-20 05:22:29.490 [info] <0.252.0> Configured peer discovery backend: rabbit_peer_discovery_classic_config
2026-04-20 05:22:29.491 [info] <0.252.0> Will try to lock with peer discovery backend rabbit_peer_discovery_classic_config
2026-04-20 05:22:29.491 [info] <0.252.0> Peer discovery backend does not support locking, falling back to randomized delay
2026-04-20 05:22:29.491 [info] <0.252.0> Peer discovery backend rabbit_peer_discovery_classic_config supports registration.
2026-04-20 05:22:29.491 [info] <0.252.0> Will wait for 38201 milliseconds before proceeding with registration...
2026-04-20 05:23:07.693 [info] <0.252.0> All discovered existing cluster peers: contrail@cn-jenkins-deploy-platform-ansible-os-5649-1, contrail@cn-jenkins-deploy-platform-ansible-os-5649-2, contrail@cn-jenkins-deploy-platform-ansible-os-5649-3
2026-04-20 05:23:07.693 [info] <0.252.0> Peer nodes we can cluster with: contrail@cn-jenkins-deploy-platform-ansible-os-5649-1, contrail@cn-jenkins-deploy-platform-ansible-os-5649-2
2026-04-20 05:23:07.694 [warning] <0.252.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,tables_not_present}
2026-04-20 05:23:07.699 [warning] <0.252.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:23:07.699 [error] <0.252.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 9 retries left...
2026-04-20 05:23:08.200 [warning] <0.252.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,tables_not_present}
2026-04-20 05:23:08.201 [warning] <0.252.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:23:08.201 [error] <0.252.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 8 retries left...
2026-04-20 05:23:08.704 [warning] <0.252.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,tables_not_present}
2026-04-20 05:23:08.704 [warning] <0.252.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:23:08.704 [error] <0.252.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 7 retries left...
2026-04-20 05:23:09.206 [warning] <0.252.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,tables_not_present}
2026-04-20 05:23:09.207 [warning] <0.252.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:23:09.207 [error] <0.252.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 6 retries left...
2026-04-20 05:23:09.709 [warning] <0.252.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,tables_not_present}
2026-04-20 05:23:09.710 [warning] <0.252.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:23:09.710 [error] <0.252.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 5 retries left...
2026-04-20 05:23:10.212 [warning] <0.252.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,tables_not_present}
2026-04-20 05:23:10.213 [warning] <0.252.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:23:10.213 [error] <0.252.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 4 retries left...
2026-04-20 05:23:10.717 [warning] <0.252.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,tables_not_present}
2026-04-20 05:23:10.718 [warning] <0.252.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:23:10.718 [error] <0.252.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 3 retries left...
2026-04-20 05:23:11.219 [warning] <0.252.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,tables_not_present}
2026-04-20 05:23:11.220 [warning] <0.252.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:23:11.221 [error] <0.252.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 2 retries left...
2026-04-20 05:23:11.722 [warning] <0.252.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,tables_not_present}
2026-04-20 05:23:11.723 [warning] <0.252.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:23:11.723 [error] <0.252.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 1 retries left...
2026-04-20 05:23:12.224 [warning] <0.252.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,tables_not_present}
2026-04-20 05:23:12.225 [warning] <0.252.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:23:12.225 [error] <0.252.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 0 retries left...
2026-04-20 05:23:12.725 [warning] <0.252.0> Could not successfully contact any node of: contrail@cn-jenkins-deploy-platform-ansible-os-5649-1,contrail@cn-jenkins-deploy-platform-ansible-os-5649-2 (as in Erlang distribution). Starting as a blank standalone node...
2026-04-20 05:23:12.728 [info] <0.43.0> Application mnesia exited with reason: stopped
2026-04-20 05:23:12.838 [info] <0.252.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2026-04-20 05:23:12.839 [info] <0.252.0> Successfully synced tables from a peer
2026-04-20 05:23:12.865 [info] <0.252.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2026-04-20 05:23:12.865 [info] <0.252.0> Successfully synced tables from a peer
2026-04-20 05:23:12.894 [info] <0.252.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2026-04-20 05:23:12.894 [info] <0.252.0> Successfully synced tables from a peer
2026-04-20 05:23:12.894 [info] <0.252.0> Will register with peer discovery backend rabbit_peer_discovery_classic_config
2026-04-20 05:23:12.894 [info] <0.252.0> Running boot step database_sync defined by app rabbit
2026-04-20 05:23:12.894 [info] <0.252.0> Running boot step feature_flags defined by app rabbit
2026-04-20 05:23:12.894 [info] <0.252.0> Running boot step codec_correctness_check defined by app rabbit
2026-04-20 05:23:12.894 [info] <0.252.0> Running boot step external_infrastructure defined by app rabbit
2026-04-20 05:23:12.894 [info] <0.252.0> Running boot step rabbit_registry defined by app rabbit
2026-04-20 05:23:12.894 [info] <0.252.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit
2026-04-20 05:23:12.894 [info] <0.252.0> Running boot step rabbit_queue_location_random defined by app rabbit
2026-04-20 05:23:12.894 [info] <0.252.0> Running boot step rabbit_event defined by app rabbit
2026-04-20 05:23:12.894 [info] <0.252.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit
2026-04-20 05:23:12.894 [info] <0.252.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit
2026-04-20 05:23:12.894 [info] <0.252.0> Running boot step rabbit_exchange_type_direct defined by app rabbit
2026-04-20 05:23:12.894 [info] <0.252.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit
2026-04-20 05:23:12.895 [info] <0.252.0> Running boot step rabbit_exchange_type_headers defined by app rabbit
2026-04-20 05:23:12.895 [info] <0.252.0> Running boot step rabbit_exchange_type_topic defined by app rabbit
2026-04-20 05:23:12.895 [info] <0.252.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit
2026-04-20 05:23:12.895 [info] <0.252.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit
2026-04-20 05:23:12.895 [info] <0.252.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit
2026-04-20 05:23:12.895 [info] <0.252.0> Running boot step rabbit_priority_queue defined by app rabbit
2026-04-20 05:23:12.895 [info] <0.252.0> Priority queues enabled, real BQ is rabbit_variable_queue
2026-04-20 05:23:12.895 [info] <0.252.0> Running boot step rabbit_queue_location_client_local defined by app rabbit
2026-04-20 05:23:12.895 [info] <0.252.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit
2026-04-20 05:23:12.895 [info] <0.252.0> Running boot step kernel_ready defined by app rabbit
2026-04-20 05:23:12.895 [info] <0.252.0> Running boot step rabbit_sysmon_minder defined by app rabbit
2026-04-20 05:23:12.895 [info] <0.252.0> Running boot step rabbit_epmd_monitor defined by app rabbit
2026-04-20 05:23:12.896 [info] <0.490.0> epmd monitor knows us, inter-node communication (distribution) port: 25673
2026-04-20 05:23:12.896 [info] <0.252.0> Running boot step guid_generator defined by app rabbit
2026-04-20 05:23:12.900 [info] <0.252.0> Running boot step rabbit_node_monitor defined by app rabbit
2026-04-20 05:23:12.900 [info] <0.494.0> Starting rabbit_node_monitor
2026-04-20 05:23:12.901 [info] <0.252.0> Running boot step delegate_sup defined by app rabbit
2026-04-20 05:23:12.901 [info] <0.252.0> Running boot step rabbit_memory_monitor defined by app rabbit
2026-04-20 05:23:12.901 [info] <0.252.0> Running boot step core_initialized defined by app rabbit
2026-04-20 05:23:12.901 [info] <0.252.0> Running boot step upgrade_queues defined by app rabbit
2026-04-20 05:23:12.925 [info] <0.252.0> message_store upgrades: 1 to apply
2026-04-20 05:23:12.925 [info] <0.252.0> message_store upgrades: Applying rabbit_variable_queue:move_messages_to_vhost_store
2026-04-20 05:23:12.925 [info] <0.252.0> message_store upgrades: No durable queues found. Skipping message store migration
2026-04-20 05:23:12.925 [info] <0.252.0> message_store upgrades: Removing the old message store data
2026-04-20 05:23:12.925 [info] <0.252.0> message_store upgrades: All upgrades applied successfully
2026-04-20 05:23:12.954 [info] <0.252.0> Running boot step rabbit_connection_tracking defined by app rabbit
2026-04-20 05:23:12.954 [info] <0.252.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit
2026-04-20 05:23:12.954 [info] <0.252.0> Running boot step rabbit_exchange_parameters defined by app rabbit
2026-04-20 05:23:12.955 [info] <0.252.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit
2026-04-20 05:23:12.955 [info] <0.252.0> Running boot step rabbit_policies defined by app rabbit
2026-04-20 05:23:12.955 [info] <0.252.0> Running boot step rabbit_policy defined by app rabbit
2026-04-20 05:23:12.955 [info] <0.252.0> Running boot step rabbit_queue_location_validator defined by app rabbit
2026-04-20 05:23:12.956 [info] <0.252.0> Running boot step rabbit_vhost_limit defined by app rabbit
2026-04-20 05:23:12.956 [info] <0.252.0> Running boot step rabbit_mgmt_reset_handler defined by app rabbitmq_management
2026-04-20 05:23:12.956 [info] <0.252.0> Running boot step rabbit_mgmt_db_handler defined by app rabbitmq_management_agent
2026-04-20 05:23:12.956 [warning] <0.252.0> force_fine_statistics set to true; ignored.
Replaced by {rates_mode, none} in the rabbitmq_management application.
2026-04-20 05:23:12.956 [info] <0.252.0> Management plugin: using rates mode 'basic'
2026-04-20 05:23:12.956 [info] <0.252.0> Running boot step recovery defined by app rabbit
2026-04-20 05:23:12.957 [info] <0.252.0> Running boot step load_definitions defined by app rabbitmq_management
2026-04-20 05:23:12.957 [info] <0.252.0> Applying definitions from /etc/rabbitmq/definitions.json
2026-04-20 05:23:12.957 [info] <0.252.0> Asked to import definitions. Acting user: rmq-internal
2026-04-20 05:23:12.957 [info] <0.252.0> Importing users...
2026-04-20 05:23:12.957 [info] <0.252.0> Creating user 'guest'
2026-04-20 05:23:12.963 [info] <0.252.0> Setting user tags for user 'guest' to [administrator]
2026-04-20 05:23:12.966 [info] <0.252.0> Importing vhosts...
2026-04-20 05:23:12.966 [info] <0.252.0> Adding vhost '/'
2026-04-20 05:23:12.988 [info] <0.542.0> Making sure data directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists
2026-04-20 05:23:12.992 [info] <0.542.0> Starting message stores for vhost '/'
2026-04-20 05:23:12.992 [info] <0.546.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index
2026-04-20 05:23:12.993 [info] <0.542.0> Started message store of type transient for vhost '/'
2026-04-20 05:23:12.993 [info] <0.549.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
2026-04-20 05:23:12.994 [warning] <0.549.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": rebuilding indices from scratch
2026-04-20 05:23:12.996 [info] <0.542.0> Started message store of type persistent for vhost '/'
2026-04-20 05:23:12.997 [info] <0.252.0> Importing user permissions...
2026-04-20 05:23:12.997 [info] <0.252.0> Setting permissions for 'guest' in '/' to '.*', '.*', '.*'
2026-04-20 05:23:13.000 [info] <0.252.0> Importing topic permissions...
2026-04-20 05:23:13.000 [info] <0.252.0> Importing parameters...
2026-04-20 05:23:13.000 [info] <0.252.0> Importing global parameters...
2026-04-20 05:23:13.000 [info] <0.252.0> Importing policies...
2026-04-20 05:23:13.007 [info] <0.252.0> Importing queues...
2026-04-20 05:23:13.007 [info] <0.252.0> Importing exchanges...
2026-04-20 05:23:13.007 [info] <0.252.0> Importing bindings...
2026-04-20 05:23:13.007 [info] <0.252.0> Running boot step empty_db_check defined by app rabbit
2026-04-20 05:23:13.007 [info] <0.252.0> Running boot step rabbit_looking_glass defined by app rabbit
2026-04-20 05:23:13.007 [info] <0.252.0> Running boot step rabbit_core_metrics_gc defined by app rabbit
2026-04-20 05:23:13.007 [info] <0.252.0> Running boot step background_gc defined by app rabbit
2026-04-20 05:23:13.007 [info] <0.252.0> Running boot step connection_tracking defined by app rabbit
2026-04-20 05:23:13.011 [info] <0.252.0> Setting up a table for connection tracking on this node: 'tracked_connection_on_node_contrail@cn-jenkins-deploy-platform-ansible-os-5649-3'
2026-04-20 05:23:13.014 [info] <0.252.0> Setting up a table for per-vhost connection counting on this node: 'tracked_connection_per_vhost_on_node_contrail@cn-jenkins-deploy-platform-ansible-os-5649-3'
2026-04-20 05:23:13.014 [info] <0.252.0> Running boot step routing_ready defined by app rabbit
2026-04-20 05:23:13.014 [info] <0.252.0> Running boot step pre_flight defined by app rabbit
2026-04-20 05:23:13.014 [info] <0.252.0> Running boot step notify_cluster defined by app rabbit
2026-04-20 05:23:13.015 [info] <0.252.0> Running boot step networking defined by app rabbit
2026-04-20 05:23:13.016 [info] <0.595.0> started TCP listener on [::]:5673
2026-04-20 05:23:13.017 [info] <0.252.0> Running boot step cluster_name defined by app rabbit
2026-04-20 05:23:13.017 [info] <0.252.0> Running boot step direct_client defined by app rabbit
2026-04-20 05:23:13.017 [info] <0.252.0> Running boot step os_signal_handler defined by app rabbit
2026-04-20 05:23:13.017 [info] <0.597.0> Swapping OS signal event handler (erl_signal_server) for our own
2026-04-20 05:23:13.046 [info] <0.647.0> Management plugin: HTTP (non-TLS) listener started on port 15673
2026-04-20 05:23:13.046 [info] <0.753.0> Statistics database started.
2026-04-20 05:23:13.046 [info] <0.752.0> Starting worker pool 'management_worker_pool' with 3 processes in it
2026-04-20 05:23:13.046 [notice] <0.106.0> Changed loghwm of /var/log/contrail/config-database-rabbitmq/rabbitmq.log to 50
2026-04-20 05:23:13.186 [info] <0.8.0> Server startup complete; 3 plugins started.
 * rabbitmq_management
 * rabbitmq_web_dispatch
 * rabbitmq_management_agent
2026-04-20 05:23:13.573 [info] <0.764.0> RabbitMQ is asked to stop...
2026-04-20 05:23:13.656 [info] <0.764.0> Stopping RabbitMQ applications and their dependencies in the following order:
    rabbitmq_management
    amqp_client
    rabbitmq_web_dispatch
    cowboy
    cowlib
    rabbitmq_management_agent
    rabbit
    mnesia
    rabbit_common
    sysmon_handler
    os_mon
2026-04-20 05:23:13.656 [info] <0.764.0> Stopping application 'rabbitmq_management'
2026-04-20 05:23:13.657 [warning] <0.643.0> RabbitMQ HTTP listener registry could not find context rabbitmq_management_tls
2026-04-20 05:23:13.659 [info] <0.43.0> Application rabbitmq_management exited with reason: stopped
2026-04-20 05:23:13.659 [info] <0.764.0> Stopping application 'amqp_client'
2026-04-20 05:23:13.660 [info] <0.43.0> Application amqp_client exited with reason: stopped
2026-04-20 05:23:13.660 [info] <0.764.0> Stopping application 'rabbitmq_web_dispatch'
2026-04-20 05:23:13.662 [info] <0.764.0> Stopping application 'cowboy'
2026-04-20 05:23:13.662 [info] <0.43.0> Application rabbitmq_web_dispatch exited with reason: stopped
2026-04-20 05:23:13.663 [info] <0.43.0> Application cowboy exited with reason: stopped
2026-04-20 05:23:13.663 [info] <0.764.0> Stopping application 'cowlib'
2026-04-20 05:23:13.663 [info] <0.43.0> Application cowlib exited with reason: stopped
2026-04-20 05:23:13.663 [info] <0.764.0> Stopping application 'rabbitmq_management_agent'
2026-04-20 05:23:13.665 [info] <0.43.0> Application rabbitmq_management_agent exited with reason: stopped
2026-04-20 05:23:13.665 [info] <0.764.0> Stopping application 'rabbit'
2026-04-20 05:23:13.665 [info] <0.252.0> Will unregister with peer discovery backend rabbit_peer_discovery_classic_config
2026-04-20 05:23:13.665 [info] <0.595.0> stopped TCP listener on [::]:5673
2026-04-20 05:23:13.669 [info] <0.519.0> Closing all connections in vhost '/' on node 'contrail@cn-jenkins-deploy-platform-ansible-os-5649-3' because the vhost is stopping
2026-04-20 05:23:13.669 [info] <0.549.0> Stopping message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent'
2026-04-20 05:23:13.674 [info] <0.549.0> Message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent' is stopped
2026-04-20 05:23:13.675 [info] <0.546.0> Stopping message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient'
2026-04-20 05:23:13.679 [info] <0.546.0> Message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient' is stopped
2026-04-20 05:23:13.682 [info] <0.43.0> Application rabbit exited with reason: stopped
2026-04-20 05:23:13.682 [info] <0.764.0> Stopping application 'mnesia'
2026-04-20 05:23:13.686 [info] <0.43.0> Application mnesia exited with reason: stopped
2026-04-20 05:23:13.686 [info] <0.764.0> Stopping application 'rabbit_common'
2026-04-20 05:23:13.686 [info] <0.43.0> Application rabbit_common exited with reason: stopped
2026-04-20 05:23:13.686 [info] <0.764.0> Stopping application 'sysmon_handler'
2026-04-20 05:23:13.688 [info] <0.43.0> Application sysmon_handler exited with reason: stopped
2026-04-20 05:23:13.688 [info] <0.764.0> Stopping application 'os_mon'
2026-04-20 05:23:13.689 [info] <0.43.0> Application os_mon exited with reason: stopped
2026-04-20 05:23:13.689 [info] <0.764.0> Successfully stopped RabbitMQ and its dependencies
2026-04-20 05:23:20.915 [info] <0.779.0> Resetting Rabbit forcefully
2026-04-20 05:23:21.932 [info] <0.791.0> RabbitMQ is asked to stop...
2026-04-20 05:23:21.983 [info] <0.791.0> Stopping RabbitMQ applications and their dependencies in the following order:
    rabbitmq_management
    amqp_client
    rabbitmq_web_dispatch
    rabbitmq_management_agent
    rabbit
    mnesia
    rabbit_common
    sysmon_handler
    os_mon
2026-04-20 05:23:21.983 [info] <0.791.0> Stopping application 'rabbitmq_management'
2026-04-20 05:23:21.983 [info] <0.791.0> Stopping application 'amqp_client'
2026-04-20 05:23:21.983 [info] <0.791.0> Stopping application 'rabbitmq_web_dispatch'
2026-04-20 05:23:21.983 [info] <0.791.0> Stopping application 'rabbitmq_management_agent'
2026-04-20 05:23:21.983 [info] <0.791.0> Stopping application 'rabbit'
2026-04-20 05:23:21.983 [info] <0.791.0> Stopping application 'mnesia'
2026-04-20 05:23:21.983 [info] <0.791.0> Stopping application 'rabbit_common'
2026-04-20 05:23:21.983 [info] <0.791.0> Stopping application 'sysmon_handler'
2026-04-20 05:23:21.983 [info] <0.791.0> Stopping application 'os_mon'
2026-04-20 05:23:21.983 [info] <0.791.0> Successfully stopped RabbitMQ and its dependencies
2026-04-20 05:23:21.983 [info] <0.791.0> Halting Erlang VM with the following applications:
    ranch
    ssl
    credentials_obfuscation
    public_key
    asn1
    observer_cli
    jsx
    tools
    crypto
    stdout_formatter
    recon
    inets
    xmerl
    lager
    goldrush
    compiler
    syntax_tools
    sasl
    stdlib
    kernel
2026-04-20 05:23:28.081 [info] <0.8.0> Log file opened with Lager
2026-04-20 05:23:29.900 [info] <0.8.0> Feature flags: list of feature flags found:
2026-04-20 05:23:29.900 [info] <0.8.0> Feature flags: feature flag states written to disk: yes
2026-04-20 05:23:29.922 [info] <0.240.0> 
 Starting RabbitMQ 3.7.28 on Erlang 21.3.8.21
 Copyright (c) 2007-2020 Pivotal Software, Inc.
 Licensed under the MPL.  See https://www.rabbitmq.com/
2026-04-20 05:23:29.923 [info] <0.240.0> 
 node           : contrail@cn-jenkins-deploy-platform-ansible-os-5649-3
 home dir       : /var/lib/rabbitmq
 config file(s) : /etc/rabbitmq/rabbitmq.config
 cookie hash    : esZsDxSN6VGbi9JkMSxNZA==
 log(s)         : /var/log/contrail/config-database-rabbitmq/rabbitmq.log
                : /var/log/rabbitmq/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3_upgrade.log
 database dir   : /var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3
2026-04-20 05:23:29.935 [info] <0.240.0> Running boot step pre_boot defined by app rabbit
2026-04-20 05:23:29.935 [info] <0.240.0> Running boot step rabbit_core_metrics defined by app rabbit
2026-04-20 05:23:29.936 [info] <0.240.0> Running boot step rabbit_alarm defined by app rabbit
2026-04-20 05:23:29.942 [info] <0.246.0> Memory high watermark set to 12637 MiB (13251081011 bytes) of 15796 MiB (16563851264 bytes) total
2026-04-20 05:23:29.947 [info] <0.248.0> Enabling free disk space monitoring
2026-04-20 05:23:29.947 [info] <0.248.0> Disk free limit set to 50MB
2026-04-20 05:23:29.951 [info] <0.240.0> Running boot step code_server_cache defined by app rabbit
2026-04-20 05:23:29.951 [info] <0.240.0> Running boot step file_handle_cache defined by app rabbit
2026-04-20 05:23:29.951 [info] <0.251.0> Limiting to approx 65436 file handles (58890 sockets)
2026-04-20 05:23:29.952 [info] <0.252.0> FHC read buffering:  OFF
2026-04-20 05:23:29.952 [info] <0.252.0> FHC write buffering: ON
2026-04-20 05:23:29.952 [info] <0.240.0> Running boot step worker_pool defined by app rabbit
2026-04-20 05:23:29.952 [info] <0.241.0> Will use 4 processes for default worker pool
2026-04-20 05:23:29.952 [info] <0.241.0> Starting worker pool 'worker_pool' with 4 processes in it
2026-04-20 05:23:29.952 [info] <0.240.0> Running boot step database defined by app rabbit
2026-04-20 05:23:29.952 [info] <0.240.0> Node database directory at /var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3 is empty. Assuming we need to join an existing cluster or initialise from scratch...
2026-04-20 05:23:29.952 [info] <0.240.0> Configured peer discovery backend: rabbit_peer_discovery_classic_config
2026-04-20 05:23:29.952 [info] <0.240.0> Will try to lock with peer discovery backend rabbit_peer_discovery_classic_config
2026-04-20 05:23:29.952 [info] <0.240.0> Peer discovery backend does not support locking, falling back to randomized delay
2026-04-20 05:23:29.952 [info] <0.240.0> Peer discovery backend rabbit_peer_discovery_classic_config supports registration.
2026-04-20 05:23:29.952 [info] <0.240.0> Will wait for 57976 milliseconds before proceeding with registration...
2026-04-20 05:24:07.179 [info] <0.294.0> RabbitMQ hasn't finished starting yet. Waiting for startup to finish before stopping...
2026-04-20 05:24:27.929 [info] <0.240.0> All discovered existing cluster peers: contrail@cn-jenkins-deploy-platform-ansible-os-5649-1, contrail@cn-jenkins-deploy-platform-ansible-os-5649-2, contrail@cn-jenkins-deploy-platform-ansible-os-5649-3
2026-04-20 05:24:27.929 [info] <0.240.0> Peer nodes we can cluster with: contrail@cn-jenkins-deploy-platform-ansible-os-5649-1, contrail@cn-jenkins-deploy-platform-ansible-os-5649-2
2026-04-20 05:24:27.931 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,tables_not_present}
2026-04-20 05:24:27.933 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,mnesia_not_running}
2026-04-20 05:24:27.933 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 9 retries left...
2026-04-20 05:24:28.435 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,tables_not_present}
2026-04-20 05:24:28.437 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {badrpc,nodedown}
2026-04-20 05:24:28.438 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 8 retries left...
2026-04-20 05:24:28.940 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,tables_not_present}
2026-04-20 05:24:28.942 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {badrpc,nodedown}
2026-04-20 05:24:28.942 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 7 retries left...
2026-04-20 05:24:29.448 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,tables_not_present}
2026-04-20 05:24:29.450 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {badrpc,nodedown}
2026-04-20 05:24:29.450 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 6 retries left...
2026-04-20 05:24:29.954 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,tables_not_present}
2026-04-20 05:24:29.956 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {badrpc,nodedown}
2026-04-20 05:24:29.956 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 5 retries left...
2026-04-20 05:24:30.460 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,tables_not_present}
2026-04-20 05:24:30.462 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {badrpc,nodedown}
2026-04-20 05:24:30.462 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 4 retries left...
2026-04-20 05:24:30.967 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,tables_not_present}
2026-04-20 05:24:30.976 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {badrpc,nodedown}
2026-04-20 05:24:30.976 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 3 retries left...
2026-04-20 05:24:31.477 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,tables_not_present}
2026-04-20 05:24:31.479 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {badrpc,nodedown}
2026-04-20 05:24:31.479 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 2 retries left...
2026-04-20 05:24:31.980 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,tables_not_present}
2026-04-20 05:24:32.017 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,mnesia_not_running}
2026-04-20 05:24:32.017 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 1 retries left...
2026-04-20 05:24:32.519 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,tables_not_present}
2026-04-20 05:24:32.522 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,mnesia_not_running}
2026-04-20 05:24:32.523 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 0 retries left...
2026-04-20 05:24:33.024 [warning] <0.240.0> Could not successfully contact any node of: contrail@cn-jenkins-deploy-platform-ansible-os-5649-1,contrail@cn-jenkins-deploy-platform-ansible-os-5649-2 (as in Erlang distribution). Starting as a blank standalone node...
2026-04-20 05:24:33.028 [info] <0.43.0> Application mnesia exited with reason: stopped
2026-04-20 05:24:33.132 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2026-04-20 05:24:33.132 [info] <0.240.0> Successfully synced tables from a peer
2026-04-20 05:24:33.158 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2026-04-20 05:24:33.158 [info] <0.240.0> Successfully synced tables from a peer
2026-04-20 05:24:33.187 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2026-04-20 05:24:33.187 [info] <0.240.0> Successfully synced tables from a peer
2026-04-20 05:24:33.187 [info] <0.240.0> Will register with peer discovery backend rabbit_peer_discovery_classic_config
2026-04-20 05:24:33.187 [info] <0.240.0> Running boot step database_sync defined by app rabbit
2026-04-20 05:24:33.187 [info] <0.240.0> Running boot step feature_flags defined by app rabbit
2026-04-20 05:24:33.187 [info] <0.240.0> Running boot step codec_correctness_check defined by app rabbit
2026-04-20 05:24:33.187 [info] <0.240.0> Running boot step external_infrastructure defined by app rabbit
2026-04-20 05:24:33.187 [info] <0.240.0> Running boot step rabbit_registry defined by app rabbit
2026-04-20 05:24:33.187 [info] <0.240.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit
2026-04-20 05:24:33.187 [info] <0.240.0> Running boot step rabbit_queue_location_random defined by app rabbit
2026-04-20 05:24:33.188 [info] <0.240.0> Running boot step rabbit_event defined by app rabbit
2026-04-20 05:24:33.188 [info] <0.240.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit
2026-04-20 05:24:33.188 [info] <0.240.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit
2026-04-20 05:24:33.188 [info] <0.240.0> Running boot step rabbit_exchange_type_direct defined by app rabbit
2026-04-20 05:24:33.188 [info] <0.240.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit
2026-04-20 05:24:33.188 [info] <0.240.0> Running boot step rabbit_exchange_type_headers defined by app rabbit
2026-04-20 05:24:33.188 [info] <0.240.0> Running boot step rabbit_exchange_type_topic defined by app rabbit
2026-04-20 05:24:33.188 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit
2026-04-20 05:24:33.188 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit
2026-04-20 05:24:33.188 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit
2026-04-20 05:24:33.188 [info] <0.240.0> Running boot step rabbit_priority_queue defined by app rabbit
2026-04-20 05:24:33.188 [info] <0.240.0> Priority queues enabled, real BQ is rabbit_variable_queue
2026-04-20 05:24:33.188 [info] <0.240.0> Running boot step rabbit_queue_location_client_local defined by app rabbit
2026-04-20 05:24:33.188 [info] <0.240.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit
2026-04-20 05:24:33.188 [info] <0.240.0> Running boot step kernel_ready defined by app rabbit
2026-04-20 05:24:33.189 [info] <0.240.0> Running boot step rabbit_sysmon_minder defined by app rabbit
2026-04-20 05:24:33.189 [info] <0.240.0> Running boot step rabbit_epmd_monitor defined by app rabbit
2026-04-20 05:24:33.189 [info] <0.488.0> epmd monitor knows us, inter-node communication (distribution) port: 25673
2026-04-20 05:24:33.189 [info] <0.240.0> Running boot step guid_generator defined by app rabbit
2026-04-20 05:24:33.195 [info] <0.240.0> Running boot step rabbit_node_monitor defined by app rabbit
2026-04-20 05:24:33.195 [info] <0.492.0> Starting rabbit_node_monitor
2026-04-20 05:24:33.195 [info] <0.240.0> Running boot step delegate_sup defined by app rabbit
2026-04-20 05:24:33.196 [info] <0.240.0> Running boot step rabbit_memory_monitor defined by app rabbit
2026-04-20 05:24:33.196 [info] <0.240.0> Running boot step core_initialized defined by app rabbit
2026-04-20 05:24:33.196 [info] <0.240.0> Running boot step upgrade_queues defined by app rabbit
2026-04-20 05:24:33.220 [info] <0.240.0> message_store upgrades: 1 to apply
2026-04-20 05:24:33.220 [info] <0.240.0> message_store upgrades: Applying rabbit_variable_queue:move_messages_to_vhost_store
2026-04-20 05:24:33.220 [info] <0.240.0> message_store upgrades: No durable queues found. Skipping message store migration
2026-04-20 05:24:33.220 [info] <0.240.0> message_store upgrades: Removing the old message store data
2026-04-20 05:24:33.220 [info] <0.240.0> message_store upgrades: All upgrades applied successfully
2026-04-20 05:24:33.248 [info] <0.240.0> Running boot step rabbit_connection_tracking defined by app rabbit
2026-04-20 05:24:33.248 [info] <0.240.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit
2026-04-20 05:24:33.248 [info] <0.240.0> Running boot step rabbit_exchange_parameters defined by app rabbit
2026-04-20 05:24:33.248 [info] <0.240.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit
2026-04-20 05:24:33.249 [info] <0.240.0> Running boot step rabbit_policies defined by app rabbit
2026-04-20 05:24:33.249 [info] <0.240.0> Running boot step rabbit_policy defined by app rabbit
2026-04-20 05:24:33.249 [info] <0.240.0> Running boot step rabbit_queue_location_validator defined by app rabbit
2026-04-20 05:24:33.249 [info] <0.240.0> Running boot step rabbit_vhost_limit defined by app rabbit
2026-04-20 05:24:33.249 [info] <0.240.0> Running boot step rabbit_mgmt_reset_handler defined by app rabbitmq_management
2026-04-20 05:24:33.249 [info] <0.240.0> Running boot step rabbit_mgmt_db_handler defined by app rabbitmq_management_agent
2026-04-20 05:24:33.249 [warning] <0.240.0> force_fine_statistics set to true; ignored.
Replaced by {rates_mode, none} in the rabbitmq_management application.
2026-04-20 05:24:33.249 [info] <0.240.0> Management plugin: using rates mode 'basic'
2026-04-20 05:24:33.250 [info] <0.240.0> Running boot step recovery defined by app rabbit
2026-04-20 05:24:33.250 [info] <0.240.0> Running boot step load_definitions defined by app rabbitmq_management
2026-04-20 05:24:33.251 [info] <0.240.0> Applying definitions from /etc/rabbitmq/definitions.json
2026-04-20 05:24:33.251 [info] <0.240.0> Asked to import definitions. Acting user: rmq-internal
2026-04-20 05:24:33.251 [info] <0.240.0> Importing users...
2026-04-20 05:24:33.251 [info] <0.240.0> Creating user 'guest'
2026-04-20 05:24:33.257 [info] <0.240.0> Setting user tags for user 'guest' to [administrator]
2026-04-20 05:24:33.260 [info] <0.240.0> Importing vhosts...
2026-04-20 05:24:33.260 [info] <0.240.0> Adding vhost '/'
2026-04-20 05:24:33.282 [info] <0.540.0> Making sure data directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists
2026-04-20 05:24:33.286 [info] <0.540.0> Starting message stores for vhost '/'
2026-04-20 05:24:33.287 [info] <0.544.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index
2026-04-20 05:24:33.287 [info] <0.540.0> Started message store of type transient for vhost '/'
2026-04-20 05:24:33.288 [info] <0.547.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
2026-04-20 05:24:33.288 [warning] <0.547.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": rebuilding indices from scratch
2026-04-20 05:24:33.289 [info] <0.540.0> Started message store of type persistent for vhost '/'
2026-04-20 05:24:33.290 [info] <0.240.0> Importing user permissions...
2026-04-20 05:24:33.290 [info] <0.240.0> Setting permissions for 'guest' in '/' to '.*', '.*', '.*'
2026-04-20 05:24:33.293 [info] <0.240.0> Importing topic permissions...
2026-04-20 05:24:33.293 [info] <0.240.0> Importing parameters...
2026-04-20 05:24:33.293 [info] <0.240.0> Importing global parameters...
2026-04-20 05:24:33.293 [info] <0.240.0> Importing policies...
2026-04-20 05:24:33.299 [info] <0.240.0> Importing queues...
2026-04-20 05:24:33.299 [info] <0.240.0> Importing exchanges...
2026-04-20 05:24:33.299 [info] <0.240.0> Importing bindings...
2026-04-20 05:24:33.299 [info] <0.240.0> Running boot step empty_db_check defined by app rabbit
2026-04-20 05:24:33.299 [info] <0.240.0> Running boot step rabbit_looking_glass defined by app rabbit
2026-04-20 05:24:33.299 [info] <0.240.0> Running boot step rabbit_core_metrics_gc defined by app rabbit
2026-04-20 05:24:33.299 [info] <0.240.0> Running boot step background_gc defined by app rabbit
2026-04-20 05:24:33.299 [info] <0.240.0> Running boot step connection_tracking defined by app rabbit
2026-04-20 05:24:33.303 [info] <0.240.0> Setting up a table for connection tracking on this node: 'tracked_connection_on_node_contrail@cn-jenkins-deploy-platform-ansible-os-5649-3'
2026-04-20 05:24:33.307 [info] <0.240.0> Setting up a table for per-vhost connection counting on this node: 'tracked_connection_per_vhost_on_node_contrail@cn-jenkins-deploy-platform-ansible-os-5649-3'
2026-04-20 05:24:33.307 [info] <0.240.0> Running boot step routing_ready defined by app rabbit
2026-04-20 05:24:33.307 [info] <0.240.0> Running boot step pre_flight defined by app rabbit
2026-04-20 05:24:33.307 [info] <0.240.0> Running boot step notify_cluster defined by app rabbit
2026-04-20 05:24:33.307 [info] <0.240.0> Running boot step networking defined by app rabbit
2026-04-20 05:24:33.309 [info] <0.593.0> started TCP listener on [::]:5673
2026-04-20 05:24:33.309 [info] <0.240.0> Running boot step cluster_name defined by app rabbit
2026-04-20 05:24:33.309 [info] <0.240.0> Running boot step direct_client defined by app rabbit
2026-04-20 05:24:33.309 [info] <0.240.0> Running boot step os_signal_handler defined by app rabbit
2026-04-20 05:24:33.309 [info] <0.595.0> Swapping OS signal event handler (erl_signal_server) for our own
2026-04-20 05:24:33.340 [info] <0.645.0> Management plugin: HTTP (non-TLS) listener started on port 15673
2026-04-20 05:24:33.340 [info] <0.751.0> Statistics database started.
2026-04-20 05:24:33.340 [info] <0.750.0> Starting worker pool 'management_worker_pool' with 3 processes in it
2026-04-20 05:24:33.341 [notice] <0.106.0> Changed loghwm of /var/log/contrail/config-database-rabbitmq/rabbitmq.log to 50
2026-04-20 05:24:33.387 [info] <0.761.0> accepting AMQP connection <0.761.0> (10.0.0.33:33514 -> 10.0.0.33:5673)
2026-04-20 05:24:33.392 [info] <0.765.0> accepting AMQP connection <0.765.0> (10.0.0.33:33524 -> 10.0.0.33:5673)
2026-04-20 05:24:33.393 [info] <0.765.0> connection <0.765.0> (10.0.0.33:33524 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:24:33.395 [info] <0.761.0> connection <0.761.0> (10.0.0.33:33514 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:24:33.444 [info] <0.8.0> Server startup complete; 3 plugins started.
 * rabbitmq_management
 * rabbitmq_web_dispatch
 * rabbitmq_management_agent
2026-04-20 05:24:33.446 [info] <0.294.0> RabbitMQ is asked to stop...
2026-04-20 05:24:33.451 [info] <0.799.0> accepting AMQP connection <0.799.0> (10.0.0.38:37556 -> 10.0.0.33:5673)
2026-04-20 05:24:33.452 [info] <0.799.0> connection <0.799.0> (10.0.0.38:37556 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:24:33.512 [info] <0.294.0> Stopping RabbitMQ applications and their dependencies in the following order:
    rabbitmq_management
    amqp_client
    rabbitmq_web_dispatch
    cowboy
    cowlib
    rabbitmq_management_agent
    rabbit
    mnesia
    rabbit_common
    sysmon_handler
    os_mon
2026-04-20 05:24:33.512 [info] <0.294.0> Stopping application 'rabbitmq_management'
2026-04-20 05:24:33.515 [warning] <0.641.0> RabbitMQ HTTP listener registry could not find context rabbitmq_management_tls
2026-04-20 05:24:33.516 [info] <0.294.0> Stopping application 'amqp_client'
2026-04-20 05:24:33.516 [info] <0.43.0> Application rabbitmq_management exited with reason: stopped
2026-04-20 05:24:33.517 [info] <0.294.0> Stopping application 'rabbitmq_web_dispatch'
2026-04-20 05:24:33.517 [info] <0.43.0> Application amqp_client exited with reason: stopped
2026-04-20 05:24:33.519 [info] <0.294.0> Stopping application 'cowboy'
2026-04-20 05:24:33.519 [info] <0.43.0> Application rabbitmq_web_dispatch exited with reason: stopped
2026-04-20 05:24:33.520 [info] <0.294.0> Stopping application 'cowlib'
2026-04-20 05:24:33.520 [info] <0.43.0> Application cowboy exited with reason: stopped
2026-04-20 05:24:33.520 [info] <0.294.0> Stopping application 'rabbitmq_management_agent'
2026-04-20 05:24:33.520 [info] <0.43.0> Application cowlib exited with reason: stopped
2026-04-20 05:24:33.522 [info] <0.294.0> Stopping application 'rabbit'
2026-04-20 05:24:33.522 [info] <0.240.0> Will unregister with peer discovery backend rabbit_peer_discovery_classic_config
2026-04-20 05:24:33.522 [info] <0.43.0> Application rabbitmq_management_agent exited with reason: stopped
2026-04-20 05:24:33.522 [info] <0.593.0> stopped TCP listener on [::]:5673
2026-04-20 05:24:33.523 [error] <0.765.0> Error on AMQP connection <0.765.0> (10.0.0.33:33524 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:24:33.523 [error] <0.761.0> Error on AMQP connection <0.761.0> (10.0.0.33:33514 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:24:33.523 [error] <0.799.0> Error on AMQP connection <0.799.0> (10.0.0.38:37556 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:24:33.524 [info] <0.517.0> Closing all connections in vhost '/' on node 'contrail@cn-jenkins-deploy-platform-ansible-os-5649-3' because the vhost is stopping
2026-04-20 05:24:33.524 [warning] <0.808.0> Mirrored queue 'contrail-dns.cn-jenkins-deploy-platform-ansible-os-5649-1' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available
2026-04-20 05:24:33.524 [warning] <0.789.0> Mirrored queue 'contrail-dns.cn-jenkins-deploy-platform-ansible-os-5649-3' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available
2026-04-20 05:24:33.524 [warning] <0.775.0> Mirrored queue 'contrail-control.cn-jenkins-deploy-platform-ansible-os-5649-3' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available
2026-04-20 05:24:33.524 [info] <0.547.0> Stopping message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent'
2026-04-20 05:24:33.531 [info] <0.547.0> Message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent' is stopped
2026-04-20 05:24:33.531 [info] <0.544.0> Stopping message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient'
2026-04-20 05:24:33.535 [info] <0.544.0> Message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient' is stopped
2026-04-20 05:24:33.542 [info] <0.294.0> Stopping application 'mnesia'
2026-04-20 05:24:33.543 [info] <0.43.0> Application rabbit exited with reason: stopped
2026-04-20 05:24:33.544 [info] <0.294.0> Stopping application 'rabbit_common'
2026-04-20 05:24:33.544 [info] <0.43.0> Application mnesia exited with reason: stopped
2026-04-20 05:24:33.544 [info] <0.294.0> Stopping application 'sysmon_handler'
2026-04-20 05:24:33.544 [info] <0.43.0> Application rabbit_common exited with reason: stopped
2026-04-20 05:24:33.546 [info] <0.294.0> Stopping application 'os_mon'
2026-04-20 05:24:33.546 [info] <0.43.0> Application sysmon_handler exited with reason: stopped
2026-04-20 05:24:33.547 [info] <0.43.0> Application os_mon exited with reason: stopped
2026-04-20 05:24:33.547 [info] <0.294.0> Successfully stopped RabbitMQ and its dependencies
2026-04-20 05:24:43.011 [info] <0.841.0> Resetting Rabbit forcefully
2026-04-20 05:24:43.917 [info] <0.853.0> RabbitMQ is asked to stop...
2026-04-20 05:24:43.958 [info] <0.853.0> Stopping RabbitMQ applications and their dependencies in the following order:
    rabbitmq_management
    amqp_client
    rabbitmq_web_dispatch
    rabbitmq_management_agent
    rabbit
    mnesia
    rabbit_common
    sysmon_handler
    os_mon
2026-04-20 05:24:43.959 [info] <0.853.0> Stopping application 'rabbitmq_management'
2026-04-20 05:24:43.959 [info] <0.853.0> Stopping application 'amqp_client'
2026-04-20 05:24:43.959 [info] <0.853.0> Stopping application 'rabbitmq_web_dispatch'
2026-04-20 05:24:43.959 [info] <0.853.0> Stopping application 'rabbitmq_management_agent'
2026-04-20 05:24:43.959 [info] <0.853.0> Stopping application 'rabbit'
2026-04-20 05:24:43.959 [info] <0.853.0> Stopping application 'mnesia'
2026-04-20 05:24:43.959 [info] <0.853.0> Stopping application 'rabbit_common'
2026-04-20 05:24:43.959 [info] <0.853.0> Stopping application 'sysmon_handler'
2026-04-20 05:24:43.959 [info] <0.853.0> Stopping application 'os_mon'
2026-04-20 05:24:43.959 [info] <0.853.0> Successfully stopped RabbitMQ and its dependencies
2026-04-20 05:24:43.959 [info] <0.853.0> Halting Erlang VM with the following applications:
    ranch
    ssl
    credentials_obfuscation
    public_key
    asn1
    observer_cli
    jsx
    tools
    crypto
    stdout_formatter
    recon
    inets
    xmerl
    lager
    goldrush
    compiler
    syntax_tools
    sasl
    stdlib
    kernel
2026-04-20 05:24:49.560 [info] <0.8.0> Log file opened with Lager
2026-04-20 05:24:51.303 [info] <0.8.0> Feature flags: list of feature flags found:
2026-04-20 05:24:51.303 [info] <0.8.0> Feature flags: feature flag states written to disk: yes
2026-04-20 05:24:51.322 [info] <0.240.0> 
 Starting RabbitMQ 3.7.28 on Erlang 21.3.8.21
 Copyright (c) 2007-2020 Pivotal Software, Inc.
 Licensed under the MPL.  See https://www.rabbitmq.com/
2026-04-20 05:24:51.323 [info] <0.240.0> 
 node           : contrail@cn-jenkins-deploy-platform-ansible-os-5649-3
 home dir       : /var/lib/rabbitmq
 config file(s) : /etc/rabbitmq/rabbitmq.config
 cookie hash    : esZsDxSN6VGbi9JkMSxNZA==
 log(s)         : /var/log/contrail/config-database-rabbitmq/rabbitmq.log
                : /var/log/rabbitmq/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3_upgrade.log
 database dir   : /var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3
2026-04-20 05:24:51.335 [info] <0.240.0> Running boot step pre_boot defined by app rabbit
2026-04-20 05:24:51.335 [info] <0.240.0> Running boot step rabbit_core_metrics defined by app rabbit
2026-04-20 05:24:51.335 [info] <0.240.0> Running boot step rabbit_alarm defined by app rabbit
2026-04-20 05:24:51.341 [info] <0.246.0> Memory high watermark set to 12637 MiB (13251081011 bytes) of 15796 MiB (16563851264 bytes) total
2026-04-20 05:24:51.351 [info] <0.248.0> Enabling free disk space monitoring
2026-04-20 05:24:51.351 [info] <0.248.0> Disk free limit set to 50MB
2026-04-20 05:24:51.361 [info] <0.240.0> Running boot step code_server_cache defined by app rabbit
2026-04-20 05:24:51.362 [info] <0.240.0> Running boot step file_handle_cache defined by app rabbit
2026-04-20 05:24:51.362 [info] <0.251.0> Limiting to approx 65436 file handles (58890 sockets)
2026-04-20 05:24:51.362 [info] <0.252.0> FHC read buffering:  OFF
2026-04-20 05:24:51.363 [info] <0.252.0> FHC write buffering: ON
2026-04-20 05:24:51.364 [info] <0.240.0> Running boot step worker_pool defined by app rabbit
2026-04-20 05:24:51.364 [info] <0.241.0> Will use 4 processes for default worker pool
2026-04-20 05:24:51.365 [info] <0.241.0> Starting worker pool 'worker_pool' with 4 processes in it
2026-04-20 05:24:51.365 [info] <0.240.0> Running boot step database defined by app rabbit
2026-04-20 05:24:51.366 [info] <0.240.0> Node database directory at /var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3 is empty. Assuming we need to join an existing cluster or initialise from scratch...
2026-04-20 05:24:51.366 [info] <0.240.0> Configured peer discovery backend: rabbit_peer_discovery_classic_config
2026-04-20 05:24:51.366 [info] <0.240.0> Will try to lock with peer discovery backend rabbit_peer_discovery_classic_config
2026-04-20 05:24:51.366 [info] <0.240.0> Peer discovery backend does not support locking, falling back to randomized delay
2026-04-20 05:24:51.366 [info] <0.240.0> Peer discovery backend rabbit_peer_discovery_classic_config supports registration.
2026-04-20 05:24:51.366 [info] <0.240.0> Will wait for 33318 milliseconds before proceeding with registration...
2026-04-20 05:25:24.684 [info] <0.240.0> All discovered existing cluster peers: contrail@cn-jenkins-deploy-platform-ansible-os-5649-1, contrail@cn-jenkins-deploy-platform-ansible-os-5649-2, contrail@cn-jenkins-deploy-platform-ansible-os-5649-3
2026-04-20 05:25:24.685 [info] <0.240.0> Peer nodes we can cluster with: contrail@cn-jenkins-deploy-platform-ansible-os-5649-1, contrail@cn-jenkins-deploy-platform-ansible-os-5649-2
2026-04-20 05:25:24.685 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,tables_not_present}
2026-04-20 05:25:24.687 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,mnesia_not_running}
2026-04-20 05:25:24.687 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 9 retries left...
2026-04-20 05:25:25.191 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,tables_not_present}
2026-04-20 05:25:25.191 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,mnesia_not_running}
2026-04-20 05:25:25.191 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 8 retries left...
2026-04-20 05:25:25.693 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,tables_not_present}
2026-04-20 05:25:25.694 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,mnesia_not_running}
2026-04-20 05:25:25.694 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 7 retries left...
2026-04-20 05:25:26.195 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,tables_not_present}
2026-04-20 05:25:26.195 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,mnesia_not_running}
2026-04-20 05:25:26.195 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 6 retries left...
2026-04-20 05:25:26.697 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,tables_not_present}
2026-04-20 05:25:26.698 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,mnesia_not_running}
2026-04-20 05:25:26.698 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 5 retries left...
2026-04-20 05:25:27.200 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,tables_not_present}
2026-04-20 05:25:27.202 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,mnesia_not_running}
2026-04-20 05:25:27.202 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 4 retries left...
2026-04-20 05:25:27.705 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,tables_not_present}
2026-04-20 05:25:27.706 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,mnesia_not_running}
2026-04-20 05:25:27.706 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 3 retries left...
2026-04-20 05:25:28.217 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,tables_not_present}
2026-04-20 05:25:28.217 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,mnesia_not_running}
2026-04-20 05:25:28.217 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 2 retries left...
2026-04-20 05:25:28.719 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,tables_not_present}
2026-04-20 05:25:28.720 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,mnesia_not_running}
2026-04-20 05:25:28.720 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 1 retries left...
2026-04-20 05:25:29.222 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,tables_not_present}
2026-04-20 05:25:29.223 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,mnesia_not_running}
2026-04-20 05:25:29.223 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 0 retries left...
2026-04-20 05:25:29.723 [warning] <0.240.0> Could not successfully contact any node of: contrail@cn-jenkins-deploy-platform-ansible-os-5649-1,contrail@cn-jenkins-deploy-platform-ansible-os-5649-2 (as in Erlang distribution). Starting as a blank standalone node...
2026-04-20 05:25:29.725 [info] <0.43.0> Application mnesia exited with reason: stopped
2026-04-20 05:25:29.896 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2026-04-20 05:25:29.896 [info] <0.240.0> Successfully synced tables from a peer
2026-04-20 05:25:29.989 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2026-04-20 05:25:29.989 [info] <0.240.0> Successfully synced tables from a peer
2026-04-20 05:25:30.028 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2026-04-20 05:25:30.028 [info] <0.240.0> Successfully synced tables from a peer
2026-04-20 05:25:30.028 [info] <0.240.0> Will register with peer discovery backend rabbit_peer_discovery_classic_config
2026-04-20 05:25:30.029 [info] <0.240.0> Running boot step database_sync defined by app rabbit
2026-04-20 05:25:30.029 [info] <0.240.0> Running boot step feature_flags defined by app rabbit
2026-04-20 05:25:30.030 [info] <0.240.0> Running boot step codec_correctness_check defined by app rabbit
2026-04-20 05:25:30.030 [info] <0.240.0> Running boot step external_infrastructure defined by app rabbit
2026-04-20 05:25:30.030 [info] <0.240.0> Running boot step rabbit_registry defined by app rabbit
2026-04-20 05:25:30.031 [info] <0.240.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit
2026-04-20 05:25:30.032 [info] <0.240.0> Running boot step rabbit_queue_location_random defined by app rabbit
2026-04-20 05:25:30.032 [info] <0.240.0> Running boot step rabbit_event defined by app rabbit
2026-04-20 05:25:30.033 [info] <0.240.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit
2026-04-20 05:25:30.033 [info] <0.240.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit
2026-04-20 05:25:30.034 [info] <0.240.0> Running boot step rabbit_exchange_type_direct defined by app rabbit
2026-04-20 05:25:30.035 [info] <0.240.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit
2026-04-20 05:25:30.036 [info] <0.240.0> Running boot step rabbit_exchange_type_headers defined by app rabbit
2026-04-20 05:25:30.036 [info] <0.240.0> Running boot step rabbit_exchange_type_topic defined by app rabbit
2026-04-20 05:25:30.037 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit
2026-04-20 05:25:30.037 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit
2026-04-20 05:25:30.037 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit
2026-04-20 05:25:30.038 [info] <0.240.0> Running boot step rabbit_priority_queue defined by app rabbit
2026-04-20 05:25:30.038 [info] <0.240.0> Priority queues enabled, real BQ is rabbit_variable_queue
2026-04-20 05:25:30.038 [info] <0.240.0> Running boot step rabbit_queue_location_client_local defined by app rabbit
2026-04-20 05:25:30.038 [info] <0.240.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit
2026-04-20 05:25:30.039 [info] <0.240.0> Running boot step kernel_ready defined by app rabbit
2026-04-20 05:25:30.039 [info] <0.240.0> Running boot step rabbit_sysmon_minder defined by app rabbit
2026-04-20 05:25:30.039 [info] <0.240.0> Running boot step rabbit_epmd_monitor defined by app rabbit
2026-04-20 05:25:30.040 [info] <0.463.0> epmd monitor knows us, inter-node communication (distribution) port: 25673
2026-04-20 05:25:30.041 [info] <0.240.0> Running boot step guid_generator defined by app rabbit
2026-04-20 05:25:30.073 [info] <0.240.0> Running boot step rabbit_node_monitor defined by app rabbit
2026-04-20 05:25:30.073 [info] <0.467.0> Starting rabbit_node_monitor
2026-04-20 05:25:30.073 [info] <0.240.0> Running boot step delegate_sup defined by app rabbit
2026-04-20 05:25:30.073 [info] <0.240.0> Running boot step rabbit_memory_monitor defined by app rabbit
2026-04-20 05:25:30.074 [info] <0.240.0> Running boot step core_initialized defined by app rabbit
2026-04-20 05:25:30.074 [info] <0.240.0> Running boot step upgrade_queues defined by app rabbit
2026-04-20 05:25:30.125 [info] <0.240.0> message_store upgrades: 1 to apply
2026-04-20 05:25:30.125 [info] <0.240.0> message_store upgrades: Applying rabbit_variable_queue:move_messages_to_vhost_store
2026-04-20 05:25:30.126 [info] <0.240.0> message_store upgrades: No durable queues found. Skipping message store migration
2026-04-20 05:25:30.126 [info] <0.240.0> message_store upgrades: Removing the old message store data
2026-04-20 05:25:30.129 [info] <0.240.0> message_store upgrades: All upgrades applied successfully
2026-04-20 05:25:30.167 [info] <0.240.0> Running boot step rabbit_connection_tracking defined by app rabbit
2026-04-20 05:25:30.167 [info] <0.240.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit
2026-04-20 05:25:30.167 [info] <0.240.0> Running boot step rabbit_exchange_parameters defined by app rabbit
2026-04-20 05:25:30.168 [info] <0.240.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit
2026-04-20 05:25:30.168 [info] <0.240.0> Running boot step rabbit_policies defined by app rabbit
2026-04-20 05:25:30.168 [info] <0.240.0> Running boot step rabbit_policy defined by app rabbit
2026-04-20 05:25:30.168 [info] <0.240.0> Running boot step rabbit_queue_location_validator defined by app rabbit
2026-04-20 05:25:30.168 [info] <0.240.0> Running boot step rabbit_vhost_limit defined by app rabbit
2026-04-20 05:25:30.169 [info] <0.240.0> Running boot step rabbit_mgmt_reset_handler defined by app rabbitmq_management
2026-04-20 05:25:30.169 [info] <0.240.0> Running boot step rabbit_mgmt_db_handler defined by app rabbitmq_management_agent
2026-04-20 05:25:30.169 [warning] <0.240.0> force_fine_statistics set to true; ignored.
Replaced by {rates_mode, none} in the rabbitmq_management application.
2026-04-20 05:25:30.169 [info] <0.240.0> Management plugin: using rates mode 'basic'
2026-04-20 05:25:30.169 [info] <0.240.0> Running boot step recovery defined by app rabbit
2026-04-20 05:25:30.178 [info] <0.240.0> Running boot step load_definitions defined by app rabbitmq_management
2026-04-20 05:25:30.179 [info] <0.240.0> Applying definitions from /etc/rabbitmq/definitions.json
2026-04-20 05:25:30.179 [info] <0.240.0> Asked to import definitions. Acting user: rmq-internal
2026-04-20 05:25:30.179 [info] <0.240.0> Importing users...
2026-04-20 05:25:30.180 [info] <0.240.0> Creating user 'guest'
2026-04-20 05:25:30.189 [info] <0.240.0> Setting user tags for user 'guest' to [administrator]
2026-04-20 05:25:30.193 [info] <0.240.0> Importing vhosts...
2026-04-20 05:25:30.193 [info] <0.240.0> Adding vhost '/'
2026-04-20 05:25:30.226 [info] <0.515.0> Making sure data directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists
2026-04-20 05:25:30.235 [info] <0.515.0> Starting message stores for vhost '/'
2026-04-20 05:25:30.235 [info] <0.519.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index
2026-04-20 05:25:30.242 [info] <0.515.0> Started message store of type transient for vhost '/'
2026-04-20 05:25:30.244 [info] <0.522.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
2026-04-20 05:25:30.250 [warning] <0.522.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": rebuilding indices from scratch
2026-04-20 05:25:30.257 [info] <0.515.0> Started message store of type persistent for vhost '/'
2026-04-20 05:25:30.258 [info] <0.240.0> Importing user permissions...
2026-04-20 05:25:30.259 [info] <0.240.0> Setting permissions for 'guest' in '/' to '.*', '.*', '.*'
2026-04-20 05:25:30.264 [info] <0.240.0> Importing topic permissions...
2026-04-20 05:25:30.264 [info] <0.240.0> Importing parameters...
2026-04-20 05:25:30.264 [info] <0.240.0> Importing global parameters...
2026-04-20 05:25:30.264 [info] <0.240.0> Importing policies...
2026-04-20 05:25:30.273 [info] <0.240.0> Importing queues...
2026-04-20 05:25:30.273 [info] <0.240.0> Importing exchanges...
2026-04-20 05:25:30.273 [info] <0.240.0> Importing bindings...
2026-04-20 05:25:30.273 [info] <0.240.0> Running boot step empty_db_check defined by app rabbit
2026-04-20 05:25:30.273 [info] <0.240.0> Running boot step rabbit_looking_glass defined by app rabbit
2026-04-20 05:25:30.273 [info] <0.240.0> Running boot step rabbit_core_metrics_gc defined by app rabbit
2026-04-20 05:25:30.274 [info] <0.240.0> Running boot step background_gc defined by app rabbit
2026-04-20 05:25:30.274 [info] <0.240.0> Running boot step connection_tracking defined by app rabbit
2026-04-20 05:25:30.281 [info] <0.240.0> Setting up a table for connection tracking on this node: 'tracked_connection_on_node_contrail@cn-jenkins-deploy-platform-ansible-os-5649-3'
2026-04-20 05:25:30.289 [info] <0.240.0> Setting up a table for per-vhost connection counting on this node: 'tracked_connection_per_vhost_on_node_contrail@cn-jenkins-deploy-platform-ansible-os-5649-3'
2026-04-20 05:25:30.289 [info] <0.240.0> Running boot step routing_ready defined by app rabbit
2026-04-20 05:25:30.290 [info] <0.240.0> Running boot step pre_flight defined by app rabbit
2026-04-20 05:25:30.290 [info] <0.240.0> Running boot step notify_cluster defined by app rabbit
2026-04-20 05:25:30.290 [info] <0.240.0> Running boot step networking defined by app rabbit
2026-04-20 05:25:30.292 [info] <0.568.0> started TCP listener on [::]:5673
2026-04-20 05:25:30.292 [info] <0.240.0> Running boot step cluster_name defined by app rabbit
2026-04-20 05:25:30.292 [info] <0.240.0> Running boot step direct_client defined by app rabbit
2026-04-20 05:25:30.292 [info] <0.240.0> Running boot step os_signal_handler defined by app rabbit
2026-04-20 05:25:30.292 [info] <0.570.0> Swapping OS signal event handler (erl_signal_server) for our own
2026-04-20 05:25:30.364 [info] <0.620.0> Management plugin: HTTP (non-TLS) listener started on port 15673
2026-04-20 05:25:30.365 [info] <0.726.0> Statistics database started.
2026-04-20 05:25:30.365 [info] <0.725.0> Starting worker pool 'management_worker_pool' with 3 processes in it
2026-04-20 05:25:30.366 [notice] <0.106.0> Changed loghwm of /var/log/contrail/config-database-rabbitmq/rabbitmq.log to 50
2026-04-20 05:25:30.601 [info] <0.736.0> accepting AMQP connection <0.736.0> (10.0.0.38:49924 -> 10.0.0.33:5673)
2026-04-20 05:25:30.608 [info] <0.736.0> connection <0.736.0> (10.0.0.38:49924 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:25:30.702 [info] <0.8.0> Server startup complete; 3 plugins started.
 * rabbitmq_management
 * rabbitmq_web_dispatch
 * rabbitmq_management_agent
2026-04-20 05:25:30.737 [info] <0.756.0> accepting AMQP connection <0.756.0> (10.0.0.50:50732 -> 10.0.0.33:5673)
2026-04-20 05:25:30.747 [info] <0.756.0> connection <0.756.0> (10.0.0.50:50732 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:25:30.763 [info] <0.774.0> accepting AMQP connection <0.774.0> (10.0.0.50:50746 -> 10.0.0.33:5673)
2026-04-20 05:25:30.767 [info] <0.774.0> connection <0.774.0> (10.0.0.50:50746 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:25:30.778 [info] <0.774.0> closing AMQP connection <0.774.0> (10.0.0.50:50746 -> 10.0.0.33:5673, vhost: '/', user: 'guest')
2026-04-20 05:25:30.783 [info] <0.785.0> accepting AMQP connection <0.785.0> (10.0.0.50:50756 -> 10.0.0.33:5673)
2026-04-20 05:25:30.793 [info] <0.785.0> connection <0.785.0> (10.0.0.50:50756 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:25:30.912 [info] <0.794.0> accepting AMQP connection <0.794.0> (10.0.0.50:50768 -> 10.0.0.33:5673)
2026-04-20 05:25:30.916 [info] <0.794.0> connection <0.794.0> (10.0.0.50:50768 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:25:30.931 [info] <0.812.0> accepting AMQP connection <0.812.0> (10.0.0.50:50772 -> 10.0.0.33:5673)
2026-04-20 05:25:30.935 [info] <0.812.0> connection <0.812.0> (10.0.0.50:50772 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:25:30.954 [info] <0.812.0> closing AMQP connection <0.812.0> (10.0.0.50:50772 -> 10.0.0.33:5673, vhost: '/', user: 'guest')
2026-04-20 05:25:30.956 [info] <0.823.0> accepting AMQP connection <0.823.0> (10.0.0.50:50774 -> 10.0.0.33:5673)
2026-04-20 05:25:30.959 [info] <0.823.0> connection <0.823.0> (10.0.0.50:50774 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:25:31.041 [info] <0.832.0> accepting AMQP connection <0.832.0> (10.0.0.50:50786 -> 10.0.0.33:5673)
2026-04-20 05:25:31.047 [info] <0.832.0> connection <0.832.0> (10.0.0.50:50786 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:25:31.075 [info] <0.850.0> accepting AMQP connection <0.850.0> (10.0.0.33:49194 -> 10.0.0.33:5673)
2026-04-20 05:25:31.077 [info] <0.853.0> accepting AMQP connection <0.853.0> (10.0.0.38:49928 -> 10.0.0.33:5673)
2026-04-20 05:25:31.079 [info] <0.850.0> connection <0.850.0> (10.0.0.33:49194 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:25:31.080 [info] <0.853.0> connection <0.853.0> (10.0.0.38:49928 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:25:31.362 [info] <0.886.0> accepting AMQP connection <0.886.0> (10.0.0.33:49208 -> 10.0.0.33:5673)
2026-04-20 05:25:31.364 [info] <0.886.0> connection <0.886.0> (10.0.0.33:49208 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:25:31.378 [info] <0.905.0> accepting AMQP connection <0.905.0> (10.0.0.33:49222 -> 10.0.0.33:5673)
2026-04-20 05:25:31.380 [info] <0.905.0> connection <0.905.0> (10.0.0.33:49222 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:25:31.389 [info] <0.905.0> closing AMQP connection <0.905.0> (10.0.0.33:49222 -> 10.0.0.33:5673, vhost: '/', user: 'guest')
2026-04-20 05:25:31.391 [info] <0.916.0> accepting AMQP connection <0.916.0> (10.0.0.33:49234 -> 10.0.0.33:5673)
2026-04-20 05:25:31.394 [info] <0.916.0> connection <0.916.0> (10.0.0.33:49234 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:25:31.496 [info] <0.925.0> accepting AMQP connection <0.925.0> (10.0.0.38:49930 -> 10.0.0.33:5673)
2026-04-20 05:25:31.498 [info] <0.925.0> connection <0.925.0> (10.0.0.38:49930 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:25:31.513 [info] <0.943.0> accepting AMQP connection <0.943.0> (10.0.0.38:49936 -> 10.0.0.33:5673)
2026-04-20 05:25:31.515 [info] <0.943.0> connection <0.943.0> (10.0.0.38:49936 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:25:31.521 [info] <0.943.0> closing AMQP connection <0.943.0> (10.0.0.38:49936 -> 10.0.0.33:5673, vhost: '/', user: 'guest')
2026-04-20 05:25:31.527 [info] <0.954.0> accepting AMQP connection <0.954.0> (10.0.0.38:49944 -> 10.0.0.33:5673)
2026-04-20 05:25:31.535 [info] <0.954.0> connection <0.954.0> (10.0.0.38:49944 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:25:31.737 [info] <0.963.0> accepting AMQP connection <0.963.0> (10.0.0.33:49240 -> 10.0.0.33:5673)
2026-04-20 05:25:31.741 [info] <0.963.0> connection <0.963.0> (10.0.0.33:49240 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:25:31.758 [info] <0.981.0> accepting AMQP connection <0.981.0> (10.0.0.33:49242 -> 10.0.0.33:5673)
2026-04-20 05:25:31.763 [info] <0.981.0> connection <0.981.0> (10.0.0.33:49242 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:25:31.781 [info] <0.981.0> closing AMQP connection <0.981.0> (10.0.0.33:49242 -> 10.0.0.33:5673, vhost: '/', user: 'guest')
2026-04-20 05:25:31.783 [info] <0.992.0> accepting AMQP connection <0.992.0> (10.0.0.33:49252 -> 10.0.0.33:5673)
2026-04-20 05:25:31.786 [info] <0.992.0> connection <0.992.0> (10.0.0.33:49252 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:25:32.944 [info] <0.1012.0> accepting AMQP connection <0.1012.0> (10.0.0.38:49958 -> 10.0.0.33:5673)
2026-04-20 05:25:32.951 [info] <0.1012.0> connection <0.1012.0> (10.0.0.38:49958 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:25:32.978 [info] <0.1030.0> accepting AMQP connection <0.1030.0> (10.0.0.38:49972 -> 10.0.0.33:5673)
2026-04-20 05:25:32.989 [info] <0.1030.0> connection <0.1030.0> (10.0.0.38:49972 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:25:32.999 [info] <0.1030.0> closing AMQP connection <0.1030.0> (10.0.0.38:49972 -> 10.0.0.33:5673, vhost: '/', user: 'guest')
2026-04-20 05:25:32.999 [info] <0.1041.0> accepting AMQP connection <0.1041.0> (10.0.0.38:49980 -> 10.0.0.33:5673)
2026-04-20 05:25:33.009 [info] <0.1041.0> connection <0.1041.0> (10.0.0.38:49980 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:25:33.052 [info] <0.1051.0> RabbitMQ is asked to stop...
2026-04-20 05:25:33.130 [info] <0.1051.0> Stopping RabbitMQ applications and their dependencies in the following order:
    rabbitmq_management
    amqp_client
    rabbitmq_web_dispatch
    cowboy
    cowlib
    rabbitmq_management_agent
    rabbit
    mnesia
    rabbit_common
    sysmon_handler
    os_mon
2026-04-20 05:25:33.131 [info] <0.1051.0> Stopping application 'rabbitmq_management'
2026-04-20 05:25:33.132 [warning] <0.616.0> RabbitMQ HTTP listener registry could not find context rabbitmq_management_tls
2026-04-20 05:25:33.134 [info] <0.43.0> Application rabbitmq_management exited with reason: stopped
2026-04-20 05:25:33.134 [info] <0.1051.0> Stopping application 'amqp_client'
2026-04-20 05:25:33.136 [info] <0.43.0> Application amqp_client exited with reason: stopped
2026-04-20 05:25:33.136 [info] <0.1051.0> Stopping application 'rabbitmq_web_dispatch'
2026-04-20 05:25:33.138 [info] <0.1051.0> Stopping application 'cowboy'
2026-04-20 05:25:33.138 [info] <0.43.0> Application rabbitmq_web_dispatch exited with reason: stopped
2026-04-20 05:25:33.140 [info] <0.43.0> Application cowboy exited with reason: stopped
2026-04-20 05:25:33.140 [info] <0.1051.0> Stopping application 'cowlib'
2026-04-20 05:25:33.141 [info] <0.43.0> Application cowlib exited with reason: stopped
2026-04-20 05:25:33.141 [info] <0.1051.0> Stopping application 'rabbitmq_management_agent'
2026-04-20 05:25:33.143 [info] <0.43.0> Application rabbitmq_management_agent exited with reason: stopped
2026-04-20 05:25:33.143 [info] <0.1051.0> Stopping application 'rabbit'
2026-04-20 05:25:33.144 [info] <0.240.0> Will unregister with peer discovery backend rabbit_peer_discovery_classic_config
2026-04-20 05:25:33.144 [info] <0.568.0> stopped TCP listener on [::]:5673
2026-04-20 05:25:33.153 [error] <0.832.0> Error on AMQP connection <0.832.0> (10.0.0.50:50786 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:25:33.153 [error] <0.853.0> Error on AMQP connection <0.853.0> (10.0.0.38:49928 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:25:33.153 [error] <0.785.0> Error on AMQP connection <0.785.0> (10.0.0.50:50756 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:25:33.153 [error] <0.992.0> Error on AMQP connection <0.992.0> (10.0.0.33:49252 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:25:33.153 [error] <0.850.0> Error on AMQP connection <0.850.0> (10.0.0.33:49194 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:25:33.153 [error] <0.1041.0> Error on AMQP connection <0.1041.0> (10.0.0.38:49980 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:25:33.153 [error] <0.756.0> Error on AMQP connection <0.756.0> (10.0.0.50:50732 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:25:33.153 [error] <0.954.0> Error on AMQP connection <0.954.0> (10.0.0.38:49944 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:25:33.154 [error] <0.823.0> Error on AMQP connection <0.823.0> (10.0.0.50:50774 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:25:33.154 [error] <0.886.0> Error on AMQP connection <0.886.0> (10.0.0.33:49208 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:25:33.154 [error] <0.1012.0> Error on AMQP connection <0.1012.0> (10.0.0.38:49958 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:25:33.154 [error] <0.925.0> Error on AMQP connection <0.925.0> (10.0.0.38:49930 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:25:33.153 [error] <0.736.0> Error on AMQP connection <0.736.0> (10.0.0.38:49924 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:25:33.153 [error] <0.794.0> Error on AMQP connection <0.794.0> (10.0.0.50:50768 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:25:33.154 [error] <0.916.0> Error on AMQP connection <0.916.0> (10.0.0.33:49234 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:25:33.154 [error] <0.963.0> Error on AMQP connection <0.963.0> (10.0.0.33:49240 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:25:33.156 [error] <0.825.0> Supervisor {<0.825.0>,rabbit_channel_sup_sup} had child channel_sup started with rabbit_channel_sup:start_link() at undefined exit with reason shutdown in context shutdown_error
2026-04-20 05:25:33.156 [error] <0.956.0> Supervisor {<0.956.0>,rabbit_channel_sup_sup} had child channel_sup started with rabbit_channel_sup:start_link() at undefined exit with reason shutdown in context shutdown_error
2026-04-20 05:25:33.156 [error] <0.918.0> Supervisor {<0.918.0>,rabbit_channel_sup_sup} had child channel_sup started with rabbit_channel_sup:start_link() at undefined exit with reason shutdown in context shutdown_error
2026-04-20 05:25:33.156 [error] <0.857.0> Supervisor {<0.857.0>,rabbit_channel_sup_sup} had child channel_sup started with rabbit_channel_sup:start_link() at undefined exit with reason shutdown in context shutdown_error
2026-04-20 05:25:33.164 [info] <0.492.0> Closing all connections in vhost '/' on node 'contrail@cn-jenkins-deploy-platform-ansible-os-5649-3' because the vhost is stopping
2026-04-20 05:25:33.165 [warning] <0.747.0> Mirrored queue 'contrail-control.cn-jenkins-deploy-platform-ansible-os-5649-1' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available
2026-04-20 05:25:33.166 [warning] <0.877.0> Mirrored queue 'cn-jenkins-deploy-platform-ansible-os-5649-1.:contrail-alarm-gen:0.cn-jenkins-deploy-platform-ansible-os-5649-1.' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available
2026-04-20 05:25:33.166 [warning] <0.765.0> Mirrored queue 'cn-jenkins-deploy-platform-ansible-os-5649-2.:contrail-snmp-collector:0.cn-jenkins-deploy-platform-ansible-os-5649-2.' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available
2026-04-20 05:25:33.166 [warning] <0.868.0> Mirrored queue 'cn-jenkins-deploy-platform-ansible-os-5649-3.:contrail-alarm-gen:0.cn-jenkins-deploy-platform-ansible-os-5649-3.' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available
2026-04-20 05:25:33.166 [warning] <0.896.0> Mirrored queue 'cn-jenkins-deploy-platform-ansible-os-5649-3.:contrail-snmp-collector:0.cn-jenkins-deploy-platform-ansible-os-5649-3.' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available
2026-04-20 05:25:33.166 [warning] <0.1021.0> Mirrored queue 'cn-jenkins-deploy-platform-ansible-os-5649-1.:contrail-topology:0.cn-jenkins-deploy-platform-ansible-os-5649-1.' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available
2026-04-20 05:25:33.167 [warning] <0.841.0> Mirrored queue 'cn-jenkins-deploy-platform-ansible-os-5649-2.:contrail-alarm-gen:0.cn-jenkins-deploy-platform-ansible-os-5649-2.' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available
2026-04-20 05:25:33.167 [warning] <0.934.0> Mirrored queue 'cn-jenkins-deploy-platform-ansible-os-5649-1.:contrail-snmp-collector:0.cn-jenkins-deploy-platform-ansible-os-5649-1.' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available
2026-04-20 05:25:33.167 [warning] <0.803.0> Mirrored queue 'cn-jenkins-deploy-platform-ansible-os-5649-2.:contrail-topology:0.cn-jenkins-deploy-platform-ansible-os-5649-2.' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available
2026-04-20 05:25:33.167 [warning] <0.972.0> Mirrored queue 'cn-jenkins-deploy-platform-ansible-os-5649-3.:contrail-topology:0.cn-jenkins-deploy-platform-ansible-os-5649-3.' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available
2026-04-20 05:25:34.239 [warning]  lager_file_backend dropped 14 messages in the last second that exceeded the limit of 50 messages/sec
2026-04-20 05:25:41.560 [info] <0.1118.0> Resetting Rabbit forcefully
2026-04-20 05:25:43.583 [info] <0.1130.0> RabbitMQ is asked to stop...
2026-04-20 05:25:43.836 [info] <0.1130.0> Stopping RabbitMQ applications and their dependencies in the following order:
    rabbitmq_management
    amqp_client
    rabbitmq_web_dispatch
    rabbitmq_management_agent
    rabbit
    mnesia
    rabbit_common
    sysmon_handler
    os_mon
2026-04-20 05:25:43.836 [info] <0.1130.0> Stopping application 'rabbitmq_management'
2026-04-20 05:25:43.836 [info] <0.1130.0> Stopping application 'amqp_client'
2026-04-20 05:25:43.836 [info] <0.1130.0> Stopping application 'rabbitmq_web_dispatch'
2026-04-20 05:25:43.836 [info] <0.1130.0> Stopping application 'rabbitmq_management_agent'
2026-04-20 05:25:43.836 [info] <0.1130.0> Stopping application 'rabbit'
2026-04-20 05:25:43.836 [info] <0.1130.0> Stopping application 'mnesia'
2026-04-20 05:25:43.836 [info] <0.1130.0> Stopping application 'rabbit_common'
2026-04-20 05:25:43.836 [info] <0.1130.0> Stopping application 'sysmon_handler'
2026-04-20 05:25:43.836 [info] <0.1130.0> Stopping application 'os_mon'
2026-04-20 05:25:43.836 [info] <0.1130.0> Successfully stopped RabbitMQ and its dependencies
2026-04-20 05:25:43.836 [info] <0.1130.0> Halting Erlang VM with the following applications:
    ranch
    ssl
    credentials_obfuscation
    public_key
    asn1
    observer_cli
    jsx
    tools
    crypto
    stdout_formatter
    recon
    inets
    xmerl
    lager
    goldrush
    compiler
    syntax_tools
    sasl
    stdlib
    kernel
2026-04-20 05:25:54.816 [info] <0.8.0> Log file opened with Lager
2026-04-20 05:25:57.654 [info] <0.8.0> Feature flags: list of feature flags found:
2026-04-20 05:25:57.654 [info] <0.8.0> Feature flags: feature flag states written to disk: yes
2026-04-20 05:25:57.691 [info] <0.240.0> 
 Starting RabbitMQ 3.7.28 on Erlang 21.3.8.21
 Copyright (c) 2007-2020 Pivotal Software, Inc.
 Licensed under the MPL.  See https://www.rabbitmq.com/
2026-04-20 05:25:57.693 [info] <0.240.0> 
 node           : contrail@cn-jenkins-deploy-platform-ansible-os-5649-3
 home dir       : /var/lib/rabbitmq
 config file(s) : /etc/rabbitmq/rabbitmq.config
 cookie hash    : esZsDxSN6VGbi9JkMSxNZA==
 log(s)         : /var/log/contrail/config-database-rabbitmq/rabbitmq.log
                : /var/log/rabbitmq/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3_upgrade.log
 database dir   : /var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3
2026-04-20 05:25:57.711 [info] <0.240.0> Running boot step pre_boot defined by app rabbit
2026-04-20 05:25:57.711 [info] <0.240.0> Running boot step rabbit_core_metrics defined by app rabbit
2026-04-20 05:25:57.712 [info] <0.240.0> Running boot step rabbit_alarm defined by app rabbit
2026-04-20 05:25:57.724 [info] <0.246.0> Memory high watermark set to 12637 MiB (13251081011 bytes) of 15796 MiB (16563851264 bytes) total
2026-04-20 05:25:57.735 [info] <0.248.0> Enabling free disk space monitoring
2026-04-20 05:25:57.735 [info] <0.248.0> Disk free limit set to 50MB
2026-04-20 05:25:57.742 [info] <0.240.0> Running boot step code_server_cache defined by app rabbit
2026-04-20 05:25:57.743 [info] <0.240.0> Running boot step file_handle_cache defined by app rabbit
2026-04-20 05:25:57.744 [info] <0.251.0> Limiting to approx 65436 file handles (58890 sockets)
2026-04-20 05:25:57.744 [info] <0.252.0> FHC read buffering:  OFF
2026-04-20 05:25:57.744 [info] <0.252.0> FHC write buffering: ON
2026-04-20 05:25:57.745 [info] <0.240.0> Running boot step worker_pool defined by app rabbit
2026-04-20 05:25:57.746 [info] <0.241.0> Will use 4 processes for default worker pool
2026-04-20 05:25:57.746 [info] <0.241.0> Starting worker pool 'worker_pool' with 4 processes in it
2026-04-20 05:25:57.747 [info] <0.240.0> Running boot step database defined by app rabbit
2026-04-20 05:25:57.748 [info] <0.240.0> Node database directory at /var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3 is empty. Assuming we need to join an existing cluster or initialise from scratch...
2026-04-20 05:25:57.748 [info] <0.240.0> Configured peer discovery backend: rabbit_peer_discovery_classic_config
2026-04-20 05:25:57.748 [info] <0.240.0> Will try to lock with peer discovery backend rabbit_peer_discovery_classic_config
2026-04-20 05:25:57.748 [info] <0.240.0> Peer discovery backend does not support locking, falling back to randomized delay
2026-04-20 05:25:57.748 [info] <0.240.0> Peer discovery backend rabbit_peer_discovery_classic_config supports registration.
2026-04-20 05:25:57.748 [info] <0.240.0> Will wait for 51028 milliseconds before proceeding with registration...
2026-04-20 05:26:39.847 [info] <0.310.0> RabbitMQ hasn't finished starting yet. Waiting for startup to finish before stopping...
2026-04-20 05:26:48.776 [info] <0.240.0> All discovered existing cluster peers: contrail@cn-jenkins-deploy-platform-ansible-os-5649-1, contrail@cn-jenkins-deploy-platform-ansible-os-5649-2, contrail@cn-jenkins-deploy-platform-ansible-os-5649-3
2026-04-20 05:26:48.777 [info] <0.240.0> Peer nodes we can cluster with: contrail@cn-jenkins-deploy-platform-ansible-os-5649-1, contrail@cn-jenkins-deploy-platform-ansible-os-5649-2
2026-04-20 05:26:48.778 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,tables_not_present}
2026-04-20 05:26:48.782 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:26:48.782 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 9 retries left...
2026-04-20 05:26:49.284 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,tables_not_present}
2026-04-20 05:26:49.285 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:26:49.285 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 8 retries left...
2026-04-20 05:26:49.786 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,tables_not_present}
2026-04-20 05:26:49.787 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:26:49.787 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 7 retries left...
2026-04-20 05:26:50.289 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,tables_not_present}
2026-04-20 05:26:50.290 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:26:50.291 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 6 retries left...
2026-04-20 05:26:50.792 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,tables_not_present}
2026-04-20 05:26:50.793 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:26:50.793 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 5 retries left...
2026-04-20 05:26:51.295 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,tables_not_present}
2026-04-20 05:26:51.296 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:26:51.297 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 4 retries left...
2026-04-20 05:26:51.798 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,tables_not_present}
2026-04-20 05:26:51.799 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:26:51.799 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 3 retries left...
2026-04-20 05:26:52.301 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,tables_not_present}
2026-04-20 05:26:52.302 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:26:52.302 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 2 retries left...
2026-04-20 05:26:52.803 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,tables_not_present}
2026-04-20 05:26:52.804 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:26:52.804 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 1 retries left...
2026-04-20 05:26:53.305 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,tables_not_present}
2026-04-20 05:26:53.305 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:26:53.306 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 0 retries left...
2026-04-20 05:26:53.806 [warning] <0.240.0> Could not successfully contact any node of: contrail@cn-jenkins-deploy-platform-ansible-os-5649-1,contrail@cn-jenkins-deploy-platform-ansible-os-5649-2 (as in Erlang distribution). Starting as a blank standalone node...
2026-04-20 05:26:53.808 [info] <0.43.0> Application mnesia exited with reason: stopped
2026-04-20 05:26:53.944 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2026-04-20 05:26:53.945 [info] <0.240.0> Successfully synced tables from a peer
2026-04-20 05:26:53.991 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2026-04-20 05:26:53.992 [info] <0.240.0> Successfully synced tables from a peer
2026-04-20 05:26:54.031 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2026-04-20 05:26:54.032 [info] <0.240.0> Successfully synced tables from a peer
2026-04-20 05:26:54.032 [info] <0.240.0> Will register with peer discovery backend rabbit_peer_discovery_classic_config
2026-04-20 05:26:54.033 [info] <0.240.0> Running boot step database_sync defined by app rabbit
2026-04-20 05:26:54.033 [info] <0.240.0> Running boot step feature_flags defined by app rabbit
2026-04-20 05:26:54.034 [info] <0.240.0> Running boot step codec_correctness_check defined by app rabbit
2026-04-20 05:26:54.035 [info] <0.240.0> Running boot step external_infrastructure defined by app rabbit
2026-04-20 05:26:54.035 [info] <0.240.0> Running boot step rabbit_registry defined by app rabbit
2026-04-20 05:26:54.035 [info] <0.240.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit
2026-04-20 05:26:54.036 [info] <0.240.0> Running boot step rabbit_queue_location_random defined by app rabbit
2026-04-20 05:26:54.036 [info] <0.240.0> Running boot step rabbit_event defined by app rabbit
2026-04-20 05:26:54.037 [info] <0.240.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit
2026-04-20 05:26:54.037 [info] <0.240.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit
2026-04-20 05:26:54.038 [info] <0.240.0> Running boot step rabbit_exchange_type_direct defined by app rabbit
2026-04-20 05:26:54.038 [info] <0.240.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit
2026-04-20 05:26:54.038 [info] <0.240.0> Running boot step rabbit_exchange_type_headers defined by app rabbit
2026-04-20 05:26:54.038 [info] <0.240.0> Running boot step rabbit_exchange_type_topic defined by app rabbit
2026-04-20 05:26:54.039 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit
2026-04-20 05:26:54.039 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit
2026-04-20 05:26:54.039 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit
2026-04-20 05:26:54.039 [info] <0.240.0> Running boot step rabbit_priority_queue defined by app rabbit
2026-04-20 05:26:54.039 [info] <0.240.0> Priority queues enabled, real BQ is rabbit_variable_queue
2026-04-20 05:26:54.039 [info] <0.240.0> Running boot step rabbit_queue_location_client_local defined by app rabbit
2026-04-20 05:26:54.039 [info] <0.240.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit
2026-04-20 05:26:54.039 [info] <0.240.0> Running boot step kernel_ready defined by app rabbit
2026-04-20 05:26:54.039 [info] <0.240.0> Running boot step rabbit_sysmon_minder defined by app rabbit
2026-04-20 05:26:54.039 [info] <0.240.0> Running boot step rabbit_epmd_monitor defined by app rabbit
2026-04-20 05:26:54.040 [info] <0.474.0> epmd monitor knows us, inter-node communication (distribution) port: 25673
2026-04-20 05:26:54.041 [info] <0.240.0> Running boot step guid_generator defined by app rabbit
2026-04-20 05:26:54.048 [info] <0.240.0> Running boot step rabbit_node_monitor defined by app rabbit
2026-04-20 05:26:54.049 [info] <0.478.0> Starting rabbit_node_monitor
2026-04-20 05:26:54.049 [info] <0.240.0> Running boot step delegate_sup defined by app rabbit
2026-04-20 05:26:54.050 [info] <0.240.0> Running boot step rabbit_memory_monitor defined by app rabbit
2026-04-20 05:26:54.050 [info] <0.240.0> Running boot step core_initialized defined by app rabbit
2026-04-20 05:26:54.050 [info] <0.240.0> Running boot step upgrade_queues defined by app rabbit
2026-04-20 05:26:54.094 [info] <0.240.0> message_store upgrades: 1 to apply
2026-04-20 05:26:54.094 [info] <0.240.0> message_store upgrades: Applying rabbit_variable_queue:move_messages_to_vhost_store
2026-04-20 05:26:54.094 [info] <0.240.0> message_store upgrades: No durable queues found. Skipping message store migration
2026-04-20 05:26:54.094 [info] <0.240.0> message_store upgrades: Removing the old message store data
2026-04-20 05:26:54.095 [info] <0.240.0> message_store upgrades: All upgrades applied successfully
2026-04-20 05:26:54.147 [info] <0.240.0> Running boot step rabbit_connection_tracking defined by app rabbit
2026-04-20 05:26:54.147 [info] <0.240.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit
2026-04-20 05:26:54.147 [info] <0.240.0> Running boot step rabbit_exchange_parameters defined by app rabbit
2026-04-20 05:26:54.147 [info] <0.240.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit
2026-04-20 05:26:54.149 [info] <0.240.0> Running boot step rabbit_policies defined by app rabbit
2026-04-20 05:26:54.150 [info] <0.240.0> Running boot step rabbit_policy defined by app rabbit
2026-04-20 05:26:54.150 [info] <0.240.0> Running boot step rabbit_queue_location_validator defined by app rabbit
2026-04-20 05:26:54.150 [info] <0.240.0> Running boot step rabbit_vhost_limit defined by app rabbit
2026-04-20 05:26:54.150 [info] <0.240.0> Running boot step rabbit_mgmt_reset_handler defined by app rabbitmq_management
2026-04-20 05:26:54.150 [info] <0.240.0> Running boot step rabbit_mgmt_db_handler defined by app rabbitmq_management_agent
2026-04-20 05:26:54.150 [warning] <0.240.0> force_fine_statistics set to true; ignored.
Replaced by {rates_mode, none} in the rabbitmq_management application.
2026-04-20 05:26:54.150 [info] <0.240.0> Management plugin: using rates mode 'basic'
2026-04-20 05:26:54.150 [info] <0.240.0> Running boot step recovery defined by app rabbit
2026-04-20 05:26:54.151 [info] <0.240.0> Running boot step load_definitions defined by app rabbitmq_management
2026-04-20 05:26:54.152 [info] <0.240.0> Applying definitions from /etc/rabbitmq/definitions.json
2026-04-20 05:26:54.152 [info] <0.240.0> Asked to import definitions. Acting user: rmq-internal
2026-04-20 05:26:54.152 [info] <0.240.0> Importing users...
2026-04-20 05:26:54.152 [info] <0.240.0> Creating user 'guest'
2026-04-20 05:26:54.171 [info] <0.240.0> Setting user tags for user 'guest' to [administrator]
2026-04-20 05:26:54.176 [info] <0.240.0> Importing vhosts...
2026-04-20 05:26:54.177 [info] <0.240.0> Adding vhost '/'
2026-04-20 05:26:54.203 [info] <0.526.0> Making sure data directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists
2026-04-20 05:26:54.213 [info] <0.526.0> Starting message stores for vhost '/'
2026-04-20 05:26:54.213 [info] <0.530.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index
2026-04-20 05:26:54.214 [info] <0.526.0> Started message store of type transient for vhost '/'
2026-04-20 05:26:54.214 [info] <0.533.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
2026-04-20 05:26:54.215 [warning] <0.533.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": rebuilding indices from scratch
2026-04-20 05:26:54.215 [info] <0.526.0> Started message store of type persistent for vhost '/'
2026-04-20 05:26:54.217 [info] <0.240.0> Importing user permissions...
2026-04-20 05:26:54.217 [info] <0.240.0> Setting permissions for 'guest' in '/' to '.*', '.*', '.*'
2026-04-20 05:26:54.225 [info] <0.240.0> Importing topic permissions...
2026-04-20 05:26:54.225 [info] <0.240.0> Importing parameters...
2026-04-20 05:26:54.225 [info] <0.240.0> Importing global parameters...
2026-04-20 05:26:54.225 [info] <0.240.0> Importing policies...
2026-04-20 05:26:54.240 [info] <0.240.0> Importing queues...
2026-04-20 05:26:54.240 [info] <0.240.0> Importing exchanges...
2026-04-20 05:26:54.240 [info] <0.240.0> Importing bindings...
2026-04-20 05:26:54.240 [info] <0.240.0> Running boot step empty_db_check defined by app rabbit
2026-04-20 05:26:54.240 [info] <0.240.0> Running boot step rabbit_looking_glass defined by app rabbit
2026-04-20 05:26:54.240 [info] <0.240.0> Running boot step rabbit_core_metrics_gc defined by app rabbit
2026-04-20 05:26:54.240 [info] <0.240.0> Running boot step background_gc defined by app rabbit
2026-04-20 05:26:54.240 [info] <0.240.0> Running boot step connection_tracking defined by app rabbit
2026-04-20 05:26:54.246 [info] <0.240.0> Setting up a table for connection tracking on this node: 'tracked_connection_on_node_contrail@cn-jenkins-deploy-platform-ansible-os-5649-3'
2026-04-20 05:26:54.250 [info] <0.240.0> Setting up a table for per-vhost connection counting on this node: 'tracked_connection_per_vhost_on_node_contrail@cn-jenkins-deploy-platform-ansible-os-5649-3'
2026-04-20 05:26:54.250 [info] <0.240.0> Running boot step routing_ready defined by app rabbit
2026-04-20 05:26:54.250 [info] <0.240.0> Running boot step pre_flight defined by app rabbit
2026-04-20 05:26:54.250 [info] <0.240.0> Running boot step notify_cluster defined by app rabbit
2026-04-20 05:26:54.250 [info] <0.240.0> Running boot step networking defined by app rabbit
2026-04-20 05:26:54.252 [info] <0.579.0> started TCP listener on [::]:5673
2026-04-20 05:26:54.253 [info] <0.240.0> Running boot step cluster_name defined by app rabbit
2026-04-20 05:26:54.253 [info] <0.240.0> Running boot step direct_client defined by app rabbit
2026-04-20 05:26:54.253 [info] <0.240.0> Running boot step os_signal_handler defined by app rabbit
2026-04-20 05:26:54.253 [info] <0.581.0> Swapping OS signal event handler (erl_signal_server) for our own
2026-04-20 05:26:54.295 [info] <0.631.0> Management plugin: HTTP (non-TLS) listener started on port 15673
2026-04-20 05:26:54.296 [info] <0.737.0> Statistics database started.
2026-04-20 05:26:54.296 [info] <0.736.0> Starting worker pool 'management_worker_pool' with 3 processes in it
2026-04-20 05:26:54.296 [notice] <0.106.0> Changed loghwm of /var/log/contrail/config-database-rabbitmq/rabbitmq.log to 50
2026-04-20 05:26:54.589 [info] <0.8.0> Server startup complete; 3 plugins started.
 * rabbitmq_management
 * rabbitmq_management_agent
 * rabbitmq_web_dispatch
2026-04-20 05:26:54.598 [info] <0.310.0> RabbitMQ is asked to stop...
2026-04-20 05:26:54.677 [info] <0.310.0> Stopping RabbitMQ applications and their dependencies in the following order:
    rabbitmq_management
    amqp_client
    rabbitmq_web_dispatch
    cowboy
    cowlib
    rabbitmq_management_agent
    rabbit
    mnesia
    rabbit_common
    sysmon_handler
    os_mon
2026-04-20 05:26:54.677 [info] <0.310.0> Stopping application 'rabbitmq_management'
2026-04-20 05:26:54.697 [warning] <0.587.0> RabbitMQ HTTP listener registry could not find context rabbitmq_management_tls
2026-04-20 05:26:54.699 [info] <0.43.0> Application rabbitmq_management exited with reason: stopped
2026-04-20 05:26:54.699 [info] <0.310.0> Stopping application 'amqp_client'
2026-04-20 05:26:54.706 [info] <0.43.0> Application amqp_client exited with reason: stopped
2026-04-20 05:26:54.706 [info] <0.310.0> Stopping application 'rabbitmq_web_dispatch'
2026-04-20 05:26:54.713 [info] <0.310.0> Stopping application 'cowboy'
2026-04-20 05:26:54.713 [info] <0.43.0> Application rabbitmq_web_dispatch exited with reason: stopped
2026-04-20 05:26:54.718 [info] <0.43.0> Application cowboy exited with reason: stopped
2026-04-20 05:26:54.718 [info] <0.310.0> Stopping application 'cowlib'
2026-04-20 05:26:54.718 [info] <0.43.0> Application cowlib exited with reason: stopped
2026-04-20 05:26:54.718 [info] <0.310.0> Stopping application 'rabbitmq_management_agent'
2026-04-20 05:26:54.720 [info] <0.43.0> Application rabbitmq_management_agent exited with reason: stopped
2026-04-20 05:26:54.720 [info] <0.310.0> Stopping application 'rabbit'
2026-04-20 05:26:54.720 [info] <0.240.0> Will unregister with peer discovery backend rabbit_peer_discovery_classic_config
2026-04-20 05:26:54.720 [info] <0.579.0> stopped TCP listener on [::]:5673
2026-04-20 05:26:54.721 [info] <0.503.0> Closing all connections in vhost '/' on node 'contrail@cn-jenkins-deploy-platform-ansible-os-5649-3' because the vhost is stopping
2026-04-20 05:26:54.721 [info] <0.533.0> Stopping message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent'
2026-04-20 05:26:54.734 [info] <0.533.0> Message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent' is stopped
2026-04-20 05:26:54.735 [info] <0.530.0> Stopping message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient'
2026-04-20 05:26:54.746 [info] <0.530.0> Message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient' is stopped
2026-04-20 05:26:54.752 [info] <0.43.0> Application rabbit exited with reason: stopped
2026-04-20 05:26:54.752 [info] <0.310.0> Stopping application 'mnesia'
2026-04-20 05:26:54.759 [info] <0.43.0> Application mnesia exited with reason: stopped
2026-04-20 05:26:54.759 [info] <0.310.0> Stopping application 'rabbit_common'
2026-04-20 05:26:54.759 [info] <0.43.0> Application rabbit_common exited with reason: stopped
2026-04-20 05:26:54.759 [info] <0.310.0> Stopping application 'sysmon_handler'
2026-04-20 05:26:54.763 [info] <0.43.0> Application sysmon_handler exited with reason: stopped
2026-04-20 05:26:54.763 [info] <0.310.0> Stopping application 'os_mon'
2026-04-20 05:26:54.769 [info] <0.43.0> Application os_mon exited with reason: stopped
2026-04-20 05:26:54.769 [info] <0.310.0> Successfully stopped RabbitMQ and its dependencies
2026-04-20 05:27:03.520 [info] <0.759.0> Resetting Rabbit forcefully
2026-04-20 05:27:05.164 [info] <0.771.0> RabbitMQ is asked to stop...
2026-04-20 05:27:05.348 [info] <0.771.0> Stopping RabbitMQ applications and their dependencies in the following order:
    rabbitmq_management
    amqp_client
    rabbitmq_management_agent
    rabbitmq_web_dispatch
    rabbit
    mnesia
    rabbit_common
    sysmon_handler
    os_mon
2026-04-20 05:27:05.348 [info] <0.771.0> Stopping application 'rabbitmq_management'
2026-04-20 05:27:05.348 [info] <0.771.0> Stopping application 'amqp_client'
2026-04-20 05:27:05.348 [info] <0.771.0> Stopping application 'rabbitmq_management_agent'
2026-04-20 05:27:05.348 [info] <0.771.0> Stopping application 'rabbitmq_web_dispatch'
2026-04-20 05:27:05.348 [info] <0.771.0> Stopping application 'rabbit'
2026-04-20 05:27:05.348 [info] <0.771.0> Stopping application 'mnesia'
2026-04-20 05:27:05.348 [info] <0.771.0> Stopping application 'rabbit_common'
2026-04-20 05:27:05.348 [info] <0.771.0> Stopping application 'sysmon_handler'
2026-04-20 05:27:05.348 [info] <0.771.0> Stopping application 'os_mon'
2026-04-20 05:27:05.348 [info] <0.771.0> Successfully stopped RabbitMQ and its dependencies
2026-04-20 05:27:05.349 [info] <0.771.0> Halting Erlang VM with the following applications:
    ranch
    ssl
    public_key
    asn1
    observer_cli
    tools
    credentials_obfuscation
    crypto
    stdout_formatter
    recon
    inets
    jsx
    xmerl
    lager
    goldrush
    compiler
    syntax_tools
    sasl
    stdlib
    kernel
2026-04-20 05:27:16.578 [info] <0.8.0> Log file opened with Lager
2026-04-20 05:27:21.071 [info] <0.8.0> Feature flags: list of feature flags found:
2026-04-20 05:27:21.071 [info] <0.8.0> Feature flags: feature flag states written to disk: yes
2026-04-20 05:27:21.102 [info] <0.240.0> 
 Starting RabbitMQ 3.7.28 on Erlang 21.3.8.21
 Copyright (c) 2007-2020 Pivotal Software, Inc.
 Licensed under the MPL.  See https://www.rabbitmq.com/
2026-04-20 05:27:21.104 [info] <0.240.0> 
 node           : contrail@cn-jenkins-deploy-platform-ansible-os-5649-3
 home dir       : /var/lib/rabbitmq
 config file(s) : /etc/rabbitmq/rabbitmq.config
 cookie hash    : esZsDxSN6VGbi9JkMSxNZA==
 log(s)         : /var/log/contrail/config-database-rabbitmq/rabbitmq.log
                : /var/log/rabbitmq/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3_upgrade.log
 database dir   : /var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3
2026-04-20 05:27:21.119 [info] <0.240.0> Running boot step pre_boot defined by app rabbit
2026-04-20 05:27:21.119 [info] <0.240.0> Running boot step rabbit_core_metrics defined by app rabbit
2026-04-20 05:27:21.120 [info] <0.240.0> Running boot step rabbit_alarm defined by app rabbit
2026-04-20 05:27:21.138 [info] <0.246.0> Memory high watermark set to 12637 MiB (13251081011 bytes) of 15796 MiB (16563851264 bytes) total
2026-04-20 05:27:21.165 [info] <0.248.0> Enabling free disk space monitoring
2026-04-20 05:27:21.166 [info] <0.248.0> Disk free limit set to 50MB
2026-04-20 05:27:21.173 [info] <0.240.0> Running boot step code_server_cache defined by app rabbit
2026-04-20 05:27:21.174 [info] <0.240.0> Running boot step file_handle_cache defined by app rabbit
2026-04-20 05:27:21.176 [info] <0.251.0> Limiting to approx 65436 file handles (58890 sockets)
2026-04-20 05:27:21.176 [info] <0.252.0> FHC read buffering:  OFF
2026-04-20 05:27:21.176 [info] <0.252.0> FHC write buffering: ON
2026-04-20 05:27:21.181 [info] <0.240.0> Running boot step worker_pool defined by app rabbit
2026-04-20 05:27:21.182 [info] <0.241.0> Will use 4 processes for default worker pool
2026-04-20 05:27:21.183 [info] <0.241.0> Starting worker pool 'worker_pool' with 4 processes in it
2026-04-20 05:27:21.183 [info] <0.240.0> Running boot step database defined by app rabbit
2026-04-20 05:27:21.185 [info] <0.240.0> Node database directory at /var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3 is empty. Assuming we need to join an existing cluster or initialise from scratch...
2026-04-20 05:27:21.186 [info] <0.240.0> Configured peer discovery backend: rabbit_peer_discovery_classic_config
2026-04-20 05:27:21.186 [info] <0.240.0> Will try to lock with peer discovery backend rabbit_peer_discovery_classic_config
2026-04-20 05:27:21.186 [info] <0.240.0> Peer discovery backend does not support locking, falling back to randomized delay
2026-04-20 05:27:21.186 [info] <0.240.0> Peer discovery backend rabbit_peer_discovery_classic_config supports registration.
2026-04-20 05:27:21.186 [info] <0.240.0> Will wait for 21517 milliseconds before proceeding with registration...
2026-04-20 05:27:42.703 [info] <0.240.0> All discovered existing cluster peers: contrail@cn-jenkins-deploy-platform-ansible-os-5649-1, contrail@cn-jenkins-deploy-platform-ansible-os-5649-2, contrail@cn-jenkins-deploy-platform-ansible-os-5649-3
2026-04-20 05:27:42.703 [info] <0.240.0> Peer nodes we can cluster with: contrail@cn-jenkins-deploy-platform-ansible-os-5649-1, contrail@cn-jenkins-deploy-platform-ansible-os-5649-2
2026-04-20 05:27:42.705 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {badrpc,nodedown}
2026-04-20 05:27:42.706 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,mnesia_not_running}
2026-04-20 05:27:42.706 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 9 retries left...
2026-04-20 05:27:43.207 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {badrpc,nodedown}
2026-04-20 05:27:43.210 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,mnesia_not_running}
2026-04-20 05:27:43.210 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 8 retries left...
2026-04-20 05:27:43.712 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {badrpc,nodedown}
2026-04-20 05:27:43.713 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,mnesia_not_running}
2026-04-20 05:27:43.713 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 7 retries left...
2026-04-20 05:27:44.215 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {badrpc,nodedown}
2026-04-20 05:27:44.216 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,mnesia_not_running}
2026-04-20 05:27:44.216 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 6 retries left...
2026-04-20 05:27:44.719 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {badrpc,nodedown}
2026-04-20 05:27:44.719 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,mnesia_not_running}
2026-04-20 05:27:44.719 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 5 retries left...
2026-04-20 05:27:45.222 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {badrpc,nodedown}
2026-04-20 05:27:45.223 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,mnesia_not_running}
2026-04-20 05:27:45.223 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 4 retries left...
2026-04-20 05:27:45.814 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,mnesia_not_running}
2026-04-20 05:27:45.815 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,mnesia_not_running}
2026-04-20 05:27:45.815 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 3 retries left...
2026-04-20 05:27:46.317 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,mnesia_not_running}
2026-04-20 05:27:46.319 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {badrpc,nodedown}
2026-04-20 05:27:46.319 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 2 retries left...
2026-04-20 05:27:46.822 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,mnesia_not_running}
2026-04-20 05:27:46.824 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {badrpc,nodedown}
2026-04-20 05:27:46.824 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 1 retries left...
2026-04-20 05:27:47.362 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,mnesia_not_running}
2026-04-20 05:27:47.365 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {badrpc,nodedown}
2026-04-20 05:27:47.366 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 0 retries left...
2026-04-20 05:27:47.866 [warning] <0.240.0> Could not successfully contact any node of: contrail@cn-jenkins-deploy-platform-ansible-os-5649-1,contrail@cn-jenkins-deploy-platform-ansible-os-5649-2 (as in Erlang distribution). Starting as a blank standalone node...
2026-04-20 05:27:47.869 [info] <0.43.0> Application mnesia exited with reason: stopped
2026-04-20 05:27:48.005 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2026-04-20 05:27:48.005 [info] <0.240.0> Successfully synced tables from a peer
2026-04-20 05:27:48.073 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2026-04-20 05:27:48.074 [info] <0.240.0> Successfully synced tables from a peer
2026-04-20 05:27:48.118 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2026-04-20 05:27:48.118 [info] <0.240.0> Successfully synced tables from a peer
2026-04-20 05:27:48.118 [info] <0.240.0> Will register with peer discovery backend rabbit_peer_discovery_classic_config
2026-04-20 05:27:48.118 [info] <0.240.0> Running boot step database_sync defined by app rabbit
2026-04-20 05:27:48.118 [info] <0.240.0> Running boot step feature_flags defined by app rabbit
2026-04-20 05:27:48.118 [info] <0.240.0> Running boot step codec_correctness_check defined by app rabbit
2026-04-20 05:27:48.119 [info] <0.240.0> Running boot step external_infrastructure defined by app rabbit
2026-04-20 05:27:48.119 [info] <0.240.0> Running boot step rabbit_registry defined by app rabbit
2026-04-20 05:27:48.119 [info] <0.240.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit
2026-04-20 05:27:48.119 [info] <0.240.0> Running boot step rabbit_queue_location_random defined by app rabbit
2026-04-20 05:27:48.119 [info] <0.240.0> Running boot step rabbit_event defined by app rabbit
2026-04-20 05:27:48.119 [info] <0.240.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit
2026-04-20 05:27:48.119 [info] <0.240.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit
2026-04-20 05:27:48.119 [info] <0.240.0> Running boot step rabbit_exchange_type_direct defined by app rabbit
2026-04-20 05:27:48.119 [info] <0.240.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit
2026-04-20 05:27:48.119 [info] <0.240.0> Running boot step rabbit_exchange_type_headers defined by app rabbit
2026-04-20 05:27:48.120 [info] <0.240.0> Running boot step rabbit_exchange_type_topic defined by app rabbit
2026-04-20 05:27:48.120 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit
2026-04-20 05:27:48.120 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit
2026-04-20 05:27:48.120 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit
2026-04-20 05:27:48.120 [info] <0.240.0> Running boot step rabbit_priority_queue defined by app rabbit
2026-04-20 05:27:48.120 [info] <0.240.0> Priority queues enabled, real BQ is rabbit_variable_queue
2026-04-20 05:27:48.120 [info] <0.240.0> Running boot step rabbit_queue_location_client_local defined by app rabbit
2026-04-20 05:27:48.123 [info] <0.240.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit
2026-04-20 05:27:48.123 [info] <0.240.0> Running boot step kernel_ready defined by app rabbit
2026-04-20 05:27:48.123 [info] <0.240.0> Running boot step rabbit_sysmon_minder defined by app rabbit
2026-04-20 05:27:48.123 [info] <0.240.0> Running boot step rabbit_epmd_monitor defined by app rabbit
2026-04-20 05:27:48.124 [info] <0.482.0> epmd monitor knows us, inter-node communication (distribution) port: 25673
2026-04-20 05:27:48.124 [info] <0.240.0> Running boot step guid_generator defined by app rabbit
2026-04-20 05:27:48.129 [info] <0.240.0> Running boot step rabbit_node_monitor defined by app rabbit
2026-04-20 05:27:48.129 [info] <0.486.0> Starting rabbit_node_monitor
2026-04-20 05:27:48.129 [info] <0.240.0> Running boot step delegate_sup defined by app rabbit
2026-04-20 05:27:48.129 [info] <0.240.0> Running boot step rabbit_memory_monitor defined by app rabbit
2026-04-20 05:27:48.130 [info] <0.240.0> Running boot step core_initialized defined by app rabbit
2026-04-20 05:27:48.130 [info] <0.240.0> Running boot step upgrade_queues defined by app rabbit
2026-04-20 05:27:48.169 [info] <0.240.0> message_store upgrades: 1 to apply
2026-04-20 05:27:48.170 [info] <0.240.0> message_store upgrades: Applying rabbit_variable_queue:move_messages_to_vhost_store
2026-04-20 05:27:48.171 [info] <0.240.0> message_store upgrades: No durable queues found. Skipping message store migration
2026-04-20 05:27:48.171 [info] <0.240.0> message_store upgrades: Removing the old message store data
2026-04-20 05:27:48.177 [info] <0.240.0> message_store upgrades: All upgrades applied successfully
2026-04-20 05:27:48.220 [info] <0.240.0> Running boot step rabbit_connection_tracking defined by app rabbit
2026-04-20 05:27:48.221 [info] <0.240.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit
2026-04-20 05:27:48.221 [info] <0.240.0> Running boot step rabbit_exchange_parameters defined by app rabbit
2026-04-20 05:27:48.221 [info] <0.240.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit
2026-04-20 05:27:48.221 [info] <0.240.0> Running boot step rabbit_policies defined by app rabbit
2026-04-20 05:27:48.222 [info] <0.240.0> Running boot step rabbit_policy defined by app rabbit
2026-04-20 05:27:48.222 [info] <0.240.0> Running boot step rabbit_queue_location_validator defined by app rabbit
2026-04-20 05:27:48.222 [info] <0.240.0> Running boot step rabbit_vhost_limit defined by app rabbit
2026-04-20 05:27:48.222 [info] <0.240.0> Running boot step rabbit_mgmt_reset_handler defined by app rabbitmq_management
2026-04-20 05:27:48.222 [info] <0.240.0> Running boot step rabbit_mgmt_db_handler defined by app rabbitmq_management_agent
2026-04-20 05:27:48.222 [warning] <0.240.0> force_fine_statistics set to true; ignored.
Replaced by {rates_mode, none} in the rabbitmq_management application.
2026-04-20 05:27:48.222 [info] <0.240.0> Management plugin: using rates mode 'basic'
2026-04-20 05:27:48.222 [info] <0.240.0> Running boot step recovery defined by app rabbit
2026-04-20 05:27:48.223 [info] <0.240.0> Running boot step load_definitions defined by app rabbitmq_management
2026-04-20 05:27:48.223 [info] <0.240.0> Applying definitions from /etc/rabbitmq/definitions.json
2026-04-20 05:27:48.223 [info] <0.240.0> Asked to import definitions. Acting user: rmq-internal
2026-04-20 05:27:48.223 [info] <0.240.0> Importing users...
2026-04-20 05:27:48.223 [info] <0.240.0> Creating user 'guest'
2026-04-20 05:27:48.234 [info] <0.240.0> Setting user tags for user 'guest' to [administrator]
2026-04-20 05:27:48.237 [info] <0.240.0> Importing vhosts...
2026-04-20 05:27:48.237 [info] <0.240.0> Adding vhost '/'
2026-04-20 05:27:48.262 [info] <0.534.0> Making sure data directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists
2026-04-20 05:27:48.267 [info] <0.534.0> Starting message stores for vhost '/'
2026-04-20 05:27:48.268 [info] <0.538.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index
2026-04-20 05:27:48.270 [info] <0.534.0> Started message store of type transient for vhost '/'
2026-04-20 05:27:48.270 [info] <0.541.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
2026-04-20 05:27:48.270 [warning] <0.541.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": rebuilding indices from scratch
2026-04-20 05:27:48.272 [info] <0.534.0> Started message store of type persistent for vhost '/'
2026-04-20 05:27:48.273 [info] <0.240.0> Importing user permissions...
2026-04-20 05:27:48.273 [info] <0.240.0> Setting permissions for 'guest' in '/' to '.*', '.*', '.*'
2026-04-20 05:27:48.277 [info] <0.240.0> Importing topic permissions...
2026-04-20 05:27:48.277 [info] <0.240.0> Importing parameters...
2026-04-20 05:27:48.277 [info] <0.240.0> Importing global parameters...
2026-04-20 05:27:48.277 [info] <0.240.0> Importing policies...
2026-04-20 05:27:48.284 [info] <0.240.0> Importing queues...
2026-04-20 05:27:48.284 [info] <0.240.0> Importing exchanges...
2026-04-20 05:27:48.284 [info] <0.240.0> Importing bindings...
2026-04-20 05:27:48.284 [info] <0.240.0> Running boot step empty_db_check defined by app rabbit
2026-04-20 05:27:48.284 [info] <0.240.0> Running boot step rabbit_looking_glass defined by app rabbit
2026-04-20 05:27:48.284 [info] <0.240.0> Running boot step rabbit_core_metrics_gc defined by app rabbit
2026-04-20 05:27:48.284 [info] <0.240.0> Running boot step background_gc defined by app rabbit
2026-04-20 05:27:48.284 [info] <0.240.0> Running boot step connection_tracking defined by app rabbit
2026-04-20 05:27:48.291 [info] <0.240.0> Setting up a table for connection tracking on this node: 'tracked_connection_on_node_contrail@cn-jenkins-deploy-platform-ansible-os-5649-3'
2026-04-20 05:27:48.296 [info] <0.240.0> Setting up a table for per-vhost connection counting on this node: 'tracked_connection_per_vhost_on_node_contrail@cn-jenkins-deploy-platform-ansible-os-5649-3'
2026-04-20 05:27:48.296 [info] <0.240.0> Running boot step routing_ready defined by app rabbit
2026-04-20 05:27:48.296 [info] <0.240.0> Running boot step pre_flight defined by app rabbit
2026-04-20 05:27:48.296 [info] <0.240.0> Running boot step notify_cluster defined by app rabbit
2026-04-20 05:27:48.296 [info] <0.240.0> Running boot step networking defined by app rabbit
2026-04-20 05:27:48.298 [info] <0.587.0> started TCP listener on [::]:5673
2026-04-20 05:27:48.298 [info] <0.240.0> Running boot step cluster_name defined by app rabbit
2026-04-20 05:27:48.298 [info] <0.240.0> Running boot step direct_client defined by app rabbit
2026-04-20 05:27:48.298 [info] <0.240.0> Running boot step os_signal_handler defined by app rabbit
2026-04-20 05:27:48.298 [info] <0.589.0> Swapping OS signal event handler (erl_signal_server) for our own
2026-04-20 05:27:48.340 [info] <0.639.0> Management plugin: HTTP (non-TLS) listener started on port 15673
2026-04-20 05:27:48.340 [info] <0.745.0> Statistics database started.
2026-04-20 05:27:48.340 [info] <0.744.0> Starting worker pool 'management_worker_pool' with 3 processes in it
2026-04-20 05:27:48.341 [notice] <0.106.0> Changed loghwm of /var/log/contrail/config-database-rabbitmq/rabbitmq.log to 50
2026-04-20 05:27:48.460 [info] <0.8.0> Server startup complete; 3 plugins started.
 * rabbitmq_management
 * rabbitmq_management_agent
 * rabbitmq_web_dispatch
2026-04-20 05:27:51.702 [info] <0.756.0> accepting AMQP connection <0.756.0> (10.0.0.50:53568 -> 10.0.0.33:5673)
2026-04-20 05:27:51.709 [info] <0.756.0> connection <0.756.0> (10.0.0.50:53568 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:27:51.747 [info] <0.756.0> closing AMQP connection <0.756.0> (10.0.0.50:53568 -> 10.0.0.33:5673, vhost: '/', user: 'guest')
2026-04-20 05:27:51.748 [info] <0.827.0> accepting AMQP connection <0.827.0> (10.0.0.50:53584 -> 10.0.0.33:5673)
2026-04-20 05:27:51.750 [info] <0.827.0> connection <0.827.0> (10.0.0.50:53584 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:27:52.386 [info] <0.850.0> accepting AMQP connection <0.850.0> (10.0.0.50:53586 -> 10.0.0.33:5673)
2026-04-20 05:27:52.387 [info] <0.853.0> accepting AMQP connection <0.853.0> (10.0.0.50:53600 -> 10.0.0.33:5673)
2026-04-20 05:27:52.388 [info] <0.856.0> accepting AMQP connection <0.856.0> (10.0.0.33:59750 -> 10.0.0.33:5673)
2026-04-20 05:27:52.388 [info] <0.859.0> accepting AMQP connection <0.859.0> (10.0.0.50:53612 -> 10.0.0.33:5673)
2026-04-20 05:27:52.388 [info] <0.862.0> accepting AMQP connection <0.862.0> (10.0.0.38:53098 -> 10.0.0.33:5673)
2026-04-20 05:27:52.388 [info] <0.865.0> accepting AMQP connection <0.865.0> (10.0.0.38:53102 -> 10.0.0.33:5673)
2026-04-20 05:27:52.389 [info] <0.868.0> accepting AMQP connection <0.868.0> (10.0.0.38:53106 -> 10.0.0.33:5673)
2026-04-20 05:27:52.389 [info] <0.871.0> accepting AMQP connection <0.871.0> (10.0.0.33:59766 -> 10.0.0.33:5673)
2026-04-20 05:27:52.390 [info] <0.874.0> accepting AMQP connection <0.874.0> (10.0.0.33:59768 -> 10.0.0.33:5673)
2026-04-20 05:27:52.390 [info] <0.850.0> connection <0.850.0> (10.0.0.50:53586 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:27:52.391 [info] <0.871.0> connection <0.871.0> (10.0.0.33:59766 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:27:52.391 [info] <0.865.0> connection <0.865.0> (10.0.0.38:53102 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:27:52.391 [info] <0.862.0> connection <0.862.0> (10.0.0.38:53098 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:27:52.391 [info] <0.853.0> connection <0.853.0> (10.0.0.50:53600 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:27:52.391 [info] <0.868.0> connection <0.868.0> (10.0.0.38:53106 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:27:52.391 [info] <0.859.0> connection <0.859.0> (10.0.0.50:53612 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:27:52.392 [info] <0.874.0> connection <0.874.0> (10.0.0.33:59768 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:27:52.396 [info] <0.856.0> connection <0.856.0> (10.0.0.33:59750 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:27:52.601 [info] <0.486.0> node 'contrail@cn-jenkins-deploy-platform-ansible-os-5649-1' down: connection_closed
2026-04-20 05:27:52.714 [info] <0.1013.0> accepting AMQP connection <0.1013.0> (10.0.0.50:53628 -> 10.0.0.33:5673)
2026-04-20 05:27:52.716 [info] <0.1013.0> connection <0.1013.0> (10.0.0.50:53628 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:27:52.735 [info] <0.1033.0> accepting AMQP connection <0.1033.0> (10.0.0.50:53644 -> 10.0.0.33:5673)
2026-04-20 05:27:52.738 [info] <0.1033.0> connection <0.1033.0> (10.0.0.50:53644 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:27:52.742 [info] <0.1033.0> closing AMQP connection <0.1033.0> (10.0.0.50:53644 -> 10.0.0.33:5673, vhost: '/', user: 'guest')
2026-04-20 05:27:52.748 [info] <0.1046.0> accepting AMQP connection <0.1046.0> (10.0.0.50:53648 -> 10.0.0.33:5673)
2026-04-20 05:27:52.751 [info] <0.1046.0> connection <0.1046.0> (10.0.0.50:53648 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:27:52.870 [info] <0.1057.0> accepting AMQP connection <0.1057.0> (10.0.0.33:59774 -> 10.0.0.33:5673)
2026-04-20 05:27:52.872 [info] <0.1057.0> connection <0.1057.0> (10.0.0.33:59774 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:27:52.925 [info] <0.1077.0> accepting AMQP connection <0.1077.0> (10.0.0.33:59784 -> 10.0.0.33:5673)
2026-04-20 05:27:52.928 [info] <0.1077.0> connection <0.1077.0> (10.0.0.33:59784 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:27:52.937 [info] <0.1077.0> closing AMQP connection <0.1077.0> (10.0.0.33:59784 -> 10.0.0.33:5673, vhost: '/', user: 'guest')
2026-04-20 05:27:52.939 [info] <0.1090.0> accepting AMQP connection <0.1090.0> (10.0.0.33:59792 -> 10.0.0.33:5673)
2026-04-20 05:27:52.944 [info] <0.1090.0> connection <0.1090.0> (10.0.0.33:59792 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:27:52.989 [info] <0.1101.0> accepting AMQP connection <0.1101.0> (10.0.0.38:53110 -> 10.0.0.33:5673)
2026-04-20 05:27:52.992 [info] <0.1101.0> connection <0.1101.0> (10.0.0.38:53110 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:27:53.013 [info] <0.1121.0> accepting AMQP connection <0.1121.0> (10.0.0.38:53114 -> 10.0.0.33:5673)
2026-04-20 05:27:53.015 [info] <0.1121.0> connection <0.1121.0> (10.0.0.38:53114 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:27:53.022 [info] <0.1121.0> closing AMQP connection <0.1121.0> (10.0.0.38:53114 -> 10.0.0.33:5673, vhost: '/', user: 'guest')
2026-04-20 05:27:53.024 [info] <0.1134.0> accepting AMQP connection <0.1134.0> (10.0.0.38:53126 -> 10.0.0.33:5673)
2026-04-20 05:27:53.029 [info] <0.1134.0> connection <0.1134.0> (10.0.0.38:53126 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:27:53.374 [warning] <0.1090.0> closing AMQP connection <0.1090.0> (10.0.0.33:59792 -> 10.0.0.33:5673, vhost: '/', user: 'guest'):
client unexpectedly closed TCP connection
2026-04-20 05:27:53.376 [warning] <0.1057.0> closing AMQP connection <0.1057.0> (10.0.0.33:59774 -> 10.0.0.33:5673, vhost: '/', user: 'guest'):
client unexpectedly closed TCP connection
2026-04-20 05:27:53.446 [warning] <0.1013.0> closing AMQP connection <0.1013.0> (10.0.0.50:53628 -> 10.0.0.33:5673, vhost: '/', user: 'guest'):
client unexpectedly closed TCP connection
2026-04-20 05:27:53.446 [warning] <0.1046.0> closing AMQP connection <0.1046.0> (10.0.0.50:53648 -> 10.0.0.33:5673, vhost: '/', user: 'guest'):
client unexpectedly closed TCP connection
2026-04-20 05:27:55.306 [info] <0.1164.0> accepting AMQP connection <0.1164.0> (10.0.0.38:38594 -> 10.0.0.33:5673)
2026-04-20 05:27:55.306 [info] <0.1167.0> accepting AMQP connection <0.1167.0> (10.0.0.38:38586 -> 10.0.0.33:5673)
2026-04-20 05:27:55.319 [info] <0.1167.0> connection <0.1167.0> (10.0.0.38:38586 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:27:55.320 [info] <0.1164.0> connection <0.1164.0> (10.0.0.38:38594 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:27:55.385 [info] <0.1167.0> closing AMQP connection <0.1167.0> (10.0.0.38:38586 -> 10.0.0.33:5673, vhost: '/', user: 'guest')
2026-04-20 05:27:55.393 [info] <0.1196.0> accepting AMQP connection <0.1196.0> (10.0.0.38:38608 -> 10.0.0.33:5673)
2026-04-20 05:27:55.400 [info] <0.1196.0> connection <0.1196.0> (10.0.0.38:38608 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:27:55.745 [info] <0.1217.0> RabbitMQ is asked to stop...
2026-04-20 05:27:55.831 [info] <0.1217.0> Stopping RabbitMQ applications and their dependencies in the following order:
    rabbitmq_management
    amqp_client
    rabbitmq_web_dispatch
    cowboy
    cowlib
    rabbitmq_management_agent
    rabbit
    mnesia
    rabbit_common
    sysmon_handler
    os_mon
2026-04-20 05:27:55.831 [info] <0.1217.0> Stopping application 'rabbitmq_management'
2026-04-20 05:27:55.832 [warning] <0.595.0> RabbitMQ HTTP listener registry could not find context rabbitmq_management_tls
2026-04-20 05:27:55.834 [info] <0.1217.0> Stopping application 'amqp_client'
2026-04-20 05:27:55.834 [info] <0.43.0> Application rabbitmq_management exited with reason: stopped
2026-04-20 05:27:55.837 [info] <0.1217.0> Stopping application 'rabbitmq_web_dispatch'
2026-04-20 05:27:55.837 [info] <0.43.0> Application amqp_client exited with reason: stopped
2026-04-20 05:27:55.840 [info] <0.1217.0> Stopping application 'cowboy'
2026-04-20 05:27:55.840 [info] <0.43.0> Application rabbitmq_web_dispatch exited with reason: stopped
2026-04-20 05:27:55.842 [info] <0.43.0> Application cowboy exited with reason: stopped
2026-04-20 05:27:55.842 [info] <0.1217.0> Stopping application 'cowlib'
2026-04-20 05:27:55.842 [info] <0.1217.0> Stopping application 'rabbitmq_management_agent'
2026-04-20 05:27:55.842 [info] <0.43.0> Application cowlib exited with reason: stopped
2026-04-20 05:27:55.845 [info] <0.43.0> Application rabbitmq_management_agent exited with reason: stopped
2026-04-20 05:27:55.845 [info] <0.1217.0> Stopping application 'rabbit'
2026-04-20 05:27:55.846 [info] <0.240.0> Will unregister with peer discovery backend rabbit_peer_discovery_classic_config
2026-04-20 05:27:55.846 [info] <0.587.0> stopped TCP listener on [::]:5673
2026-04-20 05:27:55.847 [error] <0.827.0> Error on AMQP connection <0.827.0> (10.0.0.50:53584 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:27:55.847 [error] <0.1164.0> Error on AMQP connection <0.1164.0> (10.0.0.38:38594 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:27:55.847 [error] <0.859.0> Error on AMQP connection <0.859.0> (10.0.0.50:53612 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:27:55.847 [error] <0.874.0> Error on AMQP connection <0.874.0> (10.0.0.33:59768 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:27:55.847 [error] <0.1196.0> Error on AMQP connection <0.1196.0> (10.0.0.38:38608 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:27:55.847 [error] <0.853.0> Error on AMQP connection <0.853.0> (10.0.0.50:53600 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:27:55.847 [error] <0.850.0> Error on AMQP connection <0.850.0> (10.0.0.50:53586 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:27:55.847 [error] <0.1101.0> Error on AMQP connection <0.1101.0> (10.0.0.38:53110 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:27:55.848 [error] <0.856.0> Error on AMQP connection <0.856.0> (10.0.0.33:59750 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:27:55.848 [error] <0.865.0> Error on AMQP connection <0.865.0> (10.0.0.38:53102 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:27:55.848 [error] <0.1134.0> Error on AMQP connection <0.1134.0> (10.0.0.38:53126 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:27:55.848 [error] <0.868.0> Error on AMQP connection <0.868.0> (10.0.0.38:53106 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:27:55.848 [error] <0.862.0> Error on AMQP connection <0.862.0> (10.0.0.38:53098 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:27:55.848 [error] <0.871.0> Error on AMQP connection <0.871.0> (10.0.0.33:59766 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:27:55.849 [error] <0.889.0> Supervisor {<0.889.0>,rabbit_channel_sup_sup} had child channel_sup started with rabbit_channel_sup:start_link() at undefined exit with reason shutdown in context shutdown_error
2026-04-20 05:27:55.850 [error] <0.929.0> Supervisor {<0.929.0>,rabbit_channel_sup_sup} had child channel_sup started with rabbit_channel_sup:start_link() at undefined exit with reason shutdown in context shutdown_error
2026-04-20 05:27:55.858 [info] <0.511.0> Closing all connections in vhost '/' on node 'contrail@cn-jenkins-deploy-platform-ansible-os-5649-3' because the vhost is stopping
2026-04-20 05:27:55.859 [warning] <0.805.0> Mirrored queue 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-5649-2..tftp_queue' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available
2026-04-20 05:27:55.859 [warning] <0.949.0> Mirrored queue 'contrail-control.cn-jenkins-deploy-platform-ansible-os-5649-3' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available
2026-04-20 05:27:55.859 [warning] <0.1068.0> Mirrored queue 'vnc_config.cn-jenkins-deploy-platform-ansible-os-5649-3.-8082-0' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available
2026-04-20 05:27:55.859 [warning] <0.966.0> Mirrored queue 'contrail-control.cn-jenkins-deploy-platform-ansible-os-5649-2' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available
2026-04-20 05:27:55.860 [warning] <0.961.0> Mirrored queue 'contrail-collector.cn-jenkins-deploy-platform-ansible-os-5649-3.' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available
2026-04-20 05:27:55.860 [warning] <0.786.0> Mirrored queue 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-5649-2.' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available
2026-04-20 05:27:55.860 [warning] <0.1024.0> Mirrored queue 'vnc_config.cn-jenkins-deploy-platform-ansible-os-5649-2.-8082-0' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available
2026-04-20 05:27:55.860 [warning] <0.971.0> Mirrored queue 'contrail-dns.cn-jenkins-deploy-platform-ansible-os-5649-2' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available
2026-04-20 05:27:56.610 [warning]  lager_file_backend dropped 23 messages in the last second that exceeded the limit of 50 messages/sec
2026-04-20 05:28:03.269 [info] <0.1296.0> Resetting Rabbit forcefully
2026-04-20 05:28:04.603 [info] <0.1311.0> RabbitMQ is asked to stop...
2026-04-20 05:28:04.732 [info] <0.1311.0> Stopping RabbitMQ applications and their dependencies in the following order:
    rabbitmq_management
    amqp_client
    rabbitmq_management_agent
    rabbitmq_web_dispatch
    rabbit
    mnesia
    rabbit_common
    sysmon_handler
    os_mon
2026-04-20 05:28:04.732 [info] <0.1311.0> Stopping application 'rabbitmq_management'
2026-04-20 05:28:04.732 [info] <0.1311.0> Stopping application 'amqp_client'
2026-04-20 05:28:04.732 [info] <0.1311.0> Stopping application 'rabbitmq_management_agent'
2026-04-20 05:28:04.732 [info] <0.1311.0> Stopping application 'rabbitmq_web_dispatch'
2026-04-20 05:28:04.732 [info] <0.1311.0> Stopping application 'rabbit'
2026-04-20 05:28:04.732 [info] <0.1311.0> Stopping application 'mnesia'
2026-04-20 05:28:04.732 [info] <0.1311.0> Stopping application 'rabbit_common'
2026-04-20 05:28:04.732 [info] <0.1311.0> Stopping application 'sysmon_handler'
2026-04-20 05:28:04.732 [info] <0.1311.0> Stopping application 'os_mon'
2026-04-20 05:28:04.732 [info] <0.1311.0> Successfully stopped RabbitMQ and its dependencies
2026-04-20 05:28:04.732 [info] <0.1311.0> Halting Erlang VM with the following applications:
    ranch
    ssl
    credentials_obfuscation
    public_key
    asn1
    observer_cli
    jsx
    tools
    crypto
    stdout_formatter
    recon
    inets
    xmerl
    lager
    goldrush
    compiler
    syntax_tools
    sasl
    stdlib
    kernel
2026-04-20 05:28:15.265 [info] <0.8.0> Log file opened with Lager
2026-04-20 05:28:19.053 [info] <0.8.0> Feature flags: list of feature flags found:
2026-04-20 05:28:19.053 [info] <0.8.0> Feature flags: feature flag states written to disk: yes
2026-04-20 05:28:19.117 [info] <0.240.0> 
 Starting RabbitMQ 3.7.28 on Erlang 21.3.8.21
 Copyright (c) 2007-2020 Pivotal Software, Inc.
 Licensed under the MPL.  See https://www.rabbitmq.com/
2026-04-20 05:28:19.118 [info] <0.240.0> 
 node           : contrail@cn-jenkins-deploy-platform-ansible-os-5649-3
 home dir       : /var/lib/rabbitmq
 config file(s) : /etc/rabbitmq/rabbitmq.config
 cookie hash    : esZsDxSN6VGbi9JkMSxNZA==
 log(s)         : /var/log/contrail/config-database-rabbitmq/rabbitmq.log
                : /var/log/rabbitmq/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3_upgrade.log
 database dir   : /var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3
2026-04-20 05:28:19.146 [info] <0.240.0> Running boot step pre_boot defined by app rabbit
2026-04-20 05:28:19.152 [info] <0.240.0> Running boot step rabbit_core_metrics defined by app rabbit
2026-04-20 05:28:19.153 [info] <0.240.0> Running boot step rabbit_alarm defined by app rabbit
2026-04-20 05:28:19.161 [info] <0.246.0> Memory high watermark set to 12637 MiB (13251081011 bytes) of 15796 MiB (16563851264 bytes) total
2026-04-20 05:28:19.167 [info] <0.248.0> Enabling free disk space monitoring
2026-04-20 05:28:19.167 [info] <0.248.0> Disk free limit set to 50MB
2026-04-20 05:28:19.179 [info] <0.240.0> Running boot step code_server_cache defined by app rabbit
2026-04-20 05:28:19.179 [info] <0.240.0> Running boot step file_handle_cache defined by app rabbit
2026-04-20 05:28:19.179 [info] <0.251.0> Limiting to approx 65436 file handles (58890 sockets)
2026-04-20 05:28:19.179 [info] <0.252.0> FHC read buffering:  OFF
2026-04-20 05:28:19.179 [info] <0.252.0> FHC write buffering: ON
2026-04-20 05:28:19.182 [info] <0.240.0> Running boot step worker_pool defined by app rabbit
2026-04-20 05:28:19.182 [info] <0.241.0> Will use 4 processes for default worker pool
2026-04-20 05:28:19.182 [info] <0.241.0> Starting worker pool 'worker_pool' with 4 processes in it
2026-04-20 05:28:19.182 [info] <0.240.0> Running boot step database defined by app rabbit
2026-04-20 05:28:19.184 [info] <0.240.0> Node database directory at /var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3 is empty. Assuming we need to join an existing cluster or initialise from scratch...
2026-04-20 05:28:19.184 [info] <0.240.0> Configured peer discovery backend: rabbit_peer_discovery_classic_config
2026-04-20 05:28:19.184 [info] <0.240.0> Will try to lock with peer discovery backend rabbit_peer_discovery_classic_config
2026-04-20 05:28:19.184 [info] <0.240.0> Peer discovery backend does not support locking, falling back to randomized delay
2026-04-20 05:28:19.184 [info] <0.240.0> Peer discovery backend rabbit_peer_discovery_classic_config supports registration.
2026-04-20 05:28:19.184 [info] <0.240.0> Will wait for 28571 milliseconds before proceeding with registration...
2026-04-20 05:28:47.755 [info] <0.240.0> All discovered existing cluster peers: contrail@cn-jenkins-deploy-platform-ansible-os-5649-1, contrail@cn-jenkins-deploy-platform-ansible-os-5649-2, contrail@cn-jenkins-deploy-platform-ansible-os-5649-3
2026-04-20 05:28:47.755 [info] <0.240.0> Peer nodes we can cluster with: contrail@cn-jenkins-deploy-platform-ansible-os-5649-1, contrail@cn-jenkins-deploy-platform-ansible-os-5649-2
2026-04-20 05:28:47.757 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {badrpc,nodedown}
2026-04-20 05:28:47.760 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:28:47.760 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 9 retries left...
2026-04-20 05:28:48.262 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {badrpc,nodedown}
2026-04-20 05:28:48.262 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:28:48.263 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 8 retries left...
2026-04-20 05:28:48.765 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {badrpc,nodedown}
2026-04-20 05:28:48.766 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:28:48.766 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 7 retries left...
2026-04-20 05:28:49.267 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {badrpc,nodedown}
2026-04-20 05:28:49.268 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:28:49.269 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 6 retries left...
2026-04-20 05:28:49.771 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {badrpc,nodedown}
2026-04-20 05:28:49.772 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:28:49.772 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 5 retries left...
2026-04-20 05:28:50.278 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {badrpc,nodedown}
2026-04-20 05:28:50.279 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:28:50.279 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 4 retries left...
2026-04-20 05:28:50.847 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,mnesia_not_running}
2026-04-20 05:28:50.849 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:28:50.849 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 3 retries left...
2026-04-20 05:28:51.360 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,mnesia_not_running}
2026-04-20 05:28:51.361 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:28:51.361 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 2 retries left...
2026-04-20 05:28:51.866 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,mnesia_not_running}
2026-04-20 05:28:51.867 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:28:51.867 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 1 retries left...
2026-04-20 05:28:52.374 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,mnesia_not_running}
2026-04-20 05:28:52.375 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:28:52.375 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 0 retries left...
2026-04-20 05:28:52.875 [warning] <0.240.0> Could not successfully contact any node of: contrail@cn-jenkins-deploy-platform-ansible-os-5649-1,contrail@cn-jenkins-deploy-platform-ansible-os-5649-2 (as in Erlang distribution). Starting as a blank standalone node...
2026-04-20 05:28:52.877 [info] <0.43.0> Application mnesia exited with reason: stopped
2026-04-20 05:28:53.014 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2026-04-20 05:28:53.014 [info] <0.240.0> Successfully synced tables from a peer
2026-04-20 05:28:53.041 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2026-04-20 05:28:53.041 [info] <0.240.0> Successfully synced tables from a peer
2026-04-20 05:28:53.071 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2026-04-20 05:28:53.071 [info] <0.240.0> Successfully synced tables from a peer
2026-04-20 05:28:53.072 [info] <0.240.0> Will register with peer discovery backend rabbit_peer_discovery_classic_config
2026-04-20 05:28:53.072 [info] <0.240.0> Running boot step database_sync defined by app rabbit
2026-04-20 05:28:53.072 [info] <0.240.0> Running boot step feature_flags defined by app rabbit
2026-04-20 05:28:53.073 [info] <0.240.0> Running boot step codec_correctness_check defined by app rabbit
2026-04-20 05:28:53.073 [info] <0.240.0> Running boot step external_infrastructure defined by app rabbit
2026-04-20 05:28:53.073 [info] <0.240.0> Running boot step rabbit_registry defined by app rabbit
2026-04-20 05:28:53.074 [info] <0.240.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit
2026-04-20 05:28:53.074 [info] <0.240.0> Running boot step rabbit_queue_location_random defined by app rabbit
2026-04-20 05:28:53.074 [info] <0.240.0> Running boot step rabbit_event defined by app rabbit
2026-04-20 05:28:53.075 [info] <0.240.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit
2026-04-20 05:28:53.075 [info] <0.240.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit
2026-04-20 05:28:53.075 [info] <0.240.0> Running boot step rabbit_exchange_type_direct defined by app rabbit
2026-04-20 05:28:53.076 [info] <0.240.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit
2026-04-20 05:28:53.076 [info] <0.240.0> Running boot step rabbit_exchange_type_headers defined by app rabbit
2026-04-20 05:28:53.076 [info] <0.240.0> Running boot step rabbit_exchange_type_topic defined by app rabbit
2026-04-20 05:28:53.076 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit
2026-04-20 05:28:53.076 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit
2026-04-20 05:28:53.076 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit
2026-04-20 05:28:53.076 [info] <0.240.0> Running boot step rabbit_priority_queue defined by app rabbit
2026-04-20 05:28:53.076 [info] <0.240.0> Priority queues enabled, real BQ is rabbit_variable_queue
2026-04-20 05:28:53.076 [info] <0.240.0> Running boot step rabbit_queue_location_client_local defined by app rabbit
2026-04-20 05:28:53.076 [info] <0.240.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit
2026-04-20 05:28:53.076 [info] <0.240.0> Running boot step kernel_ready defined by app rabbit
2026-04-20 05:28:53.076 [info] <0.240.0> Running boot step rabbit_sysmon_minder defined by app rabbit
2026-04-20 05:28:53.076 [info] <0.240.0> Running boot step rabbit_epmd_monitor defined by app rabbit
2026-04-20 05:28:53.078 [info] <0.469.0> epmd monitor knows us, inter-node communication (distribution) port: 25673
2026-04-20 05:28:53.078 [info] <0.240.0> Running boot step guid_generator defined by app rabbit
2026-04-20 05:28:53.088 [info] <0.240.0> Running boot step rabbit_node_monitor defined by app rabbit
2026-04-20 05:28:53.088 [info] <0.473.0> Starting rabbit_node_monitor
2026-04-20 05:28:53.088 [info] <0.240.0> Running boot step delegate_sup defined by app rabbit
2026-04-20 05:28:53.089 [info] <0.240.0> Running boot step rabbit_memory_monitor defined by app rabbit
2026-04-20 05:28:53.089 [info] <0.240.0> Running boot step core_initialized defined by app rabbit
2026-04-20 05:28:53.089 [info] <0.240.0> Running boot step upgrade_queues defined by app rabbit
2026-04-20 05:28:53.112 [info] <0.240.0> message_store upgrades: 1 to apply
2026-04-20 05:28:53.112 [info] <0.240.0> message_store upgrades: Applying rabbit_variable_queue:move_messages_to_vhost_store
2026-04-20 05:28:53.113 [info] <0.240.0> message_store upgrades: No durable queues found. Skipping message store migration
2026-04-20 05:28:53.113 [info] <0.240.0> message_store upgrades: Removing the old message store data
2026-04-20 05:28:53.114 [info] <0.240.0> message_store upgrades: All upgrades applied successfully
2026-04-20 05:28:53.139 [info] <0.240.0> Running boot step rabbit_connection_tracking defined by app rabbit
2026-04-20 05:28:53.139 [info] <0.240.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit
2026-04-20 05:28:53.139 [info] <0.240.0> Running boot step rabbit_exchange_parameters defined by app rabbit
2026-04-20 05:28:53.140 [info] <0.240.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit
2026-04-20 05:28:53.140 [info] <0.240.0> Running boot step rabbit_policies defined by app rabbit
2026-04-20 05:28:53.140 [info] <0.240.0> Running boot step rabbit_policy defined by app rabbit
2026-04-20 05:28:53.141 [info] <0.240.0> Running boot step rabbit_queue_location_validator defined by app rabbit
2026-04-20 05:28:53.141 [info] <0.240.0> Running boot step rabbit_vhost_limit defined by app rabbit
2026-04-20 05:28:53.141 [info] <0.240.0> Running boot step rabbit_mgmt_reset_handler defined by app rabbitmq_management
2026-04-20 05:28:53.141 [info] <0.240.0> Running boot step rabbit_mgmt_db_handler defined by app rabbitmq_management_agent
2026-04-20 05:28:53.141 [warning] <0.240.0> force_fine_statistics set to true; ignored.
Replaced by {rates_mode, none} in the rabbitmq_management application.
2026-04-20 05:28:53.141 [info] <0.240.0> Management plugin: using rates mode 'basic'
2026-04-20 05:28:53.141 [info] <0.240.0> Running boot step recovery defined by app rabbit
2026-04-20 05:28:53.142 [info] <0.240.0> Running boot step load_definitions defined by app rabbitmq_management
2026-04-20 05:28:53.142 [info] <0.240.0> Applying definitions from /etc/rabbitmq/definitions.json
2026-04-20 05:28:53.142 [info] <0.240.0> Asked to import definitions. Acting user: rmq-internal
2026-04-20 05:28:53.142 [info] <0.240.0> Importing users...
2026-04-20 05:28:53.142 [info] <0.240.0> Creating user 'guest'
2026-04-20 05:28:53.149 [info] <0.240.0> Setting user tags for user 'guest' to [administrator]
2026-04-20 05:28:53.152 [info] <0.240.0> Importing vhosts...
2026-04-20 05:28:53.152 [info] <0.240.0> Adding vhost '/'
2026-04-20 05:28:53.176 [info] <0.521.0> Making sure data directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists
2026-04-20 05:28:53.184 [info] <0.521.0> Starting message stores for vhost '/'
2026-04-20 05:28:53.184 [info] <0.525.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index
2026-04-20 05:28:53.185 [info] <0.521.0> Started message store of type transient for vhost '/'
2026-04-20 05:28:53.185 [info] <0.528.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
2026-04-20 05:28:53.185 [warning] <0.528.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": rebuilding indices from scratch
2026-04-20 05:28:53.186 [info] <0.521.0> Started message store of type persistent for vhost '/'
2026-04-20 05:28:53.187 [info] <0.240.0> Importing user permissions...
2026-04-20 05:28:53.187 [info] <0.240.0> Setting permissions for 'guest' in '/' to '.*', '.*', '.*'
2026-04-20 05:28:53.190 [info] <0.240.0> Importing topic permissions...
2026-04-20 05:28:53.190 [info] <0.240.0> Importing parameters...
2026-04-20 05:28:53.190 [info] <0.240.0> Importing global parameters...
2026-04-20 05:28:53.190 [info] <0.240.0> Importing policies...
2026-04-20 05:28:53.197 [info] <0.240.0> Importing queues...
2026-04-20 05:28:53.197 [info] <0.240.0> Importing exchanges...
2026-04-20 05:28:53.197 [info] <0.240.0> Importing bindings...
2026-04-20 05:28:53.197 [info] <0.240.0> Running boot step empty_db_check defined by app rabbit
2026-04-20 05:28:53.197 [info] <0.240.0> Running boot step rabbit_looking_glass defined by app rabbit
2026-04-20 05:28:53.197 [info] <0.240.0> Running boot step rabbit_core_metrics_gc defined by app rabbit
2026-04-20 05:28:53.198 [info] <0.240.0> Running boot step background_gc defined by app rabbit
2026-04-20 05:28:53.198 [info] <0.240.0> Running boot step connection_tracking defined by app rabbit
2026-04-20 05:28:53.202 [info] <0.240.0> Setting up a table for connection tracking on this node: 'tracked_connection_on_node_contrail@cn-jenkins-deploy-platform-ansible-os-5649-3'
2026-04-20 05:28:53.206 [info] <0.240.0> Setting up a table for per-vhost connection counting on this node: 'tracked_connection_per_vhost_on_node_contrail@cn-jenkins-deploy-platform-ansible-os-5649-3'
2026-04-20 05:28:53.206 [info] <0.240.0> Running boot step routing_ready defined by app rabbit
2026-04-20 05:28:53.206 [info] <0.240.0> Running boot step pre_flight defined by app rabbit
2026-04-20 05:28:53.206 [info] <0.240.0> Running boot step notify_cluster defined by app rabbit
2026-04-20 05:28:53.206 [info] <0.240.0> Running boot step networking defined by app rabbit
2026-04-20 05:28:53.208 [info] <0.574.0> started TCP listener on [::]:5673
2026-04-20 05:28:53.208 [info] <0.240.0> Running boot step cluster_name defined by app rabbit
2026-04-20 05:28:53.208 [info] <0.240.0> Running boot step direct_client defined by app rabbit
2026-04-20 05:28:53.208 [info] <0.240.0> Running boot step os_signal_handler defined by app rabbit
2026-04-20 05:28:53.208 [info] <0.576.0> Swapping OS signal event handler (erl_signal_server) for our own
2026-04-20 05:28:53.251 [info] <0.626.0> Management plugin: HTTP (non-TLS) listener started on port 15673
2026-04-20 05:28:53.251 [info] <0.732.0> Statistics database started.
2026-04-20 05:28:53.251 [info] <0.731.0> Starting worker pool 'management_worker_pool' with 3 processes in it
2026-04-20 05:28:53.251 [notice] <0.106.0> Changed loghwm of /var/log/contrail/config-database-rabbitmq/rabbitmq.log to 50
2026-04-20 05:28:53.333 [info] <0.8.0> Server startup complete; 3 plugins started.
 * rabbitmq_management
 * rabbitmq_management_agent
 * rabbitmq_web_dispatch
2026-04-20 05:28:53.887 [info] <0.473.0> rabbit on node 'contrail@cn-jenkins-deploy-platform-ansible-os-5649-2' up
2026-04-20 05:28:54.183 [info] <0.473.0> rabbit on node 'contrail@cn-jenkins-deploy-platform-ansible-os-5649-2' down
2026-04-20 05:28:54.198 [info] <0.473.0> Keeping contrail@cn-jenkins-deploy-platform-ansible-os-5649-2 listeners: the node is already back
2026-04-20 05:28:55.041 [info] <0.473.0> node 'contrail@cn-jenkins-deploy-platform-ansible-os-5649-1' down: connection_closed
2026-04-20 05:28:55.864 [info] <0.921.0> accepting AMQP connection <0.921.0> (10.0.0.50:52090 -> 10.0.0.33:5673)
2026-04-20 05:28:55.864 [info] <0.924.0> accepting AMQP connection <0.924.0> (10.0.0.33:44104 -> 10.0.0.33:5673)
2026-04-20 05:28:55.865 [info] <0.928.0> accepting AMQP connection <0.928.0> (10.0.0.38:40782 -> 10.0.0.33:5673)
2026-04-20 05:28:55.865 [info] <0.931.0> accepting AMQP connection <0.931.0> (10.0.0.50:52102 -> 10.0.0.33:5673)
2026-04-20 05:28:55.866 [info] <0.934.0> accepting AMQP connection <0.934.0> (10.0.0.33:44116 -> 10.0.0.33:5673)
2026-04-20 05:28:55.867 [info] <0.937.0> accepting AMQP connection <0.937.0> (10.0.0.50:52112 -> 10.0.0.33:5673)
2026-04-20 05:28:55.868 [info] <0.934.0> connection <0.934.0> (10.0.0.33:44116 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:28:55.868 [info] <0.928.0> connection <0.928.0> (10.0.0.38:40782 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:28:55.869 [info] <0.924.0> connection <0.924.0> (10.0.0.33:44104 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:28:55.869 [info] <0.931.0> connection <0.931.0> (10.0.0.50:52102 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:28:55.869 [info] <0.948.0> accepting AMQP connection <0.948.0> (10.0.0.33:44126 -> 10.0.0.33:5673)
2026-04-20 05:28:55.871 [info] <0.937.0> connection <0.937.0> (10.0.0.50:52112 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:28:55.872 [info] <0.948.0> connection <0.948.0> (10.0.0.33:44126 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:28:55.872 [info] <0.921.0> connection <0.921.0> (10.0.0.50:52090 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:28:55.876 [info] <0.1005.0> accepting AMQP connection <0.1005.0> (10.0.0.38:40798 -> 10.0.0.33:5673)
2026-04-20 05:28:55.878 [info] <0.1029.0> accepting AMQP connection <0.1029.0> (10.0.0.38:40808 -> 10.0.0.33:5673)
2026-04-20 05:28:55.882 [info] <0.1005.0> connection <0.1005.0> (10.0.0.38:40798 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:28:55.884 [info] <0.1029.0> connection <0.1029.0> (10.0.0.38:40808 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:28:58.115 [info] <0.1085.0> accepting AMQP connection <0.1085.0> (10.0.0.50:52122 -> 10.0.0.33:5673)
2026-04-20 05:28:58.117 [info] <0.1085.0> connection <0.1085.0> (10.0.0.50:52122 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:28:58.139 [info] <0.1105.0> accepting AMQP connection <0.1105.0> (10.0.0.50:52128 -> 10.0.0.33:5673)
2026-04-20 05:28:58.142 [info] <0.1105.0> connection <0.1105.0> (10.0.0.50:52128 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:28:58.146 [info] <0.1105.0> closing AMQP connection <0.1105.0> (10.0.0.50:52128 -> 10.0.0.33:5673, vhost: '/', user: 'guest')
2026-04-20 05:28:58.147 [info] <0.1118.0> accepting AMQP connection <0.1118.0> (10.0.0.50:52132 -> 10.0.0.33:5673)
2026-04-20 05:28:58.150 [info] <0.1118.0> connection <0.1118.0> (10.0.0.50:52132 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:28:58.407 [info] <0.1129.0> accepting AMQP connection <0.1129.0> (10.0.0.33:44128 -> 10.0.0.33:5673)
2026-04-20 05:28:58.411 [info] <0.1129.0> connection <0.1129.0> (10.0.0.33:44128 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:28:58.446 [info] <0.1149.0> accepting AMQP connection <0.1149.0> (10.0.0.33:44138 -> 10.0.0.33:5673)
2026-04-20 05:28:58.450 [info] <0.1149.0> connection <0.1149.0> (10.0.0.33:44138 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:28:58.467 [info] <0.1149.0> closing AMQP connection <0.1149.0> (10.0.0.33:44138 -> 10.0.0.33:5673, vhost: '/', user: 'guest')
2026-04-20 05:28:58.469 [info] <0.1162.0> accepting AMQP connection <0.1162.0> (10.0.0.33:44152 -> 10.0.0.33:5673)
2026-04-20 05:28:58.477 [info] <0.1162.0> connection <0.1162.0> (10.0.0.33:44152 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:28:58.976 [info] <0.1173.0> accepting AMQP connection <0.1173.0> (10.0.0.50:33190 -> 10.0.0.33:5673)
2026-04-20 05:28:58.983 [info] <0.1176.0> accepting AMQP connection <0.1176.0> (10.0.0.50:33202 -> 10.0.0.33:5673)
2026-04-20 05:28:58.985 [info] <0.1173.0> connection <0.1173.0> (10.0.0.50:33190 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:28:58.988 [info] <0.1176.0> connection <0.1176.0> (10.0.0.50:33202 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:28:59.034 [info] <0.1173.0> closing AMQP connection <0.1173.0> (10.0.0.50:33190 -> 10.0.0.33:5673, vhost: '/', user: 'guest')
2026-04-20 05:28:59.043 [info] <0.1205.0> accepting AMQP connection <0.1205.0> (10.0.0.50:33218 -> 10.0.0.33:5673)
2026-04-20 05:28:59.054 [info] <0.1205.0> connection <0.1205.0> (10.0.0.50:33218 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:28:59.295 [info] <0.1228.0> accepting AMQP connection <0.1228.0> (10.0.0.33:60342 -> 10.0.0.33:5673)
2026-04-20 05:28:59.296 [info] <0.1225.0> accepting AMQP connection <0.1225.0> (10.0.0.33:60330 -> 10.0.0.33:5673)
2026-04-20 05:28:59.300 [info] <0.1228.0> connection <0.1228.0> (10.0.0.33:60342 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:28:59.301 [info] <0.1225.0> connection <0.1225.0> (10.0.0.33:60330 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:28:59.320 [info] <0.1225.0> closing AMQP connection <0.1225.0> (10.0.0.33:60330 -> 10.0.0.33:5673, vhost: '/', user: 'guest')
2026-04-20 05:28:59.321 [info] <0.1256.0> accepting AMQP connection <0.1256.0> (10.0.0.33:60358 -> 10.0.0.33:5673)
2026-04-20 05:28:59.323 [info] <0.1256.0> connection <0.1256.0> (10.0.0.33:60358 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:28:59.576 [info] <0.1277.0> RabbitMQ is asked to stop...
2026-04-20 05:28:59.661 [info] <0.1277.0> Stopping RabbitMQ applications and their dependencies in the following order:
    rabbitmq_management
    amqp_client
    rabbitmq_web_dispatch
    cowboy
    cowlib
    rabbitmq_management_agent
    rabbit
    mnesia
    rabbit_common
    sysmon_handler
    os_mon
2026-04-20 05:28:59.662 [info] <0.1277.0> Stopping application 'rabbitmq_management'
2026-04-20 05:28:59.663 [warning] <0.582.0> RabbitMQ HTTP listener registry could not find context rabbitmq_management_tls
2026-04-20 05:28:59.665 [info] <0.43.0> Application rabbitmq_management exited with reason: stopped
2026-04-20 05:28:59.665 [info] <0.1277.0> Stopping application 'amqp_client'
2026-04-20 05:28:59.667 [info] <0.43.0> Application amqp_client exited with reason: stopped
2026-04-20 05:28:59.667 [info] <0.1277.0> Stopping application 'rabbitmq_web_dispatch'
2026-04-20 05:28:59.668 [info] <0.1277.0> Stopping application 'cowboy'
2026-04-20 05:28:59.668 [info] <0.43.0> Application rabbitmq_web_dispatch exited with reason: stopped
2026-04-20 05:28:59.670 [info] <0.43.0> Application cowboy exited with reason: stopped
2026-04-20 05:28:59.670 [info] <0.1277.0> Stopping application 'cowlib'
2026-04-20 05:28:59.670 [info] <0.43.0> Application cowlib exited with reason: stopped
2026-04-20 05:28:59.670 [info] <0.1277.0> Stopping application 'rabbitmq_management_agent'
2026-04-20 05:28:59.672 [info] <0.43.0> Application rabbitmq_management_agent exited with reason: stopped
2026-04-20 05:28:59.672 [info] <0.1277.0> Stopping application 'rabbit'
2026-04-20 05:28:59.672 [info] <0.240.0> Will unregister with peer discovery backend rabbit_peer_discovery_classic_config
2026-04-20 05:28:59.673 [info] <0.574.0> stopped TCP listener on [::]:5673
2026-04-20 05:28:59.673 [error] <0.1228.0> Error on AMQP connection <0.1228.0> (10.0.0.33:60342 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:28:59.673 [error] <0.1118.0> Error on AMQP connection <0.1118.0> (10.0.0.50:52132 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:28:59.673 [error] <0.1005.0> Error on AMQP connection <0.1005.0> (10.0.0.38:40798 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:28:59.673 [error] <0.1085.0> Error on AMQP connection <0.1085.0> (10.0.0.50:52122 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:28:59.673 [error] <0.924.0> Error on AMQP connection <0.924.0> (10.0.0.33:44104 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:28:59.673 [error] <0.937.0> Error on AMQP connection <0.937.0> (10.0.0.50:52112 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:28:59.673 [error] <0.1029.0> Error on AMQP connection <0.1029.0> (10.0.0.38:40808 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:28:59.673 [error] <0.1176.0> Error on AMQP connection <0.1176.0> (10.0.0.50:33202 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:28:59.673 [error] <0.931.0> Error on AMQP connection <0.931.0> (10.0.0.50:52102 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:28:59.673 [error] <0.921.0> Error on AMQP connection <0.921.0> (10.0.0.50:52090 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:28:59.673 [error] <0.1205.0> Error on AMQP connection <0.1205.0> (10.0.0.50:33218 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:28:59.673 [error] <0.1129.0> Error on AMQP connection <0.1129.0> (10.0.0.33:44128 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:28:59.674 [error] <0.934.0> Error on AMQP connection <0.934.0> (10.0.0.33:44116 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:28:59.674 [error] <0.948.0> Error on AMQP connection <0.948.0> (10.0.0.33:44126 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:28:59.673 [error] <0.1162.0> Error on AMQP connection <0.1162.0> (10.0.0.33:44152 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:28:59.674 [error] <0.1256.0> Error on AMQP connection <0.1256.0> (10.0.0.33:60358 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:28:59.673 [error] <0.928.0> Error on AMQP connection <0.928.0> (10.0.0.38:40782 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:28:59.688 [info] <0.498.0> Closing all connections in vhost '/' on node 'contrail@cn-jenkins-deploy-platform-ansible-os-5649-3' because the vhost is stopping
2026-04-20 05:28:59.688 [warning] <0.986.0> Mirrored queue 'contrail-control.cn-jenkins-deploy-platform-ansible-os-5649-3' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available
2026-04-20 05:28:59.689 [warning] <0.993.0> Mirrored queue 'contrail-collector.cn-jenkins-deploy-platform-ansible-os-5649-1.' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available
2026-04-20 05:28:59.689 [warning] <0.997.0> Mirrored queue 'contrail-collector.cn-jenkins-deploy-platform-ansible-os-5649-2.' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available
2026-04-20 05:28:59.689 [warning] <0.1002.0> Mirrored queue 'contrail-dns.cn-jenkins-deploy-platform-ansible-os-5649-3' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available
2026-04-20 05:29:00.706 [warning]  lager_file_backend dropped 21 messages in the last second that exceeded the limit of 50 messages/sec
2026-04-20 05:29:11.138 [info] <0.1348.0> Resetting Rabbit forcefully
2026-04-20 05:29:12.560 [info] <0.1363.0> RabbitMQ is asked to stop...
2026-04-20 05:29:12.645 [info] <0.1363.0> Stopping RabbitMQ applications and their dependencies in the following order:
    rabbitmq_management
    amqp_client
    rabbitmq_management_agent
    rabbitmq_web_dispatch
    rabbit
    mnesia
    rabbit_common
    sysmon_handler
    os_mon
2026-04-20 05:29:12.645 [info] <0.1363.0> Stopping application 'rabbitmq_management'
2026-04-20 05:29:12.645 [info] <0.1363.0> Stopping application 'amqp_client'
2026-04-20 05:29:12.645 [info] <0.1363.0> Stopping application 'rabbitmq_management_agent'
2026-04-20 05:29:12.645 [info] <0.1363.0> Stopping application 'rabbitmq_web_dispatch'
2026-04-20 05:29:12.645 [info] <0.1363.0> Stopping application 'rabbit'
2026-04-20 05:29:12.645 [info] <0.1363.0> Stopping application 'mnesia'
2026-04-20 05:29:12.645 [info] <0.1363.0> Stopping application 'rabbit_common'
2026-04-20 05:29:12.645 [info] <0.1363.0> Stopping application 'sysmon_handler'
2026-04-20 05:29:12.645 [info] <0.1363.0> Stopping application 'os_mon'
2026-04-20 05:29:12.645 [info] <0.1363.0> Successfully stopped RabbitMQ and its dependencies
2026-04-20 05:29:12.645 [info] <0.1363.0> Halting Erlang VM with the following applications:
    ranch
    ssl
    public_key
    asn1
    observer_cli
    tools
    credentials_obfuscation
    crypto
    stdout_formatter
    jsx
    recon
    inets
    xmerl
    lager
    goldrush
    compiler
    syntax_tools
    sasl
    stdlib
    kernel
2026-04-20 05:29:22.959 [info] <0.8.0> Log file opened with Lager
2026-04-20 05:29:25.388 [info] <0.8.0> Feature flags: list of feature flags found:
2026-04-20 05:29:25.388 [info] <0.8.0> Feature flags: feature flag states written to disk: yes
2026-04-20 05:29:25.409 [info] <0.240.0> 
 Starting RabbitMQ 3.7.28 on Erlang 21.3.8.21
 Copyright (c) 2007-2020 Pivotal Software, Inc.
 Licensed under the MPL.  See https://www.rabbitmq.com/
2026-04-20 05:29:25.409 [info] <0.240.0> 
 node           : contrail@cn-jenkins-deploy-platform-ansible-os-5649-3
 home dir       : /var/lib/rabbitmq
 config file(s) : /etc/rabbitmq/rabbitmq.config
 cookie hash    : esZsDxSN6VGbi9JkMSxNZA==
 log(s)         : /var/log/contrail/config-database-rabbitmq/rabbitmq.log
                : /var/log/rabbitmq/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3_upgrade.log
 database dir   : /var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3
2026-04-20 05:29:25.421 [info] <0.240.0> Running boot step pre_boot defined by app rabbit
2026-04-20 05:29:25.422 [info] <0.240.0> Running boot step rabbit_core_metrics defined by app rabbit
2026-04-20 05:29:25.422 [info] <0.240.0> Running boot step rabbit_alarm defined by app rabbit
2026-04-20 05:29:25.427 [info] <0.246.0> Memory high watermark set to 12637 MiB (13251081011 bytes) of 15796 MiB (16563851264 bytes) total
2026-04-20 05:29:25.436 [info] <0.248.0> Enabling free disk space monitoring
2026-04-20 05:29:25.436 [info] <0.248.0> Disk free limit set to 50MB
2026-04-20 05:29:25.443 [info] <0.240.0> Running boot step code_server_cache defined by app rabbit
2026-04-20 05:29:25.443 [info] <0.240.0> Running boot step file_handle_cache defined by app rabbit
2026-04-20 05:29:25.443 [info] <0.251.0> Limiting to approx 65436 file handles (58890 sockets)
2026-04-20 05:29:25.443 [info] <0.252.0> FHC read buffering:  OFF
2026-04-20 05:29:25.443 [info] <0.252.0> FHC write buffering: ON
2026-04-20 05:29:25.445 [info] <0.240.0> Running boot step worker_pool defined by app rabbit
2026-04-20 05:29:25.445 [info] <0.241.0> Will use 4 processes for default worker pool
2026-04-20 05:29:25.445 [info] <0.241.0> Starting worker pool 'worker_pool' with 4 processes in it
2026-04-20 05:29:25.446 [info] <0.240.0> Running boot step database defined by app rabbit
2026-04-20 05:29:25.447 [info] <0.240.0> Node database directory at /var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3 is empty. Assuming we need to join an existing cluster or initialise from scratch...
2026-04-20 05:29:25.447 [info] <0.240.0> Configured peer discovery backend: rabbit_peer_discovery_classic_config
2026-04-20 05:29:25.447 [info] <0.240.0> Will try to lock with peer discovery backend rabbit_peer_discovery_classic_config
2026-04-20 05:29:25.447 [info] <0.240.0> Peer discovery backend does not support locking, falling back to randomized delay
2026-04-20 05:29:25.447 [info] <0.240.0> Peer discovery backend rabbit_peer_discovery_classic_config supports registration.
2026-04-20 05:29:25.447 [info] <0.240.0> Will wait for 54130 milliseconds before proceeding with registration...
2026-04-20 05:30:14.080 [info] <0.307.0> RabbitMQ hasn't finished starting yet. Waiting for startup to finish before stopping...
2026-04-20 05:30:19.577 [info] <0.240.0> All discovered existing cluster peers: contrail@cn-jenkins-deploy-platform-ansible-os-5649-1, contrail@cn-jenkins-deploy-platform-ansible-os-5649-2, contrail@cn-jenkins-deploy-platform-ansible-os-5649-3
2026-04-20 05:30:19.578 [info] <0.240.0> Peer nodes we can cluster with: contrail@cn-jenkins-deploy-platform-ansible-os-5649-1, contrail@cn-jenkins-deploy-platform-ansible-os-5649-2
2026-04-20 05:30:19.580 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {badrpc,nodedown}
2026-04-20 05:30:19.590 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:30:19.591 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 9 retries left...
2026-04-20 05:30:20.093 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {badrpc,nodedown}
2026-04-20 05:30:20.094 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:30:20.095 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 8 retries left...
2026-04-20 05:30:20.596 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {badrpc,nodedown}
2026-04-20 05:30:20.597 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:30:20.598 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 7 retries left...
2026-04-20 05:30:21.099 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {badrpc,nodedown}
2026-04-20 05:30:21.100 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:30:21.100 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 6 retries left...
2026-04-20 05:30:21.602 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {badrpc,nodedown}
2026-04-20 05:30:21.605 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:30:21.605 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 5 retries left...
2026-04-20 05:30:22.107 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {badrpc,nodedown}
2026-04-20 05:30:22.108 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:30:22.108 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 4 retries left...
2026-04-20 05:30:22.610 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {badrpc,nodedown}
2026-04-20 05:30:22.611 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:30:22.611 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 3 retries left...
2026-04-20 05:30:23.113 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {badrpc,nodedown}
2026-04-20 05:30:23.114 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:30:23.114 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 2 retries left...
2026-04-20 05:30:23.617 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {badrpc,nodedown}
2026-04-20 05:30:23.618 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:30:23.618 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 1 retries left...
2026-04-20 05:30:24.119 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {badrpc,nodedown}
2026-04-20 05:30:24.120 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:30:24.121 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 0 retries left...
2026-04-20 05:30:24.621 [warning] <0.240.0> Could not successfully contact any node of: contrail@cn-jenkins-deploy-platform-ansible-os-5649-1,contrail@cn-jenkins-deploy-platform-ansible-os-5649-2 (as in Erlang distribution). Starting as a blank standalone node...
2026-04-20 05:30:24.624 [info] <0.43.0> Application mnesia exited with reason: stopped
2026-04-20 05:30:24.769 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2026-04-20 05:30:24.769 [info] <0.240.0> Successfully synced tables from a peer
2026-04-20 05:30:24.801 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2026-04-20 05:30:24.801 [info] <0.240.0> Successfully synced tables from a peer
2026-04-20 05:30:24.847 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2026-04-20 05:30:24.848 [info] <0.240.0> Successfully synced tables from a peer
2026-04-20 05:30:24.848 [info] <0.240.0> Will register with peer discovery backend rabbit_peer_discovery_classic_config
2026-04-20 05:30:24.849 [info] <0.240.0> Running boot step database_sync defined by app rabbit
2026-04-20 05:30:24.849 [info] <0.240.0> Running boot step feature_flags defined by app rabbit
2026-04-20 05:30:24.849 [info] <0.240.0> Running boot step codec_correctness_check defined by app rabbit
2026-04-20 05:30:24.849 [info] <0.240.0> Running boot step external_infrastructure defined by app rabbit
2026-04-20 05:30:24.849 [info] <0.240.0> Running boot step rabbit_registry defined by app rabbit
2026-04-20 05:30:24.850 [info] <0.240.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit
2026-04-20 05:30:24.850 [info] <0.240.0> Running boot step rabbit_queue_location_random defined by app rabbit
2026-04-20 05:30:24.850 [info] <0.240.0> Running boot step rabbit_event defined by app rabbit
2026-04-20 05:30:24.850 [info] <0.240.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit
2026-04-20 05:30:24.851 [info] <0.240.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit
2026-04-20 05:30:24.851 [info] <0.240.0> Running boot step rabbit_exchange_type_direct defined by app rabbit
2026-04-20 05:30:24.851 [info] <0.240.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit
2026-04-20 05:30:24.852 [info] <0.240.0> Running boot step rabbit_exchange_type_headers defined by app rabbit
2026-04-20 05:30:24.852 [info] <0.240.0> Running boot step rabbit_exchange_type_topic defined by app rabbit
2026-04-20 05:30:24.852 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit
2026-04-20 05:30:24.852 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit
2026-04-20 05:30:24.853 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit
2026-04-20 05:30:24.853 [info] <0.240.0> Running boot step rabbit_priority_queue defined by app rabbit
2026-04-20 05:30:24.853 [info] <0.240.0> Priority queues enabled, real BQ is rabbit_variable_queue
2026-04-20 05:30:24.853 [info] <0.240.0> Running boot step rabbit_queue_location_client_local defined by app rabbit
2026-04-20 05:30:24.853 [info] <0.240.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit
2026-04-20 05:30:24.853 [info] <0.240.0> Running boot step kernel_ready defined by app rabbit
2026-04-20 05:30:24.853 [info] <0.240.0> Running boot step rabbit_sysmon_minder defined by app rabbit
2026-04-20 05:30:24.854 [info] <0.240.0> Running boot step rabbit_epmd_monitor defined by app rabbit
2026-04-20 05:30:24.855 [info] <0.491.0> epmd monitor knows us, inter-node communication (distribution) port: 25673
2026-04-20 05:30:24.855 [info] <0.240.0> Running boot step guid_generator defined by app rabbit
2026-04-20 05:30:24.863 [info] <0.240.0> Running boot step rabbit_node_monitor defined by app rabbit
2026-04-20 05:30:24.863 [info] <0.495.0> Starting rabbit_node_monitor
2026-04-20 05:30:24.863 [info] <0.240.0> Running boot step delegate_sup defined by app rabbit
2026-04-20 05:30:24.863 [info] <0.240.0> Running boot step rabbit_memory_monitor defined by app rabbit
2026-04-20 05:30:24.864 [info] <0.240.0> Running boot step core_initialized defined by app rabbit
2026-04-20 05:30:24.864 [info] <0.240.0> Running boot step upgrade_queues defined by app rabbit
2026-04-20 05:30:24.892 [info] <0.240.0> message_store upgrades: 1 to apply
2026-04-20 05:30:24.893 [info] <0.240.0> message_store upgrades: Applying rabbit_variable_queue:move_messages_to_vhost_store
2026-04-20 05:30:24.893 [info] <0.240.0> message_store upgrades: No durable queues found. Skipping message store migration
2026-04-20 05:30:24.894 [info] <0.240.0> message_store upgrades: Removing the old message store data
2026-04-20 05:30:24.897 [info] <0.240.0> message_store upgrades: All upgrades applied successfully
2026-04-20 05:30:24.934 [info] <0.240.0> Running boot step rabbit_connection_tracking defined by app rabbit
2026-04-20 05:30:24.934 [info] <0.240.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit
2026-04-20 05:30:24.934 [info] <0.240.0> Running boot step rabbit_exchange_parameters defined by app rabbit
2026-04-20 05:30:24.935 [info] <0.240.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit
2026-04-20 05:30:24.935 [info] <0.240.0> Running boot step rabbit_policies defined by app rabbit
2026-04-20 05:30:24.935 [info] <0.240.0> Running boot step rabbit_policy defined by app rabbit
2026-04-20 05:30:24.935 [info] <0.240.0> Running boot step rabbit_queue_location_validator defined by app rabbit
2026-04-20 05:30:24.936 [info] <0.240.0> Running boot step rabbit_vhost_limit defined by app rabbit
2026-04-20 05:30:24.936 [info] <0.240.0> Running boot step rabbit_mgmt_reset_handler defined by app rabbitmq_management
2026-04-20 05:30:24.936 [info] <0.240.0> Running boot step rabbit_mgmt_db_handler defined by app rabbitmq_management_agent
2026-04-20 05:30:24.936 [warning] <0.240.0> force_fine_statistics set to true; ignored.
Replaced by {rates_mode, none} in the rabbitmq_management application.
2026-04-20 05:30:24.936 [info] <0.240.0> Management plugin: using rates mode 'basic'
2026-04-20 05:30:24.936 [info] <0.240.0> Running boot step recovery defined by app rabbit
2026-04-20 05:30:24.937 [info] <0.240.0> Running boot step load_definitions defined by app rabbitmq_management
2026-04-20 05:30:24.938 [info] <0.240.0> Applying definitions from /etc/rabbitmq/definitions.json
2026-04-20 05:30:24.938 [info] <0.240.0> Asked to import definitions. Acting user: rmq-internal
2026-04-20 05:30:24.938 [info] <0.240.0> Importing users...
2026-04-20 05:30:24.939 [info] <0.240.0> Creating user 'guest'
2026-04-20 05:30:24.946 [info] <0.240.0> Setting user tags for user 'guest' to [administrator]
2026-04-20 05:30:24.950 [info] <0.240.0> Importing vhosts...
2026-04-20 05:30:24.950 [info] <0.240.0> Adding vhost '/'
2026-04-20 05:30:24.981 [info] <0.543.0> Making sure data directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists
2026-04-20 05:30:24.988 [info] <0.543.0> Starting message stores for vhost '/'
2026-04-20 05:30:24.988 [info] <0.547.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index
2026-04-20 05:30:24.989 [info] <0.543.0> Started message store of type transient for vhost '/'
2026-04-20 05:30:24.990 [info] <0.550.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
2026-04-20 05:30:24.990 [warning] <0.550.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": rebuilding indices from scratch
2026-04-20 05:30:24.992 [info] <0.543.0> Started message store of type persistent for vhost '/'
2026-04-20 05:30:24.993 [info] <0.240.0> Importing user permissions...
2026-04-20 05:30:24.993 [info] <0.240.0> Setting permissions for 'guest' in '/' to '.*', '.*', '.*'
2026-04-20 05:30:24.996 [info] <0.240.0> Importing topic permissions...
2026-04-20 05:30:24.996 [info] <0.240.0> Importing parameters...
2026-04-20 05:30:24.996 [info] <0.240.0> Importing global parameters...
2026-04-20 05:30:24.996 [info] <0.240.0> Importing policies...
2026-04-20 05:30:25.003 [info] <0.240.0> Importing queues...
2026-04-20 05:30:25.003 [info] <0.240.0> Importing exchanges...
2026-04-20 05:30:25.003 [info] <0.240.0> Importing bindings...
2026-04-20 05:30:25.003 [info] <0.240.0> Running boot step empty_db_check defined by app rabbit
2026-04-20 05:30:25.003 [info] <0.240.0> Running boot step rabbit_looking_glass defined by app rabbit
2026-04-20 05:30:25.003 [info] <0.240.0> Running boot step rabbit_core_metrics_gc defined by app rabbit
2026-04-20 05:30:25.003 [info] <0.240.0> Running boot step background_gc defined by app rabbit
2026-04-20 05:30:25.003 [info] <0.240.0> Running boot step connection_tracking defined by app rabbit
2026-04-20 05:30:25.007 [info] <0.240.0> Setting up a table for connection tracking on this node: 'tracked_connection_on_node_contrail@cn-jenkins-deploy-platform-ansible-os-5649-3'
2026-04-20 05:30:25.012 [info] <0.240.0> Setting up a table for per-vhost connection counting on this node: 'tracked_connection_per_vhost_on_node_contrail@cn-jenkins-deploy-platform-ansible-os-5649-3'
2026-04-20 05:30:25.012 [info] <0.240.0> Running boot step routing_ready defined by app rabbit
2026-04-20 05:30:25.012 [info] <0.240.0> Running boot step pre_flight defined by app rabbit
2026-04-20 05:30:25.012 [info] <0.240.0> Running boot step notify_cluster defined by app rabbit
2026-04-20 05:30:25.012 [info] <0.240.0> Running boot step networking defined by app rabbit
2026-04-20 05:30:25.013 [info] <0.596.0> started TCP listener on [::]:5673
2026-04-20 05:30:25.014 [info] <0.240.0> Running boot step cluster_name defined by app rabbit
2026-04-20 05:30:25.014 [info] <0.240.0> Running boot step direct_client defined by app rabbit
2026-04-20 05:30:25.014 [info] <0.240.0> Running boot step os_signal_handler defined by app rabbit
2026-04-20 05:30:25.014 [info] <0.598.0> Swapping OS signal event handler (erl_signal_server) for our own
2026-04-20 05:30:25.057 [info] <0.648.0> Management plugin: HTTP (non-TLS) listener started on port 15673
2026-04-20 05:30:25.057 [info] <0.754.0> Statistics database started.
2026-04-20 05:30:25.057 [info] <0.753.0> Starting worker pool 'management_worker_pool' with 3 processes in it
2026-04-20 05:30:25.058 [notice] <0.106.0> Changed loghwm of /var/log/contrail/config-database-rabbitmq/rabbitmq.log to 50
2026-04-20 05:30:25.162 [info] <0.8.0> Server startup complete; 3 plugins started.
 * rabbitmq_management
 * rabbitmq_management_agent
 * rabbitmq_web_dispatch
2026-04-20 05:30:25.195 [info] <0.307.0> RabbitMQ is asked to stop...
2026-04-20 05:30:25.286 [info] <0.307.0> Stopping RabbitMQ applications and their dependencies in the following order:
    rabbitmq_management
    amqp_client
    rabbitmq_web_dispatch
    cowboy
    cowlib
    rabbitmq_management_agent
    rabbit
    mnesia
    rabbit_common
    sysmon_handler
    os_mon
2026-04-20 05:30:25.286 [info] <0.307.0> Stopping application 'rabbitmq_management'
2026-04-20 05:30:25.288 [warning] <0.604.0> RabbitMQ HTTP listener registry could not find context rabbitmq_management_tls
2026-04-20 05:30:25.289 [info] <0.43.0> Application rabbitmq_management exited with reason: stopped
2026-04-20 05:30:25.289 [info] <0.307.0> Stopping application 'amqp_client'
2026-04-20 05:30:25.291 [info] <0.307.0> Stopping application 'rabbitmq_web_dispatch'
2026-04-20 05:30:25.291 [info] <0.43.0> Application amqp_client exited with reason: stopped
2026-04-20 05:30:25.293 [info] <0.307.0> Stopping application 'cowboy'
2026-04-20 05:30:25.293 [info] <0.43.0> Application rabbitmq_web_dispatch exited with reason: stopped
2026-04-20 05:30:25.294 [info] <0.307.0> Stopping application 'cowlib'
2026-04-20 05:30:25.294 [info] <0.307.0> Stopping application 'rabbitmq_management_agent'
2026-04-20 05:30:25.294 [info] <0.43.0> Application cowboy exited with reason: stopped
2026-04-20 05:30:25.294 [info] <0.43.0> Application cowlib exited with reason: stopped
2026-04-20 05:30:25.295 [info] <0.307.0> Stopping application 'rabbit'
2026-04-20 05:30:25.295 [info] <0.43.0> Application rabbitmq_management_agent exited with reason: stopped
2026-04-20 05:30:25.295 [info] <0.240.0> Will unregister with peer discovery backend rabbit_peer_discovery_classic_config
2026-04-20 05:30:25.295 [info] <0.596.0> stopped TCP listener on [::]:5673
2026-04-20 05:30:25.296 [info] <0.520.0> Closing all connections in vhost '/' on node 'contrail@cn-jenkins-deploy-platform-ansible-os-5649-3' because the vhost is stopping
2026-04-20 05:30:25.296 [info] <0.550.0> Stopping message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent'
2026-04-20 05:30:25.305 [info] <0.550.0> Message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent' is stopped
2026-04-20 05:30:25.305 [info] <0.547.0> Stopping message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient'
2026-04-20 05:30:25.317 [info] <0.547.0> Message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient' is stopped
2026-04-20 05:30:25.321 [info] <0.43.0> Application rabbit exited with reason: stopped
2026-04-20 05:30:25.321 [info] <0.307.0> Stopping application 'mnesia'
2026-04-20 05:30:25.329 [info] <0.307.0> Stopping application 'rabbit_common'
2026-04-20 05:30:25.329 [info] <0.307.0> Stopping application 'sysmon_handler'
2026-04-20 05:30:25.331 [info] <0.43.0> Application mnesia exited with reason: stopped
2026-04-20 05:30:25.333 [info] <0.43.0> Application rabbit_common exited with reason: stopped
2026-04-20 05:30:25.333 [info] <0.43.0> Application sysmon_handler exited with reason: stopped
2026-04-20 05:30:25.333 [info] <0.307.0> Stopping application 'os_mon'
2026-04-20 05:30:25.334 [info] <0.43.0> Application os_mon exited with reason: stopped
2026-04-20 05:30:25.334 [info] <0.307.0> Successfully stopped RabbitMQ and its dependencies
2026-04-20 05:30:43.269 [info] <0.789.0> Resetting Rabbit forcefully
2026-04-20 05:30:44.776 [info] <0.803.0> RabbitMQ is asked to stop...
2026-04-20 05:30:44.902 [info] <0.803.0> Stopping RabbitMQ applications and their dependencies in the following order:
    rabbitmq_management
    amqp_client
    rabbitmq_management_agent
    rabbitmq_web_dispatch
    rabbit
    mnesia
    rabbit_common
    sysmon_handler
    os_mon
2026-04-20 05:30:44.902 [info] <0.803.0> Stopping application 'rabbitmq_management'
2026-04-20 05:30:44.902 [info] <0.803.0> Stopping application 'amqp_client'
2026-04-20 05:30:44.902 [info] <0.803.0> Stopping application 'rabbitmq_management_agent'
2026-04-20 05:30:44.903 [info] <0.803.0> Stopping application 'rabbitmq_web_dispatch'
2026-04-20 05:30:44.903 [info] <0.803.0> Stopping application 'rabbit'
2026-04-20 05:30:44.903 [info] <0.803.0> Stopping application 'mnesia'
2026-04-20 05:30:44.903 [info] <0.803.0> Stopping application 'rabbit_common'
2026-04-20 05:30:44.903 [info] <0.803.0> Stopping application 'sysmon_handler'
2026-04-20 05:30:44.903 [info] <0.803.0> Stopping application 'os_mon'
2026-04-20 05:30:44.903 [info] <0.803.0> Successfully stopped RabbitMQ and its dependencies
2026-04-20 05:30:44.903 [info] <0.803.0> Halting Erlang VM with the following applications:
    ranch
    ssl
    public_key
    asn1
    observer_cli
    tools
    credentials_obfuscation
    crypto
    stdout_formatter
    recon
    inets
    jsx
    xmerl
    lager
    goldrush
    compiler
    syntax_tools
    sasl
    stdlib
    kernel
2026-04-20 05:30:55.935 [info] <0.8.0> Log file opened with Lager
2026-04-20 05:30:58.399 [info] <0.8.0> Feature flags: list of feature flags found:
2026-04-20 05:30:58.399 [info] <0.8.0> Feature flags: feature flag states written to disk: yes
2026-04-20 05:30:58.424 [info] <0.240.0> 
 Starting RabbitMQ 3.7.28 on Erlang 21.3.8.21
 Copyright (c) 2007-2020 Pivotal Software, Inc.
 Licensed under the MPL.  See https://www.rabbitmq.com/
2026-04-20 05:30:58.424 [info] <0.240.0> 
 node           : contrail@cn-jenkins-deploy-platform-ansible-os-5649-3
 home dir       : /var/lib/rabbitmq
 config file(s) : /etc/rabbitmq/rabbitmq.config
 cookie hash    : esZsDxSN6VGbi9JkMSxNZA==
 log(s)         : /var/log/contrail/config-database-rabbitmq/rabbitmq.log
                : /var/log/rabbitmq/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3_upgrade.log
 database dir   : /var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3
2026-04-20 05:30:58.446 [info] <0.240.0> Running boot step pre_boot defined by app rabbit
2026-04-20 05:30:58.447 [info] <0.240.0> Running boot step rabbit_core_metrics defined by app rabbit
2026-04-20 05:30:58.447 [info] <0.240.0> Running boot step rabbit_alarm defined by app rabbit
2026-04-20 05:30:58.455 [info] <0.246.0> Memory high watermark set to 12637 MiB (13251081011 bytes) of 15796 MiB (16563851264 bytes) total
2026-04-20 05:30:58.462 [info] <0.248.0> Enabling free disk space monitoring
2026-04-20 05:30:58.462 [info] <0.248.0> Disk free limit set to 50MB
2026-04-20 05:30:58.468 [info] <0.240.0> Running boot step code_server_cache defined by app rabbit
2026-04-20 05:30:58.468 [info] <0.240.0> Running boot step file_handle_cache defined by app rabbit
2026-04-20 05:30:58.468 [info] <0.251.0> Limiting to approx 65436 file handles (58890 sockets)
2026-04-20 05:30:58.469 [info] <0.252.0> FHC read buffering:  OFF
2026-04-20 05:30:58.469 [info] <0.252.0> FHC write buffering: ON
2026-04-20 05:30:58.469 [info] <0.240.0> Running boot step worker_pool defined by app rabbit
2026-04-20 05:30:58.469 [info] <0.241.0> Will use 4 processes for default worker pool
2026-04-20 05:30:58.469 [info] <0.241.0> Starting worker pool 'worker_pool' with 4 processes in it
2026-04-20 05:30:58.469 [info] <0.240.0> Running boot step database defined by app rabbit
2026-04-20 05:30:58.470 [info] <0.240.0> Node database directory at /var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3 is empty. Assuming we need to join an existing cluster or initialise from scratch...
2026-04-20 05:30:58.470 [info] <0.240.0> Configured peer discovery backend: rabbit_peer_discovery_classic_config
2026-04-20 05:30:58.470 [info] <0.240.0> Will try to lock with peer discovery backend rabbit_peer_discovery_classic_config
2026-04-20 05:30:58.470 [info] <0.240.0> Peer discovery backend does not support locking, falling back to randomized delay
2026-04-20 05:30:58.470 [info] <0.240.0> Peer discovery backend rabbit_peer_discovery_classic_config supports registration.
2026-04-20 05:30:58.470 [info] <0.240.0> Will wait for 12564 milliseconds before proceeding with registration...
2026-04-20 05:31:11.035 [info] <0.240.0> All discovered existing cluster peers: contrail@cn-jenkins-deploy-platform-ansible-os-5649-1, contrail@cn-jenkins-deploy-platform-ansible-os-5649-2, contrail@cn-jenkins-deploy-platform-ansible-os-5649-3
2026-04-20 05:31:11.035 [info] <0.240.0> Peer nodes we can cluster with: contrail@cn-jenkins-deploy-platform-ansible-os-5649-1, contrail@cn-jenkins-deploy-platform-ansible-os-5649-2
2026-04-20 05:31:11.037 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {badrpc,nodedown}
2026-04-20 05:31:11.041 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,mnesia_not_running}
2026-04-20 05:31:11.041 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 9 retries left...
2026-04-20 05:31:11.544 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {badrpc,nodedown}
2026-04-20 05:31:11.545 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,mnesia_not_running}
2026-04-20 05:31:11.545 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 8 retries left...
2026-04-20 05:31:12.048 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {badrpc,nodedown}
2026-04-20 05:31:12.050 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,mnesia_not_running}
2026-04-20 05:31:12.050 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 7 retries left...
2026-04-20 05:31:12.553 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {badrpc,nodedown}
2026-04-20 05:31:12.553 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,mnesia_not_running}
2026-04-20 05:31:12.553 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 6 retries left...
2026-04-20 05:31:13.055 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {badrpc,nodedown}
2026-04-20 05:31:13.055 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,mnesia_not_running}
2026-04-20 05:31:13.056 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 5 retries left...
2026-04-20 05:31:13.557 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {badrpc,nodedown}
2026-04-20 05:31:13.558 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,mnesia_not_running}
2026-04-20 05:31:13.558 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 4 retries left...
2026-04-20 05:31:14.060 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {badrpc,nodedown}
2026-04-20 05:31:14.061 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,mnesia_not_running}
2026-04-20 05:31:14.061 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 3 retries left...
2026-04-20 05:31:14.563 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {badrpc,nodedown}
2026-04-20 05:31:14.564 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,mnesia_not_running}
2026-04-20 05:31:14.564 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 2 retries left...
2026-04-20 05:31:15.066 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {badrpc,nodedown}
2026-04-20 05:31:15.066 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,mnesia_not_running}
2026-04-20 05:31:15.067 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 1 retries left...
2026-04-20 05:31:15.568 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {badrpc,nodedown}
2026-04-20 05:31:15.569 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,mnesia_not_running}
2026-04-20 05:31:15.569 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 0 retries left...
2026-04-20 05:31:16.070 [warning] <0.240.0> Could not successfully contact any node of: contrail@cn-jenkins-deploy-platform-ansible-os-5649-1,contrail@cn-jenkins-deploy-platform-ansible-os-5649-2 (as in Erlang distribution). Starting as a blank standalone node...
2026-04-20 05:31:16.072 [info] <0.43.0> Application mnesia exited with reason: stopped
2026-04-20 05:31:16.176 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2026-04-20 05:31:16.176 [info] <0.240.0> Successfully synced tables from a peer
2026-04-20 05:31:16.233 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2026-04-20 05:31:16.234 [info] <0.240.0> Successfully synced tables from a peer
2026-04-20 05:31:16.277 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2026-04-20 05:31:16.277 [info] <0.240.0> Successfully synced tables from a peer
2026-04-20 05:31:16.278 [info] <0.240.0> Will register with peer discovery backend rabbit_peer_discovery_classic_config
2026-04-20 05:31:16.278 [info] <0.240.0> Running boot step database_sync defined by app rabbit
2026-04-20 05:31:16.278 [info] <0.240.0> Running boot step feature_flags defined by app rabbit
2026-04-20 05:31:16.279 [info] <0.240.0> Running boot step codec_correctness_check defined by app rabbit
2026-04-20 05:31:16.280 [info] <0.240.0> Running boot step external_infrastructure defined by app rabbit
2026-04-20 05:31:16.280 [info] <0.240.0> Running boot step rabbit_registry defined by app rabbit
2026-04-20 05:31:16.280 [info] <0.240.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit
2026-04-20 05:31:16.281 [info] <0.240.0> Running boot step rabbit_queue_location_random defined by app rabbit
2026-04-20 05:31:16.281 [info] <0.240.0> Running boot step rabbit_event defined by app rabbit
2026-04-20 05:31:16.282 [info] <0.240.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit
2026-04-20 05:31:16.282 [info] <0.240.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit
2026-04-20 05:31:16.282 [info] <0.240.0> Running boot step rabbit_exchange_type_direct defined by app rabbit
2026-04-20 05:31:16.283 [info] <0.240.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit
2026-04-20 05:31:16.283 [info] <0.240.0> Running boot step rabbit_exchange_type_headers defined by app rabbit
2026-04-20 05:31:16.284 [info] <0.240.0> Running boot step rabbit_exchange_type_topic defined by app rabbit
2026-04-20 05:31:16.284 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit
2026-04-20 05:31:16.285 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit
2026-04-20 05:31:16.285 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit
2026-04-20 05:31:16.286 [info] <0.240.0> Running boot step rabbit_priority_queue defined by app rabbit
2026-04-20 05:31:16.286 [info] <0.240.0> Priority queues enabled, real BQ is rabbit_variable_queue
2026-04-20 05:31:16.286 [info] <0.240.0> Running boot step rabbit_queue_location_client_local defined by app rabbit
2026-04-20 05:31:16.287 [info] <0.240.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit
2026-04-20 05:31:16.287 [info] <0.240.0> Running boot step kernel_ready defined by app rabbit
2026-04-20 05:31:16.288 [info] <0.240.0> Running boot step rabbit_sysmon_minder defined by app rabbit
2026-04-20 05:31:16.288 [info] <0.240.0> Running boot step rabbit_epmd_monitor defined by app rabbit
2026-04-20 05:31:16.290 [info] <0.456.0> epmd monitor knows us, inter-node communication (distribution) port: 25673
2026-04-20 05:31:16.290 [info] <0.240.0> Running boot step guid_generator defined by app rabbit
2026-04-20 05:31:16.296 [info] <0.240.0> Running boot step rabbit_node_monitor defined by app rabbit
2026-04-20 05:31:16.296 [info] <0.460.0> Starting rabbit_node_monitor
2026-04-20 05:31:16.297 [info] <0.240.0> Running boot step delegate_sup defined by app rabbit
2026-04-20 05:31:16.297 [info] <0.240.0> Running boot step rabbit_memory_monitor defined by app rabbit
2026-04-20 05:31:16.297 [info] <0.240.0> Running boot step core_initialized defined by app rabbit
2026-04-20 05:31:16.297 [info] <0.240.0> Running boot step upgrade_queues defined by app rabbit
2026-04-20 05:31:16.334 [info] <0.240.0> message_store upgrades: 1 to apply
2026-04-20 05:31:16.334 [info] <0.240.0> message_store upgrades: Applying rabbit_variable_queue:move_messages_to_vhost_store
2026-04-20 05:31:16.335 [info] <0.240.0> message_store upgrades: No durable queues found. Skipping message store migration
2026-04-20 05:31:16.335 [info] <0.240.0> message_store upgrades: Removing the old message store data
2026-04-20 05:31:16.340 [info] <0.240.0> message_store upgrades: All upgrades applied successfully
2026-04-20 05:31:16.381 [info] <0.240.0> Running boot step rabbit_connection_tracking defined by app rabbit
2026-04-20 05:31:16.382 [info] <0.240.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit
2026-04-20 05:31:16.382 [info] <0.240.0> Running boot step rabbit_exchange_parameters defined by app rabbit
2026-04-20 05:31:16.383 [info] <0.240.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit
2026-04-20 05:31:16.384 [info] <0.240.0> Running boot step rabbit_policies defined by app rabbit
2026-04-20 05:31:16.385 [info] <0.240.0> Running boot step rabbit_policy defined by app rabbit
2026-04-20 05:31:16.386 [info] <0.240.0> Running boot step rabbit_queue_location_validator defined by app rabbit
2026-04-20 05:31:16.386 [info] <0.240.0> Running boot step rabbit_vhost_limit defined by app rabbit
2026-04-20 05:31:16.387 [info] <0.240.0> Running boot step rabbit_mgmt_reset_handler defined by app rabbitmq_management
2026-04-20 05:31:16.388 [info] <0.240.0> Running boot step rabbit_mgmt_db_handler defined by app rabbitmq_management_agent
2026-04-20 05:31:16.388 [warning] <0.240.0> force_fine_statistics set to true; ignored.
Replaced by {rates_mode, none} in the rabbitmq_management application.
2026-04-20 05:31:16.389 [info] <0.240.0> Management plugin: using rates mode 'basic'
2026-04-20 05:31:16.390 [info] <0.240.0> Running boot step recovery defined by app rabbit
2026-04-20 05:31:16.392 [info] <0.240.0> Running boot step load_definitions defined by app rabbitmq_management
2026-04-20 05:31:16.393 [info] <0.240.0> Applying definitions from /etc/rabbitmq/definitions.json
2026-04-20 05:31:16.394 [info] <0.240.0> Asked to import definitions. Acting user: rmq-internal
2026-04-20 05:31:16.395 [info] <0.240.0> Importing users...
2026-04-20 05:31:16.396 [info] <0.240.0> Creating user 'guest'
2026-04-20 05:31:16.410 [info] <0.240.0> Setting user tags for user 'guest' to [administrator]
2026-04-20 05:31:16.416 [info] <0.240.0> Importing vhosts...
2026-04-20 05:31:16.417 [info] <0.240.0> Adding vhost '/'
2026-04-20 05:31:16.450 [info] <0.508.0> Making sure data directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists
2026-04-20 05:31:16.458 [info] <0.508.0> Starting message stores for vhost '/'
2026-04-20 05:31:16.459 [info] <0.512.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index
2026-04-20 05:31:16.461 [info] <0.508.0> Started message store of type transient for vhost '/'
2026-04-20 05:31:16.462 [info] <0.515.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
2026-04-20 05:31:16.463 [warning] <0.515.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": rebuilding indices from scratch
2026-04-20 05:31:16.465 [info] <0.508.0> Started message store of type persistent for vhost '/'
2026-04-20 05:31:16.466 [info] <0.240.0> Importing user permissions...
2026-04-20 05:31:16.467 [info] <0.240.0> Setting permissions for 'guest' in '/' to '.*', '.*', '.*'
2026-04-20 05:31:16.471 [info] <0.240.0> Importing topic permissions...
2026-04-20 05:31:16.471 [info] <0.240.0> Importing parameters...
2026-04-20 05:31:16.471 [info] <0.240.0> Importing global parameters...
2026-04-20 05:31:16.471 [info] <0.240.0> Importing policies...
2026-04-20 05:31:16.480 [info] <0.240.0> Importing queues...
2026-04-20 05:31:16.480 [info] <0.240.0> Importing exchanges...
2026-04-20 05:31:16.480 [info] <0.240.0> Importing bindings...
2026-04-20 05:31:16.481 [info] <0.240.0> Running boot step empty_db_check defined by app rabbit
2026-04-20 05:31:16.481 [info] <0.240.0> Running boot step rabbit_looking_glass defined by app rabbit
2026-04-20 05:31:16.481 [info] <0.240.0> Running boot step rabbit_core_metrics_gc defined by app rabbit
2026-04-20 05:31:16.482 [info] <0.240.0> Running boot step background_gc defined by app rabbit
2026-04-20 05:31:16.482 [info] <0.240.0> Running boot step connection_tracking defined by app rabbit
2026-04-20 05:31:16.487 [info] <0.240.0> Setting up a table for connection tracking on this node: 'tracked_connection_on_node_contrail@cn-jenkins-deploy-platform-ansible-os-5649-3'
2026-04-20 05:31:16.493 [info] <0.240.0> Setting up a table for per-vhost connection counting on this node: 'tracked_connection_per_vhost_on_node_contrail@cn-jenkins-deploy-platform-ansible-os-5649-3'
2026-04-20 05:31:16.493 [info] <0.240.0> Running boot step routing_ready defined by app rabbit
2026-04-20 05:31:16.493 [info] <0.240.0> Running boot step pre_flight defined by app rabbit
2026-04-20 05:31:16.494 [info] <0.240.0> Running boot step notify_cluster defined by app rabbit
2026-04-20 05:31:16.494 [info] <0.240.0> Running boot step networking defined by app rabbit
2026-04-20 05:31:16.497 [info] <0.561.0> started TCP listener on [::]:5673
2026-04-20 05:31:16.497 [info] <0.240.0> Running boot step cluster_name defined by app rabbit
2026-04-20 05:31:16.498 [info] <0.240.0> Running boot step direct_client defined by app rabbit
2026-04-20 05:31:16.498 [info] <0.240.0> Running boot step os_signal_handler defined by app rabbit
2026-04-20 05:31:16.498 [info] <0.563.0> Swapping OS signal event handler (erl_signal_server) for our own
2026-04-20 05:31:16.587 [info] <0.613.0> Management plugin: HTTP (non-TLS) listener started on port 15673
2026-04-20 05:31:16.587 [info] <0.719.0> Statistics database started.
2026-04-20 05:31:16.587 [info] <0.718.0> Starting worker pool 'management_worker_pool' with 3 processes in it
2026-04-20 05:31:16.588 [notice] <0.106.0> Changed loghwm of /var/log/contrail/config-database-rabbitmq/rabbitmq.log to 50
2026-04-20 05:31:16.770 [info] <0.8.0> Server startup complete; 3 plugins started.
 * rabbitmq_management
 * rabbitmq_web_dispatch
 * rabbitmq_management_agent
2026-04-20 05:31:17.900 [info] <0.743.0> RabbitMQ is asked to stop...
2026-04-20 05:31:18.000 [info] <0.746.0> accepting AMQP connection <0.746.0> (10.0.0.38:49740 -> 10.0.0.33:5673)
2026-04-20 05:31:18.009 [info] <0.750.0> accepting AMQP connection <0.750.0> (10.0.0.33:48390 -> 10.0.0.33:5673)
2026-04-20 05:31:18.009 [info] <0.753.0> accepting AMQP connection <0.753.0> (10.0.0.38:49742 -> 10.0.0.33:5673)
2026-04-20 05:31:18.009 [info] <0.756.0> accepting AMQP connection <0.756.0> (10.0.0.50:41266 -> 10.0.0.33:5673)
2026-04-20 05:31:18.009 [info] <0.759.0> accepting AMQP connection <0.759.0> (10.0.0.50:41276 -> 10.0.0.33:5673)
2026-04-20 05:31:18.009 [info] <0.762.0> accepting AMQP connection <0.762.0> (10.0.0.33:48398 -> 10.0.0.33:5673)
2026-04-20 05:31:18.009 [info] <0.746.0> connection <0.746.0> (10.0.0.38:49740 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:31:18.011 [info] <0.750.0> connection <0.750.0> (10.0.0.33:48390 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:31:18.011 [info] <0.775.0> accepting AMQP connection <0.775.0> (10.0.0.50:41286 -> 10.0.0.33:5673)
2026-04-20 05:31:18.012 [info] <0.753.0> connection <0.753.0> (10.0.0.38:49742 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:31:18.014 [info] <0.786.0> accepting AMQP connection <0.786.0> (10.0.0.38:49744 -> 10.0.0.33:5673)
2026-04-20 05:31:18.015 [info] <0.759.0> connection <0.759.0> (10.0.0.50:41276 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:31:18.015 [info] <0.756.0> connection <0.756.0> (10.0.0.50:41266 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:31:18.018 [info] <0.762.0> connection <0.762.0> (10.0.0.33:48398 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:31:18.018 [info] <0.775.0> connection <0.775.0> (10.0.0.50:41286 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:31:18.019 [info] <0.820.0> accepting AMQP connection <0.820.0> (10.0.0.33:48414 -> 10.0.0.33:5673)
2026-04-20 05:31:18.028 [info] <0.786.0> connection <0.786.0> (10.0.0.38:49744 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:31:18.033 [info] <0.820.0> connection <0.820.0> (10.0.0.33:48414 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:31:18.221 [info] <0.743.0> Stopping RabbitMQ applications and their dependencies in the following order:
    rabbitmq_management
    amqp_client
    rabbitmq_web_dispatch
    cowboy
    cowlib
    rabbitmq_management_agent
    rabbit
    mnesia
    rabbit_common
    sysmon_handler
    os_mon
2026-04-20 05:31:18.223 [info] <0.743.0> Stopping application 'rabbitmq_management'
2026-04-20 05:31:18.225 [warning] <0.609.0> RabbitMQ HTTP listener registry could not find context rabbitmq_management_tls
2026-04-20 05:31:18.227 [info] <0.743.0> Stopping application 'amqp_client'
2026-04-20 05:31:18.227 [info] <0.43.0> Application rabbitmq_management exited with reason: stopped
2026-04-20 05:31:18.231 [info] <0.43.0> Application amqp_client exited with reason: stopped
2026-04-20 05:31:18.231 [info] <0.743.0> Stopping application 'rabbitmq_web_dispatch'
2026-04-20 05:31:18.233 [info] <0.743.0> Stopping application 'cowboy'
2026-04-20 05:31:18.233 [info] <0.43.0> Application rabbitmq_web_dispatch exited with reason: stopped
2026-04-20 05:31:18.237 [info] <0.43.0> Application cowboy exited with reason: stopped
2026-04-20 05:31:18.237 [info] <0.743.0> Stopping application 'cowlib'
2026-04-20 05:31:18.237 [info] <0.43.0> Application cowlib exited with reason: stopped
2026-04-20 05:31:18.237 [info] <0.743.0> Stopping application 'rabbitmq_management_agent'
2026-04-20 05:31:18.240 [info] <0.43.0> Application rabbitmq_management_agent exited with reason: stopped
2026-04-20 05:31:18.240 [info] <0.743.0> Stopping application 'rabbit'
2026-04-20 05:31:18.240 [info] <0.240.0> Will unregister with peer discovery backend rabbit_peer_discovery_classic_config
2026-04-20 05:31:18.240 [info] <0.561.0> stopped TCP listener on [::]:5673
2026-04-20 05:31:18.247 [error] <0.750.0> Error on AMQP connection <0.750.0> (10.0.0.33:48390 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:31:18.247 [error] <0.786.0> Error on AMQP connection <0.786.0> (10.0.0.38:49744 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:31:18.247 [error] <0.759.0> Error on AMQP connection <0.759.0> (10.0.0.50:41276 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:31:18.247 [error] <0.775.0> Error on AMQP connection <0.775.0> (10.0.0.50:41286 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:31:18.247 [error] <0.762.0> Error on AMQP connection <0.762.0> (10.0.0.33:48398 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:31:18.247 [error] <0.753.0> Error on AMQP connection <0.753.0> (10.0.0.38:49742 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:31:18.247 [error] <0.756.0> Error on AMQP connection <0.756.0> (10.0.0.50:41266 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:31:18.248 [error] <0.820.0> Error on AMQP connection <0.820.0> (10.0.0.33:48414 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:31:18.248 [error] <0.746.0> Error on AMQP connection <0.746.0> (10.0.0.38:49740 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:31:18.259 [warning] <0.829.0> Mirrored queue 'contrail-dns.cn-jenkins-deploy-platform-ansible-os-5649-2' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available
2026-04-20 05:31:18.259 [info] <0.485.0> Closing all connections in vhost '/' on node 'contrail@cn-jenkins-deploy-platform-ansible-os-5649-3' because the vhost is stopping
2026-04-20 05:31:18.260 [warning] <0.877.0> Mirrored queue 'contrail-control.cn-jenkins-deploy-platform-ansible-os-5649-3' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available
2026-04-20 05:31:18.267 [warning] <0.885.0> Mirrored queue 'contrail-collector.cn-jenkins-deploy-platform-ansible-os-5649-1.' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available
2026-04-20 05:31:18.267 [warning] <0.881.0> Mirrored queue 'contrail-dns.cn-jenkins-deploy-platform-ansible-os-5649-3' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available
2026-04-20 05:31:18.267 [warning] <0.799.0> Mirrored queue 'contrail-dns.cn-jenkins-deploy-platform-ansible-os-5649-1' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available
2026-04-20 05:31:19.312 [warning]  lager_file_backend dropped 18 messages in the last second that exceeded the limit of 50 messages/sec
2026-04-20 05:37:19.066 [info] <0.1000.0> Resetting Rabbit forcefully
2026-04-20 05:37:20.771 [info] <0.1014.0> RabbitMQ is asked to stop...
2026-04-20 05:37:20.934 [info] <0.1014.0> Stopping RabbitMQ applications and their dependencies in the following order:
    rabbitmq_management
    amqp_client
    rabbitmq_web_dispatch
    rabbitmq_management_agent
    rabbit
    mnesia
    rabbit_common
    sysmon_handler
    os_mon
2026-04-20 05:37:20.934 [info] <0.1014.0> Stopping application 'rabbitmq_management'
2026-04-20 05:37:20.934 [info] <0.1014.0> Stopping application 'amqp_client'
2026-04-20 05:37:20.934 [info] <0.1014.0> Stopping application 'rabbitmq_web_dispatch'
2026-04-20 05:37:20.934 [info] <0.1014.0> Stopping application 'rabbitmq_management_agent'
2026-04-20 05:37:20.934 [info] <0.1014.0> Stopping application 'rabbit'
2026-04-20 05:37:20.934 [info] <0.1014.0> Stopping application 'mnesia'
2026-04-20 05:37:20.934 [info] <0.1014.0> Stopping application 'rabbit_common'
2026-04-20 05:37:20.934 [info] <0.1014.0> Stopping application 'sysmon_handler'
2026-04-20 05:37:20.934 [info] <0.1014.0> Stopping application 'os_mon'
2026-04-20 05:37:20.934 [info] <0.1014.0> Successfully stopped RabbitMQ and its dependencies
2026-04-20 05:37:20.934 [info] <0.1014.0> Halting Erlang VM with the following applications:
    ranch
    ssl
    credentials_obfuscation
    public_key
    asn1
    observer_cli
    jsx
    tools
    crypto
    stdout_formatter
    recon
    inets
    xmerl
    lager
    goldrush
    compiler
    syntax_tools
    sasl
    stdlib
    kernel
2026-04-20 05:37:31.206 [info] <0.8.0> Log file opened with Lager
2026-04-20 05:37:34.087 [info] <0.8.0> Feature flags: list of feature flags found:
2026-04-20 05:37:34.088 [info] <0.8.0> Feature flags: feature flag states written to disk: yes
2026-04-20 05:37:34.122 [info] <0.236.0> 
 Starting RabbitMQ 3.7.28 on Erlang 21.3.8.21
 Copyright (c) 2007-2020 Pivotal Software, Inc.
 Licensed under the MPL.  See https://www.rabbitmq.com/
2026-04-20 05:37:34.124 [info] <0.236.0> 
 node           : contrail@cn-jenkins-deploy-platform-ansible-os-5649-3
 home dir       : /var/lib/rabbitmq
 config file(s) : /etc/rabbitmq/rabbitmq.config
 cookie hash    : esZsDxSN6VGbi9JkMSxNZA==
 log(s)         : /var/log/contrail/config-database-rabbitmq/rabbitmq.log
                : /var/log/rabbitmq/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3_upgrade.log
 database dir   : /var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3
2026-04-20 05:37:34.138 [info] <0.236.0> Running boot step pre_boot defined by app rabbit
2026-04-20 05:37:34.139 [info] <0.236.0> Running boot step rabbit_core_metrics defined by app rabbit
2026-04-20 05:37:34.139 [info] <0.236.0> Running boot step rabbit_alarm defined by app rabbit
2026-04-20 05:37:34.149 [info] <0.242.0> Memory high watermark set to 12637 MiB (13251081011 bytes) of 15796 MiB (16563851264 bytes) total
2026-04-20 05:37:34.156 [info] <0.244.0> Enabling free disk space monitoring
2026-04-20 05:37:34.156 [info] <0.244.0> Disk free limit set to 50MB
2026-04-20 05:37:34.163 [info] <0.236.0> Running boot step code_server_cache defined by app rabbit
2026-04-20 05:37:34.163 [info] <0.236.0> Running boot step file_handle_cache defined by app rabbit
2026-04-20 05:37:34.164 [info] <0.247.0> Limiting to approx 65436 file handles (58890 sockets)
2026-04-20 05:37:34.164 [info] <0.248.0> FHC read buffering:  OFF
2026-04-20 05:37:34.164 [info] <0.248.0> FHC write buffering: ON
2026-04-20 05:37:34.167 [info] <0.236.0> Running boot step worker_pool defined by app rabbit
2026-04-20 05:37:34.167 [info] <0.237.0> Will use 4 processes for default worker pool
2026-04-20 05:37:34.167 [info] <0.237.0> Starting worker pool 'worker_pool' with 4 processes in it
2026-04-20 05:37:34.167 [info] <0.236.0> Running boot step database defined by app rabbit
2026-04-20 05:37:34.169 [info] <0.236.0> Node database directory at /var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3 is empty. Assuming we need to join an existing cluster or initialise from scratch...
2026-04-20 05:37:34.169 [info] <0.236.0> Configured peer discovery backend: rabbit_peer_discovery_classic_config
2026-04-20 05:37:34.169 [info] <0.236.0> Will try to lock with peer discovery backend rabbit_peer_discovery_classic_config
2026-04-20 05:37:34.169 [info] <0.236.0> Peer discovery backend does not support locking, falling back to randomized delay
2026-04-20 05:37:34.169 [info] <0.236.0> Peer discovery backend rabbit_peer_discovery_classic_config supports registration.
2026-04-20 05:37:34.169 [info] <0.236.0> Will wait for 43017 milliseconds before proceeding with registration...
2026-04-20 05:38:17.187 [info] <0.236.0> All discovered existing cluster peers: contrail@cn-jenkins-deploy-platform-ansible-os-5649-1, contrail@cn-jenkins-deploy-platform-ansible-os-5649-2, contrail@cn-jenkins-deploy-platform-ansible-os-5649-3
2026-04-20 05:38:17.187 [info] <0.236.0> Peer nodes we can cluster with: contrail@cn-jenkins-deploy-platform-ansible-os-5649-1, contrail@cn-jenkins-deploy-platform-ansible-os-5649-2
2026-04-20 05:38:17.189 [warning] <0.236.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {badrpc,nodedown}
2026-04-20 05:38:17.190 [warning] <0.236.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,mnesia_not_running}
2026-04-20 05:38:17.191 [error] <0.236.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 9 retries left...
2026-04-20 05:38:17.693 [warning] <0.236.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {badrpc,nodedown}
2026-04-20 05:38:17.694 [warning] <0.236.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,mnesia_not_running}
2026-04-20 05:38:17.694 [error] <0.236.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 8 retries left...
2026-04-20 05:38:18.196 [warning] <0.236.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {badrpc,nodedown}
2026-04-20 05:38:18.197 [warning] <0.236.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,mnesia_not_running}
2026-04-20 05:38:18.197 [error] <0.236.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 7 retries left...
2026-04-20 05:38:18.698 [warning] <0.236.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {badrpc,nodedown}
2026-04-20 05:38:18.699 [warning] <0.236.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,mnesia_not_running}
2026-04-20 05:38:18.699 [error] <0.236.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 6 retries left...
2026-04-20 05:38:19.201 [warning] <0.236.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {badrpc,nodedown}
2026-04-20 05:38:19.201 [warning] <0.236.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,mnesia_not_running}
2026-04-20 05:38:19.202 [error] <0.236.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 5 retries left...
2026-04-20 05:38:19.704 [warning] <0.236.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {badrpc,nodedown}
2026-04-20 05:38:19.705 [warning] <0.236.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,mnesia_not_running}
2026-04-20 05:38:19.705 [error] <0.236.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 4 retries left...
2026-04-20 05:38:20.211 [warning] <0.236.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {badrpc,nodedown}
2026-04-20 05:38:20.215 [warning] <0.236.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,mnesia_not_running}
2026-04-20 05:38:20.215 [error] <0.236.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 3 retries left...
2026-04-20 05:38:20.733 [warning] <0.236.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {badrpc,nodedown}
2026-04-20 05:38:20.734 [warning] <0.236.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,mnesia_not_running}
2026-04-20 05:38:20.734 [error] <0.236.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 2 retries left...
2026-04-20 05:38:21.102 [info] <0.319.0> RabbitMQ hasn't finished starting yet. Waiting for startup to finish before stopping...
2026-04-20 05:38:21.236 [warning] <0.236.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {badrpc,nodedown}
2026-04-20 05:38:21.237 [warning] <0.236.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,mnesia_not_running}
2026-04-20 05:38:21.237 [error] <0.236.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 1 retries left...
2026-04-20 05:38:21.739 [warning] <0.236.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {badrpc,nodedown}
2026-04-20 05:38:21.740 [warning] <0.236.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,mnesia_not_running}
2026-04-20 05:38:21.740 [error] <0.236.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 0 retries left...
2026-04-20 05:38:22.241 [warning] <0.236.0> Could not successfully contact any node of: contrail@cn-jenkins-deploy-platform-ansible-os-5649-1,contrail@cn-jenkins-deploy-platform-ansible-os-5649-2 (as in Erlang distribution). Starting as a blank standalone node...
2026-04-20 05:38:22.243 [info] <0.43.0> Application mnesia exited with reason: stopped
2026-04-20 05:38:22.408 [info] <0.236.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2026-04-20 05:38:22.409 [info] <0.236.0> Successfully synced tables from a peer
2026-04-20 05:38:22.437 [info] <0.236.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2026-04-20 05:38:22.437 [info] <0.236.0> Successfully synced tables from a peer
2026-04-20 05:38:22.465 [info] <0.236.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2026-04-20 05:38:22.465 [info] <0.236.0> Successfully synced tables from a peer
2026-04-20 05:38:22.465 [info] <0.236.0> Will register with peer discovery backend rabbit_peer_discovery_classic_config
2026-04-20 05:38:22.466 [info] <0.236.0> Running boot step database_sync defined by app rabbit
2026-04-20 05:38:22.466 [info] <0.236.0> Running boot step feature_flags defined by app rabbit
2026-04-20 05:38:22.466 [info] <0.236.0> Running boot step codec_correctness_check defined by app rabbit
2026-04-20 05:38:22.466 [info] <0.236.0> Running boot step external_infrastructure defined by app rabbit
2026-04-20 05:38:22.466 [info] <0.236.0> Running boot step rabbit_registry defined by app rabbit
2026-04-20 05:38:22.466 [info] <0.236.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit
2026-04-20 05:38:22.466 [info] <0.236.0> Running boot step rabbit_queue_location_random defined by app rabbit
2026-04-20 05:38:22.466 [info] <0.236.0> Running boot step rabbit_event defined by app rabbit
2026-04-20 05:38:22.466 [info] <0.236.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit
2026-04-20 05:38:22.466 [info] <0.236.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit
2026-04-20 05:38:22.466 [info] <0.236.0> Running boot step rabbit_exchange_type_direct defined by app rabbit
2026-04-20 05:38:22.466 [info] <0.236.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit
2026-04-20 05:38:22.466 [info] <0.236.0> Running boot step rabbit_exchange_type_headers defined by app rabbit
2026-04-20 05:38:22.466 [info] <0.236.0> Running boot step rabbit_exchange_type_topic defined by app rabbit
2026-04-20 05:38:22.467 [info] <0.236.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit
2026-04-20 05:38:22.467 [info] <0.236.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit
2026-04-20 05:38:22.467 [info] <0.236.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit
2026-04-20 05:38:22.467 [info] <0.236.0> Running boot step rabbit_priority_queue defined by app rabbit
2026-04-20 05:38:22.467 [info] <0.236.0> Priority queues enabled, real BQ is rabbit_variable_queue
2026-04-20 05:38:22.467 [info] <0.236.0> Running boot step rabbit_queue_location_client_local defined by app rabbit
2026-04-20 05:38:22.467 [info] <0.236.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit
2026-04-20 05:38:22.467 [info] <0.236.0> Running boot step kernel_ready defined by app rabbit
2026-04-20 05:38:22.467 [info] <0.236.0> Running boot step rabbit_sysmon_minder defined by app rabbit
2026-04-20 05:38:22.468 [info] <0.236.0> Running boot step rabbit_epmd_monitor defined by app rabbit
2026-04-20 05:38:22.468 [info] <0.484.0> epmd monitor knows us, inter-node communication (distribution) port: 25673
2026-04-20 05:38:22.468 [info] <0.236.0> Running boot step guid_generator defined by app rabbit
2026-04-20 05:38:22.472 [info] <0.236.0> Running boot step rabbit_node_monitor defined by app rabbit
2026-04-20 05:38:22.473 [info] <0.488.0> Starting rabbit_node_monitor
2026-04-20 05:38:22.473 [info] <0.236.0> Running boot step delegate_sup defined by app rabbit
2026-04-20 05:38:22.473 [info] <0.236.0> Running boot step rabbit_memory_monitor defined by app rabbit
2026-04-20 05:38:22.473 [info] <0.236.0> Running boot step core_initialized defined by app rabbit
2026-04-20 05:38:22.473 [info] <0.236.0> Running boot step upgrade_queues defined by app rabbit
2026-04-20 05:38:22.503 [info] <0.236.0> message_store upgrades: 1 to apply
2026-04-20 05:38:22.503 [info] <0.236.0> message_store upgrades: Applying rabbit_variable_queue:move_messages_to_vhost_store
2026-04-20 05:38:22.503 [info] <0.236.0> message_store upgrades: No durable queues found. Skipping message store migration
2026-04-20 05:38:22.503 [info] <0.236.0> message_store upgrades: Removing the old message store data
2026-04-20 05:38:22.504 [info] <0.236.0> message_store upgrades: All upgrades applied successfully
2026-04-20 05:38:22.631 [info] <0.236.0> Running boot step rabbit_connection_tracking defined by app rabbit
2026-04-20 05:38:22.631 [info] <0.236.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit
2026-04-20 05:38:22.631 [info] <0.236.0> Running boot step rabbit_exchange_parameters defined by app rabbit
2026-04-20 05:38:22.632 [info] <0.236.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit
2026-04-20 05:38:22.632 [info] <0.236.0> Running boot step rabbit_policies defined by app rabbit
2026-04-20 05:38:22.632 [info] <0.236.0> Running boot step rabbit_policy defined by app rabbit
2026-04-20 05:38:22.633 [info] <0.236.0> Running boot step rabbit_queue_location_validator defined by app rabbit
2026-04-20 05:38:22.633 [info] <0.236.0> Running boot step rabbit_vhost_limit defined by app rabbit
2026-04-20 05:38:22.633 [info] <0.236.0> Running boot step rabbit_mgmt_reset_handler defined by app rabbitmq_management
2026-04-20 05:38:22.633 [info] <0.236.0> Running boot step rabbit_mgmt_db_handler defined by app rabbitmq_management_agent
2026-04-20 05:38:22.633 [warning] <0.236.0> force_fine_statistics set to true; ignored.
Replaced by {rates_mode, none} in the rabbitmq_management application.
2026-04-20 05:38:22.633 [info] <0.236.0> Management plugin: using rates mode 'basic'
2026-04-20 05:38:22.633 [info] <0.236.0> Running boot step recovery defined by app rabbit
2026-04-20 05:38:22.634 [info] <0.236.0> Running boot step load_definitions defined by app rabbitmq_management
2026-04-20 05:38:22.634 [info] <0.236.0> Applying definitions from /etc/rabbitmq/definitions.json
2026-04-20 05:38:22.634 [info] <0.236.0> Asked to import definitions. Acting user: rmq-internal
2026-04-20 05:38:22.634 [info] <0.236.0> Importing users...
2026-04-20 05:38:22.634 [info] <0.236.0> Creating user 'guest'
2026-04-20 05:38:22.765 [info] <0.236.0> Setting user tags for user 'guest' to [administrator]
2026-04-20 05:38:22.802 [info] <0.236.0> Importing vhosts...
2026-04-20 05:38:22.802 [info] <0.236.0> Adding vhost '/'
2026-04-20 05:38:22.937 [info] <0.536.0> Making sure data directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists
2026-04-20 05:38:22.944 [info] <0.536.0> Starting message stores for vhost '/'
2026-04-20 05:38:22.945 [info] <0.540.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index
2026-04-20 05:38:22.946 [info] <0.536.0> Started message store of type transient for vhost '/'
2026-04-20 05:38:22.946 [info] <0.543.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
2026-04-20 05:38:22.947 [warning] <0.543.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": rebuilding indices from scratch
2026-04-20 05:38:22.948 [info] <0.536.0> Started message store of type persistent for vhost '/'
2026-04-20 05:38:22.949 [info] <0.236.0> Importing user permissions...
2026-04-20 05:38:22.949 [info] <0.236.0> Setting permissions for 'guest' in '/' to '.*', '.*', '.*'
2026-04-20 05:38:22.959 [info] <0.236.0> Importing topic permissions...
2026-04-20 05:38:22.959 [info] <0.236.0> Importing parameters...
2026-04-20 05:38:22.959 [info] <0.236.0> Importing global parameters...
2026-04-20 05:38:22.959 [info] <0.236.0> Importing policies...
2026-04-20 05:38:22.969 [info] <0.236.0> Importing queues...
2026-04-20 05:38:22.969 [info] <0.236.0> Importing exchanges...
2026-04-20 05:38:22.969 [info] <0.236.0> Importing bindings...
2026-04-20 05:38:22.969 [info] <0.236.0> Running boot step empty_db_check defined by app rabbit
2026-04-20 05:38:22.969 [info] <0.236.0> Running boot step rabbit_looking_glass defined by app rabbit
2026-04-20 05:38:22.969 [info] <0.236.0> Running boot step rabbit_core_metrics_gc defined by app rabbit
2026-04-20 05:38:22.969 [info] <0.236.0> Running boot step background_gc defined by app rabbit
2026-04-20 05:38:22.969 [info] <0.236.0> Running boot step connection_tracking defined by app rabbit
2026-04-20 05:38:22.974 [info] <0.236.0> Setting up a table for connection tracking on this node: 'tracked_connection_on_node_contrail@cn-jenkins-deploy-platform-ansible-os-5649-3'
2026-04-20 05:38:22.979 [info] <0.236.0> Setting up a table for per-vhost connection counting on this node: 'tracked_connection_per_vhost_on_node_contrail@cn-jenkins-deploy-platform-ansible-os-5649-3'
2026-04-20 05:38:22.979 [info] <0.236.0> Running boot step routing_ready defined by app rabbit
2026-04-20 05:38:22.979 [info] <0.236.0> Running boot step pre_flight defined by app rabbit
2026-04-20 05:38:22.979 [info] <0.236.0> Running boot step notify_cluster defined by app rabbit
2026-04-20 05:38:22.979 [info] <0.236.0> Running boot step networking defined by app rabbit
2026-04-20 05:38:22.982 [info] <0.589.0> started TCP listener on [::]:5673
2026-04-20 05:38:22.982 [info] <0.236.0> Running boot step cluster_name defined by app rabbit
2026-04-20 05:38:22.982 [info] <0.236.0> Running boot step direct_client defined by app rabbit
2026-04-20 05:38:22.982 [info] <0.236.0> Running boot step os_signal_handler defined by app rabbit
2026-04-20 05:38:22.982 [info] <0.591.0> Swapping OS signal event handler (erl_signal_server) for our own
2026-04-20 05:38:23.013 [info] <0.645.0> Management plugin: HTTP (non-TLS) listener started on port 15673
2026-04-20 05:38:23.014 [info] <0.751.0> Statistics database started.
2026-04-20 05:38:23.014 [info] <0.750.0> Starting worker pool 'management_worker_pool' with 3 processes in it
2026-04-20 05:38:23.014 [notice] <0.106.0> Changed loghwm of /var/log/contrail/config-database-rabbitmq/rabbitmq.log to 50
2026-04-20 05:38:23.325 [info] <0.8.0> Server startup complete; 3 plugins started.
 * rabbitmq_management
 * rabbitmq_web_dispatch
 * rabbitmq_management_agent
2026-04-20 05:38:23.331 [info] <0.319.0> RabbitMQ is asked to stop...
2026-04-20 05:38:23.440 [info] <0.319.0> Stopping RabbitMQ applications and their dependencies in the following order:
    rabbitmq_management
    amqp_client
    rabbitmq_web_dispatch
    cowboy
    cowlib
    rabbitmq_management_agent
    rabbit
    mnesia
    rabbit_common
    sysmon_handler
    os_mon
2026-04-20 05:38:23.440 [info] <0.319.0> Stopping application 'rabbitmq_management'
2026-04-20 05:38:23.442 [warning] <0.637.0> RabbitMQ HTTP listener registry could not find context rabbitmq_management_tls
2026-04-20 05:38:23.444 [info] <0.319.0> Stopping application 'amqp_client'
2026-04-20 05:38:23.444 [info] <0.43.0> Application rabbitmq_management exited with reason: stopped
2026-04-20 05:38:23.446 [info] <0.43.0> Application amqp_client exited with reason: stopped
2026-04-20 05:38:23.446 [info] <0.319.0> Stopping application 'rabbitmq_web_dispatch'
2026-04-20 05:38:23.448 [info] <0.319.0> Stopping application 'cowboy'
2026-04-20 05:38:23.448 [info] <0.43.0> Application rabbitmq_web_dispatch exited with reason: stopped
2026-04-20 05:38:23.450 [info] <0.43.0> Application cowboy exited with reason: stopped
2026-04-20 05:38:23.450 [info] <0.319.0> Stopping application 'cowlib'
2026-04-20 05:38:23.456 [info] <0.319.0> Stopping application 'rabbitmq_management_agent'
2026-04-20 05:38:23.457 [info] <0.43.0> Application cowlib exited with reason: stopped
2026-04-20 05:38:23.458 [info] <0.319.0> Stopping application 'rabbit'
2026-04-20 05:38:23.458 [info] <0.236.0> Will unregister with peer discovery backend rabbit_peer_discovery_classic_config
2026-04-20 05:38:23.458 [info] <0.43.0> Application rabbitmq_management_agent exited with reason: stopped
2026-04-20 05:38:23.459 [info] <0.589.0> stopped TCP listener on [::]:5673
2026-04-20 05:38:23.473 [info] <0.513.0> Closing all connections in vhost '/' on node 'contrail@cn-jenkins-deploy-platform-ansible-os-5649-3' because the vhost is stopping
2026-04-20 05:38:23.473 [info] <0.543.0> Stopping message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent'
2026-04-20 05:38:23.484 [info] <0.543.0> Message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent' is stopped
2026-04-20 05:38:23.484 [info] <0.540.0> Stopping message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient'
2026-04-20 05:38:23.488 [info] <0.540.0> Message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient' is stopped
2026-04-20 05:38:23.491 [info] <0.43.0> Application rabbit exited with reason: stopped
2026-04-20 05:38:23.491 [info] <0.319.0> Stopping application 'mnesia'
2026-04-20 05:38:23.498 [info] <0.43.0> Application mnesia exited with reason: stopped
2026-04-20 05:38:23.498 [info] <0.319.0> Stopping application 'rabbit_common'
2026-04-20 05:38:23.499 [info] <0.43.0> Application rabbit_common exited with reason: stopped
2026-04-20 05:38:23.499 [info] <0.319.0> Stopping application 'sysmon_handler'
2026-04-20 05:38:23.500 [info] <0.43.0> Application sysmon_handler exited with reason: stopped
2026-04-20 05:38:23.500 [info] <0.319.0> Stopping application 'os_mon'
2026-04-20 05:38:23.502 [info] <0.319.0> Successfully stopped RabbitMQ and its dependencies
2026-04-20 05:38:23.502 [info] <0.43.0> Application os_mon exited with reason: stopped
2026-04-20 05:39:25.596 [info] <0.786.0> Resetting Rabbit forcefully
2026-04-20 05:39:27.032 [info] <0.800.0> RabbitMQ is asked to stop...
2026-04-20 05:39:27.117 [info] <0.800.0> Stopping RabbitMQ applications and their dependencies in the following order:
    rabbitmq_management
    amqp_client
    rabbitmq_web_dispatch
    rabbitmq_management_agent
    rabbit
    mnesia
    rabbit_common
    sysmon_handler
    os_mon
2026-04-20 05:39:27.117 [info] <0.800.0> Stopping application 'rabbitmq_management'
2026-04-20 05:39:27.118 [info] <0.800.0> Stopping application 'amqp_client'
2026-04-20 05:39:27.118 [info] <0.800.0> Stopping application 'rabbitmq_web_dispatch'
2026-04-20 05:39:27.118 [info] <0.800.0> Stopping application 'rabbitmq_management_agent'
2026-04-20 05:39:27.118 [info] <0.800.0> Stopping application 'rabbit'
2026-04-20 05:39:27.118 [info] <0.800.0> Stopping application 'mnesia'
2026-04-20 05:39:27.118 [info] <0.800.0> Stopping application 'rabbit_common'
2026-04-20 05:39:27.118 [info] <0.800.0> Stopping application 'sysmon_handler'
2026-04-20 05:39:27.118 [info] <0.800.0> Stopping application 'os_mon'
2026-04-20 05:39:27.118 [info] <0.800.0> Successfully stopped RabbitMQ and its dependencies
2026-04-20 05:39:27.118 [info] <0.800.0> Halting Erlang VM with the following applications:
    ranch
    ssl
    public_key
    asn1
    credentials_obfuscation
    observer_cli
    jsx
    tools
    xmerl
    crypto
    stdout_formatter
    recon
    inets
    lager
    goldrush
    compiler
    syntax_tools
    sasl
    stdlib
    kernel
2026-04-20 05:39:37.134 [info] <0.8.0> Log file opened with Lager
2026-04-20 05:39:39.514 [info] <0.8.0> Feature flags: list of feature flags found:
2026-04-20 05:39:39.514 [info] <0.8.0> Feature flags: feature flag states written to disk: yes
2026-04-20 05:39:39.535 [info] <0.240.0> 
 Starting RabbitMQ 3.7.28 on Erlang 21.3.8.21
 Copyright (c) 2007-2020 Pivotal Software, Inc.
 Licensed under the MPL.  See https://www.rabbitmq.com/
2026-04-20 05:39:39.536 [info] <0.240.0> 
 node           : contrail@cn-jenkins-deploy-platform-ansible-os-5649-3
 home dir       : /var/lib/rabbitmq
 config file(s) : /etc/rabbitmq/rabbitmq.config
 cookie hash    : esZsDxSN6VGbi9JkMSxNZA==
 log(s)         : /var/log/contrail/config-database-rabbitmq/rabbitmq.log
                : /var/log/rabbitmq/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3_upgrade.log
 database dir   : /var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3
2026-04-20 05:39:39.550 [info] <0.240.0> Running boot step pre_boot defined by app rabbit
2026-04-20 05:39:39.550 [info] <0.240.0> Running boot step rabbit_core_metrics defined by app rabbit
2026-04-20 05:39:39.550 [info] <0.240.0> Running boot step rabbit_alarm defined by app rabbit
2026-04-20 05:39:39.555 [info] <0.246.0> Memory high watermark set to 12637 MiB (13251081011 bytes) of 15796 MiB (16563851264 bytes) total
2026-04-20 05:39:39.562 [info] <0.248.0> Enabling free disk space monitoring
2026-04-20 05:39:39.562 [info] <0.248.0> Disk free limit set to 50MB
2026-04-20 05:39:39.567 [info] <0.240.0> Running boot step code_server_cache defined by app rabbit
2026-04-20 05:39:39.567 [info] <0.240.0> Running boot step file_handle_cache defined by app rabbit
2026-04-20 05:39:39.568 [info] <0.251.0> Limiting to approx 65436 file handles (58890 sockets)
2026-04-20 05:39:39.568 [info] <0.252.0> FHC read buffering:  OFF
2026-04-20 05:39:39.568 [info] <0.252.0> FHC write buffering: ON
2026-04-20 05:39:39.568 [info] <0.240.0> Running boot step worker_pool defined by app rabbit
2026-04-20 05:39:39.568 [info] <0.241.0> Will use 4 processes for default worker pool
2026-04-20 05:39:39.568 [info] <0.241.0> Starting worker pool 'worker_pool' with 4 processes in it
2026-04-20 05:39:39.569 [info] <0.240.0> Running boot step database defined by app rabbit
2026-04-20 05:39:39.569 [info] <0.240.0> Node database directory at /var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3 is empty. Assuming we need to join an existing cluster or initialise from scratch...
2026-04-20 05:39:39.569 [info] <0.240.0> Configured peer discovery backend: rabbit_peer_discovery_classic_config
2026-04-20 05:39:39.569 [info] <0.240.0> Will try to lock with peer discovery backend rabbit_peer_discovery_classic_config
2026-04-20 05:39:39.569 [info] <0.240.0> Peer discovery backend does not support locking, falling back to randomized delay
2026-04-20 05:39:39.569 [info] <0.240.0> Peer discovery backend rabbit_peer_discovery_classic_config supports registration.
2026-04-20 05:39:39.569 [info] <0.240.0> Will wait for 47239 milliseconds before proceeding with registration...
2026-04-20 05:40:26.629 [info] <0.294.0> RabbitMQ hasn't finished starting yet. Waiting for startup to finish before stopping...
2026-04-20 05:40:26.808 [info] <0.240.0> All discovered existing cluster peers: contrail@cn-jenkins-deploy-platform-ansible-os-5649-1, contrail@cn-jenkins-deploy-platform-ansible-os-5649-2, contrail@cn-jenkins-deploy-platform-ansible-os-5649-3
2026-04-20 05:40:26.808 [info] <0.240.0> Peer nodes we can cluster with: contrail@cn-jenkins-deploy-platform-ansible-os-5649-1, contrail@cn-jenkins-deploy-platform-ansible-os-5649-2
2026-04-20 05:40:26.812 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {badrpc,nodedown}
2026-04-20 05:40:26.817 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:40:26.817 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 9 retries left...
2026-04-20 05:40:27.377 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,mnesia_not_running}
2026-04-20 05:40:27.378 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:40:27.379 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 8 retries left...
2026-04-20 05:40:27.885 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,mnesia_not_running}
2026-04-20 05:40:27.886 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:40:27.886 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 7 retries left...
2026-04-20 05:40:28.390 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,mnesia_not_running}
2026-04-20 05:40:28.391 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:40:28.391 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 6 retries left...
2026-04-20 05:40:28.896 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,mnesia_not_running}
2026-04-20 05:40:28.897 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:40:28.898 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 5 retries left...
2026-04-20 05:40:29.404 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,mnesia_not_running}
2026-04-20 05:40:29.405 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:40:29.405 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 4 retries left...
2026-04-20 05:40:29.918 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,mnesia_not_running}
2026-04-20 05:40:29.919 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:40:29.919 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 3 retries left...
2026-04-20 05:40:30.423 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,mnesia_not_running}
2026-04-20 05:40:30.424 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:40:30.424 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 2 retries left...
2026-04-20 05:40:30.935 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,mnesia_not_running}
2026-04-20 05:40:30.936 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:40:30.936 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 1 retries left...
2026-04-20 05:40:31.462 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,mnesia_not_running}
2026-04-20 05:40:31.463 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:40:31.463 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 0 retries left...
2026-04-20 05:40:31.964 [warning] <0.240.0> Could not successfully contact any node of: contrail@cn-jenkins-deploy-platform-ansible-os-5649-1,contrail@cn-jenkins-deploy-platform-ansible-os-5649-2 (as in Erlang distribution). Starting as a blank standalone node...
2026-04-20 05:40:31.966 [info] <0.43.0> Application mnesia exited with reason: stopped
2026-04-20 05:40:32.117 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2026-04-20 05:40:32.117 [info] <0.240.0> Successfully synced tables from a peer
2026-04-20 05:40:32.151 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2026-04-20 05:40:32.151 [info] <0.240.0> Successfully synced tables from a peer
2026-04-20 05:40:32.189 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2026-04-20 05:40:32.189 [info] <0.240.0> Successfully synced tables from a peer
2026-04-20 05:40:32.190 [info] <0.240.0> Will register with peer discovery backend rabbit_peer_discovery_classic_config
2026-04-20 05:40:32.190 [info] <0.240.0> Running boot step database_sync defined by app rabbit
2026-04-20 05:40:32.190 [info] <0.240.0> Running boot step feature_flags defined by app rabbit
2026-04-20 05:40:32.191 [info] <0.240.0> Running boot step codec_correctness_check defined by app rabbit
2026-04-20 05:40:32.191 [info] <0.240.0> Running boot step external_infrastructure defined by app rabbit
2026-04-20 05:40:32.192 [info] <0.240.0> Running boot step rabbit_registry defined by app rabbit
2026-04-20 05:40:32.192 [info] <0.240.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit
2026-04-20 05:40:32.192 [info] <0.240.0> Running boot step rabbit_queue_location_random defined by app rabbit
2026-04-20 05:40:32.193 [info] <0.240.0> Running boot step rabbit_event defined by app rabbit
2026-04-20 05:40:32.193 [info] <0.240.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit
2026-04-20 05:40:32.193 [info] <0.240.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit
2026-04-20 05:40:32.193 [info] <0.240.0> Running boot step rabbit_exchange_type_direct defined by app rabbit
2026-04-20 05:40:32.193 [info] <0.240.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit
2026-04-20 05:40:32.193 [info] <0.240.0> Running boot step rabbit_exchange_type_headers defined by app rabbit
2026-04-20 05:40:32.194 [info] <0.240.0> Running boot step rabbit_exchange_type_topic defined by app rabbit
2026-04-20 05:40:32.194 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit
2026-04-20 05:40:32.194 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit
2026-04-20 05:40:32.194 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit
2026-04-20 05:40:32.194 [info] <0.240.0> Running boot step rabbit_priority_queue defined by app rabbit
2026-04-20 05:40:32.194 [info] <0.240.0> Priority queues enabled, real BQ is rabbit_variable_queue
2026-04-20 05:40:32.194 [info] <0.240.0> Running boot step rabbit_queue_location_client_local defined by app rabbit
2026-04-20 05:40:32.194 [info] <0.240.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit
2026-04-20 05:40:32.195 [info] <0.240.0> Running boot step kernel_ready defined by app rabbit
2026-04-20 05:40:32.195 [info] <0.240.0> Running boot step rabbit_sysmon_minder defined by app rabbit
2026-04-20 05:40:32.195 [info] <0.240.0> Running boot step rabbit_epmd_monitor defined by app rabbit
2026-04-20 05:40:32.196 [info] <0.463.0> epmd monitor knows us, inter-node communication (distribution) port: 25673
2026-04-20 05:40:32.196 [info] <0.240.0> Running boot step guid_generator defined by app rabbit
2026-04-20 05:40:32.199 [info] <0.240.0> Running boot step rabbit_node_monitor defined by app rabbit
2026-04-20 05:40:32.200 [info] <0.467.0> Starting rabbit_node_monitor
2026-04-20 05:40:32.200 [info] <0.240.0> Running boot step delegate_sup defined by app rabbit
2026-04-20 05:40:32.200 [info] <0.240.0> Running boot step rabbit_memory_monitor defined by app rabbit
2026-04-20 05:40:32.201 [info] <0.240.0> Running boot step core_initialized defined by app rabbit
2026-04-20 05:40:32.201 [info] <0.240.0> Running boot step upgrade_queues defined by app rabbit
2026-04-20 05:40:32.228 [info] <0.240.0> message_store upgrades: 1 to apply
2026-04-20 05:40:32.228 [info] <0.240.0> message_store upgrades: Applying rabbit_variable_queue:move_messages_to_vhost_store
2026-04-20 05:40:32.228 [info] <0.240.0> message_store upgrades: No durable queues found. Skipping message store migration
2026-04-20 05:40:32.228 [info] <0.240.0> message_store upgrades: Removing the old message store data
2026-04-20 05:40:32.228 [info] <0.240.0> message_store upgrades: All upgrades applied successfully
2026-04-20 05:40:32.260 [info] <0.240.0> Running boot step rabbit_connection_tracking defined by app rabbit
2026-04-20 05:40:32.261 [info] <0.240.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit
2026-04-20 05:40:32.261 [info] <0.240.0> Running boot step rabbit_exchange_parameters defined by app rabbit
2026-04-20 05:40:32.261 [info] <0.240.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit
2026-04-20 05:40:32.261 [info] <0.240.0> Running boot step rabbit_policies defined by app rabbit
2026-04-20 05:40:32.262 [info] <0.240.0> Running boot step rabbit_policy defined by app rabbit
2026-04-20 05:40:32.263 [info] <0.240.0> Running boot step rabbit_queue_location_validator defined by app rabbit
2026-04-20 05:40:32.263 [info] <0.240.0> Running boot step rabbit_vhost_limit defined by app rabbit
2026-04-20 05:40:32.263 [info] <0.240.0> Running boot step rabbit_mgmt_reset_handler defined by app rabbitmq_management
2026-04-20 05:40:32.263 [info] <0.240.0> Running boot step rabbit_mgmt_db_handler defined by app rabbitmq_management_agent
2026-04-20 05:40:32.263 [warning] <0.240.0> force_fine_statistics set to true; ignored.
Replaced by {rates_mode, none} in the rabbitmq_management application.
2026-04-20 05:40:32.263 [info] <0.240.0> Management plugin: using rates mode 'basic'
2026-04-20 05:40:32.263 [info] <0.240.0> Running boot step recovery defined by app rabbit
2026-04-20 05:40:32.264 [info] <0.240.0> Running boot step load_definitions defined by app rabbitmq_management
2026-04-20 05:40:32.266 [info] <0.240.0> Applying definitions from /etc/rabbitmq/definitions.json
2026-04-20 05:40:32.266 [info] <0.240.0> Asked to import definitions. Acting user: rmq-internal
2026-04-20 05:40:32.266 [info] <0.240.0> Importing users...
2026-04-20 05:40:32.266 [info] <0.240.0> Creating user 'guest'
2026-04-20 05:40:32.273 [info] <0.240.0> Setting user tags for user 'guest' to [administrator]
2026-04-20 05:40:32.275 [info] <0.240.0> Importing vhosts...
2026-04-20 05:40:32.275 [info] <0.240.0> Adding vhost '/'
2026-04-20 05:40:32.306 [info] <0.515.0> Making sure data directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists
2026-04-20 05:40:32.312 [info] <0.515.0> Starting message stores for vhost '/'
2026-04-20 05:40:32.312 [info] <0.519.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index
2026-04-20 05:40:32.313 [info] <0.515.0> Started message store of type transient for vhost '/'
2026-04-20 05:40:32.313 [info] <0.522.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
2026-04-20 05:40:32.314 [warning] <0.522.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": rebuilding indices from scratch
2026-04-20 05:40:32.316 [info] <0.515.0> Started message store of type persistent for vhost '/'
2026-04-20 05:40:32.317 [info] <0.240.0> Importing user permissions...
2026-04-20 05:40:32.317 [info] <0.240.0> Setting permissions for 'guest' in '/' to '.*', '.*', '.*'
2026-04-20 05:40:32.322 [info] <0.240.0> Importing topic permissions...
2026-04-20 05:40:32.322 [info] <0.240.0> Importing parameters...
2026-04-20 05:40:32.322 [info] <0.240.0> Importing global parameters...
2026-04-20 05:40:32.322 [info] <0.240.0> Importing policies...
2026-04-20 05:40:32.331 [info] <0.240.0> Importing queues...
2026-04-20 05:40:32.331 [info] <0.240.0> Importing exchanges...
2026-04-20 05:40:32.331 [info] <0.240.0> Importing bindings...
2026-04-20 05:40:32.331 [info] <0.240.0> Running boot step empty_db_check defined by app rabbit
2026-04-20 05:40:32.331 [info] <0.240.0> Running boot step rabbit_looking_glass defined by app rabbit
2026-04-20 05:40:32.331 [info] <0.240.0> Running boot step rabbit_core_metrics_gc defined by app rabbit
2026-04-20 05:40:32.331 [info] <0.240.0> Running boot step background_gc defined by app rabbit
2026-04-20 05:40:32.332 [info] <0.240.0> Running boot step connection_tracking defined by app rabbit
2026-04-20 05:40:32.336 [info] <0.240.0> Setting up a table for connection tracking on this node: 'tracked_connection_on_node_contrail@cn-jenkins-deploy-platform-ansible-os-5649-3'
2026-04-20 05:40:32.344 [info] <0.240.0> Setting up a table for per-vhost connection counting on this node: 'tracked_connection_per_vhost_on_node_contrail@cn-jenkins-deploy-platform-ansible-os-5649-3'
2026-04-20 05:40:32.344 [info] <0.240.0> Running boot step routing_ready defined by app rabbit
2026-04-20 05:40:32.344 [info] <0.240.0> Running boot step pre_flight defined by app rabbit
2026-04-20 05:40:32.344 [info] <0.240.0> Running boot step notify_cluster defined by app rabbit
2026-04-20 05:40:32.344 [info] <0.240.0> Running boot step networking defined by app rabbit
2026-04-20 05:40:32.349 [info] <0.568.0> started TCP listener on [::]:5673
2026-04-20 05:40:32.350 [info] <0.240.0> Running boot step cluster_name defined by app rabbit
2026-04-20 05:40:32.350 [info] <0.240.0> Running boot step direct_client defined by app rabbit
2026-04-20 05:40:32.350 [info] <0.240.0> Running boot step os_signal_handler defined by app rabbit
2026-04-20 05:40:32.350 [info] <0.570.0> Swapping OS signal event handler (erl_signal_server) for our own
2026-04-20 05:40:32.405 [info] <0.620.0> Management plugin: HTTP (non-TLS) listener started on port 15673
2026-04-20 05:40:32.405 [info] <0.726.0> Statistics database started.
2026-04-20 05:40:32.405 [info] <0.725.0> Starting worker pool 'management_worker_pool' with 3 processes in it
2026-04-20 05:40:32.406 [notice] <0.106.0> Changed loghwm of /var/log/contrail/config-database-rabbitmq/rabbitmq.log to 50
2026-04-20 05:40:32.670 [info] <0.736.0> accepting AMQP connection <0.736.0> (10.0.0.38:47038 -> 10.0.0.33:5673)
2026-04-20 05:40:32.675 [info] <0.740.0> accepting AMQP connection <0.740.0> (10.0.0.38:47064 -> 10.0.0.33:5673)
2026-04-20 05:40:32.676 [info] <0.743.0> accepting AMQP connection <0.743.0> (10.0.0.38:47050 -> 10.0.0.33:5673)
2026-04-20 05:40:32.677 [info] <0.746.0> accepting AMQP connection <0.746.0> (10.0.0.50:59546 -> 10.0.0.33:5673)
2026-04-20 05:40:32.678 [info] <0.749.0> accepting AMQP connection <0.749.0> (10.0.0.50:59544 -> 10.0.0.33:5673)
2026-04-20 05:40:32.679 [info] <0.752.0> accepting AMQP connection <0.752.0> (10.0.0.33:50906 -> 10.0.0.33:5673)
2026-04-20 05:40:32.680 [info] <0.740.0> connection <0.740.0> (10.0.0.38:47064 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:40:32.681 [info] <0.755.0> accepting AMQP connection <0.755.0> (10.0.0.50:59556 -> 10.0.0.33:5673)
2026-04-20 05:40:32.681 [info] <0.736.0> connection <0.736.0> (10.0.0.38:47038 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:40:32.681 [info] <0.763.0> accepting AMQP connection <0.763.0> (10.0.0.33:50920 -> 10.0.0.33:5673)
2026-04-20 05:40:32.681 [info] <0.767.0> accepting AMQP connection <0.767.0> (10.0.0.33:50926 -> 10.0.0.33:5673)
2026-04-20 05:40:32.681 [info] <0.743.0> connection <0.743.0> (10.0.0.38:47050 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:40:32.681 [info] <0.746.0> connection <0.746.0> (10.0.0.50:59546 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:40:32.683 [info] <0.752.0> connection <0.752.0> (10.0.0.33:50906 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:40:32.685 [info] <0.749.0> connection <0.749.0> (10.0.0.50:59544 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:40:32.685 [info] <0.767.0> connection <0.767.0> (10.0.0.33:50926 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:40:32.685 [info] <0.755.0> connection <0.755.0> (10.0.0.50:59556 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:40:32.686 [info] <0.763.0> connection <0.763.0> (10.0.0.33:50920 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:40:32.779 [info] <0.8.0> Server startup complete; 3 plugins started.
 * rabbitmq_management
 * rabbitmq_web_dispatch
 * rabbitmq_management_agent
2026-04-20 05:40:32.801 [info] <0.294.0> RabbitMQ is asked to stop...
2026-04-20 05:40:33.092 [info] <0.294.0> Stopping RabbitMQ applications and their dependencies in the following order:
    rabbitmq_management
    amqp_client
    rabbitmq_web_dispatch
    cowboy
    cowlib
    rabbitmq_management_agent
    rabbit
    mnesia
    rabbit_common
    sysmon_handler
    os_mon
2026-04-20 05:40:33.093 [info] <0.294.0> Stopping application 'rabbitmq_management'
2026-04-20 05:40:33.095 [warning] <0.616.0> RabbitMQ HTTP listener registry could not find context rabbitmq_management_tls
2026-04-20 05:40:33.097 [info] <0.43.0> Application rabbitmq_management exited with reason: stopped
2026-04-20 05:40:33.097 [info] <0.294.0> Stopping application 'amqp_client'
2026-04-20 05:40:33.100 [info] <0.294.0> Stopping application 'rabbitmq_web_dispatch'
2026-04-20 05:40:33.100 [info] <0.43.0> Application amqp_client exited with reason: stopped
2026-04-20 05:40:33.102 [info] <0.294.0> Stopping application 'cowboy'
2026-04-20 05:40:33.102 [info] <0.43.0> Application rabbitmq_web_dispatch exited with reason: stopped
2026-04-20 05:40:33.104 [info] <0.43.0> Application cowboy exited with reason: stopped
2026-04-20 05:40:33.105 [info] <0.294.0> Stopping application 'cowlib'
2026-04-20 05:40:33.105 [info] <0.43.0> Application cowlib exited with reason: stopped
2026-04-20 05:40:33.105 [info] <0.294.0> Stopping application 'rabbitmq_management_agent'
2026-04-20 05:40:33.107 [info] <0.43.0> Application rabbitmq_management_agent exited with reason: stopped
2026-04-20 05:40:33.107 [info] <0.294.0> Stopping application 'rabbit'
2026-04-20 05:40:33.107 [info] <0.240.0> Will unregister with peer discovery backend rabbit_peer_discovery_classic_config
2026-04-20 05:40:33.108 [info] <0.568.0> stopped TCP listener on [::]:5673
2026-04-20 05:40:33.108 [error] <0.746.0> Error on AMQP connection <0.746.0> (10.0.0.50:59546 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:40:33.109 [error] <0.767.0> Error on AMQP connection <0.767.0> (10.0.0.33:50926 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:40:33.109 [error] <0.763.0> Error on AMQP connection <0.763.0> (10.0.0.33:50920 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:40:33.109 [error] <0.749.0> Error on AMQP connection <0.749.0> (10.0.0.50:59544 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:40:33.109 [error] <0.743.0> Error on AMQP connection <0.743.0> (10.0.0.38:47050 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:40:33.109 [error] <0.736.0> Error on AMQP connection <0.736.0> (10.0.0.38:47038 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:40:33.109 [error] <0.755.0> Error on AMQP connection <0.755.0> (10.0.0.50:59556 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:40:33.109 [error] <0.740.0> Error on AMQP connection <0.740.0> (10.0.0.38:47064 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:40:33.109 [error] <0.752.0> Error on AMQP connection <0.752.0> (10.0.0.33:50906 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:40:33.114 [info] <0.492.0> Closing all connections in vhost '/' on node 'contrail@cn-jenkins-deploy-platform-ansible-os-5649-3' because the vhost is stopping
2026-04-20 05:40:33.115 [warning] <0.865.0> Mirrored queue 'contrail-dns.cn-jenkins-deploy-platform-ansible-os-5649-2' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available
2026-04-20 05:40:33.115 [warning] <0.821.0> Mirrored queue 'contrail-collector.cn-jenkins-deploy-platform-ansible-os-5649-2.' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available
2026-04-20 05:40:33.115 [warning] <0.862.0> Mirrored queue 'contrail-control.cn-jenkins-deploy-platform-ansible-os-5649-2' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available
2026-04-20 05:40:33.115 [warning] <0.859.0> Mirrored queue 'contrail-dns.cn-jenkins-deploy-platform-ansible-os-5649-3' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available
2026-04-20 05:40:33.115 [warning] <0.830.0> Mirrored queue 'contrail-dns.cn-jenkins-deploy-platform-ansible-os-5649-1' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available
2026-04-20 05:40:33.115 [warning] <0.812.0> Mirrored queue 'contrail-collector.cn-jenkins-deploy-platform-ansible-os-5649-1.' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available
2026-04-20 05:40:33.115 [warning] <0.840.0> Mirrored queue 'contrail-collector.cn-jenkins-deploy-platform-ansible-os-5649-3.' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available
2026-04-20 05:40:33.115 [warning] <0.835.0> Mirrored queue 'contrail-control.cn-jenkins-deploy-platform-ansible-os-5649-1' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available
2026-04-20 05:40:33.117 [warning] <0.826.0> Mirrored queue 'contrail-control.cn-jenkins-deploy-platform-ansible-os-5649-3' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available
2026-04-20 05:40:33.118 [info] <0.522.0> Stopping message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent'
2026-04-20 05:40:33.138 [info] <0.522.0> Message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent' is stopped
2026-04-20 05:40:33.138 [info] <0.519.0> Stopping message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient'
2026-04-20 05:40:33.144 [info] <0.519.0> Message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient' is stopped
2026-04-20 05:40:33.159 [info] <0.43.0> Application rabbit exited with reason: stopped
2026-04-20 05:40:33.159 [info] <0.294.0> Stopping application 'mnesia'
2026-04-20 05:40:33.165 [info] <0.294.0> Stopping application 'rabbit_common'
2026-04-20 05:40:33.165 [info] <0.294.0> Stopping application 'sysmon_handler'
2026-04-20 05:40:33.165 [info] <0.43.0> Application mnesia exited with reason: stopped
2026-04-20 05:40:33.165 [info] <0.43.0> Application rabbit_common exited with reason: stopped
2026-04-20 05:40:33.168 [info] <0.294.0> Stopping application 'os_mon'
2026-04-20 05:40:33.168 [info] <0.43.0> Application sysmon_handler exited with reason: stopped
2026-04-20 05:40:33.174 [info] <0.294.0> Successfully stopped RabbitMQ and its dependencies
2026-04-20 05:40:33.174 [info] <0.43.0> Application os_mon exited with reason: stopped
2026-04-20 05:42:52.303 [info] <0.972.0> Resetting Rabbit forcefully
2026-04-20 05:42:53.592 [info] <0.986.0> RabbitMQ is asked to stop...
2026-04-20 05:42:53.645 [info] <0.986.0> Stopping RabbitMQ applications and their dependencies in the following order:
    rabbitmq_management
    amqp_client
    rabbitmq_web_dispatch
    rabbitmq_management_agent
    rabbit
    mnesia
    rabbit_common
    sysmon_handler
    os_mon
2026-04-20 05:42:53.645 [info] <0.986.0> Stopping application 'rabbitmq_management'
2026-04-20 05:42:53.645 [info] <0.986.0> Stopping application 'amqp_client'
2026-04-20 05:42:53.645 [info] <0.986.0> Stopping application 'rabbitmq_web_dispatch'
2026-04-20 05:42:53.645 [info] <0.986.0> Stopping application 'rabbitmq_management_agent'
2026-04-20 05:42:53.645 [info] <0.986.0> Stopping application 'rabbit'
2026-04-20 05:42:53.645 [info] <0.986.0> Stopping application 'mnesia'
2026-04-20 05:42:53.645 [info] <0.986.0> Stopping application 'rabbit_common'
2026-04-20 05:42:53.645 [info] <0.986.0> Stopping application 'sysmon_handler'
2026-04-20 05:42:53.645 [info] <0.986.0> Stopping application 'os_mon'
2026-04-20 05:42:53.645 [info] <0.986.0> Successfully stopped RabbitMQ and its dependencies
2026-04-20 05:42:53.645 [info] <0.986.0> Halting Erlang VM with the following applications:
    ranch
    ssl
    credentials_obfuscation
    public_key
    asn1
    observer_cli
    jsx
    tools
    crypto
    stdout_formatter
    recon
    inets
    xmerl
    lager
    goldrush
    compiler
    syntax_tools
    sasl
    stdlib
    kernel
2026-04-20 05:43:03.747 [info] <0.8.0> Log file opened with Lager
2026-04-20 05:43:06.351 [info] <0.8.0> Feature flags: list of feature flags found:
2026-04-20 05:43:06.352 [info] <0.8.0> Feature flags: feature flag states written to disk: yes
2026-04-20 05:43:06.400 [info] <0.240.0> 
 Starting RabbitMQ 3.7.28 on Erlang 21.3.8.21
 Copyright (c) 2007-2020 Pivotal Software, Inc.
 Licensed under the MPL.  See https://www.rabbitmq.com/
2026-04-20 05:43:06.402 [info] <0.240.0> 
 node           : contrail@cn-jenkins-deploy-platform-ansible-os-5649-3
 home dir       : /var/lib/rabbitmq
 config file(s) : /etc/rabbitmq/rabbitmq.config
 cookie hash    : esZsDxSN6VGbi9JkMSxNZA==
 log(s)         : /var/log/contrail/config-database-rabbitmq/rabbitmq.log
                : /var/log/rabbitmq/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3_upgrade.log
 database dir   : /var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3
2026-04-20 05:43:06.416 [info] <0.240.0> Running boot step pre_boot defined by app rabbit
2026-04-20 05:43:06.417 [info] <0.240.0> Running boot step rabbit_core_metrics defined by app rabbit
2026-04-20 05:43:06.418 [info] <0.240.0> Running boot step rabbit_alarm defined by app rabbit
2026-04-20 05:43:06.428 [info] <0.246.0> Memory high watermark set to 12637 MiB (13251081011 bytes) of 15796 MiB (16563851264 bytes) total
2026-04-20 05:43:06.438 [info] <0.248.0> Enabling free disk space monitoring
2026-04-20 05:43:06.438 [info] <0.248.0> Disk free limit set to 50MB
2026-04-20 05:43:06.444 [info] <0.240.0> Running boot step code_server_cache defined by app rabbit
2026-04-20 05:43:06.444 [info] <0.240.0> Running boot step file_handle_cache defined by app rabbit
2026-04-20 05:43:06.444 [info] <0.251.0> Limiting to approx 65436 file handles (58890 sockets)
2026-04-20 05:43:06.445 [info] <0.252.0> FHC read buffering:  OFF
2026-04-20 05:43:06.445 [info] <0.252.0> FHC write buffering: ON
2026-04-20 05:43:06.445 [info] <0.240.0> Running boot step worker_pool defined by app rabbit
2026-04-20 05:43:06.445 [info] <0.241.0> Will use 4 processes for default worker pool
2026-04-20 05:43:06.445 [info] <0.241.0> Starting worker pool 'worker_pool' with 4 processes in it
2026-04-20 05:43:06.445 [info] <0.240.0> Running boot step database defined by app rabbit
2026-04-20 05:43:06.446 [info] <0.240.0> Node database directory at /var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3 is empty. Assuming we need to join an existing cluster or initialise from scratch...
2026-04-20 05:43:06.446 [info] <0.240.0> Configured peer discovery backend: rabbit_peer_discovery_classic_config
2026-04-20 05:43:06.446 [info] <0.240.0> Will try to lock with peer discovery backend rabbit_peer_discovery_classic_config
2026-04-20 05:43:06.446 [info] <0.240.0> Peer discovery backend does not support locking, falling back to randomized delay
2026-04-20 05:43:06.446 [info] <0.240.0> Peer discovery backend rabbit_peer_discovery_classic_config supports registration.
2026-04-20 05:43:06.447 [info] <0.240.0> Will wait for 41956 milliseconds before proceeding with registration...
2026-04-20 05:43:48.403 [info] <0.240.0> All discovered existing cluster peers: contrail@cn-jenkins-deploy-platform-ansible-os-5649-1, contrail@cn-jenkins-deploy-platform-ansible-os-5649-2, contrail@cn-jenkins-deploy-platform-ansible-os-5649-3
2026-04-20 05:43:48.403 [info] <0.240.0> Peer nodes we can cluster with: contrail@cn-jenkins-deploy-platform-ansible-os-5649-1, contrail@cn-jenkins-deploy-platform-ansible-os-5649-2
2026-04-20 05:43:48.405 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {badrpc,nodedown}
2026-04-20 05:43:48.405 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:43:48.405 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 9 retries left...
2026-04-20 05:43:48.907 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {badrpc,nodedown}
2026-04-20 05:43:48.908 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:43:48.908 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 8 retries left...
2026-04-20 05:43:49.410 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,mnesia_not_running}
2026-04-20 05:43:49.410 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:43:49.410 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 7 retries left...
2026-04-20 05:43:49.917 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,mnesia_not_running}
2026-04-20 05:43:49.918 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:43:49.918 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 6 retries left...
2026-04-20 05:43:50.059 [info] <0.313.0> RabbitMQ hasn't finished starting yet. Waiting for startup to finish before stopping...
2026-04-20 05:43:50.420 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,mnesia_not_running}
2026-04-20 05:43:50.421 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,tables_not_present}
2026-04-20 05:43:50.421 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 5 retries left...
2026-04-20 05:43:50.925 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {error,mnesia_not_running}
2026-04-20 05:43:50.953 [info] <0.240.0> Node 'contrail@cn-jenkins-deploy-platform-ansible-os-5649-2' selected for auto-clustering
2026-04-20 05:43:50.989 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2026-04-20 05:43:50.989 [info] <0.240.0> Successfully synced tables from a peer
2026-04-20 05:43:51.254 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2026-04-20 05:43:51.254 [info] <0.240.0> Successfully synced tables from a peer
2026-04-20 05:43:51.291 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2026-04-20 05:43:51.291 [info] <0.240.0> Successfully synced tables from a peer
2026-04-20 05:43:51.303 [info] <0.240.0> Setting up a table for connection tracking on this node: 'tracked_connection_on_node_contrail@cn-jenkins-deploy-platform-ansible-os-5649-3'
2026-04-20 05:43:51.310 [info] <0.240.0> Setting up a table for per-vhost connection counting on this node: 'tracked_connection_per_vhost_on_node_contrail@cn-jenkins-deploy-platform-ansible-os-5649-3'
2026-04-20 05:43:51.310 [info] <0.240.0> Will register with peer discovery backend rabbit_peer_discovery_classic_config
2026-04-20 05:43:51.310 [info] <0.240.0> Running boot step database_sync defined by app rabbit
2026-04-20 05:43:51.310 [info] <0.240.0> Running boot step feature_flags defined by app rabbit
2026-04-20 05:43:51.311 [info] <0.240.0> Running boot step codec_correctness_check defined by app rabbit
2026-04-20 05:43:51.312 [info] <0.240.0> Running boot step external_infrastructure defined by app rabbit
2026-04-20 05:43:51.312 [info] <0.240.0> Running boot step rabbit_registry defined by app rabbit
2026-04-20 05:43:51.313 [info] <0.240.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit
2026-04-20 05:43:51.313 [info] <0.240.0> Running boot step rabbit_queue_location_random defined by app rabbit
2026-04-20 05:43:51.313 [info] <0.240.0> Running boot step rabbit_event defined by app rabbit
2026-04-20 05:43:51.314 [info] <0.240.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit
2026-04-20 05:43:51.314 [info] <0.240.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit
2026-04-20 05:43:51.315 [info] <0.240.0> Running boot step rabbit_exchange_type_direct defined by app rabbit
2026-04-20 05:43:51.315 [info] <0.240.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit
2026-04-20 05:43:51.316 [info] <0.240.0> Running boot step rabbit_exchange_type_headers defined by app rabbit
2026-04-20 05:43:51.316 [info] <0.240.0> Running boot step rabbit_exchange_type_topic defined by app rabbit
2026-04-20 05:43:51.317 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit
2026-04-20 05:43:51.317 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit
2026-04-20 05:43:51.317 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit
2026-04-20 05:43:51.318 [info] <0.240.0> Running boot step rabbit_priority_queue defined by app rabbit
2026-04-20 05:43:51.318 [info] <0.240.0> Priority queues enabled, real BQ is rabbit_variable_queue
2026-04-20 05:43:51.318 [info] <0.240.0> Running boot step rabbit_queue_location_client_local defined by app rabbit
2026-04-20 05:43:51.319 [info] <0.240.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit
2026-04-20 05:43:51.319 [info] <0.240.0> Running boot step kernel_ready defined by app rabbit
2026-04-20 05:43:51.320 [info] <0.240.0> Running boot step rabbit_sysmon_minder defined by app rabbit
2026-04-20 05:43:51.320 [info] <0.240.0> Running boot step rabbit_epmd_monitor defined by app rabbit
2026-04-20 05:43:51.321 [info] <0.498.0> epmd monitor knows us, inter-node communication (distribution) port: 25673
2026-04-20 05:43:51.322 [info] <0.240.0> Running boot step guid_generator defined by app rabbit
2026-04-20 05:43:51.328 [info] <0.240.0> Running boot step rabbit_node_monitor defined by app rabbit
2026-04-20 05:43:51.329 [info] <0.502.0> Starting rabbit_node_monitor
2026-04-20 05:43:51.330 [info] <0.240.0> Running boot step delegate_sup defined by app rabbit
2026-04-20 05:43:51.331 [info] <0.240.0> Running boot step rabbit_memory_monitor defined by app rabbit
2026-04-20 05:43:51.331 [info] <0.240.0> Running boot step core_initialized defined by app rabbit
2026-04-20 05:43:51.331 [info] <0.240.0> Running boot step upgrade_queues defined by app rabbit
2026-04-20 05:43:51.356 [info] <0.240.0> message_store upgrades: 1 to apply
2026-04-20 05:43:51.357 [info] <0.240.0> message_store upgrades: Applying rabbit_variable_queue:move_messages_to_vhost_store
2026-04-20 05:43:51.357 [info] <0.240.0> message_store upgrades: No durable queues found. Skipping message store migration
2026-04-20 05:43:51.358 [info] <0.240.0> message_store upgrades: Removing the old message store data
2026-04-20 05:43:51.361 [info] <0.240.0> message_store upgrades: All upgrades applied successfully
2026-04-20 05:43:51.390 [info] <0.240.0> Running boot step rabbit_connection_tracking defined by app rabbit
2026-04-20 05:43:51.390 [info] <0.240.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit
2026-04-20 05:43:51.390 [info] <0.240.0> Running boot step rabbit_exchange_parameters defined by app rabbit
2026-04-20 05:43:51.390 [info] <0.240.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit
2026-04-20 05:43:51.391 [info] <0.240.0> Running boot step rabbit_policies defined by app rabbit
2026-04-20 05:43:51.391 [info] <0.240.0> Running boot step rabbit_policy defined by app rabbit
2026-04-20 05:43:51.391 [info] <0.240.0> Running boot step rabbit_queue_location_validator defined by app rabbit
2026-04-20 05:43:51.391 [info] <0.240.0> Running boot step rabbit_vhost_limit defined by app rabbit
2026-04-20 05:43:51.392 [info] <0.240.0> Running boot step rabbit_mgmt_reset_handler defined by app rabbitmq_management
2026-04-20 05:43:51.392 [info] <0.240.0> Running boot step rabbit_mgmt_db_handler defined by app rabbitmq_management_agent
2026-04-20 05:43:51.392 [warning] <0.240.0> force_fine_statistics set to true; ignored.
Replaced by {rates_mode, none} in the rabbitmq_management application.
2026-04-20 05:43:51.392 [info] <0.240.0> Management plugin: using rates mode 'basic'
2026-04-20 05:43:51.393 [info] <0.240.0> Running boot step recovery defined by app rabbit
2026-04-20 05:43:51.394 [info] <0.540.0> Making sure data directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists
2026-04-20 05:43:51.402 [info] <0.540.0> Starting message stores for vhost '/'
2026-04-20 05:43:51.403 [info] <0.544.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index
2026-04-20 05:43:51.404 [info] <0.540.0> Started message store of type transient for vhost '/'
2026-04-20 05:43:51.404 [info] <0.547.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
2026-04-20 05:43:51.404 [warning] <0.547.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": rebuilding indices from scratch
2026-04-20 05:43:51.406 [info] <0.540.0> Started message store of type persistent for vhost '/'
2026-04-20 05:43:51.407 [info] <0.240.0> Running boot step load_definitions defined by app rabbitmq_management
2026-04-20 05:43:51.408 [info] <0.240.0> Applying definitions from /etc/rabbitmq/definitions.json
2026-04-20 05:43:51.408 [info] <0.240.0> Asked to import definitions. Acting user: rmq-internal
2026-04-20 05:43:51.409 [info] <0.240.0> Importing users...
2026-04-20 05:43:51.418 [info] <0.240.0> Setting user tags for user 'guest' to [administrator]
2026-04-20 05:43:51.422 [info] <0.240.0> Importing vhosts...
2026-04-20 05:43:51.422 [info] <0.240.0> Importing user permissions...
2026-04-20 05:43:51.422 [info] <0.240.0> Setting permissions for 'guest' in '/' to '.*', '.*', '.*'
2026-04-20 05:43:51.430 [info] <0.240.0> Importing topic permissions...
2026-04-20 05:43:51.430 [info] <0.240.0> Importing parameters...
2026-04-20 05:43:51.430 [info] <0.240.0> Importing global parameters...
2026-04-20 05:43:51.430 [info] <0.240.0> Importing policies...
2026-04-20 05:43:51.437 [info] <0.240.0> Importing queues...
2026-04-20 05:43:51.437 [info] <0.240.0> Importing exchanges...
2026-04-20 05:43:51.437 [info] <0.240.0> Importing bindings...
2026-04-20 05:43:51.437 [info] <0.240.0> Running boot step empty_db_check defined by app rabbit
2026-04-20 05:43:51.437 [info] <0.240.0> Running boot step rabbit_looking_glass defined by app rabbit
2026-04-20 05:43:51.437 [info] <0.240.0> Running boot step rabbit_core_metrics_gc defined by app rabbit
2026-04-20 05:43:51.437 [info] <0.240.0> Running boot step background_gc defined by app rabbit
2026-04-20 05:43:51.437 [info] <0.240.0> Running boot step connection_tracking defined by app rabbit
2026-04-20 05:43:51.441 [info] <0.240.0> Setting up a table for connection tracking on this node: 'tracked_connection_on_node_contrail@cn-jenkins-deploy-platform-ansible-os-5649-3'
2026-04-20 05:43:51.463 [info] <0.502.0> rabbit on node 'contrail@cn-jenkins-deploy-platform-ansible-os-5649-2' up
2026-04-20 05:43:51.486 [info] <0.240.0> Setting up a table for per-vhost connection counting on this node: 'tracked_connection_per_vhost_on_node_contrail@cn-jenkins-deploy-platform-ansible-os-5649-3'
2026-04-20 05:43:51.487 [info] <0.240.0> Running boot step routing_ready defined by app rabbit
2026-04-20 05:43:51.487 [info] <0.240.0> Running boot step pre_flight defined by app rabbit
2026-04-20 05:43:51.487 [info] <0.240.0> Running boot step notify_cluster defined by app rabbit
2026-04-20 05:43:51.488 [info] <0.240.0> Running boot step networking defined by app rabbit
2026-04-20 05:43:51.489 [info] <0.502.0> rabbit on node 'contrail@cn-jenkins-deploy-platform-ansible-os-5649-2' up
2026-04-20 05:43:51.496 [info] <0.600.0> started TCP listener on [::]:5673
2026-04-20 05:43:51.497 [info] <0.240.0> Running boot step cluster_name defined by app rabbit
2026-04-20 05:43:51.497 [info] <0.240.0> Running boot step direct_client defined by app rabbit
2026-04-20 05:43:51.497 [info] <0.240.0> Running boot step os_signal_handler defined by app rabbit
2026-04-20 05:43:51.497 [info] <0.602.0> Swapping OS signal event handler (erl_signal_server) for our own
2026-04-20 05:43:51.548 [info] <0.653.0> Management plugin: HTTP (non-TLS) listener started on port 15673
2026-04-20 05:43:51.548 [info] <0.759.0> Statistics database started.
2026-04-20 05:43:51.548 [info] <0.758.0> Starting worker pool 'management_worker_pool' with 3 processes in it
2026-04-20 05:43:51.549 [notice] <0.106.0> Changed loghwm of /var/log/contrail/config-database-rabbitmq/rabbitmq.log to 50
2026-04-20 05:43:51.691 [info] <0.8.0> Server startup complete; 3 plugins started.
 * rabbitmq_management
 * rabbitmq_web_dispatch
 * rabbitmq_management_agent
2026-04-20 05:43:51.780 [info] <0.313.0> RabbitMQ is asked to stop...
2026-04-20 05:43:51.836 [info] <0.502.0> rabbit on node 'contrail@cn-jenkins-deploy-platform-ansible-os-5649-2' down
2026-04-20 05:43:51.846 [info] <0.502.0> Keeping contrail@cn-jenkins-deploy-platform-ansible-os-5649-2 listeners: the node is already back
2026-04-20 05:43:51.863 [info] <0.313.0> Stopping RabbitMQ applications and their dependencies in the following order:
    rabbitmq_management
    amqp_client
    rabbitmq_web_dispatch
    cowboy
    cowlib
    rabbitmq_management_agent
    rabbit
    mnesia
    rabbit_common
    sysmon_handler
    os_mon
2026-04-20 05:43:51.863 [info] <0.313.0> Stopping application 'rabbitmq_management'
2026-04-20 05:43:51.864 [warning] <0.649.0> RabbitMQ HTTP listener registry could not find context rabbitmq_management_tls
2026-04-20 05:43:51.866 [info] <0.43.0> Application rabbitmq_management exited with reason: stopped
2026-04-20 05:43:51.866 [info] <0.313.0> Stopping application 'amqp_client'
2026-04-20 05:43:51.868 [info] <0.43.0> Application amqp_client exited with reason: stopped
2026-04-20 05:43:51.868 [info] <0.313.0> Stopping application 'rabbitmq_web_dispatch'
2026-04-20 05:43:51.869 [info] <0.43.0> Application rabbitmq_web_dispatch exited with reason: stopped
2026-04-20 05:43:51.869 [info] <0.313.0> Stopping application 'cowboy'
2026-04-20 05:43:51.871 [info] <0.313.0> Stopping application 'cowlib'
2026-04-20 05:43:51.871 [info] <0.43.0> Application cowboy exited with reason: stopped
2026-04-20 05:43:51.872 [info] <0.43.0> Application cowlib exited with reason: stopped
2026-04-20 05:43:51.872 [info] <0.313.0> Stopping application 'rabbitmq_management_agent'
2026-04-20 05:43:51.875 [info] <0.43.0> Application rabbitmq_management_agent exited with reason: stopped
2026-04-20 05:43:51.875 [info] <0.313.0> Stopping application 'rabbit'
2026-04-20 05:43:51.876 [info] <0.240.0> Will unregister with peer discovery backend rabbit_peer_discovery_classic_config
2026-04-20 05:43:51.876 [info] <0.600.0> stopped TCP listener on [::]:5673
2026-04-20 05:43:51.877 [info] <0.529.0> Closing all connections in vhost '/' on node 'contrail@cn-jenkins-deploy-platform-ansible-os-5649-3' because the vhost is stopping
2026-04-20 05:43:51.877 [info] <0.547.0> Stopping message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent'
2026-04-20 05:43:51.882 [info] <0.547.0> Message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent' is stopped
2026-04-20 05:43:51.882 [info] <0.544.0> Stopping message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient'
2026-04-20 05:43:51.887 [info] <0.544.0> Message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient' is stopped
2026-04-20 05:43:51.890 [info] <0.43.0> Application rabbit exited with reason: stopped
2026-04-20 05:43:51.890 [info] <0.313.0> Stopping application 'mnesia'
2026-04-20 05:43:51.894 [info] <0.43.0> Application mnesia exited with reason: stopped
2026-04-20 05:43:51.894 [info] <0.313.0> Stopping application 'rabbit_common'
2026-04-20 05:43:51.894 [info] <0.313.0> Stopping application 'sysmon_handler'
2026-04-20 05:43:51.894 [info] <0.43.0> Application rabbit_common exited with reason: stopped
2026-04-20 05:43:51.896 [info] <0.313.0> Stopping application 'os_mon'
2026-04-20 05:43:51.896 [info] <0.43.0> Application sysmon_handler exited with reason: stopped
2026-04-20 05:43:51.898 [info] <0.313.0> Successfully stopped RabbitMQ and its dependencies
2026-04-20 05:43:51.898 [info] <0.43.0> Application os_mon exited with reason: stopped
2026-04-20 05:44:02.985 [info] <0.785.0> Resetting Rabbit forcefully
2026-04-20 05:44:04.430 [info] <0.799.0> RabbitMQ is asked to stop...
2026-04-20 05:44:04.590 [info] <0.799.0> Stopping RabbitMQ applications and their dependencies in the following order:
    rabbitmq_management
    amqp_client
    rabbitmq_web_dispatch
    rabbitmq_management_agent
    rabbit
    mnesia
    rabbit_common
    sysmon_handler
    os_mon
2026-04-20 05:44:04.590 [info] <0.799.0> Stopping application 'rabbitmq_management'
2026-04-20 05:44:04.590 [info] <0.799.0> Stopping application 'amqp_client'
2026-04-20 05:44:04.590 [info] <0.799.0> Stopping application 'rabbitmq_web_dispatch'
2026-04-20 05:44:04.590 [info] <0.799.0> Stopping application 'rabbitmq_management_agent'
2026-04-20 05:44:04.590 [info] <0.799.0> Stopping application 'rabbit'
2026-04-20 05:44:04.590 [info] <0.799.0> Stopping application 'mnesia'
2026-04-20 05:44:04.590 [info] <0.799.0> Stopping application 'rabbit_common'
2026-04-20 05:44:04.590 [info] <0.799.0> Stopping application 'sysmon_handler'
2026-04-20 05:44:04.590 [info] <0.799.0> Stopping application 'os_mon'
2026-04-20 05:44:04.590 [info] <0.799.0> Successfully stopped RabbitMQ and its dependencies
2026-04-20 05:44:04.590 [info] <0.799.0> Halting Erlang VM with the following applications:
    ranch
    ssl
    credentials_obfuscation
    public_key
    asn1
    observer_cli
    jsx
    tools
    crypto
    stdout_formatter
    recon
    inets
    xmerl
    lager
    goldrush
    compiler
    syntax_tools
    sasl
    stdlib
    kernel
2026-04-20 05:44:14.840 [info] <0.8.0> Log file opened with Lager
2026-04-20 05:44:17.111 [info] <0.8.0> Feature flags: list of feature flags found:
2026-04-20 05:44:17.111 [info] <0.8.0> Feature flags: feature flag states written to disk: yes
2026-04-20 05:44:17.130 [info] <0.240.0> 
 Starting RabbitMQ 3.7.28 on Erlang 21.3.8.21
 Copyright (c) 2007-2020 Pivotal Software, Inc.
 Licensed under the MPL.  See https://www.rabbitmq.com/
2026-04-20 05:44:17.131 [info] <0.240.0> 
 node           : contrail@cn-jenkins-deploy-platform-ansible-os-5649-3
 home dir       : /var/lib/rabbitmq
 config file(s) : /etc/rabbitmq/rabbitmq.config
 cookie hash    : esZsDxSN6VGbi9JkMSxNZA==
 log(s)         : /var/log/contrail/config-database-rabbitmq/rabbitmq.log
                : /var/log/rabbitmq/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3_upgrade.log
 database dir   : /var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3
2026-04-20 05:44:17.143 [info] <0.240.0> Running boot step pre_boot defined by app rabbit
2026-04-20 05:44:17.143 [info] <0.240.0> Running boot step rabbit_core_metrics defined by app rabbit
2026-04-20 05:44:17.143 [info] <0.240.0> Running boot step rabbit_alarm defined by app rabbit
2026-04-20 05:44:17.149 [info] <0.246.0> Memory high watermark set to 12637 MiB (13251081011 bytes) of 15796 MiB (16563851264 bytes) total
2026-04-20 05:44:17.154 [info] <0.248.0> Enabling free disk space monitoring
2026-04-20 05:44:17.154 [info] <0.248.0> Disk free limit set to 50MB
2026-04-20 05:44:17.159 [info] <0.240.0> Running boot step code_server_cache defined by app rabbit
2026-04-20 05:44:17.159 [info] <0.240.0> Running boot step file_handle_cache defined by app rabbit
2026-04-20 05:44:17.159 [info] <0.251.0> Limiting to approx 65436 file handles (58890 sockets)
2026-04-20 05:44:17.159 [info] <0.252.0> FHC read buffering:  OFF
2026-04-20 05:44:17.159 [info] <0.252.0> FHC write buffering: ON
2026-04-20 05:44:17.160 [info] <0.240.0> Running boot step worker_pool defined by app rabbit
2026-04-20 05:44:17.160 [info] <0.241.0> Will use 4 processes for default worker pool
2026-04-20 05:44:17.160 [info] <0.241.0> Starting worker pool 'worker_pool' with 4 processes in it
2026-04-20 05:44:17.160 [info] <0.240.0> Running boot step database defined by app rabbit
2026-04-20 05:44:17.161 [info] <0.240.0> Node database directory at /var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3 is empty. Assuming we need to join an existing cluster or initialise from scratch...
2026-04-20 05:44:17.161 [info] <0.240.0> Configured peer discovery backend: rabbit_peer_discovery_classic_config
2026-04-20 05:44:17.161 [info] <0.240.0> Will try to lock with peer discovery backend rabbit_peer_discovery_classic_config
2026-04-20 05:44:17.161 [info] <0.240.0> Peer discovery backend does not support locking, falling back to randomized delay
2026-04-20 05:44:17.161 [info] <0.240.0> Peer discovery backend rabbit_peer_discovery_classic_config supports registration.
2026-04-20 05:44:17.161 [info] <0.240.0> Will wait for 31476 milliseconds before proceeding with registration...
2026-04-20 05:44:48.638 [info] <0.240.0> All discovered existing cluster peers: contrail@cn-jenkins-deploy-platform-ansible-os-5649-1, contrail@cn-jenkins-deploy-platform-ansible-os-5649-2, contrail@cn-jenkins-deploy-platform-ansible-os-5649-3
2026-04-20 05:44:48.638 [info] <0.240.0> Peer nodes we can cluster with: contrail@cn-jenkins-deploy-platform-ansible-os-5649-1, contrail@cn-jenkins-deploy-platform-ansible-os-5649-2
2026-04-20 05:44:48.640 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {badrpc,nodedown}
2026-04-20 05:44:48.643 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,mnesia_not_running}
2026-04-20 05:44:48.644 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 9 retries left...
2026-04-20 05:44:49.147 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {badrpc,nodedown}
2026-04-20 05:44:49.149 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,mnesia_not_running}
2026-04-20 05:44:49.149 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 8 retries left...
2026-04-20 05:44:49.651 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {badrpc,nodedown}
2026-04-20 05:44:49.652 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,mnesia_not_running}
2026-04-20 05:44:49.652 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 7 retries left...
2026-04-20 05:44:50.154 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {badrpc,nodedown}
2026-04-20 05:44:50.155 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,mnesia_not_running}
2026-04-20 05:44:50.155 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 6 retries left...
2026-04-20 05:44:50.657 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {badrpc,nodedown}
2026-04-20 05:44:50.658 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,mnesia_not_running}
2026-04-20 05:44:50.658 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 5 retries left...
2026-04-20 05:44:51.160 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {badrpc,nodedown}
2026-04-20 05:44:51.161 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,mnesia_not_running}
2026-04-20 05:44:51.161 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 4 retries left...
2026-04-20 05:44:51.662 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {badrpc,nodedown}
2026-04-20 05:44:51.663 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,mnesia_not_running}
2026-04-20 05:44:51.663 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 3 retries left...
2026-04-20 05:44:52.165 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {badrpc,nodedown}
2026-04-20 05:44:52.166 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,mnesia_not_running}
2026-04-20 05:44:52.166 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 2 retries left...
2026-04-20 05:44:52.668 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {badrpc,nodedown}
2026-04-20 05:44:52.669 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,mnesia_not_running}
2026-04-20 05:44:52.669 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 1 retries left...
2026-04-20 05:44:53.171 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-1: {badrpc,nodedown}
2026-04-20 05:44:53.172 [warning] <0.240.0> Could not auto-cluster with node contrail@cn-jenkins-deploy-platform-ansible-os-5649-2: {error,mnesia_not_running}
2026-04-20 05:44:53.172 [error] <0.240.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 0 retries left...
2026-04-20 05:44:53.672 [warning] <0.240.0> Could not successfully contact any node of: contrail@cn-jenkins-deploy-platform-ansible-os-5649-1,contrail@cn-jenkins-deploy-platform-ansible-os-5649-2 (as in Erlang distribution). Starting as a blank standalone node...
2026-04-20 05:44:53.675 [info] <0.43.0> Application mnesia exited with reason: stopped
2026-04-20 05:44:53.841 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2026-04-20 05:44:53.841 [info] <0.240.0> Successfully synced tables from a peer
2026-04-20 05:44:53.881 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2026-04-20 05:44:53.881 [info] <0.240.0> Successfully synced tables from a peer
2026-04-20 05:44:53.914 [info] <0.240.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2026-04-20 05:44:53.914 [info] <0.240.0> Successfully synced tables from a peer
2026-04-20 05:44:53.914 [info] <0.240.0> Will register with peer discovery backend rabbit_peer_discovery_classic_config
2026-04-20 05:44:53.914 [info] <0.240.0> Running boot step database_sync defined by app rabbit
2026-04-20 05:44:53.914 [info] <0.240.0> Running boot step feature_flags defined by app rabbit
2026-04-20 05:44:53.914 [info] <0.240.0> Running boot step codec_correctness_check defined by app rabbit
2026-04-20 05:44:53.914 [info] <0.240.0> Running boot step external_infrastructure defined by app rabbit
2026-04-20 05:44:53.914 [info] <0.240.0> Running boot step rabbit_registry defined by app rabbit
2026-04-20 05:44:53.914 [info] <0.240.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit
2026-04-20 05:44:53.914 [info] <0.240.0> Running boot step rabbit_queue_location_random defined by app rabbit
2026-04-20 05:44:53.914 [info] <0.240.0> Running boot step rabbit_event defined by app rabbit
2026-04-20 05:44:53.914 [info] <0.240.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit
2026-04-20 05:44:53.914 [info] <0.240.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit
2026-04-20 05:44:53.915 [info] <0.240.0> Running boot step rabbit_exchange_type_direct defined by app rabbit
2026-04-20 05:44:53.915 [info] <0.240.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit
2026-04-20 05:44:53.915 [info] <0.240.0> Running boot step rabbit_exchange_type_headers defined by app rabbit
2026-04-20 05:44:53.915 [info] <0.240.0> Running boot step rabbit_exchange_type_topic defined by app rabbit
2026-04-20 05:44:53.915 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit
2026-04-20 05:44:53.915 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit
2026-04-20 05:44:53.915 [info] <0.240.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit
2026-04-20 05:44:53.915 [info] <0.240.0> Running boot step rabbit_priority_queue defined by app rabbit
2026-04-20 05:44:53.915 [info] <0.240.0> Priority queues enabled, real BQ is rabbit_variable_queue
2026-04-20 05:44:53.915 [info] <0.240.0> Running boot step rabbit_queue_location_client_local defined by app rabbit
2026-04-20 05:44:53.915 [info] <0.240.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit
2026-04-20 05:44:53.915 [info] <0.240.0> Running boot step kernel_ready defined by app rabbit
2026-04-20 05:44:53.916 [info] <0.240.0> Running boot step rabbit_sysmon_minder defined by app rabbit
2026-04-20 05:44:53.916 [info] <0.240.0> Running boot step rabbit_epmd_monitor defined by app rabbit
2026-04-20 05:44:53.917 [info] <0.484.0> epmd monitor knows us, inter-node communication (distribution) port: 25673
2026-04-20 05:44:53.917 [info] <0.240.0> Running boot step guid_generator defined by app rabbit
2026-04-20 05:44:53.923 [info] <0.240.0> Running boot step rabbit_node_monitor defined by app rabbit
2026-04-20 05:44:53.923 [info] <0.488.0> Starting rabbit_node_monitor
2026-04-20 05:44:53.923 [info] <0.240.0> Running boot step delegate_sup defined by app rabbit
2026-04-20 05:44:53.923 [info] <0.240.0> Running boot step rabbit_memory_monitor defined by app rabbit
2026-04-20 05:44:53.924 [info] <0.240.0> Running boot step core_initialized defined by app rabbit
2026-04-20 05:44:53.924 [info] <0.240.0> Running boot step upgrade_queues defined by app rabbit
2026-04-20 05:44:53.950 [info] <0.240.0> message_store upgrades: 1 to apply
2026-04-20 05:44:53.950 [info] <0.240.0> message_store upgrades: Applying rabbit_variable_queue:move_messages_to_vhost_store
2026-04-20 05:44:53.950 [info] <0.240.0> message_store upgrades: No durable queues found. Skipping message store migration
2026-04-20 05:44:53.950 [info] <0.240.0> message_store upgrades: Removing the old message store data
2026-04-20 05:44:53.951 [info] <0.240.0> message_store upgrades: All upgrades applied successfully
2026-04-20 05:44:53.983 [info] <0.240.0> Running boot step rabbit_connection_tracking defined by app rabbit
2026-04-20 05:44:53.984 [info] <0.240.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit
2026-04-20 05:44:53.984 [info] <0.240.0> Running boot step rabbit_exchange_parameters defined by app rabbit
2026-04-20 05:44:53.984 [info] <0.240.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit
2026-04-20 05:44:53.984 [info] <0.240.0> Running boot step rabbit_policies defined by app rabbit
2026-04-20 05:44:53.985 [info] <0.240.0> Running boot step rabbit_policy defined by app rabbit
2026-04-20 05:44:53.985 [info] <0.240.0> Running boot step rabbit_queue_location_validator defined by app rabbit
2026-04-20 05:44:53.985 [info] <0.240.0> Running boot step rabbit_vhost_limit defined by app rabbit
2026-04-20 05:44:53.985 [info] <0.240.0> Running boot step rabbit_mgmt_reset_handler defined by app rabbitmq_management
2026-04-20 05:44:53.985 [info] <0.240.0> Running boot step rabbit_mgmt_db_handler defined by app rabbitmq_management_agent
2026-04-20 05:44:53.985 [warning] <0.240.0> force_fine_statistics set to true; ignored.
Replaced by {rates_mode, none} in the rabbitmq_management application.
2026-04-20 05:44:53.985 [info] <0.240.0> Management plugin: using rates mode 'basic'
2026-04-20 05:44:53.986 [info] <0.240.0> Running boot step recovery defined by app rabbit
2026-04-20 05:44:53.986 [info] <0.240.0> Running boot step load_definitions defined by app rabbitmq_management
2026-04-20 05:44:53.987 [info] <0.240.0> Applying definitions from /etc/rabbitmq/definitions.json
2026-04-20 05:44:53.988 [info] <0.240.0> Asked to import definitions. Acting user: rmq-internal
2026-04-20 05:44:53.989 [info] <0.240.0> Importing users...
2026-04-20 05:44:53.989 [info] <0.240.0> Creating user 'guest'
2026-04-20 05:44:53.996 [info] <0.240.0> Setting user tags for user 'guest' to [administrator]
2026-04-20 05:44:53.999 [info] <0.240.0> Importing vhosts...
2026-04-20 05:44:53.999 [info] <0.240.0> Adding vhost '/'
2026-04-20 05:44:54.032 [info] <0.536.0> Making sure data directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists
2026-04-20 05:44:54.040 [info] <0.536.0> Starting message stores for vhost '/'
2026-04-20 05:44:54.040 [info] <0.540.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index
2026-04-20 05:44:54.041 [info] <0.536.0> Started message store of type transient for vhost '/'
2026-04-20 05:44:54.041 [info] <0.543.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
2026-04-20 05:44:54.042 [warning] <0.543.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": rebuilding indices from scratch
2026-04-20 05:44:54.042 [info] <0.536.0> Started message store of type persistent for vhost '/'
2026-04-20 05:44:54.044 [info] <0.240.0> Importing user permissions...
2026-04-20 05:44:54.044 [info] <0.240.0> Setting permissions for 'guest' in '/' to '.*', '.*', '.*'
2026-04-20 05:44:54.048 [info] <0.240.0> Importing topic permissions...
2026-04-20 05:44:54.048 [info] <0.240.0> Importing parameters...
2026-04-20 05:44:54.049 [info] <0.240.0> Importing global parameters...
2026-04-20 05:44:54.049 [info] <0.240.0> Importing policies...
2026-04-20 05:44:54.057 [info] <0.240.0> Importing queues...
2026-04-20 05:44:54.057 [info] <0.240.0> Importing exchanges...
2026-04-20 05:44:54.057 [info] <0.240.0> Importing bindings...
2026-04-20 05:44:54.058 [info] <0.240.0> Running boot step empty_db_check defined by app rabbit
2026-04-20 05:44:54.058 [info] <0.240.0> Running boot step rabbit_looking_glass defined by app rabbit
2026-04-20 05:44:54.058 [info] <0.240.0> Running boot step rabbit_core_metrics_gc defined by app rabbit
2026-04-20 05:44:54.058 [info] <0.240.0> Running boot step background_gc defined by app rabbit
2026-04-20 05:44:54.058 [info] <0.240.0> Running boot step connection_tracking defined by app rabbit
2026-04-20 05:44:54.071 [info] <0.240.0> Setting up a table for connection tracking on this node: 'tracked_connection_on_node_contrail@cn-jenkins-deploy-platform-ansible-os-5649-3'
2026-04-20 05:44:54.083 [info] <0.240.0> Setting up a table for per-vhost connection counting on this node: 'tracked_connection_per_vhost_on_node_contrail@cn-jenkins-deploy-platform-ansible-os-5649-3'
2026-04-20 05:44:54.083 [info] <0.240.0> Running boot step routing_ready defined by app rabbit
2026-04-20 05:44:54.083 [info] <0.240.0> Running boot step pre_flight defined by app rabbit
2026-04-20 05:44:54.084 [info] <0.240.0> Running boot step notify_cluster defined by app rabbit
2026-04-20 05:44:54.084 [info] <0.240.0> Running boot step networking defined by app rabbit
2026-04-20 05:44:54.087 [info] <0.589.0> started TCP listener on [::]:5673
2026-04-20 05:44:54.087 [info] <0.240.0> Running boot step cluster_name defined by app rabbit
2026-04-20 05:44:54.087 [info] <0.240.0> Running boot step direct_client defined by app rabbit
2026-04-20 05:44:54.087 [info] <0.240.0> Running boot step os_signal_handler defined by app rabbit
2026-04-20 05:44:54.088 [info] <0.591.0> Swapping OS signal event handler (erl_signal_server) for our own
2026-04-20 05:44:54.125 [info] <0.641.0> Management plugin: HTTP (non-TLS) listener started on port 15673
2026-04-20 05:44:54.125 [info] <0.747.0> Statistics database started.
2026-04-20 05:44:54.125 [info] <0.746.0> Starting worker pool 'management_worker_pool' with 3 processes in it
2026-04-20 05:44:54.130 [notice] <0.106.0> Changed loghwm of /var/log/contrail/config-database-rabbitmq/rabbitmq.log to 50
2026-04-20 05:44:54.290 [info] <0.8.0> Server startup complete; 3 plugins started.
 * rabbitmq_management
 * rabbitmq_management_agent
 * rabbitmq_web_dispatch
2026-04-20 05:44:55.001 [info] <0.757.0> accepting AMQP connection <0.757.0> (10.0.0.50:42696 -> 10.0.0.33:5673)
2026-04-20 05:44:55.002 [info] <0.761.0> accepting AMQP connection <0.761.0> (10.0.0.38:46284 -> 10.0.0.33:5673)
2026-04-20 05:44:55.002 [info] <0.764.0> accepting AMQP connection <0.764.0> (10.0.0.38:46298 -> 10.0.0.33:5673)
2026-04-20 05:44:55.004 [info] <0.767.0> accepting AMQP connection <0.767.0> (10.0.0.50:42712 -> 10.0.0.33:5673)
2026-04-20 05:44:55.004 [info] <0.764.0> connection <0.764.0> (10.0.0.38:46298 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:44:55.005 [info] <0.761.0> connection <0.761.0> (10.0.0.38:46284 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:44:55.005 [info] <0.771.0> accepting AMQP connection <0.771.0> (10.0.0.50:42722 -> 10.0.0.33:5673)
2026-04-20 05:44:55.005 [info] <0.780.0> accepting AMQP connection <0.780.0> (10.0.0.38:46302 -> 10.0.0.33:5673)
2026-04-20 05:44:55.005 [info] <0.783.0> accepting AMQP connection <0.783.0> (10.0.0.33:53766 -> 10.0.0.33:5673)
2026-04-20 05:44:55.006 [info] <0.790.0> accepting AMQP connection <0.790.0> (10.0.0.33:53778 -> 10.0.0.33:5673)
2026-04-20 05:44:55.007 [info] <0.780.0> connection <0.780.0> (10.0.0.38:46302 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:44:55.007 [info] <0.797.0> accepting AMQP connection <0.797.0> (10.0.0.33:53792 -> 10.0.0.33:5673)
2026-04-20 05:44:55.007 [info] <0.767.0> connection <0.767.0> (10.0.0.50:42712 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:44:55.007 [info] <0.757.0> connection <0.757.0> (10.0.0.50:42696 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:44:55.008 [info] <0.783.0> connection <0.783.0> (10.0.0.33:53766 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:44:55.008 [info] <0.771.0> connection <0.771.0> (10.0.0.50:42722 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:44:55.009 [info] <0.797.0> connection <0.797.0> (10.0.0.33:53792 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:44:55.010 [info] <0.790.0> connection <0.790.0> (10.0.0.33:53778 -> 10.0.0.33:5673): user 'guest' authenticated and granted access to vhost '/'
2026-04-20 05:45:00.114 [info] <0.923.0> RabbitMQ is asked to stop...
2026-04-20 05:45:00.230 [info] <0.923.0> Stopping RabbitMQ applications and their dependencies in the following order:
    rabbitmq_management
    amqp_client
    rabbitmq_web_dispatch
    cowboy
    cowlib
    rabbitmq_management_agent
    rabbit
    mnesia
    rabbit_common
    sysmon_handler
    os_mon
2026-04-20 05:45:00.230 [info] <0.923.0> Stopping application 'rabbitmq_management'
2026-04-20 05:45:00.233 [warning] <0.597.0> RabbitMQ HTTP listener registry could not find context rabbitmq_management_tls
2026-04-20 05:45:00.235 [info] <0.43.0> Application rabbitmq_management exited with reason: stopped
2026-04-20 05:45:00.235 [info] <0.923.0> Stopping application 'amqp_client'
2026-04-20 05:45:00.237 [info] <0.923.0> Stopping application 'rabbitmq_web_dispatch'
2026-04-20 05:45:00.237 [info] <0.43.0> Application amqp_client exited with reason: stopped
2026-04-20 05:45:00.239 [info] <0.923.0> Stopping application 'cowboy'
2026-04-20 05:45:00.239 [info] <0.43.0> Application rabbitmq_web_dispatch exited with reason: stopped
2026-04-20 05:45:00.241 [info] <0.923.0> Stopping application 'cowlib'
2026-04-20 05:45:00.241 [info] <0.43.0> Application cowboy exited with reason: stopped
2026-04-20 05:45:00.241 [info] <0.43.0> Application cowlib exited with reason: stopped
2026-04-20 05:45:00.241 [info] <0.923.0> Stopping application 'rabbitmq_management_agent'
2026-04-20 05:45:00.243 [info] <0.43.0> Application rabbitmq_management_agent exited with reason: stopped
2026-04-20 05:45:00.243 [info] <0.923.0> Stopping application 'rabbit'
2026-04-20 05:45:00.243 [info] <0.240.0> Will unregister with peer discovery backend rabbit_peer_discovery_classic_config
2026-04-20 05:45:00.244 [info] <0.589.0> stopped TCP listener on [::]:5673
2026-04-20 05:45:00.244 [error] <0.767.0> Error on AMQP connection <0.767.0> (10.0.0.50:42712 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:45:00.244 [error] <0.783.0> Error on AMQP connection <0.783.0> (10.0.0.33:53766 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:45:00.244 [error] <0.780.0> Error on AMQP connection <0.780.0> (10.0.0.38:46302 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:45:00.245 [error] <0.757.0> Error on AMQP connection <0.757.0> (10.0.0.50:42696 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:45:00.245 [error] <0.764.0> Error on AMQP connection <0.764.0> (10.0.0.38:46298 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:45:00.245 [error] <0.771.0> Error on AMQP connection <0.771.0> (10.0.0.50:42722 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:45:00.245 [error] <0.761.0> Error on AMQP connection <0.761.0> (10.0.0.38:46284 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:45:00.245 [error] <0.790.0> Error on AMQP connection <0.790.0> (10.0.0.33:53778 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:45:00.245 [error] <0.797.0> Error on AMQP connection <0.797.0> (10.0.0.33:53792 -> 10.0.0.33:5673, vhost: '/', user: 'guest', state: running), channel 0:
 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-04-20 05:45:00.246 [error] <0.775.0> Supervisor {<0.775.0>,rabbit_channel_sup_sup} had child channel_sup started with rabbit_channel_sup:start_link() at undefined exit with reason shutdown in context shutdown_error
2026-04-20 05:45:00.254 [info] <0.513.0> Closing all connections in vhost '/' on node 'contrail@cn-jenkins-deploy-platform-ansible-os-5649-3' because the vhost is stopping
2026-04-20 05:45:00.254 [warning] <0.855.0> Mirrored queue 'contrail-dns.cn-jenkins-deploy-platform-ansible-os-5649-2' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available
2026-04-20 05:45:00.254 [warning] <0.852.0> Mirrored queue 'contrail-collector.cn-jenkins-deploy-platform-ansible-os-5649-2.' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available
2026-04-20 05:45:00.254 [warning] <0.870.0> Mirrored queue 'contrail-control.cn-jenkins-deploy-platform-ansible-os-5649-3' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available
2026-04-20 05:45:00.254 [warning] <0.830.0> Mirrored queue 'contrail-control.cn-jenkins-deploy-platform-ansible-os-5649-1' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available
2026-04-20 05:45:00.255 [warning] <0.876.0> Mirrored queue 'contrail-control.cn-jenkins-deploy-platform-ansible-os-5649-2' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available
2026-04-20 05:45:00.255 [warning] <0.859.0> Mirrored queue 'contrail-collector.cn-jenkins-deploy-platform-ansible-os-5649-3.' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available
2026-04-20 05:45:00.255 [warning] <0.863.0> Mirrored queue 'contrail-dns.cn-jenkins-deploy-platform-ansible-os-5649-3' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available
2026-04-20 05:45:00.255 [warning] <0.837.0> Mirrored queue 'contrail-collector.cn-jenkins-deploy-platform-ansible-os-5649-1.' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available
2026-04-20 05:45:00.255 [warning] <0.842.0> Mirrored queue 'contrail-dns.cn-jenkins-deploy-platform-ansible-os-5649-1' in vhost '/': Stopping all nodes on master shutdown since no synchronised slave is available
2026-04-20 05:45:00.256 [info] <0.543.0> Stopping message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent'
2026-04-20 05:45:00.284 [info] <0.543.0> Message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent' is stopped
2026-04-20 05:45:00.284 [info] <0.540.0> Stopping message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient'
2026-04-20 05:45:00.296 [info] <0.540.0> Message store for directory '/var/lib/rabbitmq/mnesia/contrail@cn-jenkins-deploy-platform-ansible-os-5649-3/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient' is stopped
2026-04-20 05:45:00.441 [info] <0.43.0> Application rabbit exited with reason: stopped
2026-04-20 05:45:00.441 [info] <0.923.0> Stopping application 'mnesia'
2026-04-20 05:45:00.443 [info] <0.43.0> Application mnesia exited with reason: stopped
2026-04-20 05:45:00.443 [info] <0.923.0> Stopping application 'rabbit_common'
2026-04-20 05:45:00.443 [info] <0.43.0> Application rabbit_common exited with reason: stopped
2026-04-20 05:45:00.443 [info] <0.923.0> Stopping application 'sysmon_handler'
2026-04-20 05:45:00.445 [info] <0.43.0> Application sysmon_handler exited with reason: stopped
2026-04-20 05:45:00.445 [info] <0.923.0> Stopping application 'os_mon'
2026-04-20 05:45:01.332 [warning]  lager_file_backend dropped 2 messages in the last second that exceeded the limit of 50 messages/sec