11/27/2025 02:00:38.125 7f2fd7d4d840 [contrail-database-nodemgr] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 11/27/2025 02:00:38.125 7f2fd7d4d840 [contrail-database-nodemgr] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 11/27/2025 02:00:38.125 7f2fd7d4d840 [contrail-database-nodemgr] [INFO]: SANDESH: Flow Logging: [None] -> [False] 11/27/2025 02:00:38.135 7f2fd5050360 [contrail-database-nodemgr] [ERROR]: Cannot write http_port 8103 to /tmp/contrail-database-nodemgr.0.http_port 11/27/2025 02:00:38.135 7f2fd5050360 [contrail-database-nodemgr] [ERROR]: Starting Introspect on HTTP Port 8103 11/27/2025 02:00:38.136 7f2fd5050720 [contrail-database-nodemgr] [INFO]: Processing event[EvStart] in state[none] 11/27/2025 02:00:38.137 7f2fd5050720 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 11/27/2025 02:00:38.137 7f2fd5050720 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/27/2025 02:00:38.137 7f2fd5050720 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 11/27/2025 02:00:38.137 7f2fd5050720 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/27/2025 02:00:38.138 7f2fd5050720 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4817-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/27/2025 02:00:38.138 7f2fd5050720 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/27/2025 02:00:38.138 7f2fd5050720 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/27/2025 02:00:38.138 7f2fd5050720 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4817-1.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1764208838137549 collector_name = collector_ip = collector_list = [ 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 11/27/2025 02:00:38.138 7f2fd5050720 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/27/2025 02:00:38.291 7f2fd7d4d840 [contrail-database-nodemgr] [DEBUG]: DBG: get_current_processes: '[{'name': 'cassandra', 'group': 'cassandra', 'pid': 12989326197423212839934474193088857156118449521262643170172618404707260753202, 'start': '1764208836000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'contrail-query-engine', 'group': 'contrail-query-engine', 'pid': 17286710462855039514018820035985870575656668448530774965960713266438321656043, 'start': '1764208836000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'contrail-database-nodemgr', 'group': 'contrail-database-nodemgr', 'pid': 45379472762256060282447266459696550450275201638003634764206054084048208119379, 'start': '1764208836000000', 'statename': 'PROCESS_STATE_RUNNING'}]' 11/27/2025 02:00:38.330 7f2fd4d5a900 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/27/2025 02:00:38.331 7f2fd4d5a900 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4817-1. build_info = {"build-info": [{"build-version": "v0+nightly", "build-time": "2025-11-27 00:50:11.016574", "build-user": "root", "build-hostname": "jenkins-build-prepare-sandbox-12893-1.novalocal", "build-id": "v0+nightly", "build-number": "202511270009"}]} system_cpu_info = << num_socket = 1 num_cpu = 8 num_core_per_socket = 8 num_thread_per_core = 1 >> >> 11/27/2025 02:00:38.331 7f2fd4d5a900 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/27/2025 02:00:38.334 7f2fd7d4d840 [contrail-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-4817-1.', _node_type='Database', _instance_id='0', _timestamp=1764208838334430, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-4817-1.', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-database-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=[''], status='Down', description='none to Idle on EvStart')]), flag_infos=None, description='Collector:Collector[none to Idle on EvStart] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 11/27/2025 02:00:38.335 7f2fd7d4d840 [contrail-database-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-4817-1.', _node_type='Database', _instance_id='0', _timestamp=1764208838334960, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-4817-1.', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='cassandra', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1764208836000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-query-engine', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1764208836000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-database-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1764208836000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info": [{"build-version": "v0+nightly", "build-time": "2025-11-27 00:50:11.016574", "build-user": "root", "build-hostname": "jenkins-build-prepare-sandbox-12893-1.novalocal", "build-id": "v0+nightly", "build-number": "202511270009"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 11/27/2025 02:00:38.337 7f2fd4d5a900 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/27/2025 02:00:38.338 7f2fd4d5a900 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4817-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/27/2025 02:00:38.338 7f2fd4d5a900 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/27/2025 02:00:38.338 7f2fd4d5a900 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/27/2025 02:00:38.338 7f2fd4d5a900 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4817-1. deleted = False process_info = [ << process_name = cassandra process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1764208836000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-query-engine process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1764208836000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-database-nodemgr process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1764208836000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, ] build_info = {"build-info": [{"build-version": "v0+nightly", "build-time": "2025-11-27 00:50:11.016574", "build-user": "root", "build-hostname": "jenkins-build-prepare-sandbox-12893-1.novalocal", "build-id": "v0+nightly", "build-number": "202511270009"}]} >> 11/27/2025 02:00:38.338 7f2fd4d5a900 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/27/2025 02:00:38.685 7f2fd7d4d840 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 11/27/2025 02:00:39.020 7f2fd7d4d840 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectDatabaseInfo:DatabaseUsage] map 11/27/2025 02:00:39.021 7f2fd4d340e0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/27/2025 02:00:39.022 7f2fd4d340e0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-ansible-os-4817-1. database_usage = [ << disk_space_used_1k = 16478616 disk_space_available_1k = 64611868 analytics_db_size_1k = 108 >>, ] >> 11/27/2025 02:00:39.022 7f2fd4d340e0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/27/2025 02:00:39.441 7f2fd7d4d840 [contrail-database-nodemgr] [ERROR]: Failed to connect to database by CQL and socket: Result: 1 11/27/2025 02:00:39.442 7f2fd7d4d840 [contrail-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-4817-1.', _node_type='Database', _instance_id='0', _timestamp=1764208839442001, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-4817-1.', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-database-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=[''], status='Down', description='none to Idle on EvStart')]), flag_infos=None, description='Cassandra state detected DOWN. Collector:Collector[none to Idle on EvStart] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 11/27/2025 02:00:39.443 7f2fd4d340e0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/27/2025 02:00:39.443 7f2fd4d340e0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4817-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Cassandra state detected DOWN. Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/27/2025 02:00:39.443 7f2fd4d340e0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/27/2025 02:00:41.910 7f2fd7d4d840 [contrail-database-nodemgr] [DEBUG]: Sending UVE: CassandraStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-4817-1.', _node_type='Database', _instance_id='0', _timestamp=1764208841909964, _versionsig=46427434, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=CassandraStatusData(name='cn-jenkins-deploy-platform-ansible-os-4817-1.', deleted=None, cassandra_compaction_task=CassandraCompactionTask(pending_compaction_tasks=0), thread_pool_stats=[CassandraThreadPoolStats(pool_name='ReadStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='CompactionExecutor', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MutationStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MemtableFlushWriter', active=0, pending=0, all_time_blocked=0)], _table='ObjectDatabaseInfo'), _seqnum=0) 11/27/2025 02:00:41.910 7f2fd7d4d840 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectDatabaseInfo:CassandraStatusUVE] map 11/27/2025 02:00:41.912 7f2fd5050d60 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/27/2025 02:00:41.913 7f2fd5050d60 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] CassandraStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4817-1. cassandra_compaction_task = << pending_compaction_tasks = 0 >> thread_pool_stats = [ << pool_name = ReadStage active = 0 pending = 0 all_time_blocked = 0 >>, << pool_name = CompactionExecutor active = 0 pending = 0 all_time_blocked = 0 >>, << pool_name = MutationStage active = 0 pending = 0 all_time_blocked = 0 >>, << pool_name = MemtableFlushWriter active = 0 pending = 0 all_time_blocked = 0 >>, ] >> 11/27/2025 02:00:41.913 7f2fd5050d60 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/27/2025 02:00:42.138 7f2fd4d34860 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/27/2025 02:00:42.141 7f2fd4d34860 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 11/27/2025 02:00:42.141 7f2fd4d34860 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/27/2025 02:00:42.142 7f2fd4d34860 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/27/2025 02:00:42.143 7f2fd4d34860 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4817-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/27/2025 02:00:42.143 7f2fd4d34860 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/27/2025 02:00:42.143 7f2fd4d34860 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/27/2025 02:00:42.143 7f2fd4d34860 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/27/2025 02:00:42.143 7f2fd4d34860 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/27/2025 02:00:42.144 7f2fd4d34860 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4817-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/27/2025 02:00:42.144 7f2fd4d34860 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/27/2025 02:00:42.144 7f2fd4d34860 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/27/2025 02:00:42.144 7f2fd4d34860 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4817-1.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1764208838137549 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/27/2025 02:00:42.144 7f2fd4d34860 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/27/2025 02:00:43.111 7f2fd7d4d840 [contrail-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7200 repair -pr ContrailAnalyticsCql'. Error is: Result: 1 11/27/2025 02:00:46.146 7f2fd4d34cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/27/2025 02:00:46.148 7f2fd4d34cc0 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 11/27/2025 02:00:46.148 7f2fd4d34cc0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/27/2025 02:00:46.148 7f2fd4d34cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/27/2025 02:00:46.148 7f2fd4d34cc0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4817-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/27/2025 02:00:46.148 7f2fd4d34cc0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/27/2025 02:00:46.148 7f2fd4d34cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/27/2025 02:00:46.149 7f2fd4d34cc0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/27/2025 02:00:46.149 7f2fd4d34cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/27/2025 02:00:46.149 7f2fd4d34cc0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4817-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/27/2025 02:00:46.149 7f2fd4d34cc0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/27/2025 02:00:46.149 7f2fd4d34cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/27/2025 02:00:46.149 7f2fd4d34cc0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4817-1.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1764208838137549 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/27/2025 02:00:46.149 7f2fd4d34cc0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/27/2025 02:00:48.152 7f2fd7d4d840 [contrail-database-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-4817-1.', _node_type='Database', _instance_id='0', _timestamp=1764208848152310, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-4817-1.', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='cassandra', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1764208836000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-query-engine', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1764208836000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-database-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1764208836000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info": [{"build-version": "v0+nightly", "build-time": "2025-11-27 00:50:11.016574", "build-user": "root", "build-hostname": "jenkins-build-prepare-sandbox-12893-1.novalocal", "build-id": "v0+nightly", "build-number": "202511270009"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 11/27/2025 02:00:48.152 7f2fd7d4d840 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-4817-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=16649144, partition_space_available_1k=64441340, percentage_partition_space_used=21, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=[], build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1307424, cpu_share=0.0, mem_res=1297572, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=7468, cpu_share=0.0, mem_res=5792, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=54536, cpu_share=0.0, mem_res=45160, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=7828644, free=7865596, buffers=670784, cached=16770340, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.6, five_min_avg=1.99, fifteen_min_avg=1.27, cpu_share=0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 11/27/2025 02:00:48.153 7f2fd5050cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/27/2025 02:00:48.153 7f2fd5050cc0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4817-1. deleted = False process_info = [ << process_name = cassandra process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1764208836000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-query-engine process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1764208836000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-database-nodemgr process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1764208836000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, ] build_info = {"build-info": [{"build-version": "v0+nightly", "build-time": "2025-11-27 00:50:11.016574", "build-user": "root", "build-hostname": "jenkins-build-prepare-sandbox-12893-1.novalocal", "build-id": "v0+nightly", "build-number": "202511270009"}]} >> 11/27/2025 02:00:48.153 7f2fd5050cc0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/27/2025 02:00:48.153 7f2fd5050cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/27/2025 02:00:48.153 7f2fd5050cc0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4817-1. disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 16649144 partition_space_available_1k = 64441340 percentage_partition_space_used = 21 >> } all_core_file_list = [ ] process_mem_cpu_usage = { cassandra : << mem_virt = 1307424 cpu_share = 0.0 mem_res = 1297572 >>contrail-query-engine : << mem_virt = 7468 cpu_share = 0.0 mem_res = 5792 >>contrail-database-nodemgr : << mem_virt = 54536 cpu_share = 0.0 mem_res = 45160 >> } system_mem_usage = << total = 32665120 used = 7828644 free = 7865596 buffers = 670784 cached = 16770340 node_type = database-node >> system_cpu_usage = << one_min_avg = 2.6 five_min_avg = 1.99 fifteen_min_avg = 1.27 cpu_share = 0 node_type = database-node >> >> 11/27/2025 02:00:48.153 7f2fd5050cc0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/27/2025 02:00:50.150 7f2fd5050cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/27/2025 02:00:50.152 7f2fd5050cc0 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 11/27/2025 02:00:50.152 7f2fd5050cc0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/27/2025 02:00:50.152 7f2fd5050cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/27/2025 02:00:50.152 7f2fd5050cc0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4817-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/27/2025 02:00:50.152 7f2fd5050cc0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/27/2025 02:00:50.152 7f2fd5050cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/27/2025 02:00:50.153 7f2fd5050cc0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/27/2025 02:00:50.153 7f2fd5050cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/27/2025 02:00:50.153 7f2fd5050cc0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4817-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/27/2025 02:00:50.153 7f2fd5050cc0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/27/2025 02:00:50.153 7f2fd5050cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/27/2025 02:00:50.153 7f2fd5050cc0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4817-1.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1764208838137549 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/27/2025 02:00:50.153 7f2fd5050cc0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/27/2025 02:00:54.153 7f2fd4d34c20 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/27/2025 02:00:54.154 7f2fd4d34c20 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 11/27/2025 02:00:54.154 7f2fd4d34c20 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/27/2025 02:00:54.154 7f2fd4d34c20 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/27/2025 02:00:54.155 7f2fd4d34c20 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4817-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/27/2025 02:00:54.155 7f2fd4d34c20 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/27/2025 02:00:54.155 7f2fd4d34c20 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/27/2025 02:00:54.155 7f2fd4d34c20 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/27/2025 02:00:54.155 7f2fd4d34c20 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/27/2025 02:00:54.156 7f2fd4d34c20 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4817-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/27/2025 02:00:54.156 7f2fd4d34c20 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/27/2025 02:00:54.156 7f2fd4d34c20 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/27/2025 02:00:54.156 7f2fd4d34c20 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4817-1.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1764208838137549 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/27/2025 02:00:54.156 7f2fd4d34c20 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/27/2025 02:00:58.157 7f2fd4d34c20 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/27/2025 02:00:58.158 7f2fd4d34c20 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 11/27/2025 02:00:58.158 7f2fd4d34c20 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/27/2025 02:00:58.159 7f2fd4d34c20 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/27/2025 02:00:58.159 7f2fd4d34c20 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4817-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/27/2025 02:00:58.159 7f2fd4d34c20 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/27/2025 02:00:58.159 7f2fd4d34c20 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/27/2025 02:00:58.159 7f2fd4d34c20 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/27/2025 02:00:58.159 7f2fd4d34c20 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/27/2025 02:00:58.159 7f2fd4d34c20 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4817-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/27/2025 02:00:58.159 7f2fd4d34c20 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/27/2025 02:00:58.160 7f2fd4d34c20 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/27/2025 02:00:58.160 7f2fd4d34c20 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4817-1.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1764208838137549 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/27/2025 02:00:58.160 7f2fd4d34c20 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/27/2025 02:01:02.160 7f2fd4d34c20 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/27/2025 02:01:02.161 7f2fd4d34c20 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 11/27/2025 02:01:02.162 7f2fd4d34c20 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/27/2025 02:01:02.162 7f2fd4d34c20 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/27/2025 02:01:02.162 7f2fd4d34c20 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4817-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/27/2025 02:01:02.162 7f2fd4d34c20 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/27/2025 02:01:02.162 7f2fd4d34c20 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/27/2025 02:01:02.162 7f2fd4d34c20 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/27/2025 02:01:02.162 7f2fd4d34c20 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/27/2025 02:01:02.163 7f2fd4d34c20 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4817-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/27/2025 02:01:02.163 7f2fd4d34c20 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/27/2025 02:01:02.163 7f2fd4d34c20 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/27/2025 02:01:02.163 7f2fd4d34c20 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4817-1.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1764208838137549 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/27/2025 02:01:02.163 7f2fd4d34c20 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/27/2025 02:01:06.164 7f2fd4d34c20 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/27/2025 02:01:06.165 7f2fd4d34c20 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 11/27/2025 02:01:06.165 7f2fd4d34c20 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/27/2025 02:01:06.165 7f2fd4d34c20 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/27/2025 02:01:06.166 7f2fd4d34c20 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4817-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/27/2025 02:01:06.166 7f2fd4d34c20 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/27/2025 02:01:06.166 7f2fd4d34c20 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 11/27/2025 02:01:06.166 7f2fd4d34c20 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 11/27/2025 02:01:06.166 7f2fd4d34c20 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/27/2025 02:01:06.167 7f2fd4d34c20 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4817-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 11/27/2025 02:01:06.167 7f2fd4d34c20 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/27/2025 02:01:06.167 7f2fd4d34c20 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 11/27/2025 02:01:06.167 7f2fd4d34c20 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4817-1.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1764208838137549 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 11/27/2025 02:01:06.167 7f2fd4d34c20 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 11/27/2025 02:01:10.167 7f2fd4d34c20 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 11/27/2025 02:01:10.168 7f2fd4d34c20 [contrail-database-nodemgr] [INFO]: Session Event: TCP Connected 11/27/2025 02:01:10.168 7f2fd4d34c20 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 11/27/2025 02:01:10.168 7f2fd4d34c20 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 11/27/2025 02:01:10.169 7f2fd4d34c20 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4817-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 11/27/2025 02:01:10.169 7f2fd4d34c20 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 11/27/2025 02:01:10.169 7f2fd4d34c20 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnected] in state[Connect] 11/27/2025 02:01:10.169 7f2fd4d34c20 [contrail-database-nodemgr] [DEBUG]: Send sandesh control message. uve type count # 6 11/27/2025 02:01:10.169 7f2fd4d34c20 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 11/27/2025 02:01:10.169 7f2fd4d34c20 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/27/2025 02:01:10.169 7f2fd4d34c20 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 11/27/2025 02:01:19.737 7f2fd4d34860 [contrail-database-nodemgr] [DEBUG]: Extracted sandesh length: 0000001016 11/27/2025 02:01:19.738 7f2fd4d34860 [contrail-database-nodemgr] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 11/27/2025 02:01:19.738 7f2fd5b724a0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 11/27/2025 02:01:19.738 7f2fd5b724a0 [contrail-database-nodemgr] [DEBUG]: Number of uve types in sandesh control message is 0 11/27/2025 02:01:19.739 7f2fd5b724a0 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:DatabaseUsage] map 11/27/2025 02:01:19.739 7f2fd5b724a0 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:CassandraStatusUVE] map 11/27/2025 02:01:19.739 7f2fd5b724a0 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 11/27/2025 02:01:19.739 7f2fd5b724a0 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/27/2025 02:01:19.739 7f2fd5b724a0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 11/27/2025 02:01:19.740 7f2fd4d34cc0 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/27/2025 02:01:19.741 7f2fd4d34cc0 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/27/2025 02:01:29.883 7f2fd4d34860 [contrail-database-nodemgr] [DEBUG]: Extracted sandesh length: 0000001582 11/27/2025 02:01:29.884 7f2fd4d34860 [contrail-database-nodemgr] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 11/27/2025 02:01:29.884 7f2fd5aacea0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[Established] 11/27/2025 02:01:29.884 7f2fd5aacea0 [contrail-database-nodemgr] [DEBUG]: Number of uve types in sandesh control message is 4 11/27/2025 02:01:29.884 7f2fd5aacea0 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:DatabaseUsage] map 11/27/2025 02:01:29.884 7f2fd5aacea0 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:CassandraStatusUVE] map 11/27/2025 02:01:29.885 7f2fd5aacea0 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 11/27/2025 02:01:29.885 7f2fd5aacea0 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 11/27/2025 02:01:29.885 7f2fd4d340e0 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 11/27/2025 02:01:29.886 7f2fd4d340e0 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 11/27/2025 02:01:38.600 7f2fd7d4d840 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 11/27/2025 02:01:39.609 7f2fd7d4d840 [contrail-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-4817-1.', _node_type='Database', _instance_id='0', _timestamp=1764208899609194, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-4817-1.', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-database-nodemgr', instance_id='0', state='Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=['10.0.0.22:8086'], status='Up', description='ClientInit to Established on EvSandeshCtrlMessageRecv')]), flag_infos=None, description='')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 11/27/2025 02:01:41.862 7f2fd7d4d840 [contrail-database-nodemgr] [DEBUG]: Sending UVE: CassandraStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-4817-1.', _node_type='Database', _instance_id='0', _timestamp=1764208901862424, _versionsig=46427434, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=CassandraStatusData(name='cn-jenkins-deploy-platform-ansible-os-4817-1.', deleted=None, cassandra_compaction_task=CassandraCompactionTask(pending_compaction_tasks=0), thread_pool_stats=[CassandraThreadPoolStats(pool_name='ReadStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='CompactionExecutor', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MutationStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='Native-Transport-Requests', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MemtableFlushWriter', active=0, pending=0, all_time_blocked=0)], _table='ObjectDatabaseInfo'), _seqnum=0) 11/27/2025 02:01:47.816 7f2fd7d4d840 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-4817-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=17719300, partition_space_available_1k=63371184, percentage_partition_space_used=22, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1341884, cpu_share=0.37, mem_res=1326300, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=11644, cpu_share=0.0, mem_res=9212, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=54688, cpu_share=0.02, mem_res=45280, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=9384976, free=4917052, buffers=698968, cached=18135656, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.61, five_min_avg=2.35, fifteen_min_avg=1.44, cpu_share=4.25, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 11/27/2025 02:02:38.825 7f2fd7d4d840 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 11/27/2025 02:02:47.830 7f2fd7d4d840 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-4817-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=19532152, partition_space_available_1k=61558332, percentage_partition_space_used=24, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1362744, cpu_share=0.13, mem_res=1331644, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=11644, cpu_share=0.0, mem_res=9588, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=47332, cpu_share=0.02, mem_res=45280, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=9357400, free=18420232, buffers=709756, cached=4648920, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.36, five_min_avg=2.52, fifteen_min_avg=1.56, cpu_share=4.62, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 11/27/2025 02:03:38.606 7f2fd7d4d840 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 11/27/2025 02:03:46.836 7f2fd7d4d840 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-4817-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=19541972, partition_space_available_1k=61548512, percentage_partition_space_used=24, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1334208, cpu_share=0.1, mem_res=1318296, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=11748, cpu_share=0.0, mem_res=9692, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=47372, cpu_share=0.02, mem_res=45280, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=9223372, free=18409436, buffers=712840, cached=4790544, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.78, five_min_avg=2.51, fifteen_min_avg=1.62, cpu_share=3.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 11/27/2025 02:04:10.982 7f2fd5050cc0 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 11/27/2025 02:04:23.346 7f2fd5050cc0 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 11/27/2025 02:04:25.625 7f2fd5050cc0 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 11/27/2025 02:04:38.628 7f2fd7d4d840 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 11/27/2025 02:04:46.850 7f2fd7d4d840 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-4817-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=19730312, partition_space_available_1k=61360172, percentage_partition_space_used=24, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1396944, cpu_share=0.13, mem_res=1379416, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=12036, cpu_share=0.0, mem_res=9976, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=47372, cpu_share=0.02, mem_res=45280, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=9981200, free=17529196, buffers=718696, cached=4906220, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.86, five_min_avg=2.58, fifteen_min_avg=1.7, cpu_share=2.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 11/27/2025 02:05:33.845 7f2fd5050ae0 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 11/27/2025 02:05:38.646 7f2fd7d4d840 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 11/27/2025 02:05:46.870 7f2fd7d4d840 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-4817-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=20997384, partition_space_available_1k=60093100, percentage_partition_space_used=26, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1354176, cpu_share=0.13, mem_res=1334416, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=12248, cpu_share=0.0, mem_res=10184, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=47412, cpu_share=0.02, mem_res=45280, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=10336156, free=15762080, buffers=753236, cached=6285688, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.83, five_min_avg=2.9, fifteen_min_avg=1.86, cpu_share=4.25, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 11/27/2025 02:06:38.633 7f2fd7d4d840 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 11/27/2025 02:06:46.887 7f2fd7d4d840 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-4817-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=21043804, partition_space_available_1k=60046680, percentage_partition_space_used=26, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1396152, cpu_share=0.2, mem_res=1369456, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=16772, cpu_share=0.0, mem_res=14616, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=47412, cpu_share=0.02, mem_res=45280, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=10609568, free=15409680, buffers=753928, cached=6361952, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.46, five_min_avg=2.95, fifteen_min_avg=1.95, cpu_share=4.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 11/27/2025 02:07:38.705 7f2fd7d4d840 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 11/27/2025 02:07:47.894 7f2fd7d4d840 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-4817-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=21154020, partition_space_available_1k=59936464, percentage_partition_space_used=26, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1420948, cpu_share=0.16, mem_res=1389516, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=16776, cpu_share=0.0, mem_res=14624, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=47412, cpu_share=0.02, mem_res=45280, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=11276976, free=14628400, buffers=754012, cached=6475920, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.35, five_min_avg=3.26, fifteen_min_avg=2.11, cpu_share=5.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 11/27/2025 02:08:38.774 7f2fd7d4d840 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 11/27/2025 02:08:47.906 7f2fd7d4d840 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-4817-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=21182064, partition_space_available_1k=59908420, percentage_partition_space_used=26, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1410476, cpu_share=0.15, mem_res=1375412, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=16844, cpu_share=0.0, mem_res=14692, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=47416, cpu_share=0.02, mem_res=45280, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=10752104, free=15106432, buffers=754568, cached=6521948, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=5.11, five_min_avg=3.62, fifteen_min_avg=2.31, cpu_share=4.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 11/27/2025 02:09:38.654 7f2fd7d4d840 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 11/27/2025 02:09:46.898 7f2fd7d4d840 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-4817-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=21209796, partition_space_available_1k=59880688, percentage_partition_space_used=26, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1422692, cpu_share=0.15, mem_res=1383128, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=16900, cpu_share=0.0, mem_res=14748, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=47416, cpu_share=0.02, mem_res=45280, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=10987972, free=14835836, buffers=754648, cached=6555952, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.81, five_min_avg=3.7, fifteen_min_avg=2.41, cpu_share=4.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 11/27/2025 02:10:38.710 7f2fd7d4d840 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 11/27/2025 02:10:46.907 7f2fd7d4d840 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-4817-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=21302996, partition_space_available_1k=59787488, percentage_partition_space_used=26, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1436392, cpu_share=0.13, mem_res=1393420, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=16904, cpu_share=0.0, mem_res=14752, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=47424, cpu_share=0.02, mem_res=45280, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=11801400, free=13977368, buffers=754752, cached=6601048, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.82, five_min_avg=3.6, fifteen_min_avg=2.46, cpu_share=4.88, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 11/27/2025 02:11:38.701 7f2fd7d4d840 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 11/27/2025 02:11:46.932 7f2fd7d4d840 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-4817-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=21299292, partition_space_available_1k=59791192, percentage_partition_space_used=26, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1451032, cpu_share=0.14, mem_res=1404008, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=16956, cpu_share=0.0, mem_res=14804, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=47424, cpu_share=0.02, mem_res=45280, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=11115344, free=14633812, buffers=754808, cached=6630392, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.5, five_min_avg=3.53, fifteen_min_avg=2.5, cpu_share=4.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 11/27/2025 02:12:38.755 7f2fd7d4d840 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 11/27/2025 02:12:47.922 7f2fd7d4d840 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-4817-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=21321724, partition_space_available_1k=59768760, percentage_partition_space_used=26, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1456988, cpu_share=0.1, mem_res=1407440, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=17028, cpu_share=0.0, mem_res=14876, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=47424, cpu_share=0.02, mem_res=45280, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=11400064, free=14325956, buffers=754868, cached=6653604, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=5.83, five_min_avg=4.07, fifteen_min_avg=2.74, cpu_share=4.12, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 11/27/2025 02:13:38.702 7f2fd7d4d840 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 11/27/2025 02:13:46.929 7f2fd7d4d840 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-4817-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=21336352, partition_space_available_1k=59754132, percentage_partition_space_used=26, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1444876, cpu_share=0.11, mem_res=1392608, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=17052, cpu_share=0.0, mem_res=14900, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=47444, cpu_share=0.02, mem_res=45280, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=11031096, free=14673316, buffers=754900, cached=6675176, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.24, five_min_avg=3.92, fifteen_min_avg=2.78, cpu_share=4.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 11/27/2025 02:14:38.691 7f2fd7d4d840 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 11/27/2025 02:14:47.945 7f2fd7d4d840 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-4817-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=21439164, partition_space_available_1k=59651320, percentage_partition_space_used=26, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1455972, cpu_share=0.12, mem_res=1400068, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=17164, cpu_share=0.0, mem_res=15012, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=47432, cpu_share=0.02, mem_res=45280, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=11010228, free=14587304, buffers=754992, cached=6780744, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=5.49, five_min_avg=4.47, fifteen_min_avg=3.05, cpu_share=5.62, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 11/27/2025 02:15:38.692 7f2fd7d4d840 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 11/27/2025 02:15:46.941 7f2fd7d4d840 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-4817-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=21449248, partition_space_available_1k=59641236, percentage_partition_space_used=26, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1465520, cpu_share=0.1, mem_res=1407496, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=17208, cpu_share=0.0, mem_res=15056, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=47432, cpu_share=0.02, mem_res=45280, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=11060952, free=14527260, buffers=755004, cached=6790244, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.56, five_min_avg=4.1, fifteen_min_avg=3.01, cpu_share=2.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 11/27/2025 02:16:32.193 7f2fd5050d60 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 11/27/2025 02:16:38.726 7f2fd7d4d840 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 11/27/2025 02:16:46.968 7f2fd7d4d840 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-4817-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=21456324, partition_space_available_1k=59634160, percentage_partition_space_used=26, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1454864, cpu_share=0.12, mem_res=1393448, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=17312, cpu_share=0.0, mem_res=15160, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=47444, cpu_share=0.02, mem_res=45280, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=10936132, free=14629468, buffers=755060, cached=6812760, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.14, five_min_avg=3.88, fifteen_min_avg=3.0, cpu_share=3.62, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 11/27/2025 02:17:22.403 7f2fd4d34680 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map