2026-04-06 03:29:14,642 - INFO - Domain Default found not creating 2026-04-06 03:29:14,800 - INFO - Project ctest-TestSvcHC-79433380 not found, creating it 2026-04-06 03:29:15,384 - INFO - Created Project:ctest-TestSvcHC-79433380, ID : 33871f1f-4880-4200-9d82-ae1986cb4ea0 2026-04-06 03:29:16,850 - INFO - ================================================================================ 2026-04-06 03:29:16,851 - INFO - STARTING TEST : test_svc_hc_basic 2026-04-06 03:29:18,129 - DEBUG - Nothing to compare xmpp stats {'10.0.0.19': {'10.20.0.193': '0', '10.20.0.129': '0'}, '10.0.0.241': {'10.20.0.18': '0', '10.20.0.129': '0'}} with 2026-04-06 03:29:18,130 - INFO - Initial checks done. Running the testcase now 2026-04-06 03:29:18,130 - INFO - 2026-04-06 03:29:18,769 - DEBUG - Response for create_network : {'network': {'id': '129f5609-753c-4f08-a2bf-74cecb3112f3', 'name': 'ctest-mgmt_vn-11503117', 'tenant_id': '33871f1f488042009d82ae1986cb4ea0', 'project_id': '33871f1f488042009d82ae1986cb4ea0', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-TestSvcHC-79433380', 'ctest-mgmt_vn-11503117'], 'port_security_enabled': True, 'description': ''}} 2026-04-06 03:29:19,068 - DEBUG - Response for create_subnet : {'subnet': {'id': '8c1eacc5-49a9-44b2-9898-4871b9e634bc', 'name': '', 'tenant_id': '33871f1f488042009d82ae1986cb4ea0', 'network_id': '129f5609-753c-4f08-a2bf-74cecb3112f3', 'ip_version': 4, 'cidr': '4.198.159.192/26', 'allocation_pools': [{'start': '4.198.159.194', 'end': '4.198.159.254'}], 'gateway_ip': '4.198.159.193', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '4.198.159.194', 'tags': [], 'project_id': '33871f1f488042009d82ae1986cb4ea0'}} 2026-04-06 03:29:19,091 - INFO - Created VN ctest-mgmt_vn-11503117 2026-04-06 03:29:19,101 - DEBUG - VN ctest-mgmt_vn-11503117 UUID is 129f5609-753c-4f08-a2bf-74cecb3112f3 2026-04-06 03:29:19,165 - DEBUG - Requesting: http://10.0.0.249:8082/domains 2026-04-06 03:29:19,456 - DEBUG - Requesting: http://10.0.0.249:8082/domains 2026-04-06 03:29:19,484 - DEBUG - Requesting: http://10.0.0.249:8082/domain/500738d8-636d-4330-98c4-d5dafb5cbb39 2026-04-06 03:29:19,497 - DEBUG - Requesting: http://10.0.0.249:8082/project/33871f1f-4880-4200-9d82-ae1986cb4ea0 2026-04-06 03:29:19,597 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/129f5609-753c-4f08-a2bf-74cecb3112f3 2026-04-06 03:29:19,607 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/129f5609-753c-4f08-a2bf-74cecb3112f3 2026-04-06 03:29:19,619 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/81a386e4-65c4-4bff-aec4-2b74cdd0a4ba 2026-04-06 03:29:19,625 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/81a386e4-65c4-4bff-aec4-2b74cdd0a4ba 2026-04-06 03:29:19,632 - DEBUG - Requesting: http://10.0.0.249:8082/route-target/d1b48f1a-befd-4dca-9e94-1792a9acabd5 2026-04-06 03:29:19,639 - DEBUG - Route Targets: ['target:64512:8000008'] 2026-04-06 03:29:19,639 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/129f5609-753c-4f08-a2bf-74cecb3112f3 2026-04-06 03:29:19,647 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/81a386e4-65c4-4bff-aec4-2b74cdd0a4ba 2026-04-06 03:29:19,662 - INFO - Verified VN network id 15 for VN 129f5609-753c-4f08-a2bf-74cecb3112f3 2026-04-06 03:29:19,662 - INFO - Verifications in API Server for VN ctest-mgmt_vn-11503117 passed 2026-04-06 03:29:19,662 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/129f5609-753c-4f08-a2bf-74cecb3112f3 2026-04-06 03:29:19,671 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/81a386e4-65c4-4bff-aec4-2b74cdd0a4ba 2026-04-06 03:29:19,678 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/81a386e4-65c4-4bff-aec4-2b74cdd0a4ba 2026-04-06 03:29:19,687 - DEBUG - Requesting: http://10.0.0.249:8082/route-target/d1b48f1a-befd-4dca-9e94-1792a9acabd5 2026-04-06 03:29:19,699 - DEBUG - Control-node 10.0.0.249 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestSvcHC-79433380:ctest-mgmt_vn-11503117', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestSvcHC-79433380', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '1341885812601278216', 'uuid-lslong': '11727220386217136883'}, 'enable': 'true', 'created': '2026-04-06T03:29:18', 'last-modified': '2026-04-06T03:29:18', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.690198'} 2026-04-06 03:29:19,704 - DEBUG - Route Targets: ['target:64512:8000008'] 2026-04-06 03:29:19,709 - DEBUG - Control-node 10.0.0.27 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestSvcHC-79433380:ctest-mgmt_vn-11503117', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestSvcHC-79433380', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '1341885812601278216', 'uuid-lslong': '11727220386217136883'}, 'enable': 'true', 'created': '2026-04-06T03:29:18', 'last-modified': '2026-04-06T03:29:18', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.702080'} 2026-04-06 03:29:19,716 - DEBUG - Route Targets: ['target:64512:8000008'] 2026-04-06 03:29:19,721 - DEBUG - Control-node 10.0.0.33 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestSvcHC-79433380:ctest-mgmt_vn-11503117', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestSvcHC-79433380', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '1341885812601278216', 'uuid-lslong': '11727220386217136883'}, 'enable': 'true', 'created': '2026-04-06T03:29:18', 'last-modified': '2026-04-06T03:29:18', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.701591'} 2026-04-06 03:29:19,728 - DEBUG - Route Targets: ['target:64512:8000008'] 2026-04-06 03:29:19,728 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-mgmt_vn-11503117 passed 2026-04-06 03:29:19,728 - DEBUG - ====Verifying policy data for ctest-mgmt_vn-11503117 in API_Server ====== 2026-04-06 03:29:19,728 - DEBUG - Requesting: http://10.0.0.249:8082/domains 2026-04-06 03:29:19,733 - DEBUG - Requesting: http://10.0.0.249:8082/domain/500738d8-636d-4330-98c4-d5dafb5cbb39 2026-04-06 03:29:19,746 - DEBUG - Requesting: http://10.0.0.249:8082/project/33871f1f-4880-4200-9d82-ae1986cb4ea0 2026-04-06 03:29:19,836 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/129f5609-753c-4f08-a2bf-74cecb3112f3 2026-04-06 03:29:19,844 - DEBUG - =>VN ctest-mgmt_vn-11503117 has no policy to be verified 2026-04-06 03:29:19,844 - DEBUG - Verifying the vn in opserver 2026-04-06 03:29:19,844 - DEBUG - Verifying the default-domain:ctest-TestSvcHC-79433380:ctest-mgmt_vn-11503117 virtual network link through opserver 10.0.0.249 2026-04-06 03:29:19,844 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/virtual-networks 2026-04-06 03:29:20,135 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/virtual-networks 2026-04-06 03:29:20,209 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestSvcHC-79433380:ctest-mgmt_vn-11503117', 'href': 'http://10.0.0.249:8081/analytics/uves/virtual-network/default-domain:ctest-TestSvcHC-79433380:ctest-mgmt_vn-11503117?flat'} 2026-04-06 03:29:20,209 - DEBUG - Verifying the default-domain:ctest-TestSvcHC-79433380:ctest-mgmt_vn-11503117 virtual network link through opserver 10.0.0.27 2026-04-06 03:29:20,209 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-networks 2026-04-06 03:29:20,516 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-networks 2026-04-06 03:29:20,592 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestSvcHC-79433380:ctest-mgmt_vn-11503117', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestSvcHC-79433380:ctest-mgmt_vn-11503117?flat'} 2026-04-06 03:29:20,592 - DEBUG - Verifying the default-domain:ctest-TestSvcHC-79433380:ctest-mgmt_vn-11503117 virtual network link through opserver 10.0.0.33 2026-04-06 03:29:20,593 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-networks 2026-04-06 03:29:20,888 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-networks 2026-04-06 03:29:20,957 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestSvcHC-79433380:ctest-mgmt_vn-11503117', 'href': 'http://10.0.0.33:8081/analytics/uves/virtual-network/default-domain:ctest-TestSvcHC-79433380:ctest-mgmt_vn-11503117?flat'} 2026-04-06 03:29:20,958 - INFO - Validated that VN default-domain:ctest-TestSvcHC-79433380:ctest-mgmt_vn-11503117 is found in opserver 2026-04-06 03:29:20,977 - DEBUG - Do not have enough data to verify VN in agent 2026-04-06 03:29:20,986 - DEBUG - VRF ids for VN ctest-mgmt_vn-11503117: {} 2026-04-06 03:29:21,256 - DEBUG - Response for create_network : {'network': {'id': '4afce4a4-c93c-4bc1-a6d5-c15f7258bfc8', 'name': 'ctest-left_vn-03129979', 'tenant_id': '33871f1f488042009d82ae1986cb4ea0', 'project_id': '33871f1f488042009d82ae1986cb4ea0', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-TestSvcHC-79433380', 'ctest-left_vn-03129979'], 'port_security_enabled': True, 'description': ''}} 2026-04-06 03:29:21,489 - DEBUG - Response for create_subnet : {'subnet': {'id': '8d2b6f5d-2067-417e-9d9a-32f5e7481d15', 'name': '', 'tenant_id': '33871f1f488042009d82ae1986cb4ea0', 'network_id': '4afce4a4-c93c-4bc1-a6d5-c15f7258bfc8', 'ip_version': 4, 'cidr': '99.211.147.64/26', 'allocation_pools': [{'start': '99.211.147.66', 'end': '99.211.147.126'}], 'gateway_ip': '99.211.147.65', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '99.211.147.66', 'tags': [], 'project_id': '33871f1f488042009d82ae1986cb4ea0'}} 2026-04-06 03:29:21,510 - INFO - Created VN ctest-left_vn-03129979 2026-04-06 03:29:21,521 - DEBUG - VN ctest-left_vn-03129979 UUID is 4afce4a4-c93c-4bc1-a6d5-c15f7258bfc8 2026-04-06 03:29:21,598 - DEBUG - Requesting: http://10.0.0.249:8082/domains 2026-04-06 03:29:21,603 - DEBUG - Requesting: http://10.0.0.249:8082/domain/500738d8-636d-4330-98c4-d5dafb5cbb39 2026-04-06 03:29:21,615 - DEBUG - Requesting: http://10.0.0.249:8082/project/33871f1f-4880-4200-9d82-ae1986cb4ea0 2026-04-06 03:29:21,707 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/4afce4a4-c93c-4bc1-a6d5-c15f7258bfc8 2026-04-06 03:29:21,718 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/4afce4a4-c93c-4bc1-a6d5-c15f7258bfc8 2026-04-06 03:29:21,727 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/f0228a8f-b431-4e53-b08f-226dc7c892a6 2026-04-06 03:29:21,737 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/f0228a8f-b431-4e53-b08f-226dc7c892a6 2026-04-06 03:29:21,755 - DEBUG - Requesting: http://10.0.0.249:8082/route-target/9ee35ab4-0acf-4de0-be77-daf639a9a751 2026-04-06 03:29:21,764 - DEBUG - Route Targets: ['target:64512:8000009'] 2026-04-06 03:29:21,764 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/4afce4a4-c93c-4bc1-a6d5-c15f7258bfc8 2026-04-06 03:29:21,778 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/f0228a8f-b431-4e53-b08f-226dc7c892a6 2026-04-06 03:29:21,795 - INFO - Verified VN network id 16 for VN 4afce4a4-c93c-4bc1-a6d5-c15f7258bfc8 2026-04-06 03:29:21,795 - INFO - Verifications in API Server for VN ctest-left_vn-03129979 passed 2026-04-06 03:29:21,795 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/4afce4a4-c93c-4bc1-a6d5-c15f7258bfc8 2026-04-06 03:29:21,806 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/f0228a8f-b431-4e53-b08f-226dc7c892a6 2026-04-06 03:29:21,814 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/f0228a8f-b431-4e53-b08f-226dc7c892a6 2026-04-06 03:29:21,821 - DEBUG - Requesting: http://10.0.0.249:8082/route-target/9ee35ab4-0acf-4de0-be77-daf639a9a751 2026-04-06 03:29:21,833 - DEBUG - Control-node 10.0.0.249 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestSvcHC-79433380:ctest-left_vn-03129979', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestSvcHC-79433380', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '5403445049339694017', 'uuid-lslong': '12021727396019879880'}, 'enable': 'true', 'created': '2026-04-06T03:29:21', 'last-modified': '2026-04-06T03:29:21', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.403642'} 2026-04-06 03:29:21,839 - DEBUG - Route Targets: ['target:64512:8000009'] 2026-04-06 03:29:21,846 - DEBUG - Control-node 10.0.0.27 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestSvcHC-79433380:ctest-left_vn-03129979', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestSvcHC-79433380', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '5403445049339694017', 'uuid-lslong': '12021727396019879880'}, 'enable': 'true', 'created': '2026-04-06T03:29:21', 'last-modified': '2026-04-06T03:29:21', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.412269'} 2026-04-06 03:29:21,852 - DEBUG - Route Targets: ['target:64512:8000009'] 2026-04-06 03:29:21,857 - DEBUG - Control-node 10.0.0.33 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestSvcHC-79433380:ctest-left_vn-03129979', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestSvcHC-79433380', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '5403445049339694017', 'uuid-lslong': '12021727396019879880'}, 'enable': 'true', 'created': '2026-04-06T03:29:21', 'last-modified': '2026-04-06T03:29:21', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.426513'} 2026-04-06 03:29:21,862 - DEBUG - Route Targets: ['target:64512:8000009'] 2026-04-06 03:29:21,862 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-left_vn-03129979 passed 2026-04-06 03:29:21,863 - DEBUG - ====Verifying policy data for ctest-left_vn-03129979 in API_Server ====== 2026-04-06 03:29:21,863 - DEBUG - Requesting: http://10.0.0.249:8082/domains 2026-04-06 03:29:21,869 - DEBUG - Requesting: http://10.0.0.249:8082/domain/500738d8-636d-4330-98c4-d5dafb5cbb39 2026-04-06 03:29:21,881 - DEBUG - Requesting: http://10.0.0.249:8082/project/33871f1f-4880-4200-9d82-ae1986cb4ea0 2026-04-06 03:29:21,991 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/4afce4a4-c93c-4bc1-a6d5-c15f7258bfc8 2026-04-06 03:29:22,003 - DEBUG - =>VN ctest-left_vn-03129979 has no policy to be verified 2026-04-06 03:29:22,003 - DEBUG - Verifying the vn in opserver 2026-04-06 03:29:22,003 - DEBUG - Verifying the default-domain:ctest-TestSvcHC-79433380:ctest-left_vn-03129979 virtual network link through opserver 10.0.0.249 2026-04-06 03:29:22,004 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/virtual-networks 2026-04-06 03:29:22,067 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestSvcHC-79433380:ctest-left_vn-03129979', 'href': 'http://10.0.0.249:8081/analytics/uves/virtual-network/default-domain:ctest-TestSvcHC-79433380:ctest-left_vn-03129979?flat'} 2026-04-06 03:29:22,067 - DEBUG - Verifying the default-domain:ctest-TestSvcHC-79433380:ctest-left_vn-03129979 virtual network link through opserver 10.0.0.27 2026-04-06 03:29:22,068 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-networks 2026-04-06 03:29:22,128 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestSvcHC-79433380:ctest-left_vn-03129979', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestSvcHC-79433380:ctest-left_vn-03129979?flat'} 2026-04-06 03:29:22,128 - DEBUG - Verifying the default-domain:ctest-TestSvcHC-79433380:ctest-left_vn-03129979 virtual network link through opserver 10.0.0.33 2026-04-06 03:29:22,128 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-networks 2026-04-06 03:29:22,193 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestSvcHC-79433380:ctest-left_vn-03129979', 'href': 'http://10.0.0.33:8081/analytics/uves/virtual-network/default-domain:ctest-TestSvcHC-79433380:ctest-left_vn-03129979?flat'} 2026-04-06 03:29:22,193 - INFO - Validated that VN default-domain:ctest-TestSvcHC-79433380:ctest-left_vn-03129979 is found in opserver 2026-04-06 03:29:22,222 - DEBUG - Do not have enough data to verify VN in agent 2026-04-06 03:29:22,234 - DEBUG - VRF ids for VN ctest-left_vn-03129979: {} 2026-04-06 03:29:22,514 - DEBUG - Response for create_network : {'network': {'id': '65a167ec-4f64-481d-a2b9-ea0426392fe6', 'name': 'ctest-right_vn-28490466', 'tenant_id': '33871f1f488042009d82ae1986cb4ea0', 'project_id': '33871f1f488042009d82ae1986cb4ea0', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-TestSvcHC-79433380', 'ctest-right_vn-28490466'], 'port_security_enabled': True, 'description': ''}} 2026-04-06 03:29:22,729 - DEBUG - Response for create_subnet : {'subnet': {'id': '5675c32a-ec04-4f5b-96b1-1269201781a7', 'name': '', 'tenant_id': '33871f1f488042009d82ae1986cb4ea0', 'network_id': '65a167ec-4f64-481d-a2b9-ea0426392fe6', 'ip_version': 4, 'cidr': '178.43.96.64/26', 'allocation_pools': [{'start': '178.43.96.66', 'end': '178.43.96.126'}], 'gateway_ip': '178.43.96.65', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '178.43.96.66', 'tags': [], 'project_id': '33871f1f488042009d82ae1986cb4ea0'}} 2026-04-06 03:29:22,749 - INFO - Created VN ctest-right_vn-28490466 2026-04-06 03:29:22,764 - DEBUG - VN ctest-right_vn-28490466 UUID is 65a167ec-4f64-481d-a2b9-ea0426392fe6 2026-04-06 03:29:22,836 - DEBUG - Requesting: http://10.0.0.249:8082/domains 2026-04-06 03:29:22,843 - DEBUG - Requesting: http://10.0.0.249:8082/domain/500738d8-636d-4330-98c4-d5dafb5cbb39 2026-04-06 03:29:22,857 - DEBUG - Requesting: http://10.0.0.249:8082/project/33871f1f-4880-4200-9d82-ae1986cb4ea0 2026-04-06 03:29:22,954 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/65a167ec-4f64-481d-a2b9-ea0426392fe6 2026-04-06 03:29:22,968 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/65a167ec-4f64-481d-a2b9-ea0426392fe6 2026-04-06 03:29:22,979 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/2b3de5d6-2b4c-4e0c-afff-421e075fd162 2026-04-06 03:29:22,987 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/2b3de5d6-2b4c-4e0c-afff-421e075fd162 2026-04-06 03:29:22,996 - DEBUG - Requesting: http://10.0.0.249:8082/route-target/38602ca4-c92e-4996-9d61-dfa3b48106f9 2026-04-06 03:29:23,005 - DEBUG - Route Targets: ['target:64512:8000010'] 2026-04-06 03:29:23,005 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/65a167ec-4f64-481d-a2b9-ea0426392fe6 2026-04-06 03:29:23,016 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/2b3de5d6-2b4c-4e0c-afff-421e075fd162 2026-04-06 03:29:23,034 - INFO - Verified VN network id 17 for VN 65a167ec-4f64-481d-a2b9-ea0426392fe6 2026-04-06 03:29:23,035 - INFO - Verifications in API Server for VN ctest-right_vn-28490466 passed 2026-04-06 03:29:23,035 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/65a167ec-4f64-481d-a2b9-ea0426392fe6 2026-04-06 03:29:23,046 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/2b3de5d6-2b4c-4e0c-afff-421e075fd162 2026-04-06 03:29:23,053 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/2b3de5d6-2b4c-4e0c-afff-421e075fd162 2026-04-06 03:29:23,062 - DEBUG - Requesting: http://10.0.0.249:8082/route-target/38602ca4-c92e-4996-9d61-dfa3b48106f9 2026-04-06 03:29:23,074 - DEBUG - Control-node 10.0.0.249 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestSvcHC-79433380:ctest-right_vn-28490466', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestSvcHC-79433380', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '7323248733723052061', 'uuid-lslong': '11725660408377847782'}, 'enable': 'true', 'created': '2026-04-06T03:29:22', 'last-modified': '2026-04-06T03:29:22', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.391128'} 2026-04-06 03:29:23,081 - DEBUG - Route Targets: ['target:64512:8000010'] 2026-04-06 03:29:23,086 - DEBUG - Control-node 10.0.0.27 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestSvcHC-79433380:ctest-right_vn-28490466', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestSvcHC-79433380', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '7323248733723052061', 'uuid-lslong': '11725660408377847782'}, 'enable': 'true', 'created': '2026-04-06T03:29:22', 'last-modified': '2026-04-06T03:29:22', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.404384'} 2026-04-06 03:29:23,094 - DEBUG - Route Targets: ['target:64512:8000010'] 2026-04-06 03:29:23,100 - DEBUG - Control-node 10.0.0.33 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestSvcHC-79433380:ctest-right_vn-28490466', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestSvcHC-79433380', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '7323248733723052061', 'uuid-lslong': '11725660408377847782'}, 'enable': 'true', 'created': '2026-04-06T03:29:22', 'last-modified': '2026-04-06T03:29:22', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.421574'} 2026-04-06 03:29:23,106 - DEBUG - Route Targets: ['target:64512:8000010'] 2026-04-06 03:29:23,106 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-right_vn-28490466 passed 2026-04-06 03:29:23,106 - DEBUG - ====Verifying policy data for ctest-right_vn-28490466 in API_Server ====== 2026-04-06 03:29:23,106 - DEBUG - Requesting: http://10.0.0.249:8082/domains 2026-04-06 03:29:23,113 - DEBUG - Requesting: http://10.0.0.249:8082/domain/500738d8-636d-4330-98c4-d5dafb5cbb39 2026-04-06 03:29:23,125 - DEBUG - Requesting: http://10.0.0.249:8082/project/33871f1f-4880-4200-9d82-ae1986cb4ea0 2026-04-06 03:29:23,216 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/65a167ec-4f64-481d-a2b9-ea0426392fe6 2026-04-06 03:29:23,225 - DEBUG - =>VN ctest-right_vn-28490466 has no policy to be verified 2026-04-06 03:29:23,225 - DEBUG - Verifying the vn in opserver 2026-04-06 03:29:23,225 - DEBUG - Verifying the default-domain:ctest-TestSvcHC-79433380:ctest-right_vn-28490466 virtual network link through opserver 10.0.0.249 2026-04-06 03:29:23,225 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/virtual-networks 2026-04-06 03:29:23,286 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestSvcHC-79433380:ctest-right_vn-28490466', 'href': 'http://10.0.0.249:8081/analytics/uves/virtual-network/default-domain:ctest-TestSvcHC-79433380:ctest-right_vn-28490466?flat'} 2026-04-06 03:29:23,286 - DEBUG - Verifying the default-domain:ctest-TestSvcHC-79433380:ctest-right_vn-28490466 virtual network link through opserver 10.0.0.27 2026-04-06 03:29:23,286 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-networks 2026-04-06 03:29:23,345 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestSvcHC-79433380:ctest-right_vn-28490466', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestSvcHC-79433380:ctest-right_vn-28490466?flat'} 2026-04-06 03:29:23,345 - DEBUG - Verifying the default-domain:ctest-TestSvcHC-79433380:ctest-right_vn-28490466 virtual network link through opserver 10.0.0.33 2026-04-06 03:29:23,345 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-networks 2026-04-06 03:29:23,402 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestSvcHC-79433380:ctest-right_vn-28490466', 'href': 'http://10.0.0.33:8081/analytics/uves/virtual-network/default-domain:ctest-TestSvcHC-79433380:ctest-right_vn-28490466?flat'} 2026-04-06 03:29:23,402 - INFO - Validated that VN default-domain:ctest-TestSvcHC-79433380:ctest-right_vn-28490466 is found in opserver 2026-04-06 03:29:23,422 - DEBUG - Do not have enough data to verify VN in agent 2026-04-06 03:29:23,431 - DEBUG - VRF ids for VN ctest-right_vn-28490466: {} 2026-04-06 03:29:23,570 - DEBUG - Services list from nova: [, , , ] 2026-04-06 03:29:24,640 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-5596-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-5596-2) 2026-04-06 03:29:25,967 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-5596-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-5596-1) 2026-04-06 03:29:25,967 - DEBUG - Creating service template: ['default-domain', 'ctest-service_template_1-74710112'] 2026-04-06 03:29:26,276 - DEBUG - Service template: ['default-domain', 'ctest-service_template_1-74710112'] created succesfully 2026-04-06 03:29:26,656 - DEBUG - Image by name tiny_nat_fw not found 2026-04-06 03:29:26,657 - DEBUG - Installing image tiny_nat_fw 2026-04-06 03:29:26,657 - DEBUG - Image info {'name': 'tinycore-in-network-nat.qcow2.gz', 'webserver': '', 'location': '/images/tinycore/', 'username': 'tc', 'password': 'secret', 'type': 'os', 'flavor': 'contrail_flavor_tiny', 'params': '--container-format bare --disk-format vmdk --property vmware_disktype="sparse" --property vmware_adaptertype="ide" --property hypervisor_type=qemu', 'name_docker': 'phusion-baseimage-enablesshd', 'vmdk': 'service_chain_images/nat_service/tiny_nat_fw/tinycore-in-network-nat-flat.vmdk', 'shrinked_vmdk': "'no'"} 2026-04-06 03:29:26,657 - DEBUG - Image build path file:///contrail-test/images/tinycore-in-network-nat.qcow2.gz 2026-04-06 03:29:26,657 - DEBUG - Download image from local file file:///contrail-test/images/tinycore-in-network-nat.qcow2.gz 2026-04-06 03:29:26,657 - DEBUG - Unzip image 2026-04-06 03:29:26,657 - DEBUG - Image download (local=True) (proxy=None): Execute cmd: gunzip -f /contrail-test/images/tinycore-in-network-nat.qcow2.gz 2026-04-06 03:29:26,678 - INFO - image /contrail-test/images/tinycore-in-network-nat.qcow2 is not found 2026-04-06 03:29:26,679 - DEBUG - Deleting service template: ['default-domain', 'ctest-service_template_1-74710112'] 2026-04-06 03:29:26,763 - INFO - Service template: ['default-domain', 'ctest-service_template_1-74710112'] deleted successfully. 2026-04-06 03:29:26,763 - INFO - Skip interface_detach for VM ctest-right_vm-73032848 in state BUILD 2026-04-06 03:29:26,763 - INFO - Deleting VM ctest-right_vm-73032848 2026-04-06 03:29:26,794 - ERROR - VM ctest-right_vm-73032848 has failed to come up 2026-04-06 03:29:26,794 - ERROR - Nova failed to get host of the VM 2026-04-06 03:29:26,794 - ERROR - VM ctest-right_vm-73032848 has failed to come up 2026-04-06 03:29:26,794 - ERROR - Nova failed to get host of the VM 2026-04-06 03:29:26,861 - INFO - Skip interface_detach for VM ctest-left_vm-05158968 in state BUILD 2026-04-06 03:29:26,861 - INFO - Deleting VM ctest-left_vm-05158968 2026-04-06 03:29:26,861 - ERROR - VM ctest-left_vm-05158968 has failed to come up 2026-04-06 03:29:26,861 - ERROR - Nova failed to get host of the VM 2026-04-06 03:29:26,861 - ERROR - VM ctest-left_vm-05158968 has failed to come up 2026-04-06 03:29:26,861 - ERROR - Nova failed to get host of the VM 2026-04-06 03:29:26,957 - INFO - Deleting VN ctest-right_vn-28490466 2026-04-06 03:29:27,000 - DEBUG - VN 65a167ec-4f64-481d-a2b9-ea0426392fe6 still in use: Unable to complete operation on network 65a167ec-4f64-481d-a2b9-ea0426392fe6. There are one or more ports still in use on the network. Neutron server returns request_ids: ['req-103c75e1-f100-49cf-af37-c78c062cc760'] 2026-04-06 03:29:27,000 - WARNING - Deleting VN ctest-right_vn-28490466 failed..Will retry 2026-04-06 03:29:29,157 - DEBUG - Response for deleting network () 2026-04-06 03:29:29,158 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/2b3de5d6-2b4c-4e0c-afff-421e075fd162 2026-04-06 03:29:29,164 - DEBUG - Response Code: 404 2026-04-06 03:29:29,164 - DEBUG - Requesting: http://10.0.0.249:8082/domains 2026-04-06 03:29:29,186 - DEBUG - Requesting: http://10.0.0.249:8082/domain/500738d8-636d-4330-98c4-d5dafb5cbb39 2026-04-06 03:29:29,207 - DEBUG - Requesting: http://10.0.0.249:8082/project/33871f1f-4880-4200-9d82-ae1986cb4ea0 2026-04-06 03:29:29,327 - INFO - Validated that VN ctest-right_vn-28490466 is not found in API Server 2026-04-06 03:29:29,340 - DEBUG - VN ctest-right_vn-28490466 is not present in Agent 10.0.0.19 2026-04-06 03:29:29,352 - DEBUG - VN ctest-right_vn-28490466 is not present in Agent 10.0.0.241 2026-04-06 03:29:29,354 - INFO - Validated that VN ctest-right_vn-28490466 is not in any agent 2026-04-06 03:29:29,395 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-right_vn-28490466 info 2026-04-06 03:29:29,396 - INFO - Deleting VN ctest-left_vn-03129979 2026-04-06 03:29:29,558 - DEBUG - Response for deleting network () 2026-04-06 03:29:29,558 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/f0228a8f-b431-4e53-b08f-226dc7c892a6 2026-04-06 03:29:29,563 - DEBUG - Response Code: 404 2026-04-06 03:29:29,563 - DEBUG - Requesting: http://10.0.0.249:8082/domains 2026-04-06 03:29:29,568 - DEBUG - Requesting: http://10.0.0.249:8082/domain/500738d8-636d-4330-98c4-d5dafb5cbb39 2026-04-06 03:29:29,591 - DEBUG - Requesting: http://10.0.0.249:8082/project/33871f1f-4880-4200-9d82-ae1986cb4ea0 2026-04-06 03:29:29,676 - INFO - Validated that VN ctest-left_vn-03129979 is not found in API Server 2026-04-06 03:29:29,689 - DEBUG - VN ctest-left_vn-03129979 is not present in Agent 10.0.0.19 2026-04-06 03:29:29,700 - DEBUG - VN ctest-left_vn-03129979 is not present in Agent 10.0.0.241 2026-04-06 03:29:29,700 - INFO - Validated that VN ctest-left_vn-03129979 is not in any agent 2026-04-06 03:29:29,739 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-left_vn-03129979 info 2026-04-06 03:29:29,739 - INFO - Deleting VN ctest-mgmt_vn-11503117 2026-04-06 03:29:29,898 - DEBUG - Response for deleting network () 2026-04-06 03:29:29,898 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/81a386e4-65c4-4bff-aec4-2b74cdd0a4ba 2026-04-06 03:29:29,910 - DEBUG - Response Code: 404 2026-04-06 03:29:29,910 - DEBUG - Requesting: http://10.0.0.249:8082/domains 2026-04-06 03:29:29,923 - DEBUG - Requesting: http://10.0.0.249:8082/domain/500738d8-636d-4330-98c4-d5dafb5cbb39 2026-04-06 03:29:29,948 - DEBUG - Requesting: http://10.0.0.249:8082/project/33871f1f-4880-4200-9d82-ae1986cb4ea0 2026-04-06 03:29:30,049 - INFO - Validated that VN ctest-mgmt_vn-11503117 is not found in API Server 2026-04-06 03:29:30,061 - DEBUG - VN ctest-mgmt_vn-11503117 is not present in Agent 10.0.0.19 2026-04-06 03:29:30,074 - DEBUG - VN ctest-mgmt_vn-11503117 is not present in Agent 10.0.0.241 2026-04-06 03:29:30,074 - INFO - Validated that VN ctest-mgmt_vn-11503117 is not in any agent 2026-04-06 03:29:30,105 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-mgmt_vn-11503117 info 2026-04-06 03:29:31,405 - DEBUG - No XMPP flaps were noticed during the test 2026-04-06 03:29:31,405 - INFO - 2026-04-06 03:29:31,405 - INFO - END TEST : test_svc_hc_basic : SKIPPED[0:00:15] 2026-04-06 03:29:31,405 - INFO - -------------------------------------------------------------------------------- 2026-04-06 03:29:31,408 - INFO - ================================================================================ 2026-04-06 03:29:31,408 - INFO - STARTING TEST : test_svc_trans_segment_right_hc_fail_svm 2026-04-06 03:29:32,681 - DEBUG - Nothing to compare xmpp stats {'10.0.0.19': {'10.20.0.193': '0', '10.20.0.129': '0'}, '10.0.0.241': {'10.20.0.18': '0', '10.20.0.129': '0'}} with 2026-04-06 03:29:32,681 - INFO - Initial checks done. Running the testcase now 2026-04-06 03:29:32,681 - INFO - 2026-04-06 03:29:32,930 - DEBUG - Response for create_network : {'network': {'id': '07206651-c885-45e9-8db3-b22bc482730b', 'name': 'ctest-mgmt_vn-60024255', 'tenant_id': '33871f1f488042009d82ae1986cb4ea0', 'project_id': '33871f1f488042009d82ae1986cb4ea0', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-TestSvcHC-79433380', 'ctest-mgmt_vn-60024255'], 'port_security_enabled': True, 'description': ''}} 2026-04-06 03:29:33,135 - DEBUG - Response for create_subnet : {'subnet': {'id': 'e0714202-ed76-4749-8746-10a3cf3f086f', 'name': '', 'tenant_id': '33871f1f488042009d82ae1986cb4ea0', 'network_id': '07206651-c885-45e9-8db3-b22bc482730b', 'ip_version': 4, 'cidr': '16.91.177.0/26', 'allocation_pools': [{'start': '16.91.177.2', 'end': '16.91.177.62'}], 'gateway_ip': '16.91.177.1', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '16.91.177.2', 'tags': [], 'project_id': '33871f1f488042009d82ae1986cb4ea0'}} 2026-04-06 03:29:33,159 - INFO - Created VN ctest-mgmt_vn-60024255 2026-04-06 03:29:33,170 - DEBUG - VN ctest-mgmt_vn-60024255 UUID is 07206651-c885-45e9-8db3-b22bc482730b 2026-04-06 03:29:33,232 - DEBUG - Requesting: http://10.0.0.249:8082/domains 2026-04-06 03:29:33,237 - DEBUG - Requesting: http://10.0.0.249:8082/domain/500738d8-636d-4330-98c4-d5dafb5cbb39 2026-04-06 03:29:33,249 - DEBUG - Requesting: http://10.0.0.249:8082/project/33871f1f-4880-4200-9d82-ae1986cb4ea0 2026-04-06 03:29:33,339 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/07206651-c885-45e9-8db3-b22bc482730b 2026-04-06 03:29:33,351 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/07206651-c885-45e9-8db3-b22bc482730b 2026-04-06 03:29:33,360 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/0883a6e4-b30c-4801-9c29-537438b6a4b0 2026-04-06 03:29:33,366 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/0883a6e4-b30c-4801-9c29-537438b6a4b0 2026-04-06 03:29:33,372 - DEBUG - Requesting: http://10.0.0.249:8082/route-target/984acfb6-4f27-4d7d-9429-c203b3bfbadc 2026-04-06 03:29:33,378 - DEBUG - Route Targets: ['target:64512:8000008'] 2026-04-06 03:29:33,378 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/07206651-c885-45e9-8db3-b22bc482730b 2026-04-06 03:29:33,386 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/0883a6e4-b30c-4801-9c29-537438b6a4b0 2026-04-06 03:29:33,400 - INFO - Verified VN network id 15 for VN 07206651-c885-45e9-8db3-b22bc482730b 2026-04-06 03:29:33,400 - INFO - Verifications in API Server for VN ctest-mgmt_vn-60024255 passed 2026-04-06 03:29:33,400 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/07206651-c885-45e9-8db3-b22bc482730b 2026-04-06 03:29:33,408 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/0883a6e4-b30c-4801-9c29-537438b6a4b0 2026-04-06 03:29:33,414 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/0883a6e4-b30c-4801-9c29-537438b6a4b0 2026-04-06 03:29:33,419 - DEBUG - Requesting: http://10.0.0.249:8082/route-target/984acfb6-4f27-4d7d-9429-c203b3bfbadc 2026-04-06 03:29:33,430 - DEBUG - Control-node 10.0.0.249 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestSvcHC-79433380:ctest-mgmt_vn-60024255', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestSvcHC-79433380', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '513522858962798057', 'uuid-lslong': '10210700681229267723'}, 'enable': 'true', 'created': '2026-04-06T03:29:32', 'last-modified': '2026-04-06T03:29:33', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.343936'} 2026-04-06 03:29:33,437 - DEBUG - Route Targets: ['target:64512:8000008'] 2026-04-06 03:29:33,442 - DEBUG - Control-node 10.0.0.27 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestSvcHC-79433380:ctest-mgmt_vn-60024255', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestSvcHC-79433380', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '513522858962798057', 'uuid-lslong': '10210700681229267723'}, 'enable': 'true', 'created': '2026-04-06T03:29:32', 'last-modified': '2026-04-06T03:29:33', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.348432'} 2026-04-06 03:29:33,448 - DEBUG - Route Targets: ['target:64512:8000008'] 2026-04-06 03:29:33,454 - DEBUG - Control-node 10.0.0.33 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestSvcHC-79433380:ctest-mgmt_vn-60024255', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestSvcHC-79433380', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '513522858962798057', 'uuid-lslong': '10210700681229267723'}, 'enable': 'true', 'created': '2026-04-06T03:29:32', 'last-modified': '2026-04-06T03:29:33', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.365097'} 2026-04-06 03:29:33,459 - DEBUG - Route Targets: ['target:64512:8000008'] 2026-04-06 03:29:33,459 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-mgmt_vn-60024255 passed 2026-04-06 03:29:33,459 - DEBUG - ====Verifying policy data for ctest-mgmt_vn-60024255 in API_Server ====== 2026-04-06 03:29:33,459 - DEBUG - Requesting: http://10.0.0.249:8082/domains 2026-04-06 03:29:33,466 - DEBUG - Requesting: http://10.0.0.249:8082/domain/500738d8-636d-4330-98c4-d5dafb5cbb39 2026-04-06 03:29:33,481 - DEBUG - Requesting: http://10.0.0.249:8082/project/33871f1f-4880-4200-9d82-ae1986cb4ea0 2026-04-06 03:29:33,570 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/07206651-c885-45e9-8db3-b22bc482730b 2026-04-06 03:29:33,578 - DEBUG - =>VN ctest-mgmt_vn-60024255 has no policy to be verified 2026-04-06 03:29:33,578 - DEBUG - Verifying the vn in opserver 2026-04-06 03:29:33,578 - DEBUG - Verifying the default-domain:ctest-TestSvcHC-79433380:ctest-mgmt_vn-60024255 virtual network link through opserver 10.0.0.249 2026-04-06 03:29:33,578 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/virtual-networks 2026-04-06 03:29:33,591 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestSvcHC-79433380:ctest-mgmt_vn-60024255', 'href': 'http://10.0.0.249:8081/analytics/uves/virtual-network/default-domain:ctest-TestSvcHC-79433380:ctest-mgmt_vn-60024255?flat'} 2026-04-06 03:29:33,591 - DEBUG - Verifying the default-domain:ctest-TestSvcHC-79433380:ctest-mgmt_vn-60024255 virtual network link through opserver 10.0.0.27 2026-04-06 03:29:33,591 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-networks 2026-04-06 03:29:33,604 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestSvcHC-79433380:ctest-mgmt_vn-60024255', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestSvcHC-79433380:ctest-mgmt_vn-60024255?flat'} 2026-04-06 03:29:33,604 - DEBUG - Verifying the default-domain:ctest-TestSvcHC-79433380:ctest-mgmt_vn-60024255 virtual network link through opserver 10.0.0.33 2026-04-06 03:29:33,605 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-networks 2026-04-06 03:29:33,617 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestSvcHC-79433380:ctest-mgmt_vn-60024255', 'href': 'http://10.0.0.33:8081/analytics/uves/virtual-network/default-domain:ctest-TestSvcHC-79433380:ctest-mgmt_vn-60024255?flat'} 2026-04-06 03:29:33,617 - INFO - Validated that VN default-domain:ctest-TestSvcHC-79433380:ctest-mgmt_vn-60024255 is found in opserver 2026-04-06 03:29:33,637 - DEBUG - Do not have enough data to verify VN in agent 2026-04-06 03:29:33,647 - DEBUG - VRF ids for VN ctest-mgmt_vn-60024255: {} 2026-04-06 03:29:33,901 - DEBUG - Response for create_network : {'network': {'id': '62b2456d-a28d-4430-ab5d-69a7e741e125', 'name': 'ctest-left_vn-75581090', 'tenant_id': '33871f1f488042009d82ae1986cb4ea0', 'project_id': '33871f1f488042009d82ae1986cb4ea0', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-TestSvcHC-79433380', 'ctest-left_vn-75581090'], 'port_security_enabled': True, 'description': ''}} 2026-04-06 03:29:34,107 - DEBUG - Response for create_subnet : {'subnet': {'id': '61d00f9e-7c07-4972-8d97-97b4846f1e76', 'name': '', 'tenant_id': '33871f1f488042009d82ae1986cb4ea0', 'network_id': '62b2456d-a28d-4430-ab5d-69a7e741e125', 'ip_version': 4, 'cidr': '105.146.159.192/26', 'allocation_pools': [{'start': '105.146.159.194', 'end': '105.146.159.254'}], 'gateway_ip': '105.146.159.193', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '105.146.159.194', 'tags': [], 'project_id': '33871f1f488042009d82ae1986cb4ea0'}} 2026-04-06 03:29:34,128 - INFO - Created VN ctest-left_vn-75581090 2026-04-06 03:29:34,141 - DEBUG - VN ctest-left_vn-75581090 UUID is 62b2456d-a28d-4430-ab5d-69a7e741e125 2026-04-06 03:29:34,206 - DEBUG - Requesting: http://10.0.0.249:8082/domains 2026-04-06 03:29:34,215 - DEBUG - Requesting: http://10.0.0.249:8082/domain/500738d8-636d-4330-98c4-d5dafb5cbb39 2026-04-06 03:29:34,225 - DEBUG - Requesting: http://10.0.0.249:8082/project/33871f1f-4880-4200-9d82-ae1986cb4ea0 2026-04-06 03:29:34,322 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/62b2456d-a28d-4430-ab5d-69a7e741e125 2026-04-06 03:29:34,337 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/62b2456d-a28d-4430-ab5d-69a7e741e125 2026-04-06 03:29:34,347 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/685d7b76-554e-420d-8c12-1c999998dd1b 2026-04-06 03:29:34,362 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/685d7b76-554e-420d-8c12-1c999998dd1b 2026-04-06 03:29:34,368 - DEBUG - Requesting: http://10.0.0.249:8082/route-target/fbacaa69-e510-45da-b63d-f79f51cd1afc 2026-04-06 03:29:34,373 - DEBUG - Route Targets: ['target:64512:8000009'] 2026-04-06 03:29:34,373 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/62b2456d-a28d-4430-ab5d-69a7e741e125 2026-04-06 03:29:34,401 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/685d7b76-554e-420d-8c12-1c999998dd1b 2026-04-06 03:29:34,416 - INFO - Verified VN network id 16 for VN 62b2456d-a28d-4430-ab5d-69a7e741e125 2026-04-06 03:29:34,416 - INFO - Verifications in API Server for VN ctest-left_vn-75581090 passed 2026-04-06 03:29:34,416 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/62b2456d-a28d-4430-ab5d-69a7e741e125 2026-04-06 03:29:34,429 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/685d7b76-554e-420d-8c12-1c999998dd1b 2026-04-06 03:29:34,445 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/685d7b76-554e-420d-8c12-1c999998dd1b 2026-04-06 03:29:34,451 - DEBUG - Requesting: http://10.0.0.249:8082/route-target/fbacaa69-e510-45da-b63d-f79f51cd1afc 2026-04-06 03:29:34,462 - DEBUG - Control-node 10.0.0.249 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestSvcHC-79433380:ctest-left_vn-75581090', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestSvcHC-79433380', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '7111823098752353328', 'uuid-lslong': '12348141923180077349'}, 'enable': 'true', 'created': '2026-04-06T03:29:33', 'last-modified': '2026-04-06T03:29:34', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.404984'} 2026-04-06 03:29:34,467 - DEBUG - Route Targets: ['target:64512:8000009'] 2026-04-06 03:29:34,471 - DEBUG - Control-node 10.0.0.27 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestSvcHC-79433380:ctest-left_vn-75581090', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestSvcHC-79433380', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '7111823098752353328', 'uuid-lslong': '12348141923180077349'}, 'enable': 'true', 'created': '2026-04-06T03:29:33', 'last-modified': '2026-04-06T03:29:34', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.413437'} 2026-04-06 03:29:34,479 - DEBUG - Route Targets: ['target:64512:8000009'] 2026-04-06 03:29:34,484 - DEBUG - Control-node 10.0.0.33 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestSvcHC-79433380:ctest-left_vn-75581090', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestSvcHC-79433380', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '7111823098752353328', 'uuid-lslong': '12348141923180077349'}, 'enable': 'true', 'created': '2026-04-06T03:29:33', 'last-modified': '2026-04-06T03:29:34', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.430390'} 2026-04-06 03:29:34,489 - DEBUG - Route Targets: ['target:64512:8000009'] 2026-04-06 03:29:34,489 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-left_vn-75581090 passed 2026-04-06 03:29:34,490 - DEBUG - ====Verifying policy data for ctest-left_vn-75581090 in API_Server ====== 2026-04-06 03:29:34,490 - DEBUG - Requesting: http://10.0.0.249:8082/domains 2026-04-06 03:29:34,495 - DEBUG - Requesting: http://10.0.0.249:8082/domain/500738d8-636d-4330-98c4-d5dafb5cbb39 2026-04-06 03:29:34,504 - DEBUG - Requesting: http://10.0.0.249:8082/project/33871f1f-4880-4200-9d82-ae1986cb4ea0 2026-04-06 03:29:34,584 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/62b2456d-a28d-4430-ab5d-69a7e741e125 2026-04-06 03:29:34,600 - DEBUG - =>VN ctest-left_vn-75581090 has no policy to be verified 2026-04-06 03:29:34,600 - DEBUG - Verifying the vn in opserver 2026-04-06 03:29:34,600 - DEBUG - Verifying the default-domain:ctest-TestSvcHC-79433380:ctest-left_vn-75581090 virtual network link through opserver 10.0.0.249 2026-04-06 03:29:34,600 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/virtual-networks 2026-04-06 03:29:34,630 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestSvcHC-79433380:ctest-left_vn-75581090', 'href': 'http://10.0.0.249:8081/analytics/uves/virtual-network/default-domain:ctest-TestSvcHC-79433380:ctest-left_vn-75581090?flat'} 2026-04-06 03:29:34,630 - DEBUG - Verifying the default-domain:ctest-TestSvcHC-79433380:ctest-left_vn-75581090 virtual network link through opserver 10.0.0.27 2026-04-06 03:29:34,630 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-networks 2026-04-06 03:29:34,648 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestSvcHC-79433380:ctest-left_vn-75581090', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestSvcHC-79433380:ctest-left_vn-75581090?flat'} 2026-04-06 03:29:34,648 - DEBUG - Verifying the default-domain:ctest-TestSvcHC-79433380:ctest-left_vn-75581090 virtual network link through opserver 10.0.0.33 2026-04-06 03:29:34,648 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-networks 2026-04-06 03:29:34,694 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestSvcHC-79433380:ctest-left_vn-75581090', 'href': 'http://10.0.0.33:8081/analytics/uves/virtual-network/default-domain:ctest-TestSvcHC-79433380:ctest-left_vn-75581090?flat'} 2026-04-06 03:29:34,694 - INFO - Validated that VN default-domain:ctest-TestSvcHC-79433380:ctest-left_vn-75581090 is found in opserver 2026-04-06 03:29:34,715 - DEBUG - Do not have enough data to verify VN in agent 2026-04-06 03:29:34,724 - DEBUG - VRF ids for VN ctest-left_vn-75581090: {} 2026-04-06 03:29:35,029 - DEBUG - Response for create_network : {'network': {'id': '502bc965-b343-4ece-9d44-de5e62b1ebb9', 'name': 'ctest-right_vn-80004709', 'tenant_id': '33871f1f488042009d82ae1986cb4ea0', 'project_id': '33871f1f488042009d82ae1986cb4ea0', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-TestSvcHC-79433380', 'ctest-right_vn-80004709'], 'port_security_enabled': True, 'description': ''}} 2026-04-06 03:29:35,308 - DEBUG - Response for create_subnet : {'subnet': {'id': '28937904-106f-4e6a-9d72-a233fa37e68b', 'name': '', 'tenant_id': '33871f1f488042009d82ae1986cb4ea0', 'network_id': '502bc965-b343-4ece-9d44-de5e62b1ebb9', 'ip_version': 4, 'cidr': '5.191.235.64/26', 'allocation_pools': [{'start': '5.191.235.66', 'end': '5.191.235.126'}], 'gateway_ip': '5.191.235.65', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '5.191.235.66', 'tags': [], 'project_id': '33871f1f488042009d82ae1986cb4ea0'}} 2026-04-06 03:29:35,333 - INFO - Created VN ctest-right_vn-80004709 2026-04-06 03:29:35,346 - DEBUG - VN ctest-right_vn-80004709 UUID is 502bc965-b343-4ece-9d44-de5e62b1ebb9 2026-04-06 03:29:35,425 - DEBUG - Requesting: http://10.0.0.249:8082/domains 2026-04-06 03:29:35,432 - DEBUG - Requesting: http://10.0.0.249:8082/domain/500738d8-636d-4330-98c4-d5dafb5cbb39 2026-04-06 03:29:35,444 - DEBUG - Requesting: http://10.0.0.249:8082/project/33871f1f-4880-4200-9d82-ae1986cb4ea0 2026-04-06 03:29:35,543 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/502bc965-b343-4ece-9d44-de5e62b1ebb9 2026-04-06 03:29:35,553 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/502bc965-b343-4ece-9d44-de5e62b1ebb9 2026-04-06 03:29:35,561 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/f0fadb26-f651-446e-b2d4-7ee2123d96d7 2026-04-06 03:29:35,567 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/f0fadb26-f651-446e-b2d4-7ee2123d96d7 2026-04-06 03:29:35,575 - DEBUG - Requesting: http://10.0.0.249:8082/route-target/48593271-9af6-4ded-8405-0182e9a1ea02 2026-04-06 03:29:35,580 - DEBUG - Route Targets: ['target:64512:8000010'] 2026-04-06 03:29:35,580 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/502bc965-b343-4ece-9d44-de5e62b1ebb9 2026-04-06 03:29:35,587 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/f0fadb26-f651-446e-b2d4-7ee2123d96d7 2026-04-06 03:29:35,601 - INFO - Verified VN network id 17 for VN 502bc965-b343-4ece-9d44-de5e62b1ebb9 2026-04-06 03:29:35,601 - INFO - Verifications in API Server for VN ctest-right_vn-80004709 passed 2026-04-06 03:29:35,601 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/502bc965-b343-4ece-9d44-de5e62b1ebb9 2026-04-06 03:29:35,608 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/f0fadb26-f651-446e-b2d4-7ee2123d96d7 2026-04-06 03:29:35,614 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/f0fadb26-f651-446e-b2d4-7ee2123d96d7 2026-04-06 03:29:35,620 - DEBUG - Requesting: http://10.0.0.249:8082/route-target/48593271-9af6-4ded-8405-0182e9a1ea02 2026-04-06 03:29:35,630 - DEBUG - Control-node 10.0.0.249 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestSvcHC-79433380:ctest-right_vn-80004709', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestSvcHC-79433380', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '5776932385669205710', 'uuid-lslong': '11332427059335130041'}, 'enable': 'true', 'created': '2026-04-06T03:29:34', 'last-modified': '2026-04-06T03:29:35', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.390273'} 2026-04-06 03:29:35,635 - DEBUG - Route Targets: ['target:64512:8000010'] 2026-04-06 03:29:35,642 - DEBUG - Control-node 10.0.0.27 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestSvcHC-79433380:ctest-right_vn-80004709', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestSvcHC-79433380', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '5776932385669205710', 'uuid-lslong': '11332427059335130041'}, 'enable': 'true', 'created': '2026-04-06T03:29:34', 'last-modified': '2026-04-06T03:29:35', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.408728'} 2026-04-06 03:29:35,648 - DEBUG - Route Targets: ['target:64512:8000010'] 2026-04-06 03:29:35,653 - DEBUG - Control-node 10.0.0.33 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestSvcHC-79433380:ctest-right_vn-80004709', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestSvcHC-79433380', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '5776932385669205710', 'uuid-lslong': '11332427059335130041'}, 'enable': 'true', 'created': '2026-04-06T03:29:34', 'last-modified': '2026-04-06T03:29:35', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.420547'} 2026-04-06 03:29:35,659 - DEBUG - Route Targets: ['target:64512:8000010'] 2026-04-06 03:29:35,659 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-right_vn-80004709 passed 2026-04-06 03:29:35,659 - DEBUG - ====Verifying policy data for ctest-right_vn-80004709 in API_Server ====== 2026-04-06 03:29:35,659 - DEBUG - Requesting: http://10.0.0.249:8082/domains 2026-04-06 03:29:35,664 - DEBUG - Requesting: http://10.0.0.249:8082/domain/500738d8-636d-4330-98c4-d5dafb5cbb39 2026-04-06 03:29:35,674 - DEBUG - Requesting: http://10.0.0.249:8082/project/33871f1f-4880-4200-9d82-ae1986cb4ea0 2026-04-06 03:29:35,768 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/502bc965-b343-4ece-9d44-de5e62b1ebb9 2026-04-06 03:29:35,778 - DEBUG - =>VN ctest-right_vn-80004709 has no policy to be verified 2026-04-06 03:29:35,778 - DEBUG - Verifying the vn in opserver 2026-04-06 03:29:35,778 - DEBUG - Verifying the default-domain:ctest-TestSvcHC-79433380:ctest-right_vn-80004709 virtual network link through opserver 10.0.0.249 2026-04-06 03:29:35,778 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/virtual-networks 2026-04-06 03:29:35,790 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestSvcHC-79433380:ctest-right_vn-80004709', 'href': 'http://10.0.0.249:8081/analytics/uves/virtual-network/default-domain:ctest-TestSvcHC-79433380:ctest-right_vn-80004709?flat'} 2026-04-06 03:29:35,790 - DEBUG - Verifying the default-domain:ctest-TestSvcHC-79433380:ctest-right_vn-80004709 virtual network link through opserver 10.0.0.27 2026-04-06 03:29:35,790 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-networks 2026-04-06 03:31:35,798 - INFO - Deleting VN ctest-right_vn-80004709 2026-04-06 03:31:35,953 - DEBUG - Response for deleting network () 2026-04-06 03:31:35,953 - INFO - Deleting VN ctest-left_vn-75581090 2026-04-06 03:31:36,131 - DEBUG - Response for deleting network () 2026-04-06 03:31:36,131 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/685d7b76-554e-420d-8c12-1c999998dd1b 2026-04-06 03:31:36,137 - DEBUG - Response Code: 404 2026-04-06 03:31:36,137 - DEBUG - Requesting: http://10.0.0.249:8082/domains 2026-04-06 03:31:36,145 - DEBUG - Requesting: http://10.0.0.249:8082/domain/500738d8-636d-4330-98c4-d5dafb5cbb39 2026-04-06 03:31:36,163 - DEBUG - Requesting: http://10.0.0.249:8082/project/33871f1f-4880-4200-9d82-ae1986cb4ea0 2026-04-06 03:31:36,253 - INFO - Validated that VN ctest-left_vn-75581090 is not found in API Server 2026-04-06 03:31:36,264 - DEBUG - VN ctest-left_vn-75581090 is not present in Agent 10.0.0.19 2026-04-06 03:31:36,274 - DEBUG - VN ctest-left_vn-75581090 is not present in Agent 10.0.0.241 2026-04-06 03:31:36,274 - INFO - Validated that VN ctest-left_vn-75581090 is not in any agent 2026-04-06 03:31:36,304 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-left_vn-75581090 info 2026-04-06 03:31:36,304 - INFO - Deleting VN ctest-mgmt_vn-60024255 2026-04-06 03:31:36,458 - DEBUG - Response for deleting network () 2026-04-06 03:31:36,458 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/0883a6e4-b30c-4801-9c29-537438b6a4b0 2026-04-06 03:31:36,464 - DEBUG - Response Code: 404 2026-04-06 03:31:36,464 - DEBUG - Requesting: http://10.0.0.249:8082/domains 2026-04-06 03:31:36,471 - DEBUG - Requesting: http://10.0.0.249:8082/domain/500738d8-636d-4330-98c4-d5dafb5cbb39 2026-04-06 03:31:36,524 - DEBUG - Requesting: http://10.0.0.249:8082/project/33871f1f-4880-4200-9d82-ae1986cb4ea0 2026-04-06 03:31:36,615 - INFO - Validated that VN ctest-mgmt_vn-60024255 is not found in API Server 2026-04-06 03:31:36,627 - DEBUG - VN ctest-mgmt_vn-60024255 is not present in Agent 10.0.0.19 2026-04-06 03:31:36,635 - DEBUG - VN ctest-mgmt_vn-60024255 is not present in Agent 10.0.0.241 2026-04-06 03:31:36,636 - INFO - Validated that VN ctest-mgmt_vn-60024255 is not in any agent 2026-04-06 03:31:36,668 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-mgmt_vn-60024255 info 2026-04-06 03:31:37,933 - ERROR - 
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 159, in _new_conn
    conn = connection.create_connection(
  File "/usr/local/lib/python3.9/site-packages/urllib3/util/connection.py", line 84, in create_connection
    raise err
  File "/usr/local/lib/python3.9/site-packages/urllib3/util/connection.py", line 74, in create_connection
    sock.connect(sa)
  File "/usr/local/lib64/python3.9/site-packages/gevent/_socketcommon.py", line 590, in connect
    self._internal_connect(address)
  File "/usr/local/lib64/python3.9/site-packages/gevent/_socketcommon.py", line 640, in _internal_connect
    self._wait(self._write_event)
  File "src/gevent/_hub_primitives.py", line 317, in gevent._gevent_c_hub_primitives.wait_on_socket
  File "src/gevent/_hub_primitives.py", line 322, in gevent._gevent_c_hub_primitives.wait_on_socket
  File "src/gevent/_hub_primitives.py", line 313, in gevent._gevent_c_hub_primitives._primitive_wait
  File "src/gevent/_hub_primitives.py", line 314, in gevent._gevent_c_hub_primitives._primitive_wait
  File "src/gevent/_hub_primitives.py", line 46, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait
  File "src/gevent/_hub_primitives.py", line 46, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait
  File "src/gevent/_hub_primitives.py", line 55, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait
  File "src/gevent/_waiter.py", line 154, in gevent._gevent_c_waiter.Waiter.get
  File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 65, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_gevent_c_greenlet_primitives.pxd", line 35, in gevent._gevent_c_greenlet_primitives._greenlet_switch
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 392, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib64/python3.9/http/client.py", line 1285, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output
    self.send(msg)
  File "/usr/lib64/python3.9/http/client.py", line 980, in send
    self.connect()
  File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 187, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 164, in _new_conn
    raise ConnectTimeoutError(
urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPConnection object at 0x7f069d7196a0>, 'Connection to 10.0.0.27 timed out. (connect timeout=120)')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 726, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.9/site-packages/urllib3/util/retry.py", line 446, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='10.0.0.27', port=8081): Max retries exceeded with url: /analytics/uves/virtual-networks (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f069d7196a0>, 'Connection to 10.0.0.27 timed out. (connect timeout=120)'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/contrail-test/tcutils/wrappers.py", line 80, in wrapper
    result = function(self, *args, **kwargs)
  File "/contrail-test/scripts/svc_health_check/test_svc_health_check.py", line 137, in test_svc_trans_segment_right_hc_fail_svm
    assert self.svc_hc_fail_svm_common()
  File "/contrail-test/scripts/svc_health_check/test_svc_health_check.py", line 179, in svc_hc_fail_svm_common
    ret_dict = self.verify_svc_chain(service_mode=svc_mode,
  File "/contrail-test/common/servicechain/verify.py", line 320, in verify_svc_chain
    ret_dict = svc_chain_info or self.config_svc_chain(*args, **kwargs)
  File "/contrail-test/common/servicechain/config.py", line 462, in config_svc_chain
    right_vn_fq_name) = self._get_vn_for_config(right_vn_name,
  File "/contrail-test/common/servicechain/config.py", line 357, in _get_vn_for_config
    vn_fixture = vn_fixture or self.config_vn(vn_name, vn_subnets, **kwargs)
  File "/contrail-test/common/servicechain/config.py", line 139, in config_vn
    assert vn_fixture.verify_on_setup()
  File "/contrail-test/fixtures/vn_test.py", line 589, in verify_on_setup
    if not self.verify_vn_in_opserver():
  File "/contrail-test/fixtures/vn_test.py", line 1122, in verify_vn_in_opserver
    res = self.analytics_obj.verify_vn_link(self.vn_fq_name)
  File "/contrail-test/tcutils/util.py", line 84, in f_retry
    result = f(*args, **kwargs)  # first attempt
  File "/contrail-test/tcutils/collector/analytics_tests.py", line 1275, in verify_vn_link
    self.links = self.ops_inspect[ip].get_hrefs_to_all_UVEs_of_a_given_UVE_type(
  File "/contrail-test/tcutils/collector/opserver_introspect_utils.py", line 145, in get_hrefs_to_all_UVEs_of_a_given_UVE_type
    dct = self.dict_get('analytics/uves/' + uveType)
  File "/contrail-test/tcutils/verification_util.py", line 245, in dict_get
    return self._drv.load(self._mk_url_str(path))
  File "/contrail-test/tcutils/verification_util.py", line 96, in load
    resp = requests.get(url,
  File "/usr/local/lib/python3.9/site-packages/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 504, in send
    raise ConnectTimeout(e, request=request)
requests.exceptions.ConnectTimeout: HTTPConnectionPool(host='10.0.0.27', port=8081): Max retries exceeded with url: /analytics/uves/virtual-networks (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f069d7196a0>, 'Connection to 10.0.0.27 timed out. (connect timeout=120)'))
2026-04-06 03:31:37,944 - DEBUG - No XMPP flaps were noticed during the test 2026-04-06 03:31:37,945 - INFO - 2026-04-06 03:31:37,945 - INFO - END TEST : test_svc_trans_segment_right_hc_fail_svm : FAILED[0:02:06] 2026-04-06 03:31:37,945 - INFO - -------------------------------------------------------------------------------- 2026-04-06 03:31:38,712 - INFO - Deleted project: ctest-TestSvcHC-79433380, ID : 33871f1f-4880-4200-9d82-ae1986cb4ea0