2026-09-17 23:07:57,286 - DEBUG - [10.0.0.177]: Running cmd : hostname 2026-09-17 23:07:58,054 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-2122-1 2026-09-17 23:07:58,054 - DEBUG - [10.0.0.177]: Running cmd : hostname -f 2026-09-17 23:07:58,099 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-2122-1 2026-09-17 23:07:58,100 - DEBUG - [10.0.0.177]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2026-09-17 23:07:58,199 - DEBUG - Output : NAMES contrail_test_40HHs6mb9 contrail-kubernetes-master-kubemanager-1 analytics_snmp-snmp-collector-1 analytics_snmp-topology-1 analytics_snmp-nodemgr-1 analytics_alarm-kafka-1 analytics_alarm-alarm-gen-1 analytics_alarm-nodemgr-1 analytics-collector-1 analytics-api-1 analytics-nodemgr-1 analytics_database-cassandra-1 analytics_database-nodemgr-1 analytics_database-query-engine-1 redis-redis-1 webui-web-1 webui-job-1 control-named-1 control-control-1 control-dns-1 control-nodemgr-1 config_database-nodemgr-1 config_database-cassandra-1 config_database-zookeeper-1 config_database-rabbitmq-1 config_api-devicemgr-1 config_api-dnsmasq-1 config_api-api-1 config_api-nodemgr-1 config_api-svcmonitor-1 config_api-schema-1 2026-09-17 23:07:58,199 - DEBUG - [10.0.0.177]: Running cmd : ip -4 -o addr show | awk '{print $4}' 2026-09-17 23:07:58,287 - DEBUG - Output : 127.0.0.1/8 10.0.0.177/24 252.177.0.1/8 10.239.127.1/24 172.17.0.1/16 2026-09-17 23:07:58,287 - DEBUG - [10.0.0.177]: Running cmd : getent hosts 10.0.0.177 | head -n 1 | awk '{print $2}' 2026-09-17 23:07:58,343 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-2122-1 2026-09-17 23:07:58,460 - DEBUG - [10.0.0.50]: Running cmd : hostname 2026-09-17 23:07:58,706 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-2122-1 2026-09-17 23:07:58,707 - DEBUG - [10.0.0.50]: Running cmd : hostname -f 2026-09-17 23:07:58,745 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-2122-1 2026-09-17 23:07:58,746 - DEBUG - [10.0.0.50]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2026-09-17 23:07:58,791 - DEBUG - Output : NAMES vrouter-vrouter-agent-1 vrouter-nodemgr-1 2026-09-17 23:07:58,791 - DEBUG - [10.0.0.50]: Running cmd : ip -4 -o addr show dev vhost0 | awk '{print $4}' 2026-09-17 23:07:58,869 - DEBUG - Output : 10.0.0.50/24 2026-09-17 23:07:58,869 - DEBUG - [10.0.0.50]: Running cmd : getent hosts 10.0.0.50 | head -n 1 | awk '{print $2}' 2026-09-17 23:07:58,907 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-2122-1 2026-09-17 23:07:59,113 - INFO - Using existing project ['default-domain', 'k8s-default'](3d3a70f3-7c44-482f-9698-c40ca1aec090) 2026-09-17 23:07:59,370 - INFO - Using existing project ['default-domain', 'k8s-default'](3d3a70f3-7c44-482f-9698-c40ca1aec090) 2026-09-17 23:07:59,719 - INFO - ================================================================================ 2026-09-17 23:07:59,719 - INFO - STARTING TEST : test_ingress_ip_assignment 2026-09-17 23:07:59,719 - INFO - TEST DESCRIPTION : Verify that Ingress gets a CLuster IP which is reachable to Pods in same namespace. Also verify that a Floating IP is assigned to the Ingress from the Public FIP poo. Steps: 1. Create a service with 2 pods running nginx 2. Create an ingress out of this service 3. From another Pod do a wget on the ingress Cluster ip Validate that Ingress get a IP from Public FIP pool which might/might not be accessible. Validate that service and its loadbalancing work 2026-09-17 23:08:00,098 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AgentXmppConnectionStatusReq? 2026-09-17 23:08:00,108 - DEBUG - Nothing to compare xmpp stats {'10.0.0.50': {'10.0.0.177': '0'}} with 2026-09-17 23:08:00,108 - INFO - Initial checks done. Running the testcase now 2026-09-17 23:08:00,108 - INFO - 2026-09-17 23:08:00,121 - INFO - Namespace default already exists 2026-09-17 23:08:00,121 - DEBUG - [10.0.0.177]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf' 2026-09-17 23:08:00,451 - DEBUG - Output : cluster_project={} 2026-09-17 23:08:00,597 - INFO - Project default-domain:k8s-default for Namespace default is seen in contrail-api 2026-09-17 23:08:00,597 - DEBUG - Project uuid in Contrail is 3d3a70f3-7c44-482f-9698-c40ca1aec090 2026-09-17 23:08:00,597 - DEBUG - [10.0.0.177]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2026-09-17 23:08:00,783 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3 2026-09-17 23:08:00,783 - DEBUG - Requesting: http://10.0.0.177:8108/Snh_NamespaceDatabaseList?namespace_uuid=6f9ad7fe-906f-44cd-a3e9-595c51443bcd 2026-09-17 23:08:00,792 - INFO - Namespace default with uuid 6f9ad7fe-906f-44cd-a3e9-595c51443bcd found in kube manager 2026-09-17 23:08:00,793 - INFO - Namespace default verification passed 2026-09-17 23:08:00,798 - DEBUG - Service ctest-nginx-svc-25872594 not present 2026-09-17 23:08:00,798 - INFO - Creating service ctest-nginx-svc-25872594 2026-09-17 23:08:00,825 - INFO - Created Service ctest-nginx-svc-25872594 2026-09-17 23:08:00,842 - DEBUG - Pod ctest-nginx-pod-47856044 not present 2026-09-17 23:08:00,843 - INFO - Creating Pod ctest-nginx-pod-47856044 2026-09-17 23:08:01,020 - DEBUG - Pod : ctest-nginx-pod-47856044 UUID is fae625c7-a1e2-489e-ad78-41738199185e 2026-09-17 23:08:01,027 - DEBUG - Pod ctest-nginx-pod-81978438 not present 2026-09-17 23:08:01,028 - INFO - Creating Pod ctest-nginx-pod-81978438 2026-09-17 23:08:01,082 - DEBUG - Pod : ctest-nginx-pod-81978438 UUID is e6607706-5893-4ed8-8c64-521effeb835f 2026-09-17 23:08:01,744 - INFO - Created VN __public__, UUID :2dd1b3b9-991e-4e80-8f3a-cf323c8e9710 2026-09-17 23:08:02,108 - DEBUG - Requesting: http://10.0.0.177:8082/domains 2026-09-17 23:08:02,120 - DEBUG - Requesting: http://10.0.0.177:8082/domain/03791da1-cab0-4609-b320-7cccee88e368 2026-09-17 23:08:02,174 - DEBUG - Requesting: http://10.0.0.177:8082/project/3d3a70f3-7c44-482f-9698-c40ca1aec090 2026-09-17 23:08:02,419 - DEBUG - Requesting: http://10.0.0.177:8082/virtual-network/2dd1b3b9-991e-4e80-8f3a-cf323c8e9710 2026-09-17 23:08:02,439 - DEBUG - Requesting: http://10.0.0.177:8082/virtual-network/2dd1b3b9-991e-4e80-8f3a-cf323c8e9710 2026-09-17 23:08:02,458 - DEBUG - Requesting: http://10.0.0.177:8082/routing-instance/98a71e3a-b74e-4c59-bc48-5149f7f79b11 2026-09-17 23:08:02,495 - DEBUG - Requesting: http://10.0.0.177:8082/routing-instance/98a71e3a-b74e-4c59-bc48-5149f7f79b11 2026-09-17 23:08:02,522 - DEBUG - Requesting: http://10.0.0.177:8082/route-target/baa8dc20-b057-4bbf-b6bb-2c44bbefa3b5 2026-09-17 23:08:02,539 - DEBUG - Route Targets: ['target:64512:8000006'] 2026-09-17 23:08:02,539 - DEBUG - Requesting: http://10.0.0.177:8082/virtual-network/2dd1b3b9-991e-4e80-8f3a-cf323c8e9710 2026-09-17 23:08:02,561 - DEBUG - Requesting: http://10.0.0.177:8082/routing-instance/98a71e3a-b74e-4c59-bc48-5149f7f79b11 2026-09-17 23:08:02,630 - INFO - Verified VN network id 13 for VN 2dd1b3b9-991e-4e80-8f3a-cf323c8e9710 2026-09-17 23:08:02,631 - INFO - Verifications in API Server for VN __public__ passed 2026-09-17 23:08:02,631 - DEBUG - Requesting: http://10.0.0.177:8082/virtual-network/2dd1b3b9-991e-4e80-8f3a-cf323c8e9710 2026-09-17 23:08:02,705 - DEBUG - Requesting: http://10.0.0.177:8082/routing-instance/98a71e3a-b74e-4c59-bc48-5149f7f79b11 2026-09-17 23:08:02,738 - DEBUG - Requesting: http://10.0.0.177:8082/routing-instance/98a71e3a-b74e-4c59-bc48-5149f7f79b11 2026-09-17 23:08:02,749 - DEBUG - Requesting: http://10.0.0.177:8082/route-target/baa8dc20-b057-4bbf-b6bb-2c44bbefa3b5 2026-09-17 23:08:02,804 - DEBUG - Requesting: http://10.0.0.177:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2026-09-17 23:08:02,812 - DEBUG - Control-node 10.0.0.177 : VN object is : {'node_name': 'virtual-network:default-domain:k8s-default:__public__', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'cloud-admin', 'owner-access': '7', 'group': 'cloud-admin-group', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '3301617611558506112', 'uuid-lslong': '10320789310744205072'}, 'enable': 'true', 'created': '2026-09-17T23:08:01', 'last-modified': '2026-09-17T23:08:01', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.848445'} 2026-09-17 23:08:02,812 - DEBUG - Requesting: http://10.0.0.177:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2026-09-17 23:08:02,833 - DEBUG - Route Targets: ['target:64512:8000006'] 2026-09-17 23:08:02,833 - INFO - On all control nodes, Config, RI and RT verification for VN __public__ passed 2026-09-17 23:08:02,834 - DEBUG - ====Verifying policy data for __public__ in API_Server ====== 2026-09-17 23:08:02,834 - DEBUG - Requesting: http://10.0.0.177:8082/domains 2026-09-17 23:08:02,843 - DEBUG - Requesting: http://10.0.0.177:8082/domain/03791da1-cab0-4609-b320-7cccee88e368 2026-09-17 23:08:02,855 - DEBUG - Requesting: http://10.0.0.177:8082/project/3d3a70f3-7c44-482f-9698-c40ca1aec090 2026-09-17 23:08:03,035 - DEBUG - Requesting: http://10.0.0.177:8082/virtual-network/2dd1b3b9-991e-4e80-8f3a-cf323c8e9710 2026-09-17 23:08:03,053 - DEBUG - =>VN __public__ has no policy to be verified 2026-09-17 23:08:03,053 - DEBUG - Verifying the vn in opserver 2026-09-17 23:08:03,053 - DEBUG - Verifying the default-domain:k8s-default:__public__ virtual network link through opserver 10.0.0.177 2026-09-17 23:08:03,053 - DEBUG - Requesting: http://10.0.0.177:8081/analytics/uves/virtual-networks 2026-09-17 23:08:03,063 - DEBUG - vn link and name as {'name': 'default-domain:k8s-default:__public__', 'href': 'http://10.0.0.177:8081/analytics/uves/virtual-network/default-domain:k8s-default:__public__?flat'} 2026-09-17 23:08:03,063 - INFO - Validated that VN default-domain:k8s-default:__public__ is found in opserver 2026-09-17 23:08:03,063 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-09-17 23:08:03,072 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-09-17 23:08:03,090 - DEBUG - Do not have enough data to verify VN in agent 2026-09-17 23:08:03,091 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-09-17 23:08:03,101 - DEBUG - VRF ids for VN __public__: {} 2026-09-17 23:08:03,295 - INFO - Creating Floating IP pool __fip_pool_public__ in API Server 2026-09-17 23:08:03,582 - DEBUG - Requesting: http://10.0.0.177:8082/domains 2026-09-17 23:08:03,606 - DEBUG - Requesting: http://10.0.0.177:8082/domain/03791da1-cab0-4609-b320-7cccee88e368 2026-09-17 23:08:03,623 - DEBUG - Requesting: http://10.0.0.177:8082/project/3d3a70f3-7c44-482f-9698-c40ca1aec090 2026-09-17 23:08:03,720 - DEBUG - Requesting: http://10.0.0.177:8082/virtual-network/2dd1b3b9-991e-4e80-8f3a-cf323c8e9710 2026-09-17 23:08:03,745 - DEBUG - Requesting: http://10.0.0.177:8082/floating-ip-pool/591bacc3-a1d0-4058-bfea-75ab44e0c35c 2026-09-17 23:08:03,756 - DEBUG - Requesting: http://10.0.0.177:8082/domains 2026-09-17 23:08:03,780 - DEBUG - Requesting: http://10.0.0.177:8082/domain/03791da1-cab0-4609-b320-7cccee88e368 2026-09-17 23:08:03,791 - DEBUG - Requesting: http://10.0.0.177:8082/project/3d3a70f3-7c44-482f-9698-c40ca1aec090 2026-09-17 23:08:03,848 - DEBUG - Requesting: http://10.0.0.177:8082/virtual-network/2dd1b3b9-991e-4e80-8f3a-cf323c8e9710 2026-09-17 23:08:03,869 - DEBUG - FIP Pool __fip_pool_public__ found in API Server 2026-09-17 23:08:03,869 - INFO - Verification for FIP pool __fip_pool_public__ in API Server passed 2026-09-17 23:08:03,914 - DEBUG - Requesting: http://10.0.0.177:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2026-09-17 23:08:03,926 - DEBUG - Control-node Ifmap-view has FIP pool __fip_pool_public__ information 2026-09-17 23:08:03,926 - INFO - Verification for FIP pool __fip_pool_public__ in Control-node passed 2026-09-17 23:08:03,931 - DEBUG - Ingress ctest-nginx-ingress-02976327 not present 2026-09-17 23:08:03,932 - INFO - Creating Ingress ctest-nginx-ingress-02976327 2026-09-17 23:08:03,954 - INFO - Created Ingress ctest-nginx-ingress-02976327 2026-09-17 23:08:03,960 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-02976327 2026-09-17 23:08:06,979 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-02976327 2026-09-17 23:08:10,000 - DEBUG - For Ingress ctest-nginx-ingress-02976327, Cluster IP: 10.47.255.249, External IPs ['43.246.134.60'] 2026-09-17 23:08:10,000 - INFO - Verifications in k8s passed for Ingress ctest-nginx-ingress-02976327 2026-09-17 23:08:10,001 - DEBUG - Requesting: http://10.0.0.177:8108/Snh_LoadbalancerDatabaseList?x=884c6200-f60c-4dd1-922d-2ed0f84ffdec 2026-09-17 23:08:10,034 - INFO - Ingress ctest-nginx-ingress-02976327 with uuid 884c6200-f60c-4dd1-922d-2ed0f84ffdec found in kube manager 2026-09-17 23:08:10,036 - INFO - Ingress ctest-nginx-ingress-02976327 verification passed 2026-09-17 23:08:10,040 - DEBUG - Pod ctest-busybox-pod-91520122 not present 2026-09-17 23:08:10,042 - INFO - Creating Pod ctest-busybox-pod-91520122 2026-09-17 23:08:10,190 - DEBUG - Pod : ctest-busybox-pod-91520122 UUID is 94f7a6ab-3298-468f-9610-a1868123c176 2026-09-17 23:08:10,220 - DEBUG - Pod ctest-nginx-pod-47856044 not in running state.Currently in Pending 2026-09-17 23:08:15,272 - DEBUG - Pod ctest-nginx-pod-47856044 not in running state.Currently in Pending 2026-09-17 23:08:20,283 - INFO - Pod ctest-nginx-pod-47856044 is in running state.Got IP 10.47.255.251 2026-09-17 23:08:20,321 - DEBUG - Pod ctest-nginx-pod-47856044 has vmi a2ffab10-b2ec-11f1-bd5f-fa163e00a419 2026-09-17 23:08:20,321 - INFO - Verified pod ctest-nginx-pod-47856044 in contrail-api 2026-09-17 23:08:20,321 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-09-17 23:08:20,339 - DEBUG - VMI a2ffab10-b2ec-11f1-bd5f-fa163e00a419 is active in agent 10.0.0.50 2026-09-17 23:08:20,343 - INFO - Verified Pod ctest-nginx-pod-47856044 in agent 10.0.0.50 2026-09-17 23:08:20,343 - DEBUG - Requesting: http://10.0.0.177:8108/Snh_VirtualMachineDatabaseList?x=fae625c7-a1e2-489e-ad78-41738199185e 2026-09-17 23:08:20,360 - INFO - Pod ctest-nginx-pod-47856044 with uuid fae625c7-a1e2-489e-ad78-41738199185e found in kube manager 2026-09-17 23:08:20,360 - INFO - Pod ctest-nginx-pod-47856044 verification passed 2026-09-17 23:08:20,441 - DEBUG - [Pod ctest-nginx-pod-47856044] Cmd: echo ctest-nginx-pod-47856044 > /usr/share/nginx/html/index.html, Output: 2026-09-17 23:08:20,459 - INFO - Pod ctest-nginx-pod-81978438 is in running state.Got IP 10.47.255.250 2026-09-17 23:08:20,484 - DEBUG - Pod ctest-nginx-pod-81978438 has vmi a3ad8596-b2ec-11f1-bd5f-fa163e00a419 2026-09-17 23:08:20,484 - INFO - Verified pod ctest-nginx-pod-81978438 in contrail-api 2026-09-17 23:08:20,484 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-09-17 23:08:20,499 - DEBUG - VMI a3ad8596-b2ec-11f1-bd5f-fa163e00a419 is active in agent 10.0.0.50 2026-09-17 23:08:20,502 - INFO - Verified Pod ctest-nginx-pod-81978438 in agent 10.0.0.50 2026-09-17 23:08:20,502 - DEBUG - Requesting: http://10.0.0.177:8108/Snh_VirtualMachineDatabaseList?x=e6607706-5893-4ed8-8c64-521effeb835f 2026-09-17 23:08:20,520 - INFO - Pod ctest-nginx-pod-81978438 with uuid e6607706-5893-4ed8-8c64-521effeb835f found in kube manager 2026-09-17 23:08:20,520 - INFO - Pod ctest-nginx-pod-81978438 verification passed 2026-09-17 23:08:20,634 - DEBUG - [Pod ctest-nginx-pod-81978438] Cmd: echo ctest-nginx-pod-81978438 > /usr/share/nginx/html/index.html, Output: 2026-09-17 23:08:20,645 - DEBUG - Pod ctest-busybox-pod-91520122 not in running state.Currently in Pending 2026-09-17 23:08:25,653 - INFO - Pod ctest-busybox-pod-91520122 is in running state.Got IP 10.47.255.248 2026-09-17 23:08:25,714 - DEBUG - Pod ctest-busybox-pod-91520122 has vmi a71b1bf8-b2ec-11f1-bd5f-fa163e00a419 2026-09-17 23:08:25,714 - INFO - Verified pod ctest-busybox-pod-91520122 in contrail-api 2026-09-17 23:08:25,714 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-09-17 23:08:25,728 - DEBUG - VMI a71b1bf8-b2ec-11f1-bd5f-fa163e00a419 is active in agent 10.0.0.50 2026-09-17 23:08:25,728 - INFO - Verified Pod ctest-busybox-pod-91520122 in agent 10.0.0.50 2026-09-17 23:08:25,728 - DEBUG - Requesting: http://10.0.0.177:8108/Snh_VirtualMachineDatabaseList?x=94f7a6ab-3298-468f-9610-a1868123c176 2026-09-17 23:08:25,737 - INFO - Pod ctest-busybox-pod-91520122 with uuid 94f7a6ab-3298-468f-9610-a1868123c176 found in kube manager 2026-09-17 23:08:25,737 - INFO - Pod ctest-busybox-pod-91520122 verification passed 2026-09-17 23:08:25,805 - DEBUG - [Pod ctest-busybox-pod-91520122] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.249:80 (10.47.255.249:80) ctest-nginx-pod-47856044 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-09-17 23:08:25,805 - DEBUG - [Pod Pod ctest-busybox-pod-91520122] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed 2026-09-17 23:08:25,805 - DEBUG - [Pod Pod ctest-busybox-pod-91520122] Cmd output: Connecting to 10.47.255.249:80 (10.47.255.249:80) ctest-nginx-pod-47856044 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-09-17 23:08:25,860 - DEBUG - [Pod ctest-busybox-pod-91520122] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: ctest-nginx-pod-81978438 Connecting to 10.47.255.249:80 (10.47.255.249:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-09-17 23:08:25,861 - DEBUG - [Pod Pod ctest-busybox-pod-91520122] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed 2026-09-17 23:08:25,861 - DEBUG - [Pod Pod ctest-busybox-pod-91520122] Cmd output: ctest-nginx-pod-81978438 Connecting to 10.47.255.249:80 (10.47.255.249:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-09-17 23:08:25,861 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-nginx-pod-47856044': 1, 'ctest-nginx-pod-81978438': 1} 2026-09-17 23:08:25,861 - WARNING - Nothing to delete parallely 2026-09-17 23:08:25,861 - INFO - Deleting pod default:ctest-busybox-pod-91520122 2026-09-17 23:08:25,883 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=94f7a6ab-3298-468f-9610-a1868123c176 2026-09-17 23:08:25,892 - WARNING - Pod uuid 94f7a6ab-3298-468f-9610-a1868123c176 is still seen in agent 10.0.0.50 VM list 2026-09-17 23:08:30,893 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=94f7a6ab-3298-468f-9610-a1868123c176 2026-09-17 23:08:30,901 - WARNING - Pod uuid 94f7a6ab-3298-468f-9610-a1868123c176 is still seen in agent 10.0.0.50 VM list 2026-09-17 23:08:35,902 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=94f7a6ab-3298-468f-9610-a1868123c176 2026-09-17 23:08:35,910 - WARNING - Pod uuid 94f7a6ab-3298-468f-9610-a1868123c176 is still seen in agent 10.0.0.50 VM list 2026-09-17 23:08:40,912 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=94f7a6ab-3298-468f-9610-a1868123c176 2026-09-17 23:08:40,920 - WARNING - Pod uuid 94f7a6ab-3298-468f-9610-a1868123c176 is still seen in agent 10.0.0.50 VM list 2026-09-17 23:08:45,920 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=94f7a6ab-3298-468f-9610-a1868123c176 2026-09-17 23:08:45,927 - WARNING - Pod uuid 94f7a6ab-3298-468f-9610-a1868123c176 is still seen in agent 10.0.0.50 VM list 2026-09-17 23:08:50,928 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=94f7a6ab-3298-468f-9610-a1868123c176 2026-09-17 23:08:50,935 - WARNING - Pod uuid 94f7a6ab-3298-468f-9610-a1868123c176 is still seen in agent 10.0.0.50 VM list 2026-09-17 23:08:55,936 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=94f7a6ab-3298-468f-9610-a1868123c176 2026-09-17 23:08:55,943 - WARNING - Pod uuid 94f7a6ab-3298-468f-9610-a1868123c176 is still seen in agent 10.0.0.50 VM list 2026-09-17 23:09:00,944 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=94f7a6ab-3298-468f-9610-a1868123c176 2026-09-17 23:09:00,954 - DEBUG - Pod 94f7a6ab-3298-468f-9610-a1868123c176 is not in agent 10.0.0.50 VM list 2026-09-17 23:09:00,954 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:a71b1bf8-b2ec-11f1-bd5f-fa163e00a419 2026-09-17 23:09:00,964 - DEBUG - VMI a71b1bf8-b2ec-11f1-bd5f-fa163e00a419 is removed from agent 10.0.0.50 2026-09-17 23:09:00,964 - INFO - Verified that pod ctest-busybox-pod-91520122 is removed in agent 2026-09-17 23:09:00,964 - INFO - Deleting Ingress : ctest-nginx-ingress-02976327 2026-09-17 23:09:00,977 - DEBUG - Requesting: http://10.0.0.177:8108/Snh_LoadbalancerDatabaseList?x=884c6200-f60c-4dd1-922d-2ed0f84ffdec 2026-09-17 23:09:00,986 - ERROR - Ingress ctest-nginx-ingress-02976327 with uuid 884c6200-f60c-4dd1-922d-2ed0f84ffdec still found in kube manager 2026-09-17 23:09:01,987 - DEBUG - Requesting: http://10.0.0.177:8108/Snh_LoadbalancerDatabaseList?x=884c6200-f60c-4dd1-922d-2ed0f84ffdec 2026-09-17 23:09:01,998 - ERROR - Ingress ctest-nginx-ingress-02976327 with uuid 884c6200-f60c-4dd1-922d-2ed0f84ffdec still found in kube manager 2026-09-17 23:09:03,005 - DEBUG - Requesting: http://10.0.0.177:8108/Snh_LoadbalancerDatabaseList?x=884c6200-f60c-4dd1-922d-2ed0f84ffdec 2026-09-17 23:09:03,012 - INFO - Ingress ctest-nginx-ingress-02976327 with uuid 884c6200-f60c-4dd1-922d-2ed0f84ffdec deleted successfully from kube manager 2026-09-17 23:09:03,013 - INFO - Verifications on Ingress %s cleanup passed 2026-09-17 23:09:03,013 - INFO - Deleting the FIP pool __fip_pool_public__ 2026-09-17 23:09:03,162 - DEBUG - Requesting: http://10.0.0.177:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2026-09-17 23:09:03,168 - DEBUG - Control-node Ifmap-view does not have FIP pool __fip_pool_public__ information 2026-09-17 23:09:03,169 - INFO - Deleting VN __public__ 2026-09-17 23:09:03,373 - DEBUG - Requesting: http://10.0.0.177:8082/routing-instance/98a71e3a-b74e-4c59-bc48-5149f7f79b11 2026-09-17 23:09:03,380 - DEBUG - Response Code: 404 2026-09-17 23:09:03,381 - DEBUG - Requesting: http://10.0.0.177:8082/domains 2026-09-17 23:09:03,403 - DEBUG - Requesting: http://10.0.0.177:8082/domain/03791da1-cab0-4609-b320-7cccee88e368 2026-09-17 23:09:03,434 - DEBUG - Requesting: http://10.0.0.177:8082/project/3d3a70f3-7c44-482f-9698-c40ca1aec090 2026-09-17 23:09:03,501 - INFO - Validated that VN __public__ is not found in API Server 2026-09-17 23:09:03,502 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VnListReq?name=default-domain:k8s-default:__public__ 2026-09-17 23:09:03,518 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-09-17 23:09:03,526 - DEBUG - VN __public__ is not present in Agent 10.0.0.50 2026-09-17 23:09:03,526 - INFO - Validated that VN __public__ is not in any agent 2026-09-17 23:09:03,527 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-09-17 23:09:03,535 - DEBUG - Requesting: http://10.0.0.177:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2026-09-17 23:09:03,543 - DEBUG - Requesting: http://10.0.0.177:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2026-09-17 23:09:03,553 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN __public__ info 2026-09-17 23:09:03,554 - INFO - Deleting pod default:ctest-nginx-pod-81978438 2026-09-17 23:09:03,604 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=e6607706-5893-4ed8-8c64-521effeb835f 2026-09-17 23:09:03,614 - WARNING - Pod uuid e6607706-5893-4ed8-8c64-521effeb835f is still seen in agent 10.0.0.50 VM list 2026-09-17 23:09:08,616 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=e6607706-5893-4ed8-8c64-521effeb835f 2026-09-17 23:09:08,623 - DEBUG - Pod e6607706-5893-4ed8-8c64-521effeb835f is not in agent 10.0.0.50 VM list 2026-09-17 23:09:08,624 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:a3ad8596-b2ec-11f1-bd5f-fa163e00a419 2026-09-17 23:09:08,632 - DEBUG - VMI a3ad8596-b2ec-11f1-bd5f-fa163e00a419 is removed from agent 10.0.0.50 2026-09-17 23:09:08,633 - INFO - Verified that pod ctest-nginx-pod-81978438 is removed in agent 2026-09-17 23:09:08,635 - INFO - Deleting pod default:ctest-nginx-pod-47856044 2026-09-17 23:09:08,666 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=fae625c7-a1e2-489e-ad78-41738199185e 2026-09-17 23:09:08,683 - WARNING - Pod uuid fae625c7-a1e2-489e-ad78-41738199185e is still seen in agent 10.0.0.50 VM list 2026-09-17 23:09:13,685 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=fae625c7-a1e2-489e-ad78-41738199185e 2026-09-17 23:09:13,692 - DEBUG - Pod fae625c7-a1e2-489e-ad78-41738199185e is not in agent 10.0.0.50 VM list 2026-09-17 23:09:13,692 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:a2ffab10-b2ec-11f1-bd5f-fa163e00a419 2026-09-17 23:09:13,698 - DEBUG - VMI a2ffab10-b2ec-11f1-bd5f-fa163e00a419 is removed from agent 10.0.0.50 2026-09-17 23:09:13,698 - INFO - Verified that pod ctest-nginx-pod-47856044 is removed in agent 2026-09-17 23:09:13,698 - INFO - Deleting service : ctest-nginx-svc-25872594 2026-09-17 23:09:13,960 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AgentXmppConnectionStatusReq? 2026-09-17 23:09:13,967 - DEBUG - No XMPP flaps were noticed during the test 2026-09-17 23:09:13,967 - INFO - END TEST : test_ingress_ip_assignment : PASSED[0:01:14] 2026-09-17 23:09:13,967 - INFO - -------------------------------------------------------------------------------- 2026-09-17 23:09:20,539 - DEBUG - [10.0.0.177]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf' 2026-09-17 23:09:20,705 - DEBUG - Output : cluster_project={} 2026-09-17 23:09:20,786 - DEBUG - [10.0.0.177]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2026-09-17 23:09:21,031 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3 2026-09-17 23:09:21,040 - DEBUG - [10.0.0.177]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf' 2026-09-17 23:09:21,373 - DEBUG - Output : cluster_project={} 2026-09-17 23:09:36,578 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-09-17 23:09:36,638 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-09-17 23:09:36,734 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-09-17 23:09:41,801 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-09-17 23:09:41,898 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-09-17 23:09:41,941 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-09-17 23:09:42,028 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-09-17 23:09:42,118 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-09-17 23:09:42,361 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AgentXmppConnectionStatusReq? 2026-09-17 23:09:45,032 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_ApplicationPolicySetReq? 2026-09-17 23:09:45,041 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AclReq? 2026-09-17 23:09:45,091 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AclReq? 2026-09-17 23:09:47,153 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AclReq? 2026-09-17 23:12:50,287 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AgentXmppConnectionStatusReq? 2026-09-17 23:12:50,339 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=28016cb5-3a90-4644-84fb-1b8234b6166f 2026-09-17 23:12:50,360 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=8980c757-47fb-46a3-b7e2-cddfcc6bdedb 2026-09-17 23:12:50,377 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=bd7d0383-31c8-45db-99d8-98c83f97fd15 2026-09-17 23:12:50,395 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=1b24fd72-19a7-4581-9ee0-822db79b9345 2026-09-17 23:12:50,416 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=8a321812-4c05-4312-be22-98acd022bc5b 2026-09-17 23:12:50,428 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=6a5a224f-90e0-4298-8f48-51f38e4b6b73 2026-09-17 23:12:50,435 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=f07e6571-7526-4853-9f30-d79569afb2e2 2026-09-17 23:12:50,448 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=60078973-d5fa-43ef-ba80-9501ecffcfc3 2026-09-17 23:12:55,452 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=8980c757-47fb-46a3-b7e2-cddfcc6bdedb 2026-09-17 23:12:55,454 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=28016cb5-3a90-4644-84fb-1b8234b6166f 2026-09-17 23:12:55,457 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=1b24fd72-19a7-4581-9ee0-822db79b9345 2026-09-17 23:12:55,459 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=bd7d0383-31c8-45db-99d8-98c83f97fd15 2026-09-17 23:12:55,461 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=f07e6571-7526-4853-9f30-d79569afb2e2 2026-09-17 23:12:55,463 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=6a5a224f-90e0-4298-8f48-51f38e4b6b73 2026-09-17 23:12:55,464 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=8a321812-4c05-4312-be22-98acd022bc5b 2026-09-17 23:12:55,470 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:d1b4288c-b2ec-11f1-bd5f-fa163e00a419 2026-09-17 23:12:55,472 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=60078973-d5fa-43ef-ba80-9501ecffcfc3 2026-09-17 23:12:55,480 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:d3aede16-b2ec-11f1-bd5f-fa163e00a419 2026-09-17 23:12:55,487 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:d313563a-b2ec-11f1-bd5f-fa163e00a419 2026-09-17 23:12:55,496 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:d24f06e0-b2ec-11f1-bd5f-fa163e00a419 2026-09-17 23:13:00,481 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=1b24fd72-19a7-4581-9ee0-822db79b9345 2026-09-17 23:13:00,483 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=bd7d0383-31c8-45db-99d8-98c83f97fd15 2026-09-17 23:13:00,492 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=f07e6571-7526-4853-9f30-d79569afb2e2 2026-09-17 23:13:00,493 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=8a321812-4c05-4312-be22-98acd022bc5b 2026-09-17 23:13:05,502 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=bd7d0383-31c8-45db-99d8-98c83f97fd15 2026-09-17 23:13:05,504 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=1b24fd72-19a7-4581-9ee0-822db79b9345 2026-09-17 23:13:05,512 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=8a321812-4c05-4312-be22-98acd022bc5b 2026-09-17 23:13:05,516 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=f07e6571-7526-4853-9f30-d79569afb2e2 2026-09-17 23:13:10,517 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=bd7d0383-31c8-45db-99d8-98c83f97fd15 2026-09-17 23:13:10,520 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=1b24fd72-19a7-4581-9ee0-822db79b9345 2026-09-17 23:13:10,526 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=8a321812-4c05-4312-be22-98acd022bc5b 2026-09-17 23:13:10,528 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=f07e6571-7526-4853-9f30-d79569afb2e2 2026-09-17 23:13:15,534 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=bd7d0383-31c8-45db-99d8-98c83f97fd15 2026-09-17 23:13:15,536 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=1b24fd72-19a7-4581-9ee0-822db79b9345 2026-09-17 23:13:15,538 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=8a321812-4c05-4312-be22-98acd022bc5b 2026-09-17 23:13:15,541 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=f07e6571-7526-4853-9f30-d79569afb2e2 2026-09-17 23:13:20,556 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=bd7d0383-31c8-45db-99d8-98c83f97fd15 2026-09-17 23:13:20,558 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=1b24fd72-19a7-4581-9ee0-822db79b9345 2026-09-17 23:13:20,561 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=f07e6571-7526-4853-9f30-d79569afb2e2 2026-09-17 23:13:20,562 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=8a321812-4c05-4312-be22-98acd022bc5b 2026-09-17 23:13:25,584 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=1b24fd72-19a7-4581-9ee0-822db79b9345 2026-09-17 23:13:25,586 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=bd7d0383-31c8-45db-99d8-98c83f97fd15 2026-09-17 23:13:25,587 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=f07e6571-7526-4853-9f30-d79569afb2e2 2026-09-17 23:13:25,589 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=8a321812-4c05-4312-be22-98acd022bc5b 2026-09-17 23:13:25,614 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:d6078e4c-b2ec-11f1-bd5f-fa163e00a419 2026-09-17 23:13:25,641 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:d45d9546-b2ec-11f1-bd5f-fa163e00a419 2026-09-17 23:13:25,654 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:d57ad1be-b2ec-11f1-bd5f-fa163e00a419 2026-09-17 23:13:25,656 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:d4e5434c-b2ec-11f1-bd5f-fa163e00a419