k3s 0.5 agent node fail to create nginx pod when starting as systemd service #478
Comments
|
I have the same issue. Funny enough the k3s server which also includes the agent works just fine. Are you sure that it is caused by systemd? |
|
I am not sure if it is directly related to systemd, maybe I can add debug level to kubelet and hopefully that could give more clue |
|
I attached the k3s agent node kubelet with log level to 4, it appears kubelet fail to create sandbox for the nignx pod, there are some errors factory.go Factory "systemd", not sure if it is related root@Office-R220-vli:/home/vincent# kubectl get po -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx-kzbq6 0/1 ContainerCreating 0 24m home-ubuntu nginx-spws4 1/1 Running 0 24m 10.42.0.23 office-r220-vli search "nginx-kzbq6" in the debug log attached systemd service to start k3s agent node [Unit] Description=Lightweight Kubernetes Documentation=https://k3s.io After=network-online.target [Service] Type=notify ExecStart=/usr/local/bin/k3s agent --kubelet-arg "v=4" -s https://192.168.1.30:6443 -t "K102fcec9c1bd7ecdeb16a471571d2fb3abe6d0a3d49cb69b029d0264ea78a71e3c::node:099ebc33d9626443a5290c3cc146602a" KillMode=process Delegate=yes LimitNOFILE=infinity LimitNPROC=infinity LimitCORE=infinity TasksMax=infinity [Install] WantedBy=multi-user.target |
|
@ibuildthecloud is this issue fixed in 0.6.0-rc2? |
|
version 0.6.0-rc3 suffer from the same problem: |
|
I am not sure I can reproduce the issue, steps:
Results: I can see all pods in a running state normally, @vincentmli can you describe the exact steps for reproducing the problem |
|
Are you also running Ubuntu 1804? Maybe it is caused by Ubuntu |
|
I tried following k3s-agent services on centos 7, with v0.5.0, it appears to be working @galal-hussein [root@k3s-agent ~]# cat /etc/systemd/system/k3s-agent.service [Unit] Description=Lightweight Kubernetes Documentation=https://k3s.io After=network-online.target [Service] Type=exec Environment="HTTP_PROXY=http://10.3.254.254:3128/" Environment="HTTPS_PROXY=http://10.3.254.254:3128/" EnvironmentFile=/etc/systemd/system/k3s-agent.service.env ExecStartPre=-/sbin/modprobe br_netfilter ExecStartPre=-/sbin/modprobe overlay ExecStart=/usr/local/bin/k3s agent --server https://10.3.72.189:6443 --token-file /usr/local/bin/node-token --flannel-iface ens224 --node-ip 10.169.72.98 KillMode=process Delegate=yes LimitNOFILE=infinity LimitNPROC=infinity LimitCORE=infinity TasksMax=infinity TimeoutStartSec=0 Restart=always [Install] WantedBy=multi-user.target [root@rancher-k3s home]# kubectl get po -o wide --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES default nginx-7rlzc 1/1 Running 0 3m51s 10.42.0.205 rancher-k3s default nginx-pjh5g 1/1 Running 0 3m51s 10.42.4.4 k3s-agent kube-system cc-cluster-574dc9565c-kmkhk 1/1 Running 0 24s 10.169.72.98 k3s-agent kube-system coredns-695688789-sjt4j 1/1 Running 0 15d 10.42.0.196 rancher-k3s |
|
k3s 0.6.1 worker still does not work with ubuntu 18.04: @galal-hussein is there anything (furhter logs or something like that) you need from us in order to fix these kind of issues with ubuntu? |
|
I fixed with network device cleanup |
|
I've the same problem with my setup. The master+agent node is able to run pods, the node started only in agent mode cannot. Here is some logs from I've updated CentOS 7 to the latest release: I'm running the same version on both master and agent. I've installed the latest version of k3s. |
|
I'm seeing this same issue with k3s v0.6.1 on Xenial and Bionic. When I switch from using the example systemd unit file in the git repo to running the command via sudo in an interactive shell, the pods immediately spring to life and I get no "context deadline exceeded" errors. |
|
After looking over the install script, I realized that there's a small difference between the systemd unit files created by the installer script and the example unit file in the root of the github repo. For the server, the installer script uses From reading the systemd manual, the only difference between these options is how/when systemd considers the service as started in order to trigger follow-up units. I'm guessing that systemd also performs some other actions that are important for the proper functioning of the agent processes, but it never does them because it's waiting to be notified by the agent process. |
|
That makes sense, thanks for pointing this out @agaffney. We should modify the install script to change the type for agents, or maybe better add some code to send a systemd notification from the agent also. |
|
The install script already does the right thing. The problem is that not everybody uses the installer (I'm not a fan of curlpipes). It would probably be a good idea to update the example systemd unit file in the repo and docs to reflect this difference, but modifying the agent so that there is no difference is also a good idea. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
A clear and concise description of what the bug is.
running k3s agent node as systemd service as below cause agent node fail to create pod
start k3s agent service as:
deploy nginx pod and service : kubectl apply -f nginx_cluster_pod_service.yaml
root@Office-R220-vli:/home/vincent# cat nginx_cluster_pod_service.yaml apiVersion: v1 kind: ReplicationController metadata: name: nginx spec: replicas: 2 selector: app: nginx template: metadata: name: nginx labels: app: nginx spec: containers: - name: nginx image: nginx ports: - containerPort: 80 --- apiVersion: v1 kind: Service metadata: labels: name: nginxservice name: nginxservice spec: ports: # The port that this service should serve on. - port: 80 selector: app: nginx type: ClusterIPnginx pod in agent node stuck in ContainerCreating
running k3s agent node from command line as below, nginx pod is able to be created
To Reproduce
Steps to reproduce the behavior:
download k3s 0.5 and run k3s agent node from systemd service as above example
Expected behavior
A clear and concise description of what you expected to happen.
k3s agent node should be able to run from systemd service and able to create pod
Screenshots
If applicable, add screenshots to help explain your problem.
k3s agent node startup logs from systemd service
May 14 20:30:51 home-ubuntu systemd[1]: Starting Lightweight Kubernetes... May 14 20:30:51 home-ubuntu k3s[11731]: time="2019-05-14T20:30:51Z" level=info msg="Preparing data dir /var/lib/rancher/k3s/data/4e1224c66a9dbb9b03daefff200f4f8eaf45590fb722b6fe2924a201d6de2e8d" May 14 20:30:52 home-ubuntu k3s[11731]: time="2019-05-14T20:30:52Z" level=debug msg="Running /var/lib/rancher/k3s/data/4e1224c66a9dbb9b03daefff200f4f8eaf45590fb722b6fe2924a201d6de2e8d/bin/k3s-agent [/usr/local/bin/k3s --debug agent -s https://192.168.1.30:6443 -t K102fcec9c1bd7ecdeb16a471571d2fb3abe6d0a3d49cb69b029d0264ea78a71e3c::node:099ebc33d9626443a5290c3cc146602a]" May 14 20:30:52 home-ubuntu k3s[11731]: time="2019-05-14T20:30:52.434902781Z" level=info msg="Starting k3s agent v0.5.0 (8c0116dd)" May 14 20:30:53 home-ubuntu k3s[11731]: time="2019-05-14T20:30:53.215686449Z" level=info msg="Running containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd" May 14 20:30:53 home-ubuntu k3s[11731]: time="2019-05-14T20:30:53.220793382Z" level=info msg="Waiting for containerd startup: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: Error while dialing dial unix /run/k3s/containerd/containerd.sock: connect: no such file or directory\"" May 14 20:30:53 home-ubuntu k3s[11731]: time="2019-05-14T20:30:53.255414211Z" level=info msg="starting containerd" revision= version=1.2.5+unknown .............CUT....... May 14 20:30:53 home-ubuntu k3s[11731]: time="2019-05-14T20:30:53.275667026Z" level=info msg="Start cri plugin with config {PluginConfig:{ContainerdConfig:{Snapshotter:overlayfs DefaultRuntime:{Type:io.containerd.runtime.v1.linux Engine: Root: Options:} UntrustedWorkloadRuntime:{Type: Engine: Root: Options:} Runtimes:map[] NoPivot:false} CniConfig:{NetworkPluginBinDir:/var/lib/rancher/k3s/data/4e1224c66a9dbb9b03daefff200f4f8eaf45590fb722b6fe2924a201d6de2e8d/bin NetworkPluginConfDir:/var/lib/rancher/k3s/agent/etc/cni/net.d NetworkPluginConfTemplate:} Registry:{Mirrors:map[docker.io:{Endpoints:[https://registry-1.docker.io]}] Auths:map[]} StreamServerAddress:home-ubuntu StreamServerPort:10010 EnableSelinux:false SandboxImage:k8s.gcr.io/pause:3.1 StatsCollectPeriod:10 SystemdCgroup:false EnableTLSStreaming:false X509KeyPairStreaming:{TLSCertFile: TLSKeyFile:} MaxContainerLogLineSize:16384 DisableCgroup:false DisableApparmor:false RestrictOOMScoreAdj:false} ContainerdRootDir:/var/lib/rancher/k3s/agent/containerd ContainerdEndpoint:/run/k3s/containerd/containerd.sock RootDir:/var/lib/rancher/k3s/agent/containerd/io.containerd.grpc.v1.cri StateDir:/run/k3s/containerd/io.containerd.grpc.v1.cri}" May 14 20:30:53 home-ubuntu k3s[11731]: time="2019-05-14T20:30:53.277992411Z" level=info msg="Connect containerd service" May 14 20:30:53 home-ubuntu k3s[11731]: time="2019-05-14T20:30:53.278197601Z" level=info msg="Get image filesystem path \"/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs\"" May 14 20:30:53 home-ubuntu k3s[11731]: time="2019-05-14T20:30:53.278553015Z" level=info msg="loading plugin \"io.containerd.grpc.v1.introspection\"..." type=io.containerd.grpc.v1 May 14 20:30:53 home-ubuntu k3s[11731]: time="2019-05-14T20:30:53.278756728Z" level=info msg=serving... address=/run/k3s/containerd/containerd.sock May 14 20:30:53 home-ubuntu k3s[11731]: time="2019-05-14T20:30:53.278847346Z" level=info msg="containerd successfully booted in 0.024563s" May 14 20:30:53 home-ubuntu k3s[11731]: time="2019-05-14T20:30:53.283062933Z" level=info msg="Start subscribing containerd event" May 14 20:30:53 home-ubuntu k3s[11731]: time="2019-05-14T20:30:53.283188201Z" level=info msg="Start recovering state" May 14 20:30:53 home-ubuntu k3s[11731]: time="2019-05-14T20:30:53.283345246Z" level=info msg="Start event monitor" May 14 20:30:53 home-ubuntu k3s[11731]: time="2019-05-14T20:30:53.283429704Z" level=info msg="Start snapshots syncer" May 14 20:30:53 home-ubuntu k3s[11731]: time="2019-05-14T20:30:53.283507970Z" level=info msg="Start streaming server" May 14 20:30:54 home-ubuntu k3s[11731]: time="2019-05-14T20:30:54.225127364Z" level=info msg="Connecting to wss://192.168.1.30:6443/v1-k3s/connect" May 14 20:30:54 home-ubuntu k3s[11731]: time="2019-05-14T20:30:54.225633260Z" level=info msg="Connecting to proxy" url="wss://192.168.1.30:6443/v1-k3s/connect" May 14 20:30:54 home-ubuntu k3s[11731]: time="2019-05-14T20:30:54.240693604Z" level=info msg="Running kubelet --resolv-conf=/run/systemd/resolve/resolv.conf --container-runtime=remote --tls-cert-file=/var/lib/rancher/k3s/agent/token-node.crt --healthz-bind-address=127.0.0.1 --authentication-token-webhook=true --seccomp-profile-root=/var/lib/rancher/k3s/agent/kubelet/seccomp --cni-bin-dir=/var/lib/rancher/k3s/data/4e1224c66a9dbb9b03daefff200f4f8eaf45590fb722b6fe2924a201d6de2e8d/bin --cluster-dns=10.43.0.10 --read-only-port=0 --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --hostname-override=home-ubuntu --allow-privileged=true --cgroup-driver=cgroupfs --kubeconfig=/var/lib/rancher/k3s/agent/kubeconfig.yaml --address=0.0.0.0 --anonymous-auth=false --root-dir=/var/lib/rancher/k3s/agent/kubelet --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.pem --tls-private-key-file=/var/lib/rancher/k3s/agent/token-node.key --kubelet-cgroups=/systemd/system.slice --fail-swap-on=false --cluster-domain=cluster.local --authorization-mode=Webhook --serialize-image-pulls=false --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --cert-dir=/var/lib/rancher/k3s/agent/kubelet/pki --cni-conf-dir=/var/lib/rancher/k3s/agent/etc/cni/net.d --runtime-cgroups=/systemd/system.slice" May 14 20:30:54 home-ubuntu k3s[11731]: Flag --allow-privileged has been deprecated, will be removed in a future version May 14 20:30:54 home-ubuntu k3s[11731]: W0514 20:30:54.259589 11731 server.go:214] WARNING: all flags other than --config, --write-config-to, and --cleanup are deprecated. Please begin using a config file ASAP. May 14 20:30:54 home-ubuntu k3s[11731]: I0514 20:30:54.265390 11731 server.go:398] Version: v1.14.1-k3s.4 May 14 20:30:54 home-ubuntu k3s[11731]: time="2019-05-14T20:30:54.293791987Z" level=info msg="waiting for node home-ubuntu: nodes \"home-ubuntu\" not found" May 14 20:30:54 home-ubuntu k3s[11731]: I0514 20:30:54.322756 11731 server.go:591] --cgroups-per-qos enabled, but --cgroup-root was not specified. defaulting to / May 14 20:30:54 home-ubuntu k3s[11731]: I0514 20:30:54.323120 11731 container_manager_linux.go:261] container manager verified user specified cgroup-root exists: [] May 14 20:30:54 home-ubuntu k3s[11731]: I0514 20:30:54.323205 11731 container_manager_linux.go:266] Creating Container Manager object based on Node Config: {RuntimeCgroupsName:/systemd/system.slice SystemCgroupsName: KubeletCgroupsName:/systemd/system.slice ContainerRuntime:remote CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:cgroupfs KubeletRootDir:/var/lib/rancher/k3s/agent/kubelet ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:imagefs.available Operator:LessThan Value:{Quantity: Percentage:0.05} GracePeriod:0s MinReclaim:} {Signal:nodefs.available Operator:LessThan Value:{Quantity: Percentage:0.05} GracePeriod:0s MinReclaim:}]} QOSReserved:map[] ExperimentalCPUManagerPolicy:none ExperimentalCPUManagerReconcilePeriod:10s ExperimentalPodPidsLimit:-1 EnforceCPULimits:true CPUCFSQuotaPeriod:100ms} May 14 20:30:54 home-ubuntu k3s[11731]: I0514 20:30:54.323345 11731 container_manager_linux.go:286] Creating device plugin manager: true May 14 20:30:54 home-ubuntu k3s[11731]: I0514 20:30:54.323463 11731 state_mem.go:36] [cpumanager] initializing new in-memory state store May 14 20:30:54 home-ubuntu k3s[11731]: W0514 20:30:54.330725 11731 node.go:113] Failed to retrieve node info: nodes "home-ubuntu" not found May 14 20:30:54 home-ubuntu k3s[11731]: I0514 20:30:54.330741 11731 server_others.go:148] Using iptables Proxier. May 14 20:30:54 home-ubuntu k3s[11731]: W0514 20:30:54.330798 11731 proxier.go:314] invalid nodeIP, initializing kube-proxy with 127.0.0.1 as nodeIP May 14 20:30:54 home-ubuntu k3s[11731]: I0514 20:30:54.334052 11731 server.go:555] Version: v1.14.1-k3s.4 May 14 20:30:54 home-ubuntu k3s[11731]: I0514 20:30:54.344672 11731 conntrack.go:103] Set sysctl 'net/netfilter/nf_conntrack_max' to 131072 May 14 20:30:54 home-ubuntu k3s[11731]: I0514 20:30:54.344733 11731 conntrack.go:52] Setting nf_conntrack_max to 131072 May 14 20:30:54 home-ubuntu k3s[11731]: I0514 20:30:54.344792 11731 conntrack.go:103] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400 May 14 20:30:54 home-ubuntu k3s[11731]: I0514 20:30:54.344815 11731 conntrack.go:103] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600 May 14 20:30:54 home-ubuntu k3s[11731]: I0514 20:30:54.345128 11731 config.go:202] Starting service config controller May 14 20:30:54 home-ubuntu k3s[11731]: I0514 20:30:54.345142 11731 controller_utils.go:1027] Waiting for caches to sync for service config controller May 14 20:30:54 home-ubuntu k3s[11731]: I0514 20:30:54.345184 11731 config.go:102] Starting endpoints config controller May 14 20:30:54 home-ubuntu k3s[11731]: I0514 20:30:54.345204 11731 controller_utils.go:1027] Waiting for caches to sync for endpoints config controller May 14 20:30:54 home-ubuntu k3s[11731]: I0514 20:30:54.345659 11731 kubelet.go:299] Watching apiserver May 14 20:30:54 home-ubuntu k3s[11731]: I0514 20:30:54.369664 11731 kuberuntime_manager.go:210] Container runtime containerd initialized, version: 1.2.5+unknown, apiVersion: v1alpha2 May 14 20:30:54 home-ubuntu k3s[11731]: I0514 20:30:54.383909 11731 server.go:987] Started kubelet May 14 20:30:54 home-ubuntu systemd[1]: Started Lightweight Kubernetes. May 14 20:30:54 home-ubuntu k3s[11731]: I0514 20:30:54.387048 11731 fs_resource_analyzer.go:64] Starting FS ResourceAnalyzer May 14 20:30:54 home-ubuntu k3s[11731]: I0514 20:30:54.387288 11731 status_manager.go:152] Starting to sync pod status with apiserver May 14 20:30:54 home-ubuntu k3s[11731]: I0514 20:30:54.387298 11731 kubelet.go:1774] Starting kubelet main sync loop. May 14 20:30:54 home-ubuntu k3s[11731]: I0514 20:30:54.387307 11731 kubelet.go:1791] skipping pod synchronization - [container runtime status check may not have completed yet., PLEG is not healthy: pleg has yet to be successful.] May 14 20:30:54 home-ubuntu k3s[11731]: I0514 20:30:54.387354 11731 server.go:137] Starting to listen on 0.0.0.0:10250 May 14 20:30:54 home-ubuntu k3s[11731]: I0514 20:30:54.387747 11731 server.go:328] Adding debug handlers to kubelet server. May 14 20:30:54 home-ubuntu k3s[11731]: I0514 20:30:54.388350 11731 volume_manager.go:248] Starting Kubelet Volume Manager May 14 20:30:54 home-ubuntu k3s[11731]: I0514 20:30:54.391035 11731 desired_state_of_world_populator.go:130] Desired state populator starts to run May 14 20:30:54 home-ubuntu k3s[11731]: E0514 20:30:54.406753 11731 cri_stats_provider.go:373] Failed to get the info of the filesystem with mountpoint "/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs": unable to find data in memory cache. May 14 20:30:54 home-ubuntu k3s[11731]: E0514 20:30:54.406777 11731 kubelet.go:1250] Image garbage collection failed once. Stats initialization may not have completed yet: invalid capacity 0 on image filesystem May 14 20:30:54 home-ubuntu k3s[11731]: I0514 20:30:54.531837 11731 kubelet_node_status.go:281] Setting node annotation to enable volume controller attach/detach May 14 20:30:54 home-ubuntu k3s[11731]: I0514 20:30:54.547893 11731 cpu_manager.go:155] [cpumanager] starting with none policy May 14 20:30:54 home-ubuntu k3s[11731]: I0514 20:30:54.548002 11731 cpu_manager.go:156] [cpumanager] reconciling every 10s May 14 20:30:54 home-ubuntu k3s[11731]: I0514 20:30:54.548091 11731 policy_none.go:42] [cpumanager] none policy: Start May 14 20:30:54 home-ubuntu k3s[11731]: E0514 20:30:54.549209 11731 controller.go:194] failed to get node "home-ubuntu" when trying to set owner ref to the node lease: nodes "home-ubuntu" not found May 14 20:30:54 home-ubuntu k3s[11731]: W0514 20:30:54.565052 11731 manager.go:537] Failed to retrieve checkpoint for "kubelet_internal_checkpoint": checkpoint is not found May 14 20:30:54 home-ubuntu k3s[11731]: I0514 20:30:54.585609 11731 kubelet_node_status.go:70] Attempting to register node home-ubuntu May 14 20:30:54 home-ubuntu k3s[11731]: E0514 20:30:54.593410 11731 eviction_manager.go:247] eviction manager: failed to get summary stats: failed to get node info: node "home-ubuntu" not found May 14 20:30:54 home-ubuntu k3s[11731]: I0514 20:30:54.596512 11731 kubelet_node_status.go:73] Successfully registered node home-ubuntu May 14 20:30:54 home-ubuntu k3s[11731]: I0514 20:30:54.625293 11731 kuberuntime_manager.go:946] updating runtime config through cri with podcidr 10.42.1.0/24 May 14 20:30:54 home-ubuntu k3s[11731]: time="2019-05-14T20:30:54.625888861Z" level=info msg="No cni config template is specified, wait for other system components to drop the config." May 14 20:30:54 home-ubuntu k3s[11731]: I0514 20:30:54.626302 11731 kubelet_network.go:69] Setting Pod CIDR: -> 10.42.1.0/24 May 14 20:30:54 home-ubuntu k3s[11731]: I0514 20:30:54.805829 11731 reconciler.go:154] Reconciler: start to sync state May 14 20:30:56 home-ubuntu k3s[11731]: I0514 20:30:56.299737 11731 flannel.go:89] Determining IP address of default interface May 14 20:30:56 home-ubuntu k3s[11731]: I0514 20:30:56.300471 11731 flannel.go:102] Using interface with name enp0s3 and address 192.168.1.31 May 14 20:30:56 home-ubuntu k3s[11731]: I0514 20:30:56.301525 11731 kube.go:127] Waiting 10m0s for node controller to sync May 14 20:30:56 home-ubuntu k3s[11731]: I0514 20:30:56.301647 11731 kube.go:306] Starting kube subnet manager May 14 20:30:57 home-ubuntu k3s[11731]: I0514 20:30:57.302469 11731 kube.go:134] Node controller sync successful May 14 20:30:57 home-ubuntu k3s[11731]: I0514 20:30:57.303067 11731 vxlan.go:120] VXLAN config: VNI=1 Port=0 GBP=false DirectRouting=false May 14 20:30:57 home-ubuntu k3s[11731]: I0514 20:30:57.317812 11731 flannel.go:75] Wrote subnet file to /run/flannel/subnet.env May 14 20:30:57 home-ubuntu k3s[11731]: I0514 20:30:57.317825 11731 flannel.go:79] Running backend. May 14 20:30:57 home-ubuntu k3s[11731]: I0514 20:30:57.317830 11731 vxlan_network.go:60] watching for new subnet leases May 14 20:30:57 home-ubuntu k3s[11731]: I0514 20:30:57.318481 11731 iptables.go:145] Some iptables rules are missing; deleting and recreating rules May 14 20:30:57 home-ubuntu k3s[11731]: I0514 20:30:57.318488 11731 iptables.go:167] Deleting iptables rule: -s 10.42.0.0/16 -d 10.42.0.0/16 -j RETURN May 14 20:30:57 home-ubuntu k3s[11731]: I0514 20:30:57.318758 11731 iptables.go:167] Deleting iptables rule: -s 10.42.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE --random-fully May 14 20:30:57 home-ubuntu k3s[11731]: I0514 20:30:57.319044 11731 iptables.go:167] Deleting iptables rule: ! -s 10.42.0.0/16 -d 10.42.1.0/24 -j RETURN May 14 20:30:57 home-ubuntu k3s[11731]: I0514 20:30:57.319311 11731 iptables.go:167] Deleting iptables rule: ! -s 10.42.0.0/16 -d 10.42.0.0/16 -j MASQUERADE --random-fully May 14 20:30:57 home-ubuntu k3s[11731]: I0514 20:30:57.319592 11731 iptables.go:155] Adding iptables rule: -s 10.42.0.0/16 -d 10.42.0.0/16 -j RETURN May 14 20:30:57 home-ubuntu k3s[11731]: I0514 20:30:57.321515 11731 iptables.go:155] Adding iptables rule: -s 10.42.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE --random-fully May 14 20:30:57 home-ubuntu k3s[11731]: I0514 20:30:57.322163 11731 iptables.go:155] Adding iptables rule: ! -s 10.42.0.0/16 -d 10.42.1.0/24 -j RETURN May 14 20:30:57 home-ubuntu k3s[11731]: I0514 20:30:57.325100 11731 iptables.go:155] Adding iptables rule: ! -s 10.42.0.0/16 -d 10.42.0.0/16 -j MASQUERADE --random-fully May 14 20:30:57 home-ubuntu k3s[11731]: I0514 20:30:57.326550 11731 iptables.go:145] Some iptables rules are missing; deleting and recreating rules May 14 20:30:57 home-ubuntu k3s[11731]: I0514 20:30:57.326560 11731 iptables.go:167] Deleting iptables rule: -s 10.42.0.0/16 -j ACCEPT May 14 20:30:57 home-ubuntu k3s[11731]: I0514 20:30:57.329360 11731 iptables.go:167] Deleting iptables rule: -d 10.42.0.0/16 -j ACCEPT May 14 20:30:57 home-ubuntu k3s[11731]: I0514 20:30:57.329703 11731 iptables.go:155] Adding iptables rule: -s 10.42.0.0/16 -j ACCEPT May 14 20:30:57 home-ubuntu k3s[11731]: I0514 20:30:57.330390 11731 iptables.go:155] Adding iptables rule: -d 10.42.0.0/16 -j ACCEPT May 14 20:31:48 home-ubuntu k3s[11731]: I0514 20:31:48.395415 11731 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "default-token-2f6rr" (UniqueName: "kubernetes.io/secret/4c202600-7687-11e9-aa0a-000af77c5868-default-token-2f6rr") pod "nginx-jn2mw" (UID: "4c202600-7687-11e9-aa0a-000af77c5868") May 14 20:31:48 home-ubuntu k3s[11731]: W0514 20:31:48.512614 11731 raw.go:87] Error while processing event ("/sys/fs/cgroup/cpu,cpuacct/system.slice/run-re59e9cc84ea14690841a3e3d69bb4d4b.scope": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/cpu,cpuacct/system.slice/run-re59e9cc84ea14690841a3e3d69bb4d4b.scope: no such file or directory May 14 20:31:48 home-ubuntu k3s[11731]: W0514 20:31:48.512657 11731 raw.go:87] Error while processing event ("/sys/fs/cgroup/blkio/system.slice/run-re59e9cc84ea14690841a3e3d69bb4d4b.scope": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/blkio/system.slice/run-re59e9cc84ea14690841a3e3d69bb4d4b.scope: no such file or directory May 14 20:31:48 home-ubuntu k3s[11731]: W0514 20:31:48.512674 11731 raw.go:87] Error while processing event ("/sys/fs/cgroup/memory/system.slice/run-re59e9cc84ea14690841a3e3d69bb4d4b.scope": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/memory/system.slice/run-re59e9cc84ea14690841a3e3d69bb4d4b.scope: no such file or directory May 14 20:31:48 home-ubuntu k3s[11731]: W0514 20:31:48.512685 11731 raw.go:87] Error while processing event ("/sys/fs/cgroup/devices/system.slice/run-re59e9cc84ea14690841a3e3d69bb4d4b.scope": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/devices/system.slice/run-re59e9cc84ea14690841a3e3d69bb4d4b.scope: no such file or directory May 14 20:31:48 home-ubuntu k3s[11731]: time="2019-05-14T20:31:48.675197793Z" level=info msg="RunPodsandbox for &PodSandboxMetadata{Name:nginx-jn2mw,Uid:4c202600-7687-11e9-aa0a-000af77c5868,Namespace:default,Attempt:0,}" May 14 20:31:49 home-ubuntu k3s[11731]: time="2019-05-14T20:31:49.239254167Z" level=debug msg="Wrote ping" May 14 20:31:50 home-ubuntu k3s[11731]: time="2019-05-14T20:31:50.963468649Z" level=info msg="ImageCreate event &ImageCreate{Name:k8s.gcr.io/pause:3.1,Labels:map[string]string{},}" May 14 20:31:51 home-ubuntu k3s[11731]: time="2019-05-14T20:31:51.007659199Z" level=info msg="ImageCreate event &ImageCreate{Name:sha256:da86e6ba6ca197bf6bc5e9d900febd906b133eaa4750e6bed647b0fbe50ed43e,Labels:map[string]string{io.cri-containerd.image: managed,},}" May 14 20:31:51 home-ubuntu k3s[11731]: time="2019-05-14T20:31:51.008251852Z" level=info msg="ImageUpdate event &ImageUpdate{Name:k8s.gcr.io/pause:3.1,Labels:map[string]string{io.cri-containerd.image: managed,},}" May 14 20:31:51 home-ubuntu k3s[11731]: time="2019-05-14T20:31:51.031802056Z" level=info msg="ImageUpdate event &ImageUpdate{Name:sha256:da86e6ba6ca197bf6bc5e9d900febd906b133eaa4750e6bed647b0fbe50ed43e,Labels:map[string]string{io.cri-containerd.image: managed,},}" May 14 20:31:51 home-ubuntu k3s[11731]: time="2019-05-14T20:31:51.033982425Z" level=info msg="ImageUpdate event &ImageUpdate{Name:k8s.gcr.io/pause:3.1,Labels:map[string]string{io.cri-containerd.image: managed,},}" May 14 20:31:51 home-ubuntu k3s[11731]: time="2019-05-14T20:31:51.041692134Z" level=info msg="ImageCreate event &ImageCreate{Name:k8s.gcr.io/pause@sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea,Labels:map[string]string{io.cri-containerd.image: managed,},}" May 14 20:31:51 home-ubuntu k3s[11731]: time="2019-05-14T20:31:51.134885515Z" level=info msg="shim containerd-shim started" address=/containerd-shim/k8s.io/02697cc2519c0c5fb404b3a255822bc5a71ecb9da54bf6d68d0f6be17cde5242/shim.sock debug=false pid=12085 May 14 20:33:51 home-ubuntu k3s[11731]: E0514 20:33:51.506213 11731 remote_runtime.go:186] ListPodSandbox with filter nil from runtime service failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded May 14 20:33:51 home-ubuntu k3s[11731]: E0514 20:33:51.506293 11731 kuberuntime_sandbox.go:210] ListPodSandbox failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded May 14 20:33:51 home-ubuntu k3s[11731]: E0514 20:33:51.506311 11731 generic.go:205] GenericPLEG: Unable to retrieve pods: rpc error: code = DeadlineExceeded desc = context deadline exceeded May 14 20:33:52 home-ubuntu k3s[11731]: E0514 20:33:52.392974 11731 remote_runtime.go:186] ListPodSandbox with filter &PodSandboxFilter{Id:,State:&PodSandboxStateValue{State:SANDBOX_READY,},LabelSelector:map[string]string{},} from runtime service failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded May 14 20:33:52 home-ubuntu k3s[11731]: E0514 20:33:52.393361 11731 kuberuntime_sandbox.go:210] ListPodSandbox failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded May 14 20:33:52 home-ubuntu k3s[11731]: E0514 20:33:52.393732 11731 kubelet_pods.go:1019] Error listing containers: &status.statusError{Code:4, Message:"context deadline exceeded", Details:[]*any.Any(nil)} May 14 20:33:52 home-ubuntu k3s[11731]: E0514 20:33:52.393913 11731 kubelet.go:1942] Failed cleaning pods: rpc error: code = DeadlineExceeded desc = context deadline exceeded May 14 20:33:54 home-ubuntu k3s[11731]: time="2019-05-14T20:33:54.239713492Z" level=debug msg="Wrote ping" May 14 20:33:54 home-ubuntu k3s[11731]: E0514 20:33:54.468564 11731 remote_runtime.go:186] ListPodSandbox with filter nil from runtime service failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded May 14 20:33:54 home-ubuntu k3s[11731]: E0514 20:33:54.469246 11731 kuberuntime_sandbox.go:210] ListPodSandbox failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded May 14 20:33:54 home-ubuntu k3s[11731]: E0514 20:33:54.469463 11731 kubelet.go:1222] Container garbage collection failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded May 14 20:33:54 home-ubuntu k3s[11731]: E0514 20:33:54.662278 11731 remote_runtime.go:186] ListPodSandbox with filter &PodSandboxFilter{Id:,State:nil,LabelSelector:map[string]string{},} from runtime service failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded May 14 20:33:54 home-ubuntu k3s[11731]: E0514 20:33:54.662629 11731 eviction_manager.go:247] eviction manager: failed to get summary stats: failed to list pod stats: failed to list all pod sandboxes: rpc error: code = DeadlineExceeded desc = context deadline exceededAdditional context
Add any other context about the problem here.
The text was updated successfully, but these errors were encountered: