Examination of the Closed Column of the ss Command

Since the netstat (actually net-tools) is deprecated, people are moving from netstat to the ss command.

I see that a lot of people get confused with the output of ss -s command.

The closed column is especially mind-blowing.

Total: 246
TCP: 129 (estab 87, closed 28, orphaned 4, timewait 28)
Transport Total IP IPv6
RAW 1 0 1
UDP 6 4 2
TCP 101 94 7
INET 108 98 10
FRAG 0 0 0

This output is from my test server. The test server’s OS is Ubuntu 20.04.1 LTS, Linux Kernel version is 5.4.0 and iproute2 package version is 5.5.0

You may get different output in older/newer Linux servers.

The ss stands for socket statistics.

How does ss collect/calculate the socket statistics?

A screenshot from ss’ source code

As you have probably seen, ss calculates the number of closed connections with this simple mathematical operation:

s.tcp_total - (s.tcp4_hashed + s.tcp6_hashed - s.tcp_tws)

Let’s stop here.

The ss is actually parsing the output of /proc/net/sockstat and /proc/net/sockstat6:

The ss is parsing the output of the /proc/net/sockstat and proc/net/sockstat6

tcp_total is actually “alloc” in the output of /proc/net/sockstat.
tcp4_hashed is actually “inuse” in the output of /proc/net/sockstat
tcp6_hashed is actually “inuse” in the output of /proc/net/sockstat6
tcp_tws is actually “tw” in the output of /proc/net/sockstat

P.S. TWS stands for “timewaits”

So, the output of /proc/net/sockstat and sockstat6 must be consistent with the output of ss -s:

root@adil:~# cat /proc/net/sockstat && echo "---" && cat /proc/net/sockstat6 && echo "---" && ss -ssockets: used 400
TCP: inuse 143 orphan 1 tw 38 alloc 247 mem 86
UDP: inuse 4 mem 3
UDPLITE: inuse 0
RAW: inuse 0
FRAG: inuse 0 memory 0
---
TCP6: inuse 4
UDP6: inuse 2
UDPLITE6: inuse 0
RAW6: inuse 1
FRAG6: inuse 0 memory 0
---
Total: 400
TCP: 285 (estab 136, closed 138, orphaned 1, timewait 38)
Transport Total IP IPv6
RAW 1 0 1
UDP 6 4 2
TCP 147 143 4
INET 154 147 7
FRAG 0 0 0

Let’s parse the output of /proc/net/sockstat and sockstat6 by manually:

s.tcp_total   = 247
s.tcp4_hashed = 143
s.tcp6_hashed = 4
s.tcp_tws = 38

Let’s remember the mathematical operational that calculates the number of closed connections:

s.tcp_total - (s.tcp4_hashed + s.tcp6_hashed - s.tcp_tws)

Let’s put the values to correct places:

247 - (143 + 4 - 38)
Result (the number of closed connections): 138

38 of 138 closed TCP sockets are actually in the TIME_WAIT state.

How about other 100 TCP sockets?

We know that the closed connections are calculated based on the “alloc” column. Then, we should examine the “alloc” column that is in the output of /proc/net/sockstat now.

What is the alloc column in the sockstat output?

There are 2 types of the TCP sockets in the socket statistics: allocated and in use.

All TCP socket states count as alloc.
All TCP socket states except TCP_CLOSE count as inuse.

So, we found out that other 100 TCP sockets are in the TCP_CLOSE state.

A TCP socket can be flagged as TCP_CLOSE in many cases. However, I’d like to point out a common case.

On the TCP creation, the necessary fields are set in the sock_init_data function in the source code

It may sound strange that Kernel sets the initial state of a TCP socket as “TCP_CLOSE

Let’s test it with a simple PHP script:

<?php
$socket = [];
for($i = 0; $i < 100; $i++){
$socket[$i] = socket_create(AF_INET, SOCK_STREAM, 0);
}
sleep (30);

As you can see, we create 100 sockets and we do nothing else.

The output of ss -s is before this script is like this:

root@adil:~# ss -s
Total: 145
TCP: 4 (estab 2, closed 0, orphaned 0, timewait 0)
Transport Total IP IPv6
RAW 1 0 1
UDP 1 1 0
TCP 4 3 1
INET 6 4 2
FRAG 0 0 0

The output of ss -s has changed when I ran the script:

root@adil:~# ss -s
Total: 245
TCP: 104 (estab 2, closed 100, orphaned 0, timewait 0)
Transport Total IP IPv6
RAW 1 0 1
UDP 1 1 0
TCP 4 3 1
INET 6 4 2
FRAG 0 0 0

We have seen that the kernel creates the TCP socket with the TCP_CLOSE state.

So, if you have high numbers on the closed column and low numbers on the timewait column, your application might create TCP sockets and do nothing.

However, don’t forget that the kernel may flag a TCP socket as TCP_CLOSE in many cases also. This case is one of those cases and the common one.