isisd: Missing routes/corrupted LSDB with large complex networks

[mwinter-osr]

This seems to happen with ISIS topologies of more than approx 6000 routes and > 3000 routers.

The issue seems to be timing related - if debug isis snp-packets is configured, then everything works fine.

Output without the debug:

osr-perf3# show ip route summ
Route Source         Routes               FIB  (vrf Default-IP-Routing-Table)
kernel               1                    1                    
connected            5                    5                    
static               1                    1                    
isis                 6970                 6969                 
------
Totals               6977                 6976                 

and with the debug (correct):

osr-perf3# show ip route summ
Route Source         Routes               FIB  (vrf Default-IP-Routing-Table)
kernel               1                    1                    
connected            5                    5                    
static               1                    1                    
isis                 7073                 7072                 
------
Totals               7080                 7079

The issue can be tracked down to debug call in isis_pdu.c near lines 1780-1792:

if (isis->debugs & DEBUG_SNP_PACKETS) {
	zlog_debug(
		"ISIS-Snp (%s): Sending L%d CSNP on %s, length %zd",
			circuit->area->area_tag, level,
			circuit->interface->name,
			stream_get_endp(circuit->snd_stream));
	log_multiline(LOG_DEBUG, "              ", "%s",
		isis_format_tlvs(tlvs));
	if (isis->debugs & DEBUG_PACKET_DUMP)
		zlog_dump_data(
			STREAM_DATA(circuit->snd_stream),
			stream_get_endp(circuit->snd_stream));
}

Just adding a simple usleep(5000) in front of this will avoid the issue (and make it work without the debug enabled).

[mwinter-osr]

Christian is working on this and has the information and testbed to reproduce the issue.

[cfra]

Got full 7072 routes with master-2017-08-23 when I started frr while ixia was already running. Still investigating for other sequences and for stable/3.0

[donaldsharp]

This has been a long-standing issue, should we just document this as opposed to making this a 3.0 release gating issue?