Files
lcpng/lcpng_netlink.h
Pim van Pelt 7a76498277 Add NEWROUTE/DELROUTE handler
This is super complicated work, taken mostly verbatim from the upstream
linux-cp Gerrit, with due credit mgsmith@netgate.com neale@grafiant.com

First, add main handler lcp_nl_route_add() and lcp_nl_route_del()

Introduce two FIB sources: one for manual routes, one for dynamic
routes.  See lcp_nl_proto_fib_source() fo details.

Add a bunch of helpers that translate Netlink message data into VPP
primitives:
- lcp_nl_mk_addr46()
    converts a Netlink nl_addr to a VPP ip46_address_t.

- lcp_nl_mk_route_prefix()
    converts a Netlink rtnl_route to a VPP fib_prefix_t.

- lcp_nl_mk_route_mprefix()
     converts a Netlink rtnl_route to a VPP mfib_prefix_t.

- lcp_nl_proto_fib_source()
    selects the most appropciate fib_src by looking at the rt_proto
    (see /etc/iproute2/rt_protos for a hint). Anything RTPROT_STATIC or
    better is 'fib_src', while anything above that becomes fib_src_dynamic.

- lcp_nl_mk_route_entry_flags()
    generates fib_entry_flag_t from the Netlink route type,
    table and proto metadata.

- lcp_nl_route_path_parse()
    converts a Netlink rtnl_nexthop to VPP fib_route_path_t and adds
    that to a growing list of paths.

- lcp_nl_route_path_add_special()
    adds a blackhole/unreach/prohibit route to the list of paths, in
    the special-case there is not yet a path for the destination.

Now we're ready to insert FIB entries:
- lcp_nl_table_find()
    selects the matching table-id,protocol(v4/v6) from a hash of tables.

- lcp_nl_table_add_or_lock()
    if at table-id,protocol(v4/v6) hasn't been used yet, create one,
    otherwise increment a table reference counter so we know how many
    FIB entries we have in this table. Then, return it.

- lcp_nl_table_unlock()
    Decrease the refcount on a table, and if no more prefixes are in
    the table, remove it from VPP.

- lcp_nl_route_del()
    Remove a route from the given table-id/protocol. Do this by applying
    rtnl_route_foreach_nexthop() to the list of Netlink nexthops,
    converting them into VPP paths in a lcp_nl_route_path_parse_t
    structure. If the route is for unreachable/blackhole/prohibit in
    Linux, add that path too.
    Then, remove the VPP paths from the FIB and reduce refcnt or
    remove the table if it's empty using table_unlock().

- lcp_nl_route_add()
    Not all routes are relevant for VPP. Those in table 255 are 'local'
    routes, already set up by ip[46]_address_add(), and some other route
    types are invalid, skip those. Link-local IPv6 and IPv6 multicast is
    also skipped. Then, construct lcp_nl_route_path_parse_t by walking
    the Netlink nexthops, and optionally add a special (in case the
    route was for unreachable/blackhole/prohibit in Linux -- those won't
    have a nexthop).
    Then, insert the VPP paths found in the Netlink message into the FIB
    or the multicast FIB, respectively.

And with that, Bird shoots to life. Both IPv4 and IPv6 OSPF interior
gateway protocol and BGP full tables can be consumed, on my bench in
about 9 seconds:
- A batch of 2048 Netlink messages is handled in 9-11ms, so we can do
  approx 200K messages/sec at peak (and this will consume 50% CPU due
  to the yielding logic in lcp_nl_process() (see the 'case
  NL_EVENT_READ' block that adds a cooldown period of
  LCP_NL_PROCESS_WAIT milliseconds between batches.
- With 3 route reflectors and 2 full BGP peers, at peak I could see
  309K messages left in the producer queue.

- All IPv4 and IPv6 prefixes made their way into the FIB
pim@hippo:~/src/lcpng$ echo -n "IPv6: "; vppctl sh ip6 fib summary | awk '$1~/[0-9]+/ { total += $2 } END { print total }'
IPv6: 132506
pim@hippo:~/src/lcpng$ echo -n "IPv4: "; vppctl sh ip fib summary | awk '$1~/[0-9]+/ { total += $2 } END { print total }'
IPv4: 869966

- Compared to Bird2's view:
pim@hippo:~/src/lcpng$ birdc show route count
BIRD 2.0.7 ready.
3477845 of 3477845 routes for 869942 networks in table master4
527887 of 527887 routes for 132484 networks in table master6
Total: 4005732 of 4005732 routes for 1002426 networks in 2 tables

- Flipping one of the full feeds to another, forcing a reconvergence
  of every prefix in the FIB took about 8 seconds, peaking at 242K
  messages in the queue, with again an average consumption of 2048
  messages per 9-10ms.

- All of this was done while iperf'ing 6Gbps to and from the
  controlplane.
---

Because handling full BGP table is O(1M) messages, I will have to make
some changes in the logging:
- all neigh/route messages become DBG/INFO at best
- all addr/link messages become INFO/NOTICE at best
- when we overflow time/msgs, turn process_msgs into a WARN, otherwise
  keep it at INFO so as not to spam.

In lcpng_interface.c:
- Log NOTICE for pair_add() and pair_del() call;
- Log NOTICE for set_interface_addr() call;

With this approach, setting the logging level of the linux-cp/nl plugin
to 'notice' hits the sweet spot: with things that the operator has
~explicitly done, leaving implicit actions (BGP route adds/dels, ARP/ND)
to stay below the NOTICE level.
2021-08-29 14:57:21 +02:00

128 lines
3.6 KiB
C

/*
* Copyright (c) 2021 Cisco and/or its affiliates.
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at:
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
#include <vlib/vlib.h>
#include <plugins/lcpng/lcpng.h>
#include <netlink/msg.h>
#include <netlink/netlink.h>
#include <netlink/socket.h>
#include <netlink/route/link.h>
#include <netlink/route/route.h>
#include <netlink/route/neighbour.h>
#include <netlink/route/addr.h>
#include <netlink/route/link/vlan.h>
#include <vnet/fib/fib_table.h>
#include <vnet/mfib/mfib_table.h>
typedef enum nl_event_type_t_
{
NL_EVENT_READ,
NL_EVENT_READ_ERR,
} nl_event_type_t;
#define NL_RX_BUF_SIZE_DEF (1 << 27) /* 128 MB */
#define NL_TX_BUF_SIZE_DEF (1 << 18) /* 256 kB */
#define NL_BATCH_SIZE_DEF (1 << 11) /* 2048 */
#define NL_BATCH_DELAY_MS_DEF 50 /* 50 ms, max 20 batch/s */
#define NL_DBG(...) vlib_log_debug (lcp_nl_main.nl_logger, __VA_ARGS__);
#define NL_INFO(...) vlib_log_info (lcp_nl_main.nl_logger, __VA_ARGS__);
#define NL_NOTICE(...) vlib_log_notice (lcp_nl_main.nl_logger, __VA_ARGS__);
#define NL_WARN(...) vlib_log_warn (lcp_nl_main.nl_logger, __VA_ARGS__);
#define NL_ERROR(...) vlib_log_err (lcp_nl_main.nl_logger, __VA_ARGS__);
/* struct type to hold context on the netlink message being processed.
*/
typedef struct nl_msg_info
{
struct nl_msg *msg;
f64 ts;
} nl_msg_info_t;
typedef struct lcp_nl_netlink_namespace
{
struct nl_sock *sk_route;
nl_msg_info_t *nl_msg_queue;
uword clib_file_index; // clib file that holds the netlink socket for this
// namespace
u32 clib_file_lcp_refcnt; // number of interfaces watched in the this netlink
// namespace
u8 netns_name[LCP_NS_LEN]; // namespace name (can be empty, for 'self')
} lcp_nl_netlink_namespace_t;
typedef struct lcp_nl_table_t_
{
uint32_t nlt_id;
fib_protocol_t nlt_proto;
u32 nlt_fib_index;
u32 nlt_mfib_index;
u32 nlt_refs;
} lcp_nl_table_t;
typedef struct lcp_nl_main
{
vlib_log_class_t nl_logger;
/* TODO(pim): nl_ns should become a list, one for each unique namespace we
* created LCP pairs in.
*/
lcp_nl_netlink_namespace_t nl_ns;
fib_source_t fib_src; // For static routes set manually
fib_source_t
fib_src_dynamic; // For routes set by routing software (Bird, FRR, etc)
uword *table_db[FIB_PROTOCOL_MAX];
lcp_nl_table_t *table_pool;
u32 rx_buf_size;
u32 tx_buf_size;
u32 batch_size;
u32 batch_delay_ms;
} lcp_nl_main_t;
extern lcp_nl_main_t lcp_nl_main;
typedef struct lcp_nl_route_path_parse_t_
{
fib_route_path_t *paths;
fib_protocol_t route_proto;
bool is_mcast;
fib_route_path_flags_t type_flags;
u8 preference;
} lcp_nl_route_path_parse_t;
u8 *format_nl_object (u8 *s, va_list *args);
/* Functions from lcpng_nl_sync.c
*/
void lcp_nl_neigh_add (struct rtnl_neigh *rn);
void lcp_nl_neigh_del (struct rtnl_neigh *rn);
void lcp_nl_addr_add (struct rtnl_addr *ra);
void lcp_nl_addr_del (struct rtnl_addr *ra);
void lcp_nl_link_add (struct rtnl_link *rl, void *ctx);
void lcp_nl_link_del (struct rtnl_link *rl);
void lcp_nl_route_add (struct rtnl_route *rr);
void lcp_nl_route_del (struct rtnl_route *rr);
/*
* fd.io coding-style-patch-verification: ON
*
* Local Variables:
* eval: (c-set-style "gnu")
* End:
*/