Cisco Nexus 9000 Series NX-OS Programmability Guide, Release 9.3(x) - gNMI - gRPC Network Management Interface [Cisco Nexus 9000 Series Switches] (2025)

Table of Contents

About gNMI VRF Contexts for gNMI gNMI Subscribe RPC Optional SUBSCRIBE Flags Guidelines and Limitations for gNMI Configuring gNMI Procedure Example: Example: Example: Example: Example: Example: gNMI - gRPC Network Management Interface Configuring Server Certificate Generating Key/Certificate Examples Generating and Configuring Key/Certificate Examples for Cisco NX-OS Release 9.3(2) and Earlier Procedure Examples for Generating and Configuring Key/Certificate for Cisco NX-OS Release 9.3(3) and Later Procedure Verifying gNMI show grpc gnmi service statistics Example show grpc gnmi rpc summary Example show grpc gnmi transactions Example Clients Sample DME Subscription - PROTO Encoding About Capabilities Guidelines and Limitations for Capabilities Example Client Output for Capabilities About Get Guidelines and Limitations for Get About Set Guidelines and Limitations for Set Guidelines and Limitations for Subscribe gNMI Payload Sample gNMI Payload for DME Stream Sample gNMI Payload YANG Stream About Streaming Syslog for gNMI Guidelines and Limitations for Streaming Syslog - gNMI Syslog Native YANG Model Subscribe Request Example Sample PROTO Output Sample JSON Output Gathering TM-Trace Logs Gathering MTX-Internal Logs

This chapter contains the following topics:

About gNMI

gNMI uses gRPC (Google Remote Procedure Call) as its transport protocol.

Cisco NX-OS supports gNMI for dial-in subscription to telemetry applications running on the Cisco Nexus 9000 Series switches. Although past release supported telemetry events over gRPC, the switch pushed the telemetry data to the telemetry receivers. This method was called dial out.

With gNMI, applications can pull information from the switch. They subscribe to specific telemetry services by learning the supported telemetry capabilities and subscribing to only the telemetry services that it needs.

Table 1. Supported gNMI RPCs
gNMI RPC	Supported
Capabilities	Yes
Get	Yes
Set	Yes
Subscribe	Yes

VRF Contexts for gNMI

In previous releases, the gRPC server ran in the management VRF only. As a result, the gRPC process could communicate only in this VRF, forcing the management interface to support all gRPC calls. This configuration is still supported in release 9.3(1).

Beginning in release 9.3(1), gRPC functionality now includes the default VRF for a total of 2 gRPC servers on each Cisco Nexus 9000 switch. You can run one gRPC server in each VRF, or run only one gRPC server in the management VRF. Supporting a gRPC in the default VRF adds flexibility to offload processing gRPC calls from the management VRF, where significant traffic load might not be desirable.

If two gRPC servers are configured, be aware of the following:

VRF boundaries are strictly enforced, so each gRPC server processes requests independent of the other, and requests do not cross between VRFs.
The two servers are not HA or fault tolerant. One gRPC server does not back up the other, and there is no switchover or switchback between them.
Any limits for the gRPC server are per VRF.

gNMI Subscribe RPC

The Cisco NX-OS 9.3(1) release and later support the following gNMI Subscription features:

Table 2. Subscribe Options
Type	Sub Type	Supported?	Description
Once		Yes	Switch sends current values only once for all specified paths
Poll		Yes	Whenever the switch receives a Poll message, the switch sends the current values for all specified paths.
Stream	Sample	Yes	Once per stream sample interval, the switch sends the current values for all specified paths. The supported sample interval range is from 1 through 604800 seconds. The default sample interval is 10 seconds.
	On_Change	Yes	The switch sends current values as its initial state, but then updates the values only when changes, such as create, modify, or delete occur to any of the specified paths.
	Target_Defined	No

Optional SUBSCRIBE Flags

For the SUBSCRIBE option, some optional flags are available that modify the response to the options listed in the table. In Cisco NX-OS release 9.3(1) and later, the updates_only optional flag is supported, which is applicable to ON_CHANGE subscriptions. If this flag is set, the switch suppresses the initial snapshot data (current state) that is normally sent with the first response.

The following flags are not supported:

aliases
allow_aggregation
extensions
prefix
qos

The following is the support metrics for the subscribe flags:

Table 3. Support Metrics for SUBSCRIBE flags
Subscription Type	heartbeat_interval	suppress_redundant
On_Change	Origin: Device YANG, OpenConfig YANG, DME	N/A
Sample	Origin: Device YANG, OpenConfig YANG, DME	Origin: Device YANG, OpenConfig YANG

Guidelines and Limitations for gNMI

Following are the guidelines and limitations for gNMI:

Beginning with Cisco NX-OS Release 9.3(5), Get and Set are supported.
gNMI queries do not support wildcards in paths.
gRPC traffic destined for a Nexus device will hit the control-plane policer (CoPP) in the default class. To limit the possibility of gRPC drops, configure a custom CoPP policy using the gRPC configured port in the management class.
When you enable gRPC on both the management VRF and default VRF and later disable on the default VRF, the gNMI notifications on the management VRF stop working.

As a workaround, disable gRPC completely by entering the no feature grpc command and reprovision it by entering the feature grpc command and any existing gRPC configuration commands. For example, grpc certificate or grpc port . You must also resubscribe to any existing notifications on the management VRF.

When you attempt to subscribe an OpenConfig routing policy with a preexisting CLI configuration like the following, it returns empty values due to the current implementation of the OpenConfig model.

ip prefix-list bgp_v4_drop seq 5 deny 125.2.0.0/16 le 32ipv6 prefix-list bgp_v6_drop seq 5 deny cafe:125:2::/48 le 128using the xpathopenconfig-routing-policy:/routing-policy/defined-sets/prefix-sets/prefix-set[name=bgp_v4_drop]/configopenconfig-routing-policy:/routing-policy/defined-sets/prefix-sets/prefix-set[name=bgp_v6_drop]/config

Only server certificate authentication takes place. The client certificate is not authenticated by the server.
If the gRPC certificate is explicitly configured, after a reload with the saved startup configuration to a prior Cisco NX-OS 9.3(x) image, the gRPC feature does not accept connections. To confirm this issue, enter the show grpc gnmi service statistics command and the status line displays an error like the following:
```
Status: Not running - Initializing...Port not available or certificate invalid.
```
Unconfigure and configure the proper certificate command to restore the service.
Beginning with Cisco NX-OS Release 9.3(3), if you have configured a custom gRPC certificate, upon entering the reload ascii command the configuration is lost. It reverts to the default day-1 certificate. After entering the reload ascii command, the switch reloads. Once the switch is up again, you must reconfigure the gRPC custom certificate.

Note

This applies when entering the grpc certificate command.
Use of origin, use_models, or both, is optional for gNMI subscriptions.
gNMI Subscription supports Cisco DME and Device YANG data models. Beginning with Cisco NX-OS Release 9.3(3), Subscribe supports the OpenConfig model.
For Cisco NX-OS prior to 9.3(x), information about supported platforms, see Platform Support for Programmability Features in the guide for that release. Starting with Cisco NX-OS release 9.3(x), for information about supported platforms, see the Nexus Switch Platform Matrix.
The gNMI feature supports the Subscribe and Capability gNMI RPCs.
The feature supports JSON and gnmi.proto encoding. The feature does not support protobuf.any encoding.
Each gNMI message has a maximum size of 12 MB. If the amount of collected data exceeds the 12 MB maximum, the collected data is dropped. Applies to gNMI ON_CHANGE mode only.

You can avoid this situation by creating more focused subscriptions that handle smaller, more granular data-collection sets. So, instead of subscribing to one higher-level path, create multiple subscriptions for different, lower-level parts of the path.
Across all subscriptions, there is support of up to 150K aggregate MOs. Subscribing to more MOs can lead to collection data drops.
The feature does not support a path prefix in the Subscription request, but the Subscription can contain an empty prefix field.
The gRPC process that supports gNMI uses the HIGH_PRIO control group, which limits the CPU usage to 75% of CPU and memory to 1.5 GB.
The show grpc gnmi command has the following considerations:
- The gRPC agent retains gNMI calls for a maximum of one hour after the call has ended.
- If the total number of calls exceeds 2000, the gRPC agent purges ended calls based on the internal cleanup routine.

The gRPC server runs in the management VRF. As a result, the gRPC process communicates only in this VRF forcing the management interface to support all gRPC calls.

gRPC functionality now includes the default VRF for a total of two gRPC servers on each switch. You can run one gRPC server in each VRF, or run only one gRPC server in the management VRF. Supporting a gRPC in the default VRF adds flexibility to offload processing gRPC calls from the management VRF, where significant traffic load is not desirable.

If two gRPC servers are configured, be aware of the following:

VRF boundaries are strictly enforced, so each gRPC server process requests independent of the other. Requests do not cross between VRFs.
The two servers are not HA or fault tolerant. One gRPC server does not back up the other, and there is no switchover or switchback between them.
Any limits for the gRPC server are per VRF.

Configuring gNMI

Configure the gNMI feature through the grpc gnmi commands.

Procedure

Command or Action Purpose

Step1

configure terminal

Example:

switch-1# configure terminalswitch-1(config)#

Enters global configuration mode.

Step2

feature grpc

Example:

switch-1# feature grpcswitch-1(config)#

Enables the gRPC agent, which supports the gNMI interface for dial-in.

Step3

(Optional) grpc port port-id

Example:

switch-1(config)# grpc port 50051

(Optional)

Configure the port number. The range of port-id is from 1024 to 65535. 50051 is the default.

Note

This command is available beginning with Cisco NX-OS Release 9.3(3).

Step4

grpc certificate certificate-id

Example:

switch-1(config)# grpc certificate cert-1

Specify the certificate trustpoint ID. For more information, see the Installing Identity Certificates section of the Cisco Nexus 9000 Series NX-OS Security Configuration Guide, Release 9.3(x) for importing the certificate to the switch.

Note

This command is available beginning with Cisco NX-OS Release 9.3(3).

Step5

grpc gnmi max-concurrent-call number

Example:

switch-1(config)# grpc gnmi max-concurrent-call 16switch-1(config)#

Sets the limit of simultaneous dial-in calls to the gNMI server on the switch. Configure a limit from 1 through 16. The default limit is 8.

The maximum value that you configure is for each VRF. If you set a limit of 16 and gNMI is configured for both management and default VRFs, each VRF supports 16 simultaneous gNMI calls.

This command does not affect and ongoing or in-progress gNMI calls. Instead, gRPC enforces the limit on new calls, so any in-progress calls are unaffected and allowed to complete.

Note

The configured limit does not affect the gRPCConfigOper service.

Step6

(Optional) grpc use-vrf default

Example:

switch(config)# grpc use-vrf default

(Optional)

Enables the gRPC agent to accept incoming (dial-in) RPC requests from the default VRF. This step enables the default VRF to process incoming RPC requests. By default, the management VRF processes incoming RPC requests when the gRPC feature is enabled.

Note

Both VRFs process requests individually, so that requests do not cross between VRFs.

gNMI - gRPC Network Management Interface

This section defines a gRPC-based protocol for the modification and retrieval of the configuration from a target device, as well as, the control and generation of telemetry streams from a target device to a data collection system. The intention is that a single gRPC service definition can cover both configuration and telemetry - allowing a single implementation on the target, as well as a single NMS element to interact with the device via telemetry and configuration RPCs.

Configuring Server Certificate

When you configured a TLS certificate and imported successfully onto the switch, the following is an example of the show grpc gnmi service statistics command output.

#show grpc gnmi service statistics=============gRPC Endpoint=============Vrf : managementServer address : [::]:50051Cert notBefore : Mon Jan 27 15:34:08 PDT 2020Cert notAfter : Tue Jan 26 15:34:08 PDT 2021Max concurrent calls : 8Listen calls : 1Active calls : 0Number of created calls : 1Number of bad calls : 0Subscription stream/once/poll : 0/0/0

gNMI communicates over gRPC and uses TLS to secure the channel between the switch and the client. The default hard-coded gRPC certificate is no longer shipped with the switch. The default behavior is a self-signed key and certificate which is generated on the switch as shown below with an expiration date of one day.

When the certificate is expired or failed to install successfully, you will see the 1-D default certificate. The following is an example of the show grpc gnmi service statistics command output.

#show grpc gnmi service statistics=============gRPC Endpoint=============Vrf : managementServer address : [::]:50051Cert notBefore : Wed Mar 11 19:43:01 PDT 2020Cert notAfter : Thu Mar 12 19:43:01 PDT 2020Max concurrent calls : 8Listen calls : 1Active calls : 0Number of created calls : 1Number of bad calls : 0Subscription stream/once/poll : 0/0/0

With an expiration of one day, you can use this temporary certificate for quick testing. For long term a new key/certificate must be generated.

Cisco Nexus 9000 Series NX-OS Programmability Guide, Release 9.3(x) - gNMI - gRPC Network Management Interface [Cisco Nexus 9000 Series Switches] (3)

Note

After the certificate expires, there are two ways to have the key/certificate to regenerate:

Reload the switch.
Manually delete the key/certificate in the /opt/mtx/etc folder and enter the no feature grpc and feature grpc commands.

Generating Key/Certificate Examples

Follow these examples to generate Key/Certificates:

Generating and Configuring Key/Certificate Examples for Cisco NX-OS Release 9.3(2) and Earlier

Generating and Configuring Key/Certificate Examples for Cisco NX-OS Release 9.3(2) and Earlier

The following is an example for generating key/certificate:

For more information on generating identify certificates, see the Installing Identity Certificates section of the Cisco Nexus 9000 Series NX-OS Security Configuration Guide, Release 9.3(x).

Procedure

Step1	Generate the selfsigned key and pem files. `switch# run bash sudo subash-4.3# openssl req -x509 -newkey rsa:2048 -keyout self_sign2048.key -out self_sign2048.pem -days 365 -nodes`
Step2	After generating the key and pem files, modify the mtx.conf.user files in the Bash shell to have the gRPC service pick up the certificates. `[grpc]key = /bootflash/self-sign2048.keycert = /bootflash/self-sign2048.pem`
Step3	Reload the box to have the gRPC service pick up the certificate.
Step4	Verify gRPC is now using the certificate. switch# show grpc gnmi service statistics=============gRPC Endpoint=============Vrf : managementServer address : [::]:50051Cert notBefore : Nov 5 16:48:58 2015 GMTCert notAfter : Nov 5 16:48:58 2035 GMTMax concurrent calls : 16Listen calls : 1Active calls : 0Number of created calls : 953Number of bad calls : 0Subscription stream/once/poll : 476/238/238Max gNMI::Get concurrent : 5Max grpc message size : 8388608gNMI Synchronous calls : 10gNMI Synchronous errors : 0gNMI Adapter errors : 0gNMI Dtx errors : 0

Examples for Generating and Configuring Key/Certificate for Cisco NX-OS Release 9.3(3) and Later

The following is an example for generating key/certificate.

Cisco Nexus 9000 Series NX-OS Programmability Guide, Release 9.3(x) - gNMI - gRPC Network Management Interface [Cisco Nexus 9000 Series Switches] (4)

Note

This task is an example of how a certificate can be generated on a switch. You can also generate a certificate in any Linux environment. In a production environment, you should consider using a CA signed certificate.

For more information on generating identity certificates, see the Installing Identity Certificates section of the Cisco Nexus 9000 Series NX-OS Security Configuration Guide, Release 9.3(x).

Procedure

Step1	Generate the selfsigned key and pem files. `switch# run bash sudo subash-4.3# openssl req -x509 -newkey rsa:2048 -keyout self_sign2048.key -out self_sign2048.pem -days 365 -nodes`
Step2	After generating the key and pem files, you must bundle the key and pem files for use in the trustpoint CA Association. `switch# run bash sudo subash-4.3# cd /bootflash/bash-4.3# openssl pkcs12 -export -out self_sign2048.pfx -inkey self_sign2048.key -in self_sign2048.pem -certfile self_sign2048.pem -password pass:Ciscolab123!bash-4.3# exit`
Step3	Verify the setup. switch(config)# show crypto ca certificatesTrustpoint: mytrustpointcertificate:subject= /C=US/O=Cisco Systems, Inc./OU=CSG/L=San Jose/ST=CA/street=3700 Cisco Way/postalCode=95134/CN=ems.cisco.com/serialNumber=FGE18420K0Rissuer= /C=US/O=Cisco Systems, Inc./OU=CSG/L=San Jose/ST=CA/street=3700 Cisco Way/postalCode=95134/CN=ems.cisco.com/serialNumber=FGE18420K0Rserial=0413notBefore=Nov 5 16:48:58 2015 GMTnotAfter=Nov 5 16:48:58 2035 GMTSHA1 Fingerprint=2E:99:2C:CE:2F:C3:B4:EC:C7:E2:52:3A:19:A2:10:D0:54:CA:79:3Epurposes: sslserver sslclient CA certificate 0:subject= /C=US/O=Cisco Systems, Inc./OU=CSG/L=San Jose/ST=CA/street=3700 Cisco Way/postalCode=95134/CN=ems.cisco.com/serialNumber=FGE18420K0Rissuer= /C=US/O=Cisco Systems, Inc./OU=CSG/L=San Jose/ST=CA/street=3700 Cisco Way/postalCode=95134/CN=ems.cisco.com/serialNumber=FGE18420K0Rserial=0413notBefore=Nov 5 16:48:58 2015 GMTnotAfter=Nov 5 16:48:58 2035 GMTSHA1 Fingerprint=2E:99:2C:CE:2F:C3:B4:EC:C7:E2:52:3A:19:A2:10:D0:54:CA:79:3Epurposes: sslserver sslclient
Step4	Configure gRPC to use the trustpoint. `switch(config)# grpc certificate mytrustpointswitch(config)# show run grpc !Command: show running-config grpc!Running configuration last done at: Thu Jul 2 12:24:02 2020!Time: Thu Jul 2 12:24:05 2020 version 9.3(5) Bios:version 05.38feature grpc grpc gnmi max-concurrent-calls 16grpc use-vrf defaultgrpc certificate mytrustpoint`
Step5	Verify gRPC is now using the certificate. switch# show grpc gnmi service statistics=============gRPC Endpoint=============Vrf : managementServer address : [::]:50051Cert notBefore : Nov 5 16:48:58 2015 GMTCert notAfter : Nov 5 16:48:58 2035 GMTMax concurrent calls : 16Listen calls : 1Active calls : 0Number of created calls : 953Number of bad calls : 0Subscription stream/once/poll : 476/238/238Max gNMI::Get concurrent : 5Max grpc message size : 8388608gNMI Synchronous calls : 10gNMI Synchronous errors : 0gNMI Adapter errors : 0gNMI Dtx errors : 0

Verifying gNMI

To verify the gNMI configuration, enter the following command:

Command

Description

show grpc gnmi service statistics

Displays a summary of the agent running status, respectively for the management VRF, or the default VRF (if configured). It also displays:

Basic overall counters
Certificate expiration time

Note

If the certificate is expired, the agent cannot accept requests.

show grpc gnmi rpc summary

Displays the following:

Number of capability RPCs received.
Capability RPC errors.
Number of Get RPCs received.
Get RPC errors.
Number of Set RPCs received.
Set RPC errors.
More error types and counts.

show grpc gnmi transactions

The show grpc gnmi transactions command is the most dense and contains considerable information. It is a history buffer of the most recent 50 gNMI transactions that are received by the switch. As new RPCs come in, the oldest history entry is removed from the end. The following explains what is displayed:

RPC – This shows the type of RPC that was received (Get, Set, Capabilities)
DataType – For a Get only. Has values ALL, CONFIG, and STATE.
Session – shows the unique session-id that is assigned to this transaction. It can be used to correlate data that is found in other log files.
Time In -- shows timestamp of when the RPC was received by the gNMI handler.
Duration – time delta in ms from receiving the request to giving response.
Status – the status code of the operation returned to the client (0 = Success, !0 == error)

This section is data that is kept per path within a single gNMI transaction. For example, a single Get or Set

subtype – for a Set RPC, shows the specific operation that is requested per path (Delete, Update, Replace). For Get, there is no subtype.
dtx – shows that this path was processed in DTX “fast” path or not. A dash ‘-‘ means no, an asterisk ‘*’ means yes.
st – Status for this path. The meaning is as follows:
- OK: path is valid and processed by infra successfully.
- ERR: path is either invalid or generated error by infra
- --: path not processed yet, might or might not be valid and has not been sent to infra yet.
path – the path

show grpc gnmi service statistics Example

=============gRPC Endpoint=============Vrf : managementServer address : [::]:50051Cert notBefore : Mar 13 19:05:24 2020 GMTCert notAfter : Nov 20 19:05:24 2033 GMTMax concurrent calls : 8Listen calls : 1Active calls : 0Number of created calls : 1Number of bad calls : 0Subscription stream/once/poll : 0/0/0Max gNMI::Get concurrent : 5Max grpc message size : 8388608gNMI Synchronous calls : 74gNMI Synchronous errors : 0gNMI Adapter errors : 0gNMI Dtx errors : 0

show grpc gnmi rpc summary Example

=============gRPC Endpoint============= Vrf : managementServer address : [::]:50051 Cert notBefore : Mar 31 20:55:02 2020 GMTCert notAfter : Apr 1 20:55:02 2020 GMT Capability rpcs : 1 Capability errors : 0 Get rpcs : 53 Get errors : 19 Set rpcs : 23 Set errors : 8 Resource Exhausted : 0 Option Unsupported : 6Invalid Argument : 18Operation Aborted : 1Internal Error : 2Unknown Error : 0 RPC Type State Last Activity Cnt Req Cnt Resp Client--------------- ---------- -------------- ---------- ---------- ----------------------------------------Subscribe Listen 04/01 07:39:21 0 0

show grpc gnmi transactions Example

=============gRPC Endpoint============= Vrf : managementServer address : [::]:50051 Cert notBefore : Mar 31 20:55:02 2020 GMTCert notAfter : Apr 1 20:55:02 2020 GMT RPC DataType Session Time In Duration(ms) Status------------ ---------- --------------- -------------------- ------------ ------Set - 2361443608 04/01 07:43:49 173 0 subtype: dtx: st: path: Delete - OK /System/intf-items/lb-items/LbRtdIf-list[id=lo789] Set - 2293989720 04/01 07:43:45 183 0subtype: dtx: st: path: Replace - OK /System/intf-items/lb-items/LbRtdIf-list[id=lo6] Set - 2297110560 04/01 07:43:41 184 0subtype: dtx: st: path: Update - OK /System/intf-items/lb-items/LbRtdIf-list[id=lo7] Set - 0 04/01 07:43:39 0 10 Set - 3445444384 04/01 07:43:33 3259 0subtype: dtx: st: path: Delete - OK /System/intf-items/lb-items/LbRtdIf-list[id=lo789] Delete - OK /System/intf-items/lb-items/LbRtdIf-list[id=lo790] Delete - OK /System/intf-items/lb-items/LbRtdIf-list[id=lo791] Delete - OK /System/intf-items/lb-items/LbRtdIf-list[id=lo792] Delete - OK /System/intf-items/lb-items/LbRtdIf-list[id=lo793] Delete - OK /System/intf-items/lb-items/LbRtdIf-list[id=lo794] Delete - OK /System/intf-items/lb-items/LbRtdIf-list[id=lo795] Delete - OK /System/intf-items/lb-items/LbRtdIf-list[id=lo796] Delete - OK /System/intf-items/lb-items/LbRtdIf-list[id=lo797] Delete - OK /System/intf-items/lb-items/LbRtdIf-list[id=lo798] Delete - OK /System/intf-items/lb-items/LbRtdIf-list[id=lo799] Delete - OK /System/intf-items/lb-items/LbRtdIf-list[id=lo800] Delete - OK /System/intf-items/lb-items/LbRtdIf-list[id=lo801] Delete - OK /System/intf-items/lb-items/LbRtdIf-list[id=lo802] Delete - OK /System/intf-items/lb-items/LbRtdIf-list[id=lo803] Delete - OK /System/intf-items/lb-items/LbRtdIf-list[id=lo804] Delete - OK /System/intf-items/lb-items/LbRtdIf-list[id=lo805] Delete - OK /System/intf-items/lb-items/LbRtdIf-list[id=lo806] Delete - OK /System/intf-items/lb-items/LbRtdIf-list[id=lo807] Delete - OK /System/intf-items/lb-items/LbRtdIf-list[id=lo808] Set - 2297474560 04/01 07:43:26 186 0subtype: dtx: st: path: Update - OK /System/ipv4-items/inst-items/dom-items/Dom-list[name=foo]/rt-items/Route-list[prefix=0.0.0.0/0]/nh-items/Nexthop-list[nhAddr=192.168.1.1/32][nhVrf=foo][nhIf=unspecified]/tag Set - 2294408864 04/01 07:43:17 176 13subtype: dtx: st: path: Delete - ERR /System/intf-items/lb-items/LbRtdIf-list/descr Set - 0 04/01 07:43:11 0 3subtype: dtx: st: path: Update - -- /System/intf-items/lb-items/LbRtdIf-list[id=lo4]/descr Update - ERR /system/processes Set - 2464255200 04/01 07:43:05 708 0subtype: dtx: st: path: Delete - OK /System/intf-items/lb-items/LbRtdIf-list[id=lo2] Delete - OK /System/intf-items/lb-items/LbRtdIf-list[id=lo777] Delete - OK /System/intf-items/lb-items/LbRtdIf-list[id=lo778] Delete - OK /System/intf-items/lb-items/LbRtdIf-list[id=lo779] Delete - OK /System/intf-items/lb-items/LbRtdIf-list[id=lo780] Replace - OK /System/intf-items/lb-items/LbRtdIf-list[id=lo3]/descr Replace - OK /System/intf-items/lb-items/LbRtdIf-list[id=lo4]/descr Replace - OK /System/intf-items/lb-items/LbRtdIf-list[id=lo5]/descr Update - OK /System/intf-items/lb-items/LbRtdIf-list[id=lo3]/descr Update - OK /System/intf-items/lb-items/LbRtdIf-list[id=lo4]/descr Update - OK /System/intf-items/lb-items/LbRtdIf-list[id=lo5]/descr Set - 3491213208 04/01 07:42:58 14 0subtype: dtx: st: path: Replace - OK /System/intf-items/lb-items/LbRtdIf-list[id=lo3]/descr Set - 3551604840 04/01 07:42:54 35 0subtype: dtx: st: path: Delete - OK /System/intf-items/lb-items/LbRtdIf-list[id=lo1] Set - 2362201592 04/01 07:42:52 13 13subtype: dtx: st: path: Delete - ERR /System/intf-items/lb-items/LbRtdIf-list[id=lo3]/lbrtdif-items/operSt Set - 0 04/01 07:42:47 0 3subtype: dtx: st: path: Delete - ERR /System/* Set - 2464158360 04/01 07:42:46 172 3subtype: dtx: st: path: Delete - ERR /system/processes/shabang Set - 2295440864 04/01 07:42:46 139 3subtype: dtx: st: path: Delete - ERR /System/invalid/path Set - 3495739048 04/01 07:42:44 10 0 Get ALL 3444580832 04/01 07:42:40 3 0subtype: dtx: st: path: - - OK /System/bgp-items/inst-items/disPolBatch Get ALL 0 04/01 07:42:36 0 3subtype: dtx: st: path: - - -- /system/processes/process[pid=1] Get ALL 3495870472 04/01 07:42:36 2 0subtype: dtx: st: path: - * OK /system/processes/process[pid=1] Get ALL 2304485008 04/01 07:42:36 33 0subtype: dtx: st: path: - * OK /system/processes Get ALL 2464159088 04/01 07:42:36 251 0subtype: dtx: st: path: - - OK /system Get ALL 2293232352 04/01 07:42:35 258 0subtype: dtx: st: path: - - OK /system Get ALL 0 04/01 07:42:33 0 12subtype: dtx: st: path: - - -- /intf-items

Clients

There are available clients for gNMI subscription. One such client is located at https://github.com/influxdata/telegraf/tree/master/plugins/inputs/cisco_telemetry_gnmi.

Sample DME Subscription - PROTO Encoding

gnmi-console --host >iip> --port 50051 -u <user> -p <pass> --tls --operation=Subscribe --rpc /root/gnmi-console/testing_bl/once/61_subscribe_bgp_dme_gpb.json[Subscribe]-------------------------------### Reading from file ' /root/gnmi-console/testing_bl/once/61_subscribe_bgp_dme_gpb.json 'Wed Jun 26 11:49:17 2019### Generating request : 1 -----------### Comment : ONCE request### Delay : 2 sec(s) ...### Delay : 2 sec(s) DONEsubscribe {subscription {path {origin: "DME"elem {name: "sys"}elem {name: "bgp"}}mode: SAMPLE}mode: ONCEuse_models {name: "DME"organization: "Cisco Systems, Inc."version: "1.0.0"}encoding: PROTO}Wed Jun 26 11:49:19 2019Received response 1 --------------------------update {timestamp: 1561574967761prefix {elem {name: "sys"}elem {name: "bgp"}}update {path {elem {}elem {name: "version_str"}}val {string_val: "1.0.0"}}update {path {elem {}elem {name: "node_id_str"}}val {string_val: "n9k-tm2"}}update {path {elem {}elem {name: "encoding_path"}}val {string_val: "sys/bgp"}}update {path {elem {}elem {/Received -------------------------------------Wed Jun 26 11:49:19 2019Received response 2 --------------------------sync_response: true/Received -------------------------------------(_gnmi) [root@tm-ucs-1 gnmi-console]#

Capabilities

About Capabilities

The Capabilities RPC returns the list of capabilities of the gNMI service. The response message to the RPC request includes the gNMI service version, the versioned data models, and data encodings supported by the server.

Guidelines and Limitations for Capabilities

Following are the guidelines and limitations for Capabilities:

Beginning with Cisco NX-OS Release 9.3(3), Capabilities supports the OpenConfig model.
For information about supported platforms, see Nexus Switch Platform Matrix.
The gNMI feature supports Subscribe and Capability as options of the gNMI service.
The feature supports JSON and gnmi.proto encoding. The feature does not support protobuf.any encoding.
Each gNMI message has a maximum size of 12 MB. If the amount of collected data exceeds the 12-MB maximum, the collected data is dropped.

You can avoid this situation by creating more focused subscriptions that handle smaller, more granular data-collection sets. So, instead of subscribing to one higher-level path, create multiple subscriptions for different, lower-level parts of the path.
All paths within the same subscription request must have the same sample interval. If the same path requires different sample intervals, create multiple subscriptions.
The feature does not support a path prefix in the Subscription request, but the Subscription can contain an empty prefix field.
The feature supports Cisco DME and Device YANG data models.
The gRPC process that supports gNMI uses the HIGH_PRIO cgroup, which limits the CPU usage to 75% of CPU and memory to 1.5 GB.
The show grpc gnmi command has the following considerations:
- The commands are not XMLized in this release.
- The gRPC agent retains gNMI calls for a maximum of 1 hour after the call has ended.
- If the total number of calls exceeds 2000, the gRPC agent purges ended calls based an internal cleanup routine.

The gRPC server runs in the management VRF. As a result, the gRPC process communicates only in this VRF forcing the management interface to support all gRPC calls.

gRPC functionality now includes the default VRF for a total of 2 gRPC servers on each Cisco Nexus 9000 switch. You can run one gRPC server in each VRF, or run only one gRPC server in the management VRF. Supporting a gRPC in the default VRF adds flexibility to offload processing gRPC calls from the management VRF, where significant traffic load might not be desirable.

If two gRPC servers are configured, be aware of the following:

VRF boundaries are strictly enforced, so each gRPC server processes requests independent of the other, and requests do not cross between VRFs.
The two servers are not HA or fault tolerant. One gRPC server does not back up the other, and there is no switchover or switchback between them.
Any limits for the gRPC server are per VRF.

Example Client Output for Capabilities

In this example, all the OpenConfig model RPMs have been installed on the switch.

The following is an example of client output for Capabilities.

hostname user$ ./gnmi_cli -a 172.19.193.166:50051 -ca_crt ./grpc.pem -insecure -capabilitiessupported_models: < name: "Cisco-NX-OS-device" organization: "Cisco Systems, Inc." version: "2019-11-13"> supported_models: < name: "openconfig-acl" organization: "OpenConfig working group" version: "1.0.0"> supported_models: < name: "openconfig-bgp-policy" organization: "OpenConfig working group" version: "4.0.1"> supported_models: < name: "openconfig-interfaces" organization: "OpenConfig working group" version: "2.0.0"> supported_models: < name: "openconfig-if-aggregate" organization: "OpenConfig working group" version: "2.0.0"> supported_models: < name: "openconfig-if-ethernet" organization: "OpenConfig working group" version: "2.0.0"> supported_models: < name: "openconfig-if-ip" organization: "OpenConfig working group" version: "2.3.0"> supported_models: < name: "openconfig-if-ip-ext" organization: "OpenConfig working group" version: "2.3.0"> supported_models: < name: "openconfig-lacp" organization: "OpenConfig working group" version: "1.0.2"> supported_models: < name: "openconfig-lldp" organization: "OpenConfig working group" version: "0.2.1"> supported_models: < name: "openconfig-network-instance" organization: "OpenConfig working group" version: "0.11.1"> supported_models: < name: "openconfig-network-instance-policy" organization: "OpenConfig working group" version: "0.1.1"> supported_models: < name: "openconfig-ospf-policy" organization: "OpenConfig working group" version: "0.1.1"> supported_models: < name: "openconfig-platform" organization: "OpenConfig working group" version: "0.12.2"> supported_models: < name: "openconfig-platform-cpu" organization: "OpenConfig working group" version: "0.1.1"> supported_models: < name: "openconfig-platform-fan" organization: "OpenConfig working group" version: "0.1.1"> supported_models: < name: "openconfig-platform-linecard" organization: "OpenConfig working group" version: "0.1.1"> supported_models: < name: "openconfig-platform-port" organization: "OpenConfig working group" version: "0.3.2"> supported_models: < name: "openconfig-platform-psu" organization: "OpenConfig working group" version: "0.2.1"> supported_models: < name: "openconfig-platform-transceiver" organization: "OpenConfig working group" version: "0.7.0"> supported_models: < name: "openconfig-relay-agent" organization: "OpenConfig working group" version: "0.1.0"> supported_models: < name: "openconfig-routing-policy" organization: "OpenConfig working group" version: "2.0.1"> supported_models: < name: "openconfig-spanning-tree" organization: "OpenConfig working group" version: "0.2.0"> supported_models: < name: "openconfig-system" organization: "OpenConfig working group" version: "0.3.0"> supported_models: < name: "openconfig-telemetry" organization: "OpenConfig working group" version: "0.5.1"> supported_models: < name: "openconfig-vlan" organization: "OpenConfig working group" version: "3.0.2"> supported_models: < name: "DME" organization: "Cisco Systems, Inc."> supported_models: < name: "Cisco-NX-OS-Syslog-oper" organization: "Cisco Systems, Inc." version: "2019-08-15"> supported_encodings: JSONsupported_encodings: PROTOgNMI_version: "0.5.0" hostname user$

Get

About Get

The purpose of the Get RPC is to allow a client to retrieve a snapshot of the data tree from the device. Multiple paths may be requested in a single request. A simplified form of XPATH according to the gNMI Path Conventions, Schema path encoding conventions for gNMI are used for the path.

For detailed information on the Get operation, refer to the Retrieving Snapshots of State Information section in the gNMI specification: gRPC Network Management Interface (gNMI)

Guidelines and Limitations for Get

The following are guidelines and limitations for Get and Set:

GetRequest.encoding supports only JSON.
For GetRequest.type, only DataType CONFIG and STATE have direct correlation and expression in YANG. OPERATIONAL is not supported.
A single request cannot have both OpenConfig (OC) YANG and device YANG paths. A request must have only OC YANG paths or device YANG paths, but not both.
GetRequest for root path (“/”: everything from all models) is not allowed.
GetRequest for the top level of the device model (“/System”) is not allowed.
gNMI Get returns all default values (ref. report-all mode in RFC 6243 [4]).
Subscribe supports the model Cisco-NX-OS-syslog-oper.
See Also
Yang, OpenConfig and gNMI
Get does not support the model Cisco-NX-OS-syslog-oper.
Query from the path /system does not return data from the path /system/processes. The specific path /system/processes should be used to query openconfig-procmon data.
The following optional items are not supported:
- Path prefix
- Path alias
- Wildcards in path
A single GetRequest can have up to 10 paths.
If the size of value field to be returned in GetResponse is over 12 MB, the system returns error statusgrpc::RESOURCE_EXHAUSTED.
The maximum gRPC receive buffer size is set to 8 MB.
The number of total concurrent sessions for Get is limited to five.
Performing a Get operation when a large configuration is applied to the switch might cause the gRPC process to consume all available memory. If a memory exhaustion condition is hit, the following syslog is generated:
```
MTX-API: The memory usage is reaching the max memory resource limit (3072) MB
```
If this condition is hit several times consecutively, the following syslog is generated:
```
The process has become unstable and the feature should be restarted.
```
We recommend that you restart the gRPC feature at this point to continue normal processing of gNMI transactions.
The combined number of concurrent sessions for Get and Set is the currently configured max gNMI concurrent-1. For instance, if gnmi concurrent calls are configured to 16, the maximum number of total concurrent sessions for Get and Set will be 15.
Performing a Get operation when a large configuration is applied to the switch might cause the gRPC process to be unable to be process the request . At that point, the following error is returned:

There is insufficient memory available on the device to process the subscription.

Set

About Set

The Set RPC is used by a client to change the configuration of the device. The operations, which may be applied to the device data, are (in order) delete, replace, and update. All operations in a single Set request are treated as a transaction, meaning that all operations are successful or the device is rolled-back to the original state. The Set operations are applied in the order that is specified in the SetRequest. If a path is mentioned multiple times, the changes are applied even if they overwrite each other. The final state of the data is achieved with the final operation in the transaction. It is assumed that all paths specified in the SetRequest::delete, replace, update fields are CONFIG data paths and writable by the client.

For detailed information on the Set operation, refer to the Modifying State section of the gNMI Specification https://github.com/openconfig/reference/blob/1cf43d2146f9ba70abb7f04f6b0f6eaa504cef05/rpc/gnmi/gnmi-specification.md.

Guidelines and Limitations for Set

The following are guidelines and limitations for Set:

SetRequest.encoding supports only JSON.
A single request cannot have both OpenConfig (OC) YANG and device YANG paths. A request must have only OC YANG paths or device YANG paths, but not both.
Subscribe supports the model Cisco-NX-OS-syslog-oper.
Query from the path /system does not return data from the path /system/processes. The specific path /system/processes should be used to query openconfig-procmon data.
The following optional items are not supported:
- Path prefix
- Path alias
- Wildcards in path
A single SetRequest can have up to 20 paths.
The maximum gRPC receive buffer size is set to 8 MB.
The combined number of concurrent sessions for Get and Set is the currently configured max gNMI concurrent-1. For instance, if gNMI concurrent calls are configured to 16, the maximum number of total concurrent sessions for Get and Set will be 15.

For the Set::Delete RPC, an MTX log message warns if the configuration being operated on may be too large:

Configuration size for this namespace exceeds operational limit. Feature maybecome unstable and require restart.

Guidelines and Limitations for Subscribe

Following are the guidelines and limitations for Subscribe:

If you configure a routing-policy prefix-list using the CLI and request gNMI Subscription for the routing-policy OpenConfig model, it is not supported. For example, when you attempt to subscribe an OpenConfig routing policy with a preexisting CLI configuration like the following, it returns empty values due to the current implementation of the OpenConfig model.
```
ip prefix-list bgp_v4_drop seq 5 deny 125.2.0.0/16 le 32ipv6 prefix-list bgp_v6_drop seq 5 deny cafe:125:2::/48 le 128Using the example paths,openconfig-routing-policy:/routing-policy/defined-sets/prefix-sets/prefix-set[name=bgp_v4_drop]/configopenconfig-routing-policy:/routing-policy/defined-sets/prefix-sets/prefix-set[name=bgp_v6_drop]/config
```
Beginning with Cisco NX-OS Release 9.3(3), Subscribe supports the OpenConfig model.
For information about supported platforms, see the Nexus Switch Platform Matrix.
The gNMI feature supports Subscribe and Capability RPCs.
The feature supports JSON and gnmi.proto encoding. The feature does not support protobuf.any encoding.
Each gNMI message has a maximum size of 12 MB. If the amount of collected data exceeds the 12 MB maximum, the collected data is dropped.

You can avoid this situation by creating more focused subscriptions that handle smaller, more granular data-collection sets. So, instead of subscribing to one higher-level path, create multiple subscriptions for different, lower-level parts of the path.
All paths within the same subscription request must have the same sample interval. If the same path requires different sample intervals, create multiple subscriptions.
The feature does not support a path prefix in the Subscription request, but the Subscription can contain an empty prefix field.
The feature supports Cisco DME and Device YANG data models.
The gRPC process that supports gNMI uses the HIGH_PRIO cgroup, which limits the CPU usage to 75% of CPU and memory to 1.5 GB.
The show grpc gnmi command has the following considerations:
- The commands are not XMLized in this release.
- The gRPC agent retains gNMI calls for a maximum of 1 hour after the call has ended.
- If the total number of calls exceeds 2000, the gRPC agent purges ended calls based an internal cleanup routine.

The gRPC server runs in the management VRF. As a result, the gRPC process communicates only in this VRF forcing the management interface to support all gRPC calls.

If two gRPC servers are configured, be aware of the following:

VRF boundaries are strictly enforced, so each gRPC server processes requests independent of the other, and requests do not cross between VRFs.
The two servers are not HA or fault tolerant. One gRPC server does not back up the other, and there is no switchover or switchback between them.
Any limits for the gRPC server are per VRF.

gNMI Payload

gNMI uses a specific payload format to subscribe to:

DME Streams
YANG Streams

Subscribe operations are supported with the following modes:

ONCE: Subscribe and receive data once and close session.
POLL: Subscribe and keep session open, client sends poll request each time data is needed.
STREAM: Subscribe and receive data at specific cadence. The payload accepts values in nanoseconds 1 second = 1000000000.
ON_CHANGE: Subscribe, receive a snapshot, and only receive data when something changes in the tree.

Setting modes:

Each mode requires 2 settings, inside sub and outside sub
ONCE: SAMPLE, ONCE
POLL: SAMPLE, POLL
STREAM: SAMPLE, STREAM
ON_CHANGE: ON_CHANGE, STREAM

Origin

DME: Subscribing to DME model
device: Subscribing to YANG model

Name

DME = subscribing to DME model
Cisco-NX-OS-device = subscribing to YANG model

Encoding

JSON = Stream will be send in JSON format.
PROTO = Stream will be sent in protobuf.any format.

Sample gNMI Payload for DME Stream

Cisco Nexus 9000 Series NX-OS Programmability Guide, Release 9.3(x) - gNMI - gRPC Network Management Interface [Cisco Nexus 9000 Series Switches] (5)

Note

Different clients have their own input format.

{ "SubscribeRequest": [ { "_comment" : "ONCE request", "_delay" : 2, "subscribe": { "subscription": [ { "_comment" : "1st subscription path", "path": { "origin": "DME", "elem": [ { "name": "sys" }, { "name": "bgp" } ] }, "mode": "SAMPLE" } ], "mode": "ONCE", "allow_aggregation" : false, "use_models": [ { "_comment" : "1st module", "name": "DME", "organization": "Cisco Systems, Inc.", "version": "1.0.0" } ], "encoding": "JSON" } } ]}

Sample gNMI Payload YANG Stream

{ "SubscribeRequest": [ { "_comment" : "ONCE request", "_delay" : 2, "subscribe": { "subscription": [ { "_comment" : "1st subscription path", "path": { "origin": "device", "elem": [ { "name": "System" }, { "name": "bgp-items" } ] }, "mode": "SAMPLE" } ], "mode": "ONCE", "allow_aggregation" : false, "use_models": [ { "_comment" : "1st module", "name": "Cisco-NX-OS-device", "organization": "Cisco Systems, Inc.", "version": "0.0.0" } ], "encoding": "JSON" } } ]}

Streaming Syslog

About Streaming Syslog for gNMI

gNMI Subscribe is a new way of monitoring the network as it provides a real-time view of what's going on in your system by pushing the structured data as per gNMI Subscribe request.

Beginning with the Cisco NX-OS Release 9.3(3), support is added for gNMI Subscribe functionality.

gNMI Subscribe Support Detail

Syslog-oper model streaming
- stream_on_change

This feature applies to Cisco Nexus 9000 Series switches with 8 GB or more of memory.

Guidelines and Limitations for Streaming Syslog - gNMI

The following are guidelines and limitations for Streaming Syslog:

An invalid syslog is not supported. For example, a syslog with a filter or query condition
Only the following paths are supported:
- Cisco-NX-OS-Syslog-oper:syslog
- Cisco-NX-OS-Syslog-oper:syslog/messages
The following modes are not supported:
- Stream sample
- POLL
A request must be in the YANG model format.
You can use the internal application or write your own application.
The payload comes from the controller and gNMI sends a response.
Encoding formats are JSON and PROTO.

Syslog Native YANG Model

The YangModels are located here.

Cisco Nexus 9000 Series NX-OS Programmability Guide, Release 9.3(x) - gNMI - gRPC Network Management Interface [Cisco Nexus 9000 Series Switches] (6)

Note

The time-zone field is set only when the clock format show-timezone syslog is entered. By default, it's not set, therefore the time-zone field is empty.

PYANG Tree for Syslog Native Yang Model:>>> pyang -f tree Cisco-NX-OS-infra-syslog-oper.yangmodule: Cisco-NX-OS-syslog-oper+--ro syslog+--ro messages+--ro message* [message-id]+--ro message-id int32+--ro node-name? string+--ro time-stamp? uint64+--ro time-of-day? string+--ro time-zone? string+--ro category? string+--ro group? string+--ro message-name? string+--ro severity? System-message-severity+--ro text? string

Subscribe Request Example

The following is an example of a Subscribe request:

{ "SubscribeRequest": [ { "_comment" : "STREAM request", "_delay" : 2, "subscribe": { "subscription": [ { "_comment" : "1st subscription path", "path": { "origin": "syslog-oper", "elem": [ { "name": "syslog" }, { "name":"messages" } ] }, "mode": "ON_CHANGE" } ], "mode": "ON_CHANGE", "allow_aggregation" : false, "use_models": [ { "_comment" : "1st module", "name": "Cisco-NX-OS-Syslog-oper", "organization": "Cisco Systems, Inc.", "version": "0.0.0" } ], "encoding":"JSON" } } ]}

Sample PROTO Output

This is a sample of PROTO output.

############################[Subscribe]-------------------------------### Reading from file ' /root/gnmi-console/testing_bl/stream_on_change/OC_SYSLOG.json 'Sat Aug 24 14:38:06 2019### Generating request : 1 -----------### Comment : STREAM request### Delay : 2 sec(s) ...### Delay : 2 sec(s) DONEsubscribe {subscription {path {origin: "syslog-oper"elem {name: "syslog"}elem {name: "messages"}}mode: ON_CHANGE}use_models {name: "Cisco-NX-OS-Syslog-oper"organization: "Cisco Systems, Inc."version: "0.0.0"}encoding: PROTO}Thu Nov 21 14:26:41 2019Received response 3 --------------------------update {timestamp: 1574375201665688000prefix {origin: "Syslog-oper"elem {name: "syslog"}elem {name: "messages"}}update {path {elem {name: "message-id"}}val {uint_val: 529}}update {path {elem {name: "node-name"}}val {string_val: "task-n9k-1"}}update {path {elem {name: "message-name"}}val {string_val: "VSHD_SYSLOG_CONFIG_I"}}update {path {elem {name: "text"}}val {string_val: "Configured from vty by admin on console0"}}update {path {elem {name: "group"}}val {string_val: "VSHD"}}update {path {elem {name: "category"}}val {string_val: "VSHD"}}update {path {elem {name: "time-of-day"}}val {string_val: "Nov 21 2019 14:26:40"}}update {path {elem {name: "time-zone"}}val {string_val: ""}}update {path {elem {name: "time-stamp"}}val {uint_val: 1574375200000}}update {path {elem {name: "severity"}}val {uint_val: 5}}}/Received -------------------------------------

Sample JSON Output

This is a sample JSON output.

[Subscribe]-------------------------------### Reading from file ' testing_bl/stream_on_change/OC_SYSLOG.json 'Tue Nov 26 11:47:00 2019### Generating request : 1 -----------### Comment : STREAM request### Delay : 2 sec(s) ...### Delay : 2 sec(s) DONEsubscribe {subscription {path {origin: "syslog-oper"elem {name: "syslog"}elem {name: "messages"}}mode: ON_CHANGE}use_models {name: "Cisco-NX-OS-Syslog-oper"organization: "Cisco Systems, Inc."version: "0.0.0"}}Tue Nov 26 11:47:15 2019Received response 5 --------------------------update {timestamp: 1574797636002053000prefix {}update {path {origin: "Syslog-oper"elem {name: "syslog"}}val {json_val: "[ { \"messages\" : [[ {\"message-id\":657},{\"node-name\":\"task-n9k-1\",\"time-stamp\":\"1574797635000\",\"time-of-day\":\"Nov 26 2019 11:47:15\",\"severity\":3,\"message-name\":\"HDR_L2LEN_ERR\",\"category\":\"ARP\",\"group\":\"ARP\",\"text\":\"arp [30318] Received packet with incorrect layer 2 address length (8 bytes), Normal pkt with S/D MAC: 003a.7d21.d55e ffff.ffff.ffff eff_ifc mgmt0(9), log_ifc mgmt0(9), phy_ifc mgmt0(9)\",\"time-zone\":\"\"} ]] } ]"}}}/Received -------------------------------------

Troubleshooting

Gathering TM-Trace Logs

1. tmtrace.bin -f gnmi-logs gnmi-events gnmi-errors following are available2. Usage:bash-4.3# tmtrace.bin -d gnmi-events | tail -30 Gives the last 30}}}[06/21/19 15:58:38.969 PDT f8f 3133] [3981658944][tm_transport_internal.c:43] dn: Cisco-NX-OS-device:System/cdp-items, sub_id: 0,sub_id_str: 2329, dc_start_time: 0, length: 124, sync_response:1[06/21/19 15:58:43.210 PDT f90 3133] [3621780288][tm_ec_yang_data_processor.c:93] TM_EC: [Y] Data received for 2799743488: 49{"cdp-items" : {"inst-items" : {"if-items" : {"If-list" : [{"id" : "mgmt0","ifstats-items" : {"v2Sent" : "74","validV2Rcvd" : "79"}}]}}}}[06/21/19 15:58:43.210 PDT f91 3133] [3981658944][tm_transport_internal.c:43] dn: Cisco-NX-OS-device:System/cdp-items, sub_id: 0,sub_id_str: 2329, dc_start_time: 0, length: 141, sync_response:1[06/21/19 15:59:01.341 PDT f92 3133] [3981658944][tm_transport_internal.c:43] dn: Cisco-NX-OS-device:System/intf-items, sub_id:4091, sub_id_str: , dc_start_time: 1561157935518, length: 3063619, sync_response:0[06/21/19 15:59:03.933 PDT f93 3133] [3981658944][tm_transport_internal.c:43] dn: Cisco-NX-OS-device:System/cdp-items, sub_id:4091, sub_id_str: , dc_start_time: 1561157940881, length: 6756, sync_response:0[06/21/19 15:59:03.940 PDT f94 3133] [3981658944][tm_transport_internal.c:43] dn: Cisco-NX-OS-device:System/lldp-items, sub_id:4091, sub_id_str: , dc_start_time: 1561157940912, length: 8466, sync_response:1bash-4.3#

Gathering MTX-Internal Logs

1. Modify the following file with below /opt/mtx/conf/mtxlogger.cfg<config name="nxos-device-mgmt"> <container name="mgmtConf"> <container name="logging"> <leaf name="enabled" type="boolean" default="false">true</leaf> <leaf name="allActive" type="boolean" default="false">true</leaf> <container name="format"> <leaf name="content" type="string" default="$DATETIME$$COMPONENTID$ $TYPE$: $MSG$">$DATETIME$ $COMPONENTID$ $TYPE$$SRCFILE$ @ $SRCLINE$ $FCNINFO$:$MSG$</leaf> <container name="componentID"> <leaf name="enabled" type="boolean" default="true"></leaf> </container> <container name="dateTime"> <leaf name="enabled" type="boolean" default="true"></leaf> <leaf name="format" type="string" default="%y%m%d.%H%M%S"></leaf> </container> <container name="fcn"> <leaf name="enabled" type="boolean" default="true"></leaf> <leaf name="format" type="string"default="$CLASS$::$FCNNAME$($ARGS$)@$LINE$"></leaf> </container> </container> <container name="facility"> <leaf name="info" type="boolean" default="true">true</leaf> <leaf name="warning" type="boolean" default="true">true</leaf> <leaf name="error" type="boolean" default="true">true</leaf>Note: Beginning with Cisco NX-OS Release 9.3(4), the following default configuration is changed from default true to false. To investigate an issue which requires the debug messages, edit the following configuration and toggle it to true. <leaf name="debug" type="boolean" default="false">true</leaf> </container> <container name="dest"> <container name="console"> <leaf name="enabled" type="boolean" default="false">true</leaf> </container> <container name="file"> <leaf name="enabled" type="boolean" default="false">true</leaf> <leaf name="name" type="string" default="mtx-internal.log"></leaf> <leaf name="location" type="string" default="./mtxlogs">/volatile</leaf> <leaf name="mbytes-rollover" type="uint32" default="10">50</leaf> <leaf name="hours-rollover" type="uint32" default="24">24</leaf> <leaf name="startup-rollover" type="boolean" default="false">true</leaf> <leaf name="max-rollover-files" type="uint32" default="10">10</leaf> </container> </container> <list name="logitems" key="id"> <listitem> <leaf name="id" type="string">*</leaf> <leaf name="active" type="boolean" default="false">false</leaf> </listitem> <listitem> <leaf name="id" type="string">MTX-EvtMgr</leaf> <leaf name="active" type="boolean" default="true">true</leaf> </listitem> <listitem> <leaf name="id" type="string">TM-ADPT</leaf> <leaf name="active" type="boolean" default="true">false</leaf> </listitem> <listitem> <leaf name="id" type="string">TM-ADPT-JSON</leaf> <leaf name="active" type="boolean" default="true">false</leaf> </listitem > <listitem> <leaf name="id" type="string">SYSTEM</leaf> <leaf name="active" type="boolean" default="true">true</leaf> </listitem> <listitem> <leaf name="id" type="string">LIBUTILS</leaf> <leaf name="active" type="boolean" default="true">true</leaf> </listitem> <listitem> <leaf name="id" type="string">MTX-API</leaf> <leaf name="active" type="boolean" default="true">true</leaf> </listitem> <listitem> <leaf name="id" type="string">Model-*</leaf> <leaf name="active" type="boolean" default="true">true</leaf> </listitem> <listitem> <leaf name="id" type="string">Model-Cisco-NX-OS-device</leaf> <leaf name="active" type="boolean" default="true">false</leaf> </listitem> <listitem> <leaf name="id" type="string">Model-openconfig-bgp</leaf> <leaf name="active" type="boolean" default="true">false</leaf> </listitem> <listitem> <leaf name="id" type="string">INST-MTX-API</leaf> <leaf name="active" type="boolean" default="true">true</leaf> </listitem> <listitem> <leaf name="id" type="string">INST-ADAPTER-NC</leaf> <leaf name="active" type="boolean" default="true">true</leaf> </listitem> <listitem> <leaf name="id" type="string">INST-ADAPTER-RC</leaf> <leaf name="active" type="boolean" default="true">true</leaf> </listitem> <listitem> <leaf name="id" type="string">INST-ADAPTER-GRPC</leaf> <leaf name="active" type="boolean" default="true">true</leaf> </listitem> </list> </container> </container></config>2. Run "no feature grpc" / "feature grpc"3. The /volatile directory houses the mtx-internal.log, the log rolls over time so be sure to grab what you need before then.bash-4.3# cd /volatile/bash-4.3# cd /volatile -altotal 148drwxrwxrwx 4 root root 340 Jun 21 15:47 .drwxrwxr-t 64 root network-admin 1600 Jun 21 14:45 ..-rw-rw-rw- 1 root root 103412 Jun 21 16:14 grpc-internal-log-rw-r--r-- 1 root root 24 Jun 21 14:44 mtx-internal-19-06-21-14-46-21.log-rw-r--r-- 1 root root 24 Jun 21 14:46 mtx-internal-19-06-21-14-46-46.log-rw-r--r-- 1 root root 175 Jun 21 15:11 mtx-internal-19-06-21-15-11-57.log-rw-r--r-- 1 root root 175 Jun 21 15:12 mtx-internal-19-06-21-15-12-28.log-rw-r--r-- 1 root root 175 Jun 21 15:13 mtx-internal-19-06-21-15-13-17.log-rw-r--r-- 1 root root 175 Jun 21 15:13 mtx-internal-19-06-21-15-13-42.log-rw-r--r-- 1 root root 24 Jun 21 15:13 mtx-internal-19-06-21-15-14-22.log-rw-r--r-- 1 root root 24 Jun 21 15:14 mtx-internal-19-06-21-15-19-05.log-rw-r--r-- 1 root root 24 Jun 21 15:19 mtx-internal-19-06-21-15-47-09.log-rw-r--r-- 1 root root 24 Jun 21 15:47 mtx-internal.log-rw-rw-rw- 1 root root 355 Jun 21 14:44 netconf-internal-log-rw-rw-rw- 1 root root 0 Jun 21 14:45 nginx_logflagdrwxrwxrwx 3 root root 60 Jun 21 14:45 uwsgipydrwxrwxrwx 2 root root 40 Jun 21 14:43 virtual-instancebash-4.3#.