Saturday, May 25, 2013

NTP

I try to dig pretty hard in my blog posts.  In this case, I found out that there's a lot to NTP!  In fact, there's a lot that would be a real stretch to get in to the CCIE lab.  I feel bad about writing a "high level" post, but I also want to save both my time and my readers' time - a lot of the topics I could've dug harder on are really unlikely to be of any use, either production or lab-wise.

That said, here's a list of things I will not be covering:
 - Any versions besides NTP v3.  v4 should be out-of-scope for the lab exam, as it's not supported on 12.4(15)T.
- The NTP query system.  This is normally SNMP based and there's very little information on Cisco's implementation.  It has nothing to do with syncing time.

Let's start by discussing the stratum system.

The stratum is treated by NTP similar to a hop system in a distance vector protocol.  Stratum 1 servers should be best, stratum 2 servers are one NTP hop away from stratum 1.  The servers speaking to stratum 2s become stratum 3s, and so on and so forth.

Let's look at some easy concepts that will help lead into some more advanced topics.


That's simple enough.  We gain a stratum the further we get from the most trusted time servers. 

As an attempt at being somewhat realistic, I'm going to start another diagram at stratum 5, because a stratum 1 router should have a hyper-accurate clock attached (such as a GPS or atomic clock).



Our stratum 5 router would be configured like this:

ntp master 5

Yep, that's it.  Of interesting note, it actually synchronizes with itself at either 127.127.1.1 or 127.127.7.1. Newer IOSes use 127.127.1.1, older use 127.127.7.1.  The master's "source" at 127.127.x.1 is always one lower than what you configured the master level at.  In this router's case, you'd be synchronizing to stratum 4:

Router(config)#do show ntp association detail
127.127.1.1 configured, our_master, sane, valid, stratum 4

Out of the box, NTP allows anyone to receive time from it, so no more config is required on our stratum 5 device.

Let's say our stratum 5 device is at 5.5.5.5:

The next row down, on stratum 6:

ntp server 5.5.5.5

once again, that's it.  Before synchronization happens, it will be stratum 16 (maximum stratum), and refuse to serve time to others.  Afterwards it will be stratum 6, as it's receiving time from stratum 5.

Now we'll work at the bottom row, on the "stratum ??" device:

ntp server 6.6.6.6
ntp server 5.5.5.5

So what stratum would we be?  It's probably relatively easy to figure out that we'll still end up stratum 6, as it naturally prefers the "better" time from the stratum 5 router.  Better is determined by the stratum hop-count.

What if we did this?:

ntp server 6.6.6.6 prefer
ntp server 5.5.5.5

Still stratum 6!  Why not 7?  Because prefer, contrary to what some other documentation indicates, doesn't actually statically prefer that host.  If two systems had the same stratum, prefer does function, but not for two separate stratums.

Now let's take a look at the ntp peer command.  The premise isn't that hard, it means sync time from that router, but also accept time from it.  Wait a minute!  How do we determine who has the right time in that case?

It's stratum-based, of course!  But wait, why not just make the router with the better time the server.  Well, that's great until a network failure occurs....

Let's take this scenario:


Let's say our two stratum 6 servers are using the ntp server command against the stratum 5 router.  That's great until the link between the right stratum 6 router and top stratum 5 router fails.  You'd isolate time services from the right side of the diagram.

ntp peer to the rescue!  By telling the two stratum 6 routers to peer off each other, they can both act as either a client or a server at the same time.  The router experiencing the failure would have lost its link to stratum 5, and would accept the stratum 6 as it's master, on the fly.  If either the left or right link failed, the other could fill in as its server.  When the failed link comes back up, the router would automatically reconverge to the better stratum master.

Now that that's all explained, let's swap to a simpler topology, using physical gear, to look at the CLI output.  We'll have three routers, R1, R2, and R3.  They all have layer 3 reachability to each other. 

Let's make R1 a stratum 5 master:

R1(config)#ntp master 5
R1(config)#exit
R1#show ntp status
Clock is synchronized, stratum 5, reference is 127.127.7.1
nominal freq is 250.0000 Hz, actual freq is 249.9857 Hz, precision is 2**18
reference time is D54801E4.4BC4EBC0 (22:48:04.295 BOB Thu May 23 2013)
clock offset is 0.0000 msec, root delay is 0.00 msec
root dispersion is 0.02 msec, peer dispersion is 0.02 msec

Above we see R1 is synchronized to its internal clock at 127.127.7.1

R1#show ntp association detail
127.127.7.1 configured, our_master, sane, valid, stratum 4
ref ID 127.127.7.1, time D54801E4.4BC4EBC0 (22:48:04.295 BOB Thu May 23 2013)
our mode active, peer mode passive, our poll intvl 64, peer poll intvl 64
root delay 0.00 msec, root disp 0.00, reach 377, sync dist 0.015
delay 0.00 msec, offset 0.0000 msec, dispersion 0.02
precision 2**18, version 3
<output omitted>

There's a lot of information on this command, but I've highlighted the important parts.  We can see who we're peered with, whether the peering is "sane" and "valid", and what the stratum is of our neighbor (note the "neighbor's" value of stratum 4, although we configured master 5!). 

Now let's setup R2 to poll off this device.

R2(config)#ntp server 1.1.1.1     ! 1.1.1.1 is R1's Loopback address

R2#show ntp status
Clock is unsynchronized, stratum 16, no reference clock
nominal freq is 250.0000 Hz, actual freq is 249.9999 Hz, precision is 2**16
reference time is D52B5B95.62DBC65D (09:15:01.386 UTC Wed May 1 2013)
clock offset is 1878824324.3843 msec, root delay is 57.97 msec
root dispersion is 21735550.67 msec, peer dispersion is 16000.00 msec
uh-oh, we're out of sync!

R2#show ntp association detail
1.1.1.1 configured, insane, invalid, stratum 5
ref ID 127.127.7.1, time D54806A4.5D738DD4 (03:08:20.365 UTC Thu May 23 2013)
our mode client, peer mode server, our poll intvl 64, peer poll intvl 64
root delay 0.00 msec, root disp 0.03, reach 0, sync dist 15904.022
delay 57.97 msec, offset 1878824324.3843 msec, dispersion 16000.00
precision 2**18, version 3
org time 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
rcv time 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
xmt time 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
filtdelay =     0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00
filtoffset =    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00
filterror =  16000.0 16000.0 16000.0 16000.0 16000.0 16000.0 16000.0 16000.0

Oh, the first time I configured NTP, how I came to hate you, "insane, invalid" error message!

This output is very common for NTP, and can be broken down easily if you disregard most of the message.

offset = This is the time difference in milliseconds between our local clock and the ntp server's reference clock.
insane, invalid = I'm sure this has more meaning, but in short, this basically says look at the offset value.  The offset must be < 1000 msec (1 second) off in order for the insane, invalid error to go away.

The problem is, after NTP kicks in, it doesn't just shift the clock instantaneously.  It slowly drifts towards it.  If your router is stuck in 1993 and you're trying to shift to 2013, this will take a long time. You can speed it up by manually adjusting the clock closer:

R2#clock set 03:17:00 May 23 2013

After you're < 1 second off, the router will adjust its stratum from 16 to the appropriate stratum (in our case, 6). 

R2#show ntp status
Clock is synchronized, stratum 6, reference is 1.1.1.1
<output omitted>

R2#show ntp association detail
1.1.1.1 configured, our_master, sane, valid, stratum 5
ref ID 127.127.7.1, time D5480624.5B94958C (03:06:12.357 UTC Thu May 23 2013)
our mode client, peer mode server, our poll intvl 64, peer poll intvl 64
root delay 0.00 msec, root disp 0.03, reach 377, sync dist 29.129
delay 57.39 msec, offset 0.9640 msec, dispersion 0.41
<output omitted>

Note our offset of .9640, less than 1 second.  At this point, R2 would now be willing to be an NTP server as well.  Let's verify on R3.

R3(config)#ntp server 2.2.2.2

R3#show ntp status | i Clock
Clock is synchronized, stratum 7, reference is 2.2.2.2
Again, any router that's > stratum 16 is automatically happy to be an NTP server. 

Let's look at some random features before moving on to authentication & access control.

NTP updates are always sent in UTC/GMT; time zone information is adapted to the local time with:
clock timezone PST +7

If your router has a hardware clock, you can use NTP to synchronize to it:
ntp update-calendar

Want to control how many associations it can make?
ntp max-associations 20

Disable NTP from answering on a per-interface basis?
ntp disable

Authentication & Access Control

It can be a bad idea to have a promiscuous NTP process.  While I haven't found any flaws that would cause significant damage to the router directly, attacking NTP can be used as a way to obscure what really happened during another attack.  Logs rely heavily on the time being right so that they can be correlated against specific events; if an attacker managed to skew all your logging times it could make it far more difficult to track down what happened.

So how exactly could NTP be abused?

Two ways:
1) A malicious NTP peer relationship
2) A malicious, fake NTP server

I've pre-setup R2 to treat R1 as it's NTP server.  R1 is a stratum 5 master.

R2#show ntp association detail
1.1.1.1 configured, our_master, sane, valid, stratum 5
<output omitted>

Now let's have R3 hijack R2's time:
R3(config)#ntp master 1
R3(config)#ntp source lo0
R3(config)#ntp peer 2.2.2.2

R2#show ntp association detail
1.1.1.1 configured, insane, invalid, stratum 5
<output omitted>

3.3.3.3 dynamic, our_master, sane, valid, stratum 1<output omitted>

Yes, it's just that easy to hijack NTP. Note I don't reference 3.3.3.3 anywhere in R2's config.  R3 simply announced itself as a peer with a better stratum and it wins.

You'd use either access controls or authentication to fix this.

The other potential "hack" is to impersonate a valid NTP server.  Let's say the clients already have access controls to prevent a new NTP server from taking over as shown above.  An impersonation might be from a rogue server posing as having the same IP address as the legitimate NTP server. If it gets polled for time instead, or worse yet, it announced time via broadcast or multicast (discussed later), then its time may be assumed as the correct time.  This can be fixed with authentication.

Let's look at access controls first.  These have a funny order-of-operations and can be confusing.

R3(config)#ntp access-group ?
  peer        Provide full access
  query-only  Allow only control queries
  serve       Provide server and query access
  serve-only  Provide only server access

peer = I allow every operation from this set of IPs
query-only = control queries only, nothing to do with time sync, and outside the scope of this article
serve = I serve time + allow control queries
serve-only = I serve time only

So here's the funny rules.
1) Rules are matched in the order shown above: peer, query-only, serve, serve-only.
2) Once you've put an access-list with an implicit deny in place, there's an implicit deny for all the other mechanisms.  So you need to permit everything you want after that point.

Let's walk through some examples.

First, let's stop that rogue peer on R2:

access-list 90 deny   any
ntp access-group peer 90

We'll eventually see the R3 peer fall off:

But wait, there's more!  .. well, problems that is.

sh ntp association detail
1.1.1.1 configured, insane, invalid, stratum 5

Well we lost 3.3.3.3, but we also lost 1.1.1.1.  We need to trust 1.1.1.1 to set our time:

access-list 90 permit 1.1.1.1
access-list 90 deny   any

R2(config)#do sh ntp association detail
1.1.1.1 configured, our_master, sane, valid, stratum 5

We recover pretty quickly.

Now let's have R3 get it's time from R2.

R3(config)#no ntp
R3(config)#ntp source lo0
R3(config)#ntp server 2.2.2.2

On R3, note that we specify a loopback address as source.  This is best practice when you're restricting access on the server, as NTP clients will automatically source their packets from the nearest interface.  If you lose an interface, you may lose your time, because it's a fair bet you only permitted the closest interface in your access list on your NTP server.

R3(config)#do show ntp association detail
2.2.2.2 configured, insane, invalid, unsynced, stratum 16

And, predictably, it doesn't work.  We restricted peering in the first example, which also restricted every other type of communication automatically.  As mentioned above, you get an implicit deny as soon as you add an access list.

R2(config)#access-list 91 permit 3.3.3.3
R2(config)#ntp access-group serve-only 91
R3(config)#do show ntp association detail
2.2.2.2 configured, our_master, sane, valid, stratum 6

Let's try an order-of-operations example.

Let's permit 3.3.3.3 on peer, and then deny it in serve-only.

R2(config)#no ntp
R2(config)#access-list 91 permit 3.3.3.3
R2(config)#access-list 92 deny   3.3.3.3
R2(config)#ntp master 5
R2(config)#ntp access-group serve-only 92
R2(config)#ntp access-group peer 91

R3#show ntp association detail
2.2.2.2 configured, insane, invalid, unsynced, stratum 16

Well, that didn't work, but it should have.  We expected to have the peer access-list take priority over the serve-only.  Let's examine R2:

R2(config)#do show ntp association detail
127.127.7.1 configured, insane, invalid, unsynced, stratum 4

Uh-oh, R2 thinks its own clock is insane/invalid.
This is a big gotcha on older 12.4(x) code.  A master syncs to itself on 127.127.7.1.  If we don't allow it as a peer, we won't be getting time from ourselves!  Of important note, newer versions of 12.4(x) don't have this problem.  You can easily spot the difference, as modern versions sync to 127.127.1.1 instead.  So if you see that 7 in the 3rd octet, it's a warning sign to not forget to permit it.

R2(config)#access-list 91 permit 127.127.7.1

<wait a while>

R2(config)#do sh ntp association detail
127.127.7.1 configured, our_master, sane, valid, stratum 4

Now R2 is happy again.

R3(config)#do show ntp association detail
2.2.2.2 configured, our_master, sane, valid, stratum 5

And so is R3!

Now let's try authentication.
First rule of authentication: the client is authenticating the server.

There's not a whole lot of harm of giving time to someone, so there's no need for the server to authenticate the client -- if you want to control that, then use access groups.  However, the client needs to know they're not talking to a fake server, and that's where authentication comes in.

Let's have R3 authenticate R2.

The server-side configuration is rather light:

R2(config)#ntp authentication-key 1 md5 CISCO
(note, some newer versions of IOS, including ones potentially used on the lab exam, require "ntp trusted-key <key number>" on the server as well)

The client side configuration:

R3(config)#ntp authentication-key 1 md5 CISCO
R3(config)#ntp authenticate
R3(config)#ntp trusted-key 1
R3(config)#ntp server 2.2.2.2 key 1

The trusted-key command allows you to trust a variety of keys instead of just one.  You could, for example, trust keys 1-3 if you had keys 1, 2, & 3 defined.

And the outcome?

R3(config)#do show ntp association detail
2.2.2.2 configured, authenticated, our_master, sane, valid, stratum 5

Of important note, this doesn't in any fashion force authentication for the other clients.  Other NTP clients will still be able to request time without authentication..

And last, let's look at broadcast and multicast NTP.  We're going to have R2 broadcast time to R3 on a serial link connecting them.

Not shown here, I removed the static configuration on R3 that was referencing R2.

R3(config)#interface Serial1/3
R3(config-if)#ntp broadcast client
R3(config-if)#ntp broadcast key 1

We'll also keep authenticating the packets.  This is particularly important with broadcast or multicast NTP deployments.

R2(config)#interface Serial0/1
 ntp broadcast key 1

and the outcome?

R3(config)#do show ntp association detail
155.1.23.2 dynamic, authenticated, our_master, sane, valid, stratum 5
<output omitted>

Note the dynamic peering this time.

To be fair, it wasn't quite as easy as I show above.  What looks like an IOS bug had R3 remembering that R2 was a configured peer, and it wasn't happy about getting broadcasts from it.  I found that with these commands:

debug ntp packet
debug ntp authentication

After doing a "no ntp" and re-entering just the vital commands, it was happy again.

I'm not going to do a thorough write-up of multicast basics here, so the multicast configuration is very similar to the broadcast:

R2(config)#interface Serial0/1
R2(config-if)#no ntp broadcast
R2(config-if)#ntp multicast 239.0.0.1 key 1

R3(config)#interface Serial1/3
R3(config-if)#no ntp broadcast client
R3(config-if)#no ntp broadcast key 1
R3(config-if)#ntp multicast client 239.0.0.1
R3(config-if)#ntp multicast key 1

And the outcome?

R3(config-if)#do show ntp association detail
155.1.23.2 dynamic, authenticated, our_master, sane, valid, stratum 5

There's a potential gotcha on multicast NTP - the default TTL, at least on 12.4T, is 1.  That means it's basically only valid for that segment.  You can't multicast route it without changing the TTL:

R2(config-if)#ntp multicast ttl ?
  <1-255>  TTL
There you have it folks!  Hope you enjoyed,

Jeff Kronlage

Thursday, May 16, 2013

SNMP & RMON

There's quite a bit to Cisco's SNMP implementation.  Cisco also has a partial implementation of the RMON specification that, among other functions, allows monitoring local SNMP values and generates SNMP traps or syslog as a result.

I'm only going to give a high-level of how SNMP works and what the basic differences are in the versions.

SNMP has three basic functions:
- To expose system information that can be polled by a network monitoring station (NMS) to get information from the device (SNMP GET)
- To allow a NMS to make changes to the device (SNMP SET)
- To directly send urgent information to the NMS (SNMP TRAP or INFORM)

SNMP has four versions:
- v1 - The original specification. Has security problems.
- v2 - The original specification, plus 64-bit counters, informs, and a complex user authentication and data encryption system.  v2 is not backwards compatible with v1, it uses a separate packet structure.  v2 is not widely used, and is not supported on IOS.  This was largely due to the complexity of the security system.
- v2c - 64-bit counters and informs, new packet structure, but without the authentication and data encryption.  This format is widely used and supported on IOS.
- v3 - Similar to v2c except it re-adds user authentication and data encryption.

v1 and v2c both use "communities" for authentication, which is basically a weak, plain-text password system.  v3 uses users instead.

All systems uses the concept of MIBs and OIDs.  OIDs are hierarchical identifiers in numerical format that represent some near-arbitrary value on the device. Most OIDs you'll see on a Cisco device will start with one of the following OIDs:
1.3.6.1.2.1 "Interfaces"
1.3.6.1.4.1.9 "Enterprises - Cisco"

MIBs are dictionaries that give a very primitive text description of an OID.  MIBs are normally installed on your NMS.  If you're querying something like clogHistoryEntry.4.58, you're querying by MIB instead of by OID.  MIBs translate directly to an OID on the target device.

Basic tutorial over, let's look at Cisco's implementation.  Here's our very simple lab:


The Ubuntu VM will act as our NMS. 

R1(config)#snmp-server community public RO
R1(config)#snmp-server community private RW

This is probably the most basic (and miserably insecure) setup for demoing SNMP GET and SET.  The first command associates the community "public" with full read-only access to every OID used on the router.  The second command associates the community "private" with full read & write access to every OID used on the router.

I'm going to use snmpwalk on the Ubuntu box.  Snmpwalk is basically a macro to perform a series of SNMP GETs for all OIDs down the tree from that point.

ubuntu@ubuntu:~$ snmpwalk -v2c -c public 10.0.0.1 .1.3.6.1.2.1

iso.3.6.1.2.1.1.1.0 = STRING: "Cisco IOS Software, 3700 Software (C3725-ADVENTERPRISEK9-M), Version 12.4(15)T14, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2010 by Cisco Systems, Inc.
Compiled Tue 17-Aug-10 12:08 by prod_rel_team"
iso.3.6.1.2.1.1.2.0 = OID: iso.3.6.1.4.1.9.1.122
iso.3.6.1.2.1.1.3.0 = Timeticks: (87351) 0:14:33.51
iso.3.6.1.2.1.1.4.0 = ""
iso.3.6.1.2.1.1.5.0 = STRING: "R1.lab.local"
iso.3.6.1.2.1.1.6.0 = ""
iso.3.6.1.2.1.1.7.0 = INTEGER: 78
iso.3.6.1.2.1.1.8.0 = Timeticks: (0) 0:00:00.00
<majority of output omitted>

This would've produced tens of thousands of lines of output, so I just showed the first few.  The "public" community is both an identifier and a password.  Problem is, it's sent in plain-text, so anyone able to intercept an SNMP packet can make this same query.

Being able to find out router variables may be fairly innocuous, but let's look at a truly risky scenario.

R1(config)#snmp-server system-shutdown

This will allow an snmp-set to reboot the router.

ubuntu@ubuntu:~$ snmpset -v2c -c private 10.0.0.1 1.3.6.1.4.1.9.2.9.9.0 i 2
R1(config)#

***
*** --- SHUTDOWN in

***
*** --- SHUTDOWN NOW ---
*** Message from network to all terminals:
***

Now that would be truly awful. 

So how do you secure this thing?  There are a few options:
- Apply an access-list to the community
- Limit what MIBs the community can access
- Use SNMPv3 (covered later)

R1(config)#access-list 52 permit 10.0.0.5
R1(config)#snmp-server community private RW 52

ubuntu@ubuntu:~$ snmpset -v2c -c private 10.0.0.1 1.3.6.1.4.1.9.2.9.9.0 i 2
Timeout: No Response from 10.0.0.1

Well that solved the problem.  Sort of...

SNMP SET operations are application-layer ACKed, but there's no need for the ACK to actually arrive.  The target will act on the SET operation regardless.  This means that if you can spoof a source IP address to one the router will accept, you can still send SETs at it and it will react!  The attacker will presumably not get the ACK, the real NMS would, but it doesn't matter. Spoofing source addresses is generally very easy, so this is still a major flaw.

On the bright side, this method is still better than ACLing SNMP at the interface.  This way the router can still transit SNMP traffic for other devices.  At the interface, it would block SNMP for devices downstream, as well.

Shy of going to SNMPv3, the best option is only allowing specific SNMP SETs to be sent to specific MIBs.  That is done with views.

First determine what you want to include:

R1#show snmp mib
dot1xPaeSystemAuthControl
dot1xPaePortProtocolVersion
dot1xPaePortCapabilities
dot1xPaePortInitialize
dot1xPaePortReauthenticate
dot1xAuthPaeState
dot1xAuthBackendAuthState
dot1xAuthAdminControlledDirections
dot1xAuthOperControlledDirections
dot1xAuthAuthControlledPortStatus
dot1xAuthAuthControlledPortControl
dot1xAuthQuietPeriod
dot1xAuthTxPeriod

This list is really, really long, and can even hang low-end routers while you're trying to display it.  Let's use a few I've pre-picked.

R1(config)#snmp-server view MYVIEW system.1 included
R1(config)#snmp-server view MYVIEW ifType included
R1(config)#snmp-server community private view MYVIEW RW

ubuntu@ubuntu:~$ snmpwalk -v2c -c private 10.0.0.1 .1.3.6

iso.3.6.1.2.1.1.1.0 = STRING: "Cisco IOS Software, 3700 Software (C3725-ADVENTERPRISEK9-M), Version 12.4(15)T14, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2010 by Cisco Systems, Inc.
Compiled Tue 17-Aug-10 12:08 by prod_rel_team"
iso.3.6.1.2.1.2.2.1.3.1 = INTEGER: 6
iso.3.6.1.2.1.2.2.1.3.2 = INTEGER: 6
iso.3.6.1.2.1.2.2.1.3.3 = INTEGER: 6
iso.3.6.1.2.1.2.2.1.3.5 = INTEGER: 1
iso.3.6.1.2.1.2.2.1.3.5 = No more variables left in this MIB View (It is past the end of the MIB tree)
ubuntu@ubuntu:~$

Now we have a very short list of R/W variables.

Of note, you can also add to SNMP views by OID.  For example:
snmp-server view MYVIEW 1.3.6.1.4.1.9.10.91.1.1.1.1 included

The router will translate this back to the MIB:
snmp-server view MYVIEW ceemEventMapEntry included

Might also be a handy way to acquire a MIB value if given an OID?

There are a few values that are commonly set by the config that are important to be aware of:
R1(config)#snmp-server contact Your Admins Name
R1(config)#snmp-server location San Jose, CA
R1(config)#snmp-server chassis-id 11223344

ubuntu@ubuntu:~$ snmpget -v2c -c public 10.0.0.1 1.3.6.1.2.1.1.6.0
iso.3.6.1.2.1.1.6.0 = STRING: "San Jose, CA"

Thus far I haven't spent much time explaining which version we're using.  We're still going to avoid v3 until later, but let's look at v1 and v2c.

To simplify our output, let's cut this down to just the read-only community, public.

Now let's run show snmp group:



We see "public" twice - security model "v1" and security model "v2c".  Just turning on the community enables both v1 and v2c by default.

If you just wanted v2c, you can do this... rather awkward... command.

R1(config)#no snmp-server group public v1

So what's so awkward about it?

R1(config)#do sh run | i snmp-server group
R1(config)#

It doesn't show up in the config.  If you do a wr mem and reboot, you get both v1 and v2c again.  Not sure how that plays out for the CCIE lab, but it sure makes me uneasy.

That all said, before the reboot, it does work:



You might wonder what else the snmp-server group command can do.  Best I can tell, before considering v3, there's not much else you can use it for besides selectively disabling v1 or v2c.

Before we consider SNMP traps or v3, let's look at some SNMP router tricks you should know about.

Did you know you could acquire the router config via SNMP?  Not so much by SNMP GETs, but by SETs!  How, you may ask?  You can instruct the router to TFTP the config somewhere!

I've pre-setup a TFTP server on the Ubuntu box.  I've also removed the view from the "private" community.  I'm not going to go over these commands one-by-one, as there's no way this could possibly be on the CCIE lab, but you may want to try them for yourself:

These prep the transfer:
snmpset -c private -v 1 10.0.0.1 1.3.6.1.4.1.9.9.96.1.1.1.1.2.111 i 1  
snmpset -c private -v 1 10.0.0.1 1.3.6.1.4.1.9.9.96.1.1.1.1.3.111 i 4
snmpset -c private -v 1 10.0.0.1 1.3.6.1.4.1.9.9.96.1.1.1.1.4.111 i 1
snmpset -c private -v 1 10.0.0.1 1.3.6.1.4.1.9.9.96.1.1.1.1.5.111 a 10.0.0.2
snmpset -c private -v 1 10.0.0.1 1.3.6.1.4.1.9.9.96.1.1.1.1.6.111 s running.cfg

This initiates:
snmpset -c private -v 1 10.0.0.1 1.3.6.1.4.1.9.9.96.1.1.1.1.14.111 i 1  

This resets the prep commands above:
snmpset -c private -v 1 10.0.0.1 1.3.6.1.4.1.9.9.96.1.1.1.1.14.111 i 6

Afterwards, you'll have a file called "running.cfg" on the TFTP server on 10.0.0.2.

Well that's kind of scary, too.  Someone could send the config to any TFTP server.
Here's how you leverage this feature but limit your risk:

R1(config)#snmp-server file-transfer access-group 10 protocol ?
  ftp   Configure acl for ftp transfer protocol
  rcp   Configure acl for rcp transfer protocol
  scp   Configure acl for scp transfer protocol
  sftp  Configure acl for sftp transfer protocol
  tftp  Configure acl for tftp transfer protocol

Configure an access-list, specify your protocol, and you can control where the router will send files to via SNMP.

On another topic, most items that are monitored on a Cisco device are interface-related.  Did you know the interface OIDs can shift between reboots?  If a new interface is added, the interfaces can be re-indexed when the router comes back up.  This is not helpful if you're trying to reference them by either MIB OR OID.  To prevent this default behavior, you have two options:

Globally:
snmp-server ifindex persist

Per-interface:
snmp ifindex persist

Either of these will store the interface index in NVRAM, and load them on reboots.

Now we'll take a look at SNMP v3.  We'll follow that up with traps and informs, and then RMON last.

SNMP has three security models:
noAuthNoPriv - This basically disables the security features of SNMPv3, and there's almost no reason to use it.
authNoPriv - This will use a hash to authenticate the user, but not encrypt the SNMP data
authPriv - This will use a hash to authenticate the user, as well as encrypt the SNMP data

First and foremost, SNMP v3 drops the community used by v1 and v2c.  It uses groups and users instead.

Let's start with the most basic noAuthNoPriv example:

R1(config)#snmp-server user LABUSER LABGROUP v3
R1(config)#snmp-server group LABGROUP v3 noauth

ubuntu@ubuntu:~$ snmpwalk -v3 -u LABUSER -l noAuthNoPriv 10.0.0.1 1.3.6.1.2.1.1.9
iso.3.6.1.2.1.1.9.1.2.1 = OID: iso.3.6.1.4.1.9.7.129
iso.3.6.1.2.1.1.9.1.2.2 = OID: iso.3.6.1.4.1.9.7.115
iso.3.6.1.2.1.1.9.1.2.3 = OID: iso.3.6.1.4.1.9.7.265
<output omitted>

As you can see, that's really not any better than SNMP v2c.  Basically we substituted an non-secure community for a non-secure username.  Of note, you can still use views, just like with previous versions of SNMP

snmp-server view MYVIEW system.1 included
snmp-server view MYVIEW ifType included
snmp-server group LABGROUP v3 noauth read MYVIEW write MYVIEW

ubuntu@ubuntu:~$ snmpwalk -v3 -u LABUSER -l noAuthNoPriv 10.0.0.1 1.3.6.1.2
<output omitted>
iso.3.6.1.2.1.2.2.1.3.1 = INTEGER: 6
iso.3.6.1.2.1.2.2.1.3.2 = INTEGER: 6
iso.3.6.1.2.1.2.2.1.3.3 = INTEGER: 6
iso.3.6.1.2.1.2.2.1.3.5 = INTEGER: 1
iso.3.6.1.2.1.2.2.1.3.5 = No more variables left in this MIB View (It is past the end of the MIB tree)

Just for comparison purposes moving forward, let's take a look at the debug from debug snmp headers on R1:

Incoming SNMP packet
*Mar  1 00:26:29.311: v3 packet         security model: v3       security level: noauth
*Mar  1 00:26:29.311: username: LABUSER
<output omitted>

The important part here is the noauth, which is noAuthNoPriv in Cisco-speak.

Let's upgrade our security from noAuthNoPriv to authNoPriv

R1(config)#snmp-server user LABUSER LABGROUP v3 auth sha MYSECRET
R1(config)#snmp-server group LABGROUP v3 noauth  ! NOTE, I am not changing this yet

ubuntu@ubuntu:~$ snmpwalk -v3 -u LABUSER -l authNoPriv -a sha -A MYSECRET 10.0.0.1 1.3.6.1.2
<output omitted>
iso.3.6.1.2.1.2.2.1.3.1 = INTEGER: 6
iso.3.6.1.2.1.2.2.1.3.2 = INTEGER: 6
iso.3.6.1.2.1.2.2.1.3.3 = INTEGER: 6
iso.3.6.1.2.1.2.2.1.3.5 = INTEGER: 1
iso.3.6.1.2.1.2.2.1.3.5 = No more variables left in this MIB View (It is past the end of the MIB tree)

We've asked for authNoPriv, SHA authentication, with the appropriate password.

Incoming SNMP packet
*Mar  1 00:29:52.723: v3 packet         security model: v3       security level: auth
*Mar  1 00:29:52.723: username: LABUSER

And we got auth (authNoPriv) per the debug.  So what's this actually do?:

snmp-server group LABGROUP v3 noAuth

This doesn't force noAuth.  It just doesn't enforce a higher security level.  In other words, this group will accept authentication, no authentication, or authentication + encryption.  Now we did have to include a username and hash method on the snmp-server user command, or there would be no way to authenticate.

Of course the catch with this, you can still authenticate without the password:

ubuntu@ubuntu:~$ snmpwalk -v3 -u LABUSER -l noAuthNoPriv 10.0.0.1 1.3.6.1.2
<output omitted>
iso.3.6.1.2.1.2.2.1.3.1 = INTEGER: 6
iso.3.6.1.2.1.2.2.1.3.2 = INTEGER: 6
iso.3.6.1.2.1.2.2.1.3.3 = INTEGER: 6
iso.3.6.1.2.1.2.2.1.3.5 = INTEGER: 1
iso.3.6.1.2.1.2.2.1.3.5 = No more variables left in this MIB View (It is past the end of the MIB tree)

Incoming SNMP packet
*Mar  1 00:37:46.367: v3 packet         security model: v3       security level: noauth
*Mar  1 00:37:46.367: username: LABUSER

So that doesn't really protect anything.  As always, my focus is regarding the CCIE lab, and it's about knowing how to do things, not about producing good production configs.  So knowing this can be done is important.

Let's make this work in a more logical fashion:

R1(config)#no snmp-server group LABGROUP v3 noauth
R1(config)#snmp-server group LABGROUP v3 auth

ubuntu@ubuntu:~$ snmpwalk -v3 -u LABUSER -l noAuthNoPriv 10.0.0.1 1.3.6.1.2
Error in packet.
Reason: authorizationError (access denied to that object)

The door's not wide up any longer...  let's try it with auth.

ubuntu@ubuntu:~$ snmpwalk -v3 -u LABUSER -l authNoPriv -a sha -A MYSECRET 10.0.0.1 1.3.6.1.2.1
<output omitted>
iso.3.6.1.2.1.1.2.0 = OID: iso.3.6.1.4.1.9.1.122
iso.3.6.1.2.1.1.3.0 = Timeticks: (271023) 0:45:10.23
iso.3.6.1.2.1.1.4.0 = ""
iso.3.6.1.2.1.1.5.0 = STRING: "R1.lab.local"
iso.3.6.1.2.1.1.6.0 = ""
iso.3.6.1.2.1.1.7.0 = INTEGER: 78
iso.3.6.1.2.1.1.8.0 = Timeticks: (0) 0:00:00.00
iso.3.6.1.2.1.1.9.1.2.1 = OID: iso.3.6.1.4.1.9.7.129
<output omitted>

Now we have a true authentication requirement on our SNMP process.
Let's add encryption.

R1(config)#snmp-server group LABGROUP v3 priv
R1(config)#snmp-server user LABUSER LABGROUP v3 auth sha MYSECRET priv aes 128 PW4ENCRYPTION

ubuntu@ubuntu:~$ snmpwalk -v3 -u LABUSER -l authPriv -a sha -A MYSECRET -x aes -X PW4ENCRYPTION 10.0.0.1 1.3.6.1.2
<output omitted>
iso.3.6.1.2.1.1.2.0 = OID: iso.3.6.1.4.1.9.1.122
iso.3.6.1.2.1.1.3.0 = Timeticks: (359166) 0:59:51.66
iso.3.6.1.2.1.1.4.0 = ""
iso.3.6.1.2.1.1.5.0 = STRING: "R1.lab.local"
iso.3.6.1.2.1.1.6.0 = ""
iso.3.6.1.2.1.1.7.0 = INTEGER: 78
<output omitted>

Incoming SNMP packet
*Mar  1 00:59:52.687: v3 packet         security model: v3       security level: priv
*Mar  1 00:59:52.687: username: LABUSER

Did you know you could mix the v3 authentication methods on the same group?

snmp-server group LABGROUP v3 noauth read MYVIEW
snmp-server group LABGROUP v3 priv

This will permit noAuthNoPriv read access to the MYVIEW view, and authPriv to everything else.

ubuntu@ubuntu:~$ snmpwalk -v3 -u LABUSER -l authNoPriv -a sha -A MYSECRET 10.0.0.1 1.3.6.1.2.1
<output omitted>
iso.3.6.1.2.1.2.2.1.3.1 = INTEGER: 6
iso.3.6.1.2.1.2.2.1.3.2 = INTEGER: 6
iso.3.6.1.2.1.2.2.1.3.3 = INTEGER: 6
iso.3.6.1.2.1.2.2.1.3.5 = INTEGER: 1
iso.3.6.1.2.1.2.2.1.3.5 = No more variables left in this MIB View (It is past the end of the MIB tree

authNoPriv, we get the small sampling... what about with priv?

ubuntu@ubuntu:~$ snmpwalk -v3 -u LABUSER -l authPriv -a sha -A MYSECRET -x aes -X PW4ENCRYPTION 10.0.0.1 1.3.6.1.2.1
<output omitted>
iso.3.6.1.2.1.1.2.0 = OID: iso.3.6.1.4.1.9.1.122
iso.3.6.1.2.1.1.3.0 = Timeticks: (511295) 1:25:12.95
iso.3.6.1.2.1.1.4.0 = ""
iso.3.6.1.2.1.1.5.0 = STRING: "R1.lab.local"
iso.3.6.1.2.1.1.6.0 = ""
iso.3.6.1.2.1.1.7.0 = INTEGER: 78
iso.3.6.1.2.1.1.8.0 = Timeticks: (0) 0:00:00.00
iso.3.6.1.2.1.1.9.1.2.1 = OID: iso.3.6.1.4.1.9.7.129
iso.3.6.1.2.1.1.9.1.2.2 = OID: iso.3.6.1.4.1.9.7.115
iso.3.6.1.2.1.1.9.1.2.3 = OID: iso.3.6.1.4.1.9.7.265
iso.3.6.1.2.1.1.9.1.2.4 = OID: iso.3.6.1.4.1.9.7.112
iso.3.6.1.2.1.1.9.1.2.5 = OID: iso.3.6.1.4.1.9.7.106
iso.3.6.1.2.1.1.9.1.2.6 = OID: iso.3.6.1.4.1.9.7.47
iso.3.6.1.2.1.1.9.1.2.7 = OID: iso.3.6.1.4.1.9.7.122
iso.3.6.1.2.1.1.9.1.2.8 = OID: iso.3.6.1.4.1.9.7.135
iso.3.6.1.2.1.1.9.1.2.9 = OID: iso.3.6.1.4.1.9.7.43
iso.3.6.1.2.1.1.9.1.2.10 = OID: iso.3.6.1.4.1.9.7.37
iso.3.6.1.2.1.1.9.1.2.11 = OID: iso.3.6.1.4.1.9.7.92
iso.3.6.1.2.1.1.9.1.2.12 = OID: iso.3.6.1.4.1.9.7.53
iso.3.6.1.2.1.1.9.1.2.13 = OID: iso.3.6.1.4.1.9.7.54
iso.3.6.1.2.1.1.9.1.2.14 = OID: iso.3.6.1.4.1.9.7.52
^C
Obviously, I killed it there, so it didn't go on forever.

Here's an oddity...

R1#sh run | i snmp-server user
R1#

You saw me create the users.

They're not shown in the running-config for security reasons.  Annoying, in my opinion.  You can get some limited information from:

R1#show snmp user

User name: LABUSER
Engine ID: 800000090300C2000BF40000
storage-type: nonvolatile        active
Authentication Protocol: SHA
Privacy Protocol: AES128
Group-name: LABGROUP

Now let's take a look at traps.

An SNMP trap is sent to the NMS when an "urgent" event occurs on the device.  The idea is that the NMS can only poll the target device every so often, and the event may have cleared before NMS station revisits the target (my personal opinion is use a faster NMS so you can poll more often!).  Anyway, Cisco devices can send a plethora of events to the NMS, and here's how you configure it.

R1(config)#snmp-server enable traps
R1(config)#snmp-server host 10.0.0.2 traps version 2c MYCOMMUNITY

This is the most basic setup. Enable traps globally, then enable them per-NMS.  Note, if you just use the host command, and don't enable traps globally, you will get no results.

snmp-server enable traps enables all possible traps.  For example:

snmp-server enable traps snmp authentication linkdown linkup coldstart warmstart
snmp-server enable traps vrrp
snmp-server enable traps ds1
snmp-server enable traps tty
snmp-server enable traps eigrp
snmp-server enable traps xgcp
snmp-server enable traps flash insertion removal
<output omitted>

The usage of the snmp-server host command is pretty simple:
snmp-server host 10.0.0.2 traps version 2c MYCOMMUNITY

traps indicates traps instead of informs (covered below), version indicates the trap format (defaults to v1 if unspecified), and the community gives a way of grouping the traps on the NMS.  Note, the trap community does not have to correspond with your "GET" communities whatsoever.

Now our config above produces an awful lot of traps.  You have two options for limiting them:

Globally:
no snmp-server enable traps   ! turns off the mass of traps we already enabled
snmp-server enable traps eigrp
snmp-server enable traps bgp
snmp-server enable traps ospf

That will globally send those traps towards any configured hosts.

Now if we had another host we only wanted BGP traps to, but not eigrp and ospf, you could additionally:

Per-host:
snmp-server host 10.0.0.2 version 2c MYCOMMUNITY bgp

Default destination port for traps is UDP 162, if you want to change it:
snmp-server host 10.0.0.2 version 2c MYCOMMUNITY bgp udp-port 100

One of the catches of traps is that they're an un-ACKed UDP packet.  The network device has no way of knowing whether or not they arrive.  Introducing informs.

Informs are basically ACKed traps.  They were introduced in SNMP v2.

Usage is:
snmp-server enable traps
snmp-server host 10.0.0.2 informs version 2c MYCOMMUNITY

Usage is exactly the same as traps except you put the word "informs" in the host statement.

The syslog can be encapsulated into SNMP traps and sent to an NMS.

R1(config)#snmp-server enable traps syslog
R1(config)#snmp-server host 10.0.0.2 MYCOMMUNITY syslog

Both of these above options would be enabled by default if you just enabled trapping unfiltered globally and on the host, but here's the magic command:

R1(config)#logging history debugging

Logging history is the key.  You can enable whatever level of logging you like.  Let's give this a try.

Note I wasn't up for installing an NMS product, so we'll be using the magic of tcpdump to see this happen on 10.0.0.2.

R1(config)#exit
R1#
*Mar  3 00:25:46.499: %SYS-5-CONFIG_I: Configured from console by console

(exiting config mode is an easy way to generate syslog on debugging)

04:40:57.603943 IP 10.0.0.1.64738 > 10.0.0.2.snmp-trap:  C=MYCOMMUNITY Trap(182)  E:cisco.9.41.2 10.0.0.1 enterpriseSpecific s=1 17434657 E:cisco.9.41.1.2.3.1.2.4="SYS" E:cisco.9.41.1.2.3.1.3.4=6 E:cisco.9.41.1.2.3.1.4.4="CONFIG_I" E:cisco.9.41.1.2.3.1.5.4="Configured from console by console" E:cisco.9.41.1.2.3.1.6.4=17434651

And some miscellaneous items:

How to change the SNMP packet size:
snmp-server packetsize <size> ! defaults to 1500 bytes

How to change the source interface:
snmp-server trap-source <interface>

By default, all interfaces send link-status traps.  To individually disable them:
interface Fa0/0
  no snmp trap link-status

SNMP traps are rate-limited on egress.  If you have a large number of NMS stations or expect a great deal of traps, you may need to queue traps for delivery.  By default, the IOS device will hold 10 traps in its queue.  To change how many traps IOS will queue:
snmp-server queue-length <qty of traps>

SNMP informs have a similar queue-length, but it holds items in queue until they're acknowledged, meaning the queue-length needs to potentially be longer.  The default is 25.
snmp-server inform pending <qty of informs>

When to consider a lack of ACK a failure on an inform:
snmp-server inform timeout <seconds>

How many times to retry an un-acked inform:
snmp-server inform retries <qty of retries>

RMON - Defined over RFC 2819 and 4502, RMON is a large, complex network monitoring suite.  Only a couple of Cisco devices implement the entire suite, and none of them are on the CCIE lab.  The big important piece that is implemented on all IOSes and is on the lab is the ability to monitor a local SNMP variable and generate syslog or an SNMP trap.

The easiest way for me to think about RMON's function on IOS is to consider an SNMP GET -> TRAP converter.  It looks at it's own SNMP variable and generates TRAPs if certain criteria are met.

With RMON, you generate alarms and events.  Alarms call the events.  Events can trigger logging, traps, or both.

rmon event 1 trap MYCOMMUNITY owner config
rmon alarm 1 ip.3.0 15 delta rising-threshold 10 1 falling-threshold 5 1 owner config


This is the most basic usage possible.  Let's run over some of the variables.

rmon event 1 trap MYCOMMUNITY owner config

- The "1" in "event 1" is what the alarm will reference.
- The trap indicates sending a trap. 
- MYCOMMUNITY is the community string to send to.  Note, this must match a host statement's community, or no one will get these traps, i.e:
snmp-server host 10.0.0.2 informs version 2c MYCOMMUNITY
- I searched high and low for a better explanation on this, but best I can tell, the "owner" variable is just a text label.  It doesn't appear to do anything at all.  It's not included in the trap or with the log.  "config" is the default value for "owner" if you don't specify anything.

rmon alarm 1 ip.3.0 15 delta rising-threshold 10 1 falling-threshold 5 1 owner config

- The "1" in "alarm 1" is just an identifier, it doesn't reference anything
- "ip.3.0" is an arbitrary SNMP MIB value I picked, it can be any value exposed by the system
- "15" is the number of seconds between polling attempts
- "delta" will be explained later (options are either delta or absolute)
- The "10" in "rising threshold 10" is a high-water mark to trigger an event when hit.
- The following "1" is which event to call if the rising-threshold is triggered
- The "5" in "falling-threshold 5" is the low-water mark to trigger an event when hit
- The following "1" is which event to call if the falling-threshold is triggered
- "owner config", as referenced above, doesn't appear to actually have any meaning.

Of important note is the command show snmp mib ifmib ifindex, which will help you find the index number assigned to a specific interface.  Sample questions often have you measuring interface load, and while many questions provide you with the general value, you have to find the index for yourself.  Remember the snmp-server ifindex persist command from above, to make these values static between reboots.

As mentioned above, there are two operators that can be applied to alarms - delta and absolute.  These can be confusing.  Absolute is easier to work with, so I will start with it first.  Absolute should be used with values that can increase or decrease, such as CPU load.  If you wanted to get an alert when CPU load hit 80%, and then wanted to get notified when it settled back down to 10%, you would use absolute.

That's great in production, but it turns out getting a virtualized router to 80% CPU load in such a way that it can fall back down to a reasonable number afterwards without crashing actually takes quite a lot of effort.  So for our purposes, I'm going to test with CPU load at 10% being the high-water mark, and 5% being the low-water mark.  CPU load will be created on the device by enabling debug ip packet and flooding it with pings from another terminal (not pictured here).

This is our config:
rmon event 1 trap MYCOMMUNITY description "HIGH CPU LOAD" owner config
rmon event 2 trap MYCOMMUNITY description "CPU LOAD OK" owner config
rmon alarm 1 cpmCPUTotalTable.1.6.1 10 absolute rising-threshold 10 1 falling-threshold 5 2 owner config

When cpmCPUTotalTable.1.6.1 (5 second average of router CPU) hits 10%, event 1 will be called.  When it gets back down to 5 percent or lower, event 2 will be called.  It is important to note that events are not called more than once in a row.  If the CPU load is over 10% for more than two 2-second cycles, only one trap will be generated.  CPU load must fall below 5% again, event 2 will be called, and then if CPU load goes over 10% again, then event 1 can be called again.

I've turned on debug ip packet on R1, and I'm now firing away at it with a huge number of pings.

<large amounts of debug output omitted>
*Mar  3 02:13:56.711: %RMON-5-RISINGTRAP: Rising trap is generated because the value of cpmCPUTotalTable.1.6.1 exceeded the rising-threshold value 10

Let's take a look at the output on our Linux box, monitoring traps via tcpdump:

15:01:55.204278 IP 10.0.0.1.64738 > 10.0.0.2.snmp-trap:  C=MYCOMMUNITY Trap(258)  E:cisco.9.41.2 10.0.0.1 enterpriseSpecific s=1 18142683 E:cisco.9.41.1.2.3.1.2.25="RMON" E:cisco.9.41.1.2.3.1.3.25=6 E:cisco.9.41.1.2.3.1.4.25="RISINGTRAP" E:cisco.9.41.1.2.3.1.5.25="Rising trap is generated because the value of cpmCPUTotalTable.1.6.1 exceeded the rising-threshold value 10" E:cisco.9.41.1.2.3.1.6.25=18142676

Terminating the ping stream, giving it a second...

<more debug omitted>
*Mar  3 02:14:06.731: %RMON-5-FALLINGTRAP: Falling trap is generated because the value of cpmCPUTotalTable.1.6.1 has fallen below the falling-threshold value 5

And on the Linux box:

15:02:05.225729 IP 10.0.0.1.64738 > 10.0.0.2.snmp-trap:  C=MYCOMMUNITY Trap(269)  E:cisco.9.41.2 10.0.0.1 enterpriseSpecific s=1 18143682 E:cisco.9.41.1.2.3.1.2.26="RMON" E:cisco.9.41.1.2.3.1.3.26=6 E:cisco.9.41.1.2.3.1.4.26="FALLINGTRAP" E:cisco.9.41.1.2.3.1.5.26="Falling trap is generated because the value of cpmCPUTotalTable.1.6.1 has fallen below the falling-threshold value 5" E:cisco.9.41.1.2.3.1.6.26=18143678

It works as expected.  Well, sort of.  If you were paying close attention, you'll notice I put descriptions in with the trap events:

rmon event 1 trap MYCOMMUNITY description "HIGH CPU LOAD" owner config
rmon event 2 trap MYCOMMUNITY description "CPU LOAD OK" owner config

I haven't found a use for these other than to provide a text description in the config.  I've read elsewhere that these are supposed to be included in the trap, but as you can plainly see from the tcpdump output, this text isn't passed along anywhere.  You'll also notice the "owner" value isn't passed in the trap, either.

RMON events can either log, trap, or log & trap. Let's try swapping our output to logging.

R1(config)#rmon event 1 log description "HIGH CPU LOAD"
R1(config)#rmon event 2 log description "CPU LOAD OK"
R1(config)#logging trap notifications
R1(config)#logging 10.0.0.2

I'm firing up the same ping flood to create cpu load.

R1#
*Mar  1 00:08:33.711: %RMON-5-RISINGTRAP: Rising trap is generated because the value of cpmCPUTotalTable.1.6.1 exceeded the rising-threshold value 10

And on our tcpdump "collector":

15:39:04.438924 IP (tos 0x0, ttl 255, id 45646, offset 0, flags [none], proto UDP (17), length 190)
    10.0.0.1.55641 > 10.0.0.2.syslog: [udp sum ok] SYSLOG, length: 162
        Facility local7 (23), Severity notice (5)
        Msg: 176719: *Mar  1 00:08:33.711: %RMON-5-RISINGTRAP: Rising trap is generated because the value of cpmCPUTotalTable.1.6.1 exceeded the rising-threshold value 10

There it is, facility local7, severity 5.
For brevity I'm excluding the falling output, but suffice to say it works as expected.

One would think this is a slam-dunk, but not so fast.

Let's put it back to trap, but leave the generalized syslog config in place:

R1(config)#no rmon event 1 log description "HIGH CPU LOAD"
R1(config)#no rmon event 2 log description "CPU LOAD OK"
R1(config)#rmon event 1 trap BOGUS description "HIGH CPU LOAD"
R1(config)#rmon event 2 trap BOGUS description "CPU LOAD OK"

Now we won't get the traps any longer, but check out what happened on our syslog output after I fired up the pings again:

16:03:49.160416 IP (tos 0x0, ttl 255, id 21, offset 0, flags [none], proto UDP (17), length 186)
    10.0.0.1.49286 > 10.0.0.2.syslog: [udp sum ok] SYSLOG, length: 158
        Facility local7 (23), Severity notice (5)
        Msg: 22: *Mar  1 00:07:08.415: %RMON-5-RISINGTRAP: Rising trap is generated because the value of cpmCPUTotalTable.1.6.1 exceeded the rising-threshold value 10

Best I can tell, the "log" keyword does absolutely nothing.  You get logging automatically when you enable an RMON event, even if you only specify traps.  Could be a bug in my version of IOS.  Testing on 12.4(15)T14.

Now that we've mastered absolute, let's look at the trickier delta.

Delta is best used for SNMP values that only climb.  For instance, number of packets across an interface will only go up, barring clearing the counters.  The command syntax is identical, which adds to the difficulty of getting your head around this, because the variables actually act differently.

Let's go back to a similar example to my original:

rmon alarm 1 ip.3.0 15 delta rising-threshold 10 1 falling-threshold 5 2 owner config

In this case, we're measuring the change per-measurement of ip.3.0.  Let's replace with some pseudo-variables for ease of reference:

rmon alarm 1 ip.3.0 $SEC$ delta rising-threshold $BIG$ 1 falling-threshold $SMALL$ 2

During period $SEC$, if ip.3.0 increased < $SMALL$, trigger event 2
During period $SEC$, if ip.3.0 increased > $BIG$, trigger event 1
During period $SEC$, if ip.3.0 increased > $SMALL$ but < $BIG$, do nothing

So let's walk through this.  I'm not exactly sure what ip.3.0 references, but it appears to be a packet counter of some sort. 

rmon alarm 1 ip.3.0 15 delta rising-threshold 10 1 falling-threshold 5 2 owner config
$SMALL$ = 2
$BIG$ = 10

For simplicity, let's assume we are turning the router on fresh, and RMON kicks in at the same time IP forwarding does.

On seconds 0-9, if no packets are received whatsoever, our value for ip.3.0 = 0, and event 2 is triggered due to "ip.3.0 increased < 2".

On seconds 10-19, if five packets are received, our value for ip.3.0 = 5, our previous value for ip.3.0 = 0.  5 - 0 = 5, so we increased 5 packets.  Per my rule above:
"During period $SEC$, if ip.3.0 increased > $SMALL$ but < $BIG$, do nothing"
We defined $SMALL$ (falling-threshold) as 2 and $BIG$ (rising-threshold) as 10, so:
"During period 10 seconds, if ip.3.0 increased > 2 but < 10, do nothing"
No action is triggered.

On seconds 20-29, if fifty packets are received, our value for ip.3.0 = 55, our previous value for ip.3.0 = 5.  50-5 = 50, so we increased 50 packets.  Per my rule above:
During period 10 seconds, if ip.3.0 increased > 10, trigger event 1
Event 1 is triggered

On seconds 30-39, if one hundred packets are received, our value for ip.3.0 = 155, our previous value for ip.3.0 = 55.  155-55 = 100, so we increased 50 packets.  Per my rule above:
During period 10 seconds, if ip.3.0 increased > 10, trigger event 1
CATCH: rising-events and falling-events cannot be triggered back to back, so actually, no action is triggered.

On seconds 30-39, if three packets are received, our value for ip.3.0 = 158, our previous value for ip.3.0 = 155.  158-155 = 3, so we increased 3 packets.  Per my rule above:
During period 10 seconds, if ip.3.0 increased < $SMALL$, trigger event 2
Event 2 is triggered

I had a complex lab solution worked out for delta, but it's so tied to being able to follow the timer that I've decided it didn't work well visually and went with the above example instead.

I really hope you enjoyed!

Jeff