Category Archives: Routing / switching

Routing / switching

QOS-4-QLIMIT_HQUEUE_VALUE_SYNC_ISSUE.. eh what?

Earlier this week I added a class to a already existing policy which was being used for outbound traffic-shaping. Nothing excited you would say? Well, not really. However.. it turned out that when I created the 15th class under the policy a worrying message appeared in the logging of the router:

%QOS-4-QLIMIT_HQUEUE_VALUE_SYNC_ISSUE

The policy did seem to work but this message was not giving me a warm feeling. So as a typical network engineer of the 21th century, I copied the message into Google in the hope to find out what the poor router was trying to tell me. Literately zero results, so that was very promising.

I knew that it had to be something with queues and the size of it (until the adding of this new class, everything was working fine). So I searched for some basic information about hold queues on ISR routers and how to configure them. I stumbled upon some interesting information, but still no satisfying answer on how to calculate which “hold queue size” was appropriated for my situation. So I reached out to TAC and the information they give me was very helpful, so I want to share it with the Internet 🙂

Once you configure class-based queuing on a ISR router, the default interface hold-queue is being changed to 1000 packets. Every class has a default queue-limit of 64, so once you are configuring the 15th class (which is the 16th if you count the default class, which is always there) you should have a hold-queue of 1024 packets (16*64). The hold-queue is 1000 packets so that is the reason why the message is being displayed. So what size should it be? As always it depends, but in consultation with with TAC I configured it to a size of 1984 packets without any problems and with some grow in mind. This was on a 2901 ISR with 50~70Mbit/s traffic placed as a Internet router (so just some basic routing, ACL’s and traffic-shaping). I hope this information can help you!

Routing / switching, Wireless

Default QoS (CoS) 802.1p values on a WLC and the IEEE/IETF standards

Right now I’m preparing for the 642-742 IUWVN (implementing Cisco Unified Wireless Voice Networks) exam. In all the learning material, you will find that the authors say that Cisco is using a confusing way of configure the QoS markings for bridging wireless traffic to the wired side. The issue is that when you configure the QoS profiles, the default 802.1p (CoS) values that will be shown are wrong if you follow the IEEE/IETF standards. The explanation from Cisco for this is that when you configure the QoS profiles, you work with the WMM user priority value and not the “real” 802.1p value. Very confusing if you ask me 😕 .

However. Since software 8.0 the previous described behavior got changed to match with the IEEE/IETF standards (see bug CSCui22330 for more information). So if you do an upgrade the values you supposed to use are wrong and you need to correct them to the “real” 802.1p values.

Routing / switching, Security

Cisco AnyConnect slow authentication prompt

Last month I deployed a remote access VPN solution based on Cisco IOS routers with webvpn and Cisco’s AnyConnect client. After some basic tests everything looked good until someone with Windows 8.1 tried to make a connection. The user complained that it took at least 20 seconds before the authentication prompt appeared after making the initial connection. And guess what, the user was right! It was even worse -> also Android and iOS showed the same behavior.

So we fired up wireshark and looked at the packets to see what was going. We discovered that AnyConnect sends a client hello message using TLS 1.2, the IOS routers send a [FIN, ACK], terminating the session. AnyConnect eventually tries to connect using TLS 1.0, but that process took over 20 seconds to complete. So I contacted TAC and they filled a bug for this case (CSCun89616).

The good news is that Cisco fixed this bug with creating a new software release. The bad news is that at the this very moment there is not a software release out where bug CSCun89616 is fixed and the router does not crash when using webvpn because of bug CSCug17485… 🙁

So to make a long story short; if you are running into this issue you can’t fix it without giving up stability. I’m still in contact with TAC, so when there is a stable release where both bugs are fixed, I will update this post.

Update may: In IOS versions 15.3(3)M3 and 15.4(3)M both bugs should be fixed.

Routing / switching, Security, Wireless

Cisco Prime Infrastructure import certificates made easy

Working with certificates time to time can be a little difficult, implementations differ and there are more than one certificate “formats”. This is also the case when it comes to importing a “pfx” archive to Cisco Prime Infrastructure (1.x or 2.x). If you search the web you will find a lot topics, but you will have to mix those together to get the job really done.

In this scenario you have a wildcard certificate and the private key combined together in a pfx archive (cert.pfx). You will need a openssl installation, the CA bundle (Root certificate and intermediates certificates, “CA-CERTS.CER”) which you can download from the certificate authorities website and these six steps:

1. Export the pfx archive to a standalone certificate and private key
openssl pkcs12 -in cert.pfx -nocerts -out key.pem
openssl pkcs12 -in cert.pfx -clcerts -nokeys -out cert.pem
2. Decrypt the private key
openssl rsa -in key.pem -out key2.pem
3. Convert certificate from pem to der format
openssl x509 -outform der -in cert.pem -out cert.der
4. Download the “CA-CERTS.CER”, “key2.pem” and “cert.der” files with FTP from the Prime CLI
5. Import the CA certs in Prime:
PRIME/admin# ncs key importcacert CA-CERTS CA-CERTS.cer repository defaultRepo
INFO: no staging url defined, using local space. rval:2
truststore used is /opt/CSCOlumos/conf/truststore
The NCS server is running
Changes will take affect on the next server restart
Importing certificate to trust store
6. Import certificate and private key
PRIME/admin# ncs key importkey key2.pem cert.der repository defaultRepo
INFO: no staging url defined, using local space. rval:2
INFO: no staging url defined, using local space. rval:2
truststore used is /opt/CSCOlumos/conf/truststore
The NCS server is running
Changes will take affect on the next server restart
Importing RSA key and matching certificate
7. Restart the NCS application, and drink a cup of coffee.. 😉
PRIME/admin# ncs stop
PRIME/admin# ncs start

Routing / switching, Security

Cisco AnyConnect VPN connected through a firewall

Most Cisco AnyConnect VPN configurations I see in the field, or have deployment myself, are terminated on a Cisco ASA firewall who is directly connected to the internet. However, in some bigger networks it is not uncommon to have another firewall in front of the remote access / VPN block in your network or to have an access-list on the routers in the internet edge.

Everybody knows the story about the biggest pro which the Cisco AnyConnect solution has if you compare it to the old IPSEC remote access based solution –> “it just works everywhere™”. That story is based on the fact that in most guest and mobile networks SSL network traffic (TCP/443) is allowed. This is true; AnyConnect will work fine if DNS is working and TCP port 443 is open. However, AnyConnect will try to use the DTLS protocol first which uses UDP port 443, if it fails than the client will fall back to use SSL for the transport of user data. The reason that AnyConnect prefers DTLS is that DTLS has less delay because of the connectionless nature of UDP and thus performance is better then with a SSL tunnel.

It is very easy to check if you are actually using DTLS in the AnyConnect client:

Cisco AnyConnect DTLSConclusion:
 If you filter the network traffic destinate to a Cisco IOS webvpn router or Cisco ASA firewall in the remote access / VPN block in your network don’t forget to open UDP port 443 also.. 🙂

Routing / switching, Study

CCNP: TSHOOT rocks!

301 Comments

Never in my life I left a testing facility with a smile that big like I did on 16 august last year. It was just minutes ago that I received my score on the Cisco CCNP Troubleshoot (TSHOOT) exam. It was the magic 1000 points, incredible high! Not that the a lower score should had matter, the reason behind the smile was the fun I had in answering the tickets. All the pain of doing the CCNP route exam twice earlier that year was suddenly disappeared.. 🙂

Cisco did a very good job in creating this exam. I have to troubleshoot a lot in my daily work, so for me troubleshooting is like a second nature. However I do have a few recommendation to make it more fun;
– Create a few tickets where more than one thing is broken, stuff like that happens in the real world!
– I missed the option “it is not the network, it is the server..” sounds like a joke but with the ability to look into the network traffic and spot a “tcp reset” from the server could prove a configuration failure on the server. (yes, that kind of prove is sometimes necessary…)
– Where was the option “It is a IOS bug, create a TAC case”? 😉