Tune Interrupt and Process CPU affinity

When an interrupt signal arrives, the CPU must stop what it's currently doing and switch to a new activity. Interrupts consume CPU time, too many interrupts could cause high system CPU usage as observed from mpstat. In Symmetric Multiprocessing model (SMP ), balancing IRQ is NOT usually needed, because  the  /proc/irq/default_smp_affinity or irqbalance daemon can distribute IRQ signals  among  CPUs automatically.
If you find a particular IRQ is unevenly distributed from the output of “cat /proc/interrupts “, then you can try to Set smp_affinity manually.
Tunning IRQ affinity with smp_affinity flag
It is pointless to set smp_affinity manually with  irqbalance running.

##Check current smp_affinity setting
$ grep -H * /proc/irq/*/smp_affinity
default_smp_affinity:00000000,00000000,ffffffff,ffffffff
/proc/irq/0/smp_affinity:00000000,00000000,00000000,00000020
..
/proc/irq/24/smp_affinity:00000000,00000000,00000000,00000010
##The value of smp_affinity is hex format, which is 128 bit in binary in my host. (It might be  32bit or 256bit depending kernel version  ).  It is called bit mask, 128 represent the position of 128 CPUs, if the value of the position is “1”, then the CPU is allowed to accept the IRQ.
##e.g to allow CPU 0,1,3, the binary presentation is “1011”, which is “b” in hex.
$echo -e "obase=16 \n ibase=2 \n 1011 \n" | bc
B
##Changing IRQ 24 from CPU 4 to CPU 0,1,3
$cat /proc/irq/24/smp_affinity 
00000000,00000000,00000000,00000010
$ echo 0b >/proc/irq/24/smp_affinity
$ cat /proc/irq/24/smp_affinity 
00000000,00000000,00000000,0000000b

Tunning process affinity with taskset 
Pin a process to a particular CPU  to improve cache hit
$taskset -p $$

pid 21698's current affinity mask: ff
$taskset -cp 0,1,3  $$
pid 21698's current affinity list: 0-7
pid 21698's new affinity list: 0,1,3
$taskset -p $$
pid 21698's current affinity mask: b
##you can check which CPU a process is running on with "psr" parameter
$ ps axo  psr,pid,cmd | grep $$
  0 21698 -bash

Isolate CPUs to run particular process only.
Kernel parameter “isolcpus” can isolate particular CPUs from doing other tasks. Together with taskset, you can have particular CPUs to run designated tasks only.
E.g put “isolcpus=2,3 “ in grub.conf will isolate CPU 2 and 3.