Finding switches with snoop

I’ve been playing with snoop to try and find out the details of network topology…

A useful tip, that I’m still working on is

snoop -x 1 -d qfe0

If you run this for a while, eventually it’ll come back with some packets that display the switch name, and port it’s connected to…

? -> (multicast)  ETHER Type=2000 (Unknown), size = 447 bytes

0: 0100 0ccc cccc 0012 8037 1b9d 01b1 aaaa    ………7……
16: 0300 000c 2000 02b4 f6c7 0001 000f 4c49    …. ………MY
32: 565f 3435 3036 3a30 3100 0501 2443 6973    S_4506:01…$Cis
48: 636f 2049 6e74 6572 6e65 7477 6f72 6b20    co Internetwork
64: 4f70 6572 6174 696e 6720 5379 7374 656d    Operating System
80: 2053 6f66 7477 6172 6520 0a49 4f53 2028     Software .IOS (
96: 746d 2920 4361 7461 6c79 7374 2034 3030    tm) Catalyst 400
112: 3020 4c33 2053 7769 7463 6820 536f 6674    0 L3 Switch Soft
128: 7761 7265 2028 6361 7434 3030 302d 4939    ware (cat4000-I9
144: 532d 4d29 2c20 5665 7273 696f 6e20 3132    S-M), Version 12
160: 2e32 2831 3829 4557 322c 2052 454c 4541    .2(18)EW2, RELEA
176: 5345 2053 4f46 5457 4152 4520 2866 6331    SE SOFTWARE (fc1
192: 290a 5465 6368 6e69 6361 6c20 5375 7070    ).Technical Supp
208: 6f72 743a 2068 7474 703a 2f2f 7777 772e    ort: http://www.
224: 6369 7363 6f2e 636f 6d2f 7465 6368 7375    cisco.com/techsu
240: 7070 6f72 740a 436f 7079 7269 6768 7420    pport.Copyright
256: 2863 2920 3139 3836 2d32 3030 3420 6279    (c) 1986-2004 by
272: 2063 6973 636f 2053 7973 7465 6d73 2c20     cisco Systems,
288: 496e 632e 0a43 6f6d 7069 6c65 6420 4d6f    Inc..Compiled Mo
304: 6e20 3031 2d4e 6f76 2d30 3420 3136 3a32    n 01-Nov-04 16:2
320: 3320 6279 206b 656c 6c79 7468 7700 0600    3 by kellythw…
336: 1263 6973 636f 2057 532d 4334 3530 3600    .cisco WS-C4506.
352: 0200 1100 0000 0101 01cc 0004 ac15 63a6    …………..c.
368: 0003 0014 4661 7374 4574 6865 726e 6574    ….FastEthernet
384: 332f 3436 0004 0008 0000 0028 0009 000e    2/23…….(….
400: 6c69 7669 6e67 7374 6f6e 000a 0006 0015    location……
416: 000b 0005 0100 1200 0500 0013 0005 0000    …………….

So, the switch is “MYS_4506″ and it’s connected to FastEthernet 2/23 (fa2\23 in switch parlance)

I’ve been trying to extend this into a script - which kinda nearly works.

# more find_switch.sh
#!/bin/ksh

# First, the theory -
# We can (eventually) get some output from snoop -x that
# tells us the switch, port and port type that the interface is
# connected to. This does mean running snoop for a while, so
# I’ll pipe this to a file and kill the snoop once we get a match.

do_snoop()
{
pid=$(ps -ef | grep snoop | grep -v grep | awk ‘{print $2}’)
if [[ $pid == "" ]]
then
snoop -x 1 -d $interface 2>&1 >> /tmp/snoop.$interface &
sleep 1
pid=$(ps -ef | grep snoop | grep -v grep | awk ‘{print $2}’)
echo “Process is $pid”
else
echo “Snoop already running, bailing out.”
fi
}

parse_output()
{
match=0
cat /tmp/snoop.$interface|while read line
do
if [[ $match -eq 2 ]]
then
if  [[ $line == "" ]]
then
match=3
else
print -n “$(echo $line | cut -d\   -f10-)”
fi
fi

if [[ $match -eq 1 ]]
then
if  [[ $line == "" ]]
then
match=2
fi
fi
if [[ $(echo $line|grep -c Unknown) -eq 1 ]]
then
echo “Match”
match=1
fi

if [[ $match -eq 3 ]]
then
echo “Done”
fi
done
}

# Get the interfaces

ifconfig -a |grep “:”|grep -v lo0|while read line
do
interface=$(echo $line | cut -d\: -f1)
echo “Checking interface $interface”
do_snoop
# Go grep the output until theres a match
found=0
while [[ $found -eq 0 ]]
do
found=$(grep -c Eth /tmp/snoop.$interface)
sleep 1
done
echo “Killing $pid”
kill $pid
# parse the output
parse_output
done

Doesn’t quite work - the parsing part is a bit on the dodgy side, and if it doesn’t work, it could fill the /tmp fileystem.

Seems to be effective for Cisco switches, not sure about anything else.

Adding HDS LUNs to existing system.

System needed some extra disk space, so had some LUNs created on the HDS and presented to them.

Problem was, couldn’t see them on the hosts -

Tried the devfsadm, which didn’t work, then moved on to some more obscure (at least to me) methods.

Checking the sd.conf, we were using JNI cards, with auto_mapping, which meant no need to update that. I tried reloading (update_drv -f sd), which had no effect.

I did find a reference to a re-probe specifically with JNI, which required a package to be installed into /opt/JNIC146x - and gave a command jnic146x_update_drv, which would appear to be appropriate.

The jnic146x_update_drv command is used to request that one or all jnic146x driver
instances perform LUN rediscovery.
This is useful when new LUNs are added to an existing RAID that are already LLZ mapped to
the HBA. Normally the RAID target would need to go offline and back online before LUN
discovery is repeated, but this command eliminates that need.
LUN rediscovery is only available on targets with disk devices; it is not supported on targets
with tape devices.
The ‘-t’ option selects the target id that is effected; when not specified, all targets are effected.
The driver instance(s) and target(s) need not be idle to use this option.

No luck with that either - just wouldn’t run without errors, so went for a reconfiguration reboot (boot -r).

The LUNs were then available to ‘format’, but wouldn’t go into Veritas. Turns out that both the paths were disbabled and had to be enabled manually. I had written the labels to the LUNs on another system, so don’t know whether this was the problem. Haven’t seen this behaviour before either.

bash-2.05# vxdmpadm getsubpaths dmpnodename=HDS9500V0_2
NAME         STATE      PATH-TYPE[M] CTLR-NAME  ENCLR-TYPE   ENCLR-NAME    ATTRS
================================================================================
c2t12345678d7s2 DISABLED   PRIMARY      c2         HDS9500V     HDS9500V0        -
c3t12345679d7s2 DISABLED   SECONDARY    c3         HDS9500V     HDS9500V0        -

bash-2.05# vxdmpadm enable path=c2t12345678d7s2
bash-2.05# vxdmpadm getsubpaths dmpnodename=HDS9500V0_2
NAME         STATE      PATH-TYPE[M] CTLR-NAME  ENCLR-TYPE   ENCLR-NAME    ATTRS
================================================================================
c2t12345678d7s2 ENABLED    PRIMARY      c2         HDS9500V     HDS9500V0        -
c3t12345679d7s2 DISABLED   SECONDARY    c3         HDS9500V     HDS9500V0        -
bash-2.05# vxdmpadm enable path=c3t12345678d7s2
bash-2.05# vxdmpadm getsubpaths dmpnodename=HDS9500V0_2
NAME         STATE      PATH-TYPE[M] CTLR-NAME  ENCLR-TYPE   ENCLR-NAME    ATTRS
================================================================================
c2t12345678d7s2 ENABLED    PRIMARY      c2         HDS9500V     HDS9500V0        -
c3t12345679d7s2 ENABLED    SECONDARY    c3         HDS9500V     HDS9500V0        -
bash-2.05#
bash-2.05# vxdisk list
DEVICE       TYPE            DISK         GROUP        STATUS
HDS9500V0_0  auto:cdsdisk    HDS9500V0_1  mydg2      online
HDS9500V0_1  auto:none       -            -            online invalid
HDS9500V0_2  auto            -            -            error
HDS9500V0_3  auto:cdsdisk    -            -            online
HDS9500V0_4  auto            -            -            error
HDS9500V0_5  auto            -            -            error
HDS9500V0_6  auto            -            -            error
HDS9500V0_7  auto            -            -            error
c0t0d0s2     auto:none       -            -            online invalid
c0t1d0s2     auto:none       -            -            online invalid
bash-2.05# vxdctl enable
bash-2.05# vxdisk list
DEVICE       TYPE            DISK         GROUP        STATUS
HDS9500V0_0  auto:cdsdisk    HDS9500V0_1  mydg2      online
HDS9500V0_1  auto:none       -            -            online invalid
HDS9500V0_2  auto:none       -            -            online invalid
HDS9500V0_3  auto:cdsdisk    -            -            online
HDS9500V0_4  auto            -            -            error
HDS9500V0_5  auto            -            -            error
HDS9500V0_6  auto            -            -            error
HDS9500V0_7  auto            -            -            error
c0t0d0s2     auto:none       -            -            online invalid
c0t1d0s2     auto:none       -            -            online invalid

bash-2.05# /usr/lib/vxvm/bin/vxdisksetup -i HDS9500V0_2

Back to the normal process of adding to disk group, etc.

Linux Raid

Been messing about with Linux RAID - software one that is.

Seemed to lose a disk out of a RAID 5, 3 disk setup after reboot.

Since the disk was there, I tried this

“/sbin/mdadm /dev/md0 -a /dev/sdd1″ to re-add the RAID partition on the disk to the array.

Then, “watch -n .1 cat /proc/mdstat”, to follow the progress

Checking Versions

Discovered this little gem trying to find the version of in.named we were running…

/usr/ccs/bin/what /usr/sbin/in.named
/usr/sbin/in.named:
in.named BIND 8.1.2 Wed Jan  5 17:46:35 PST 2000 Generic 107018-02-5.7-September 1998
db_dump.c       4.33 (Berkeley) 3/3/91
db_load.c       4.38 (Berkeley) 3/2/91
db_lookup.c     4.18 (Berkeley) 3/21/91
db_save.c       4.16 (Berkeley) 3/21/91
db_update.c     4.28 (Berkeley) 3/21/91
db_glue.c       4.4 (Berkeley) 6/1/90
ns_forw.c       4.32 (Berkeley) 3/3/91
ns_init.c       4.38 (Berkeley) 3/21/91
ns_main.c       4.55 (Berkeley) 7/1/91
Copyright (c) 1986, 1989, 1990 The Regents of the University of California.
ns_maint.c      4.39 (Berkeley) 3/2/91
ns_req.c        4.47 (Berkeley) 7/1/91
ns_resp.c       4.65 (Berkeley) 3/3/91
ns_stats.c      4.10 (Berkeley) 6/27/90
SunOS 5.7 Generic 107018-02 January 2000

screen

ftp://ftp.sunfreeware.com/pub/freeware/sparc/10/screen-4.0.2-sol10-sparc-local.gz

http://www.gnu.org/software/screen/

If you’ve not come across it, it’s a wee tool that allows you to have many sessions in one session, i.e.
Fire up putty to  hop off point, eval `ssh-agent` <other environment stuff you need> screen

After that :

CTRL-A-c - Create a new session
CTRL-A-n - Next session
p = previous
? = Help

you get the idea.
Handy if you have loads of sessions, as seems par for the course for me here.
Also handy for firewalls - if timeout happens, just log back in and do :
screen -d
screen -r

More Storage Stuff

[:root]# luxadm display /dev/rdsk/c4t6006048000028775197156434D303030d0s2
DEVICE PROPERTIES for disk: /dev/rdsk/c4t6006048000028775197156434D303030d0s2
Vendor:               EMC
Product ID:           SYMMETRIX
Revision:             5671
Serial Num:           751971000000
Unformatted capacity: 45.000 MBytes
Read Cache:           Enabled
Minimum prefetch:   0×0
Maximum prefetch:   0xffff
Device Type:          Disk device
Path(s):

/dev/rdsk/c4t6006048000028775197156434D303030d0s2
/devices/scsi_vhci/ssd@g6006048000028775197156434d303030:c,raw
Controller           /devices/pci@1f,700000/pci@0/SUNW,emlxs@2/fp@0,0
Device Address              5006048449afc8c6,0
Host controller port WWN    10000000c9727f4d
Class                       primary
State                       ONLINE
Controller           /devices/pci@1f,700000/pci@0,2/SUNW,emlxs@1/fp@0,0
Device Address              5006048449afc8e9,0
Host controller port WWN    10000000c9728210
Class                       primary
State                       ONLINE

[:root]# luxadm probe
No Network Array enclosures found in /dev/es

Found Fibre Channel device(s):
Node WWN:5006048449afc8c6  Device Type:Disk device
Logical Path:/dev/rdsk/c4t6006048000028775197147544B313444d0s2
Node WWN:5006048449afc8c6  Device Type:Disk device
Logical Path:/dev/rdsk/c4t6006048000028775197156434D303030d0s2

[:root]# luxadm -e dump_map /dev/rdsk/c4t6006048000028775197147544B313444d0s2
Pos  Port_ID Hard_Addr Port WWN         Node WWN         Type
0    91b00   0         5006048449afc8e9 5006048449afc8e9 0×0  (Disk device)
1    90400   0         10000000c9728210 20000000c9728210 0×1f (Unknown Type,Host Bus Adapter)

Multi-path disable/enable is in :

/kernel/drv/fp.conf -

mpxio-disable=”no”;

Get WWN on Solaris

# luxadm -e port

Found path to 2 HBA ports

/devices/pci@1d,700000/SUNW,jfca@1/fp@0,0:devctl                   CONNECTED
/devices/pci@1d,700000/SUNW,jfca@1,1/fp@0,0:devctl                 CONNECTED

# luxadm -e dump_map /devices/pci@1d,700000/SUNW,jfca@1/fp@0,0:devctl
Pos  Port_ID Hard_Addr Port WWN         Node WWN         Type
0    a0000   0         50060e8000437210 50061e8000437210 0×0  (Disk device)
1    a0600   0         2000000173012f67 1000000173012f67 0×1f (Unknown Type)
2    a0500   0         2000000173013515 1000000173013515 0×1f (Unknown Type,Host Bus Adapter)
# luxadm -e dump_map /devices/pci@1d,700000/SUNW,jfca@1,1/fp@0,0:devctl
Pos  Port_ID Hard_Addr Port WWN         Node WWN         Type
0    a0000   0         50060e8000437212 50061e8000437212 0×0  (Disk device)
1    a0600   0         2000000173812f67 1000000173812f67 0×1f (Unknown Type)
2    a0500   0         2000000173813515 1000000173813515 0×1f (Unknown Type,Host Bus Adapter)

Connected disks / Luns

List all the disks Veritas knows about, including reporting deported diskgroups

vxdisk -o alldgs list

Check device names using -

vxdisk list <device>

Check OS connectivity using

cfgadm -al

Limit coredump size

ulimit -c <size in 512 byte blocks>

ulimit -c 204800 - limit to 100MB

vxstat

vxstat -g <diskgroup> -i 10

Output the stats every 10 seconds - the number of blocks can be useful in estimating how long a job will take.

e.g. 20000 blocks a second would be

“20000*8192/1024/1024=156.25MB/s”