ISILON

2017년 8월 30일 수요일

hangdump

isi_for_array -n 1 cp /var/crash/isi_hangdump_2017_08_29_22_13_30.log.gz /ifs/data/Isilon_Support/dump/node01_isi_hangdump_2017_08_29_22_13_30.log.gz
isi_for_array -n 2 cp /var/crash/isi_hangdump_2017_08_29_22_13_30.log.gz /ifs/data/Isilon_Support/dump/node02_isi_hangdump_2017_08_29_22_13_30.log.gz
isi_for_array -n 3 cp /var/crash/isi_hangdump_2017_08_29_22_13_30.log.gz /ifs/data/Isilon_Support/dump/node03_isi_hangdump_2017_08_29_22_13_30.log.gz
isi_for_array -n 4 cp /var/crash/isi_hangdump_2017_08_29_22_13_30.log.gz /ifs/data/Isilon_Support/dump/node04_isi_hangdump_2017_08_29_22_13_30.log.gz
isi_for_array -n 5 cp /var/crash/isi_hangdump_2017_08_29_22_13_30.log.gz /ifs/data/Isilon_Support/dump/node05_isi_hangdump_2017_08_29_22_13_30.log.gz
isi_for_array -n 6 cp /var/crash/isi_hangdump_2017_08_29_22_13_30.log.gz /ifs/data/Isilon_Support/dump/node06_isi_hangdump_2017_08_29_22_13_30.log.gz
isi_for_array -n 7 cp /var/crash/isi_hangdump_2017_08_29_22_13_30.log.gz /ifs/data/Isilon_Support/dump/node07_isi_hangdump_2017_08_29_22_13_30.log.gz
isi_for_array -n 8 cp /var/crash/isi_hangdump_2017_08_29_22_13_30.log.gz /ifs/data/Isilon_Support/dump/node08_isi_hangdump_2017_08_29_22_13_30.log.gz
isi_for_array -n 9 cp /var/crash/isi_hangdump_2017_08_29_22_13_30.log.gz /ifs/data/Isilon_Support/dump/node09_isi_hangdump_2017_08_29_22_13_30.log.gz
isi_for_array -n 10 cp /var/crash/isi_hangdump_2017_08_29_22_13_30.log.gz /ifs/data/Isilon_Support/dump/node10_isi_hangdump_2017_08_29_22_13_30.log.gz
isi_for_array -n 11 cp /var/crash/isi_hangdump_2017_08_29_22_13_30.log.gz /ifs/data/Isilon_Support/dump/node11_isi_hangdump_2017_08_29_22_13_30.log.gz

2017년 8월 9일 수요일

[ Isilon ] PERFORMANCE METRICS HINTS

# Start
cd /ifs/data/Isilon_Support
screen -L

# health of the Clusters and nodes
Capacity:
isi status --quiet
isi storagepool list
isi status -p

CPU:
isi_for_array -s -X 'top -n –S 10'

Memory:
isi statistics query current --nodes all --degraded --stats node.memory.used
isi statistics query current --nodes all --degraded --stats node.memory.free
isi_for_array -s -X 'sysctl kern.malloc_pigs' | less

Load average:
isi statistics query current --nodes all --degraded --stats node.cpu.count
isi_for_array -s 'uptime'

disk operations across Balance across nodes
isi statistics query current --nodes=all --stats=node.disk.xfers.rate.sum

# SmartConnect and load balance
isi network interfaces list
isi status

isi_for_array -sX 'isi devices|grep -vi healthy'

# SmartConnect Suspend check
isi networks list pools --verbose
isi networks modify pool --sc-suspend-node=5 --name=172_28_30_x:Service_Net
isi networks modify pool --sc-resume-node=5 --name=172_28_30_x:Service_Net

# File Server Protocol Operational Rates
isi statistics protocol –-nodes all –-top –-orderby=timeavg

# Disk Hardware Latency
isi statistics drive –-nodes all –-top –-long –-orderby timeavg

# Node CPU Utilization
isi statistics system –-nodes --top

# isi statistics client
isi statistics client --orderby=ops -nall --top --numeric output
isi statistics client --orderby=ops -nall --top --numeric
isi statistics client --long -i 5

# isi statistics system
isi statistics system --timestamp --nodes -i 5
isi statistics system --nodes --interval=1 --top

# isi statistics query
isi statistics query --stats=node.disk.xfers.rate.5 -i 10

# SMB open files list
isi smb session list
isi_for_array -X 'isi smb openfiles list -v --format=csv --no-header --no-footer'

# isi statistics pstat (<--SMB)
isi statistics pstat --protocol smb2

# NFS locks list
isi_for_array -X isi nfs nlm locks list -v --format=csv --no-header

# Users by protocol
isi statistics client --protocols external --no-footer | awk '{print $1 " "$6 " " $8}' | head -22

# Busiest files and paths
isi statistics heat --nodes all --totalby path | awk '{print $1 " " $5}' | sort -n -r | head -20
isi statistics heat --nodes 1 | awk '{print $1 " " $4 " " $5}' | sort -n -r | head -20

# Blocked, contended, and deadlocked events
isi statistics heat --totalby event,lin,path --limit 50

# Disk activity
Time in Queue: isi statistics drive --nodes=all --degraded --no-header --no-footer | awk ' /SATA/{sum+=$8; max=0; min=1000} {if ($8>max) max=$8; if ($8<min) min=$8} END {print "Min =",min; print "Max = ",max; print "Average = ",sum/NR}'
Number in Queue: isi statistics drive --nodes=all --degraded --no-header --no-footer | awk ' /SATA/{sum+=$9; max=0; min=1000} {if ($9>max) max=$9; if ($9<min) min=$9} END {print "Min =",min; print "Max = ",max; print "Average = ",sum/NR}'

# Network factors

Latency and hops:
traceroute –q 5 <IP > Issue from client to a node
traceroute –q 5 <IP > Issue from node to another node

Latency and packet loss:
ping -i .1 -c 250 <IP > Issue from client to a node
ping -i .1 -c 250 <IP > Issue from node to another node

Bandwidth:
iperf -s On target node
iperf -c <IP of Target Host> On source host

Hostcache.list:
isi_for_array -X sysctl net.inet.tcp.hostcache.list

Cache response times and effectiveness:
isi_cbind show dns
isi_cbind show cluster

# Protocol operations
Protocol operations most used:
isi statistics protocol list --sort Ops --degraded

Protocol operations taking the most time
isi statistics protocol list --sort TimeAvg --degraded

Clients most-to-least demanding:
isi statistics client list --sort Ops --degraded

Connection distribution:
isi statistics query current --nodes all --degraded --stats node.clientstats.connected.nfs,node.clientstats.active.nfs
isi statistics query current --nodes all --degraded --stats node.clientstats.connected.smb,node.clientstats.active.smb2

Slow authentication:
isi statistics protocol list --protocols lsass_out --degraded

Protocol latency:
isi statistics protocol list --totalby class

# L3 Cache
isi_cache_stats
isi_cache_stats –c (*reset command)
isi_cache_stats –v
isi_for_array –s isi_cache_stats –v | grep –A3 l3

# SMB Config
isi_for_array sysctl isi.stats.client.smb2.max_clients

# Netstat
SMB: isi_for_array -s "netstat -an |grep 445 | grep -i ESTABLISHED |wc -l"

# tcpdump
tcpdump -i vlan0 -s 0 -w /ifs/data/nasssip30.pcap host 172.28.30.15

# IP move
isi networks support sc_put_ip --sc-iface=5:10gige-agg-1 --sc-ip=172.28.31.16

2017년 5월 31일 수요일

[ISILON] 운영자용 Command

# 상태 모니터링

# isi status

# isi devices

# isi auth status

# isi event list

# isi event show [이벤트 Number]

* isi_for_array à Cluster wide

#SMB 생성 삭제

1) SMB Share 생성

# mkdir /ifs/data/smb-1;

# isi smb shares create --zone=system --name=smb-1 --path=/ifs/data/smb-1 --create-path;

# isi smb shares permission create --share=smb-1 --permission-type=allow --permission= ['full', 'change', 'read'] --group='LAB\domain users';

# isi smb shares permission delete --share=smb-1 --wellknown=everyone –force;

# isi smb shares list

# ls –led /ifs/data/smb-1

* ls의 -w option à 한글 파일

2) SMB Share 삭제

# isi smb share list

# isi smb share view smb-1

#네트워크

# isi networks list ifaces

#로그 수집

# isi_gather_info

2016년 11월 8일 화요일

[ ISILON ] SSH Ciphers: 3des-cbc 설정 방법

<어떨때 쓰나!>

1. 고객사에 Isilon에 SSH command 기반으로 Provisioning 하는 시스템이 있을 경우

2. Isilon에 SSH 접속이 느린 경우

<확인사항>

그때 보니 SSH Client 에서 3des-cbc 암호화를 사용하였던 상황이고,

SSH 설정의 Ciphers에 해당 항목을 추가하면 되더라.

* Isilon은 Cluster로 구성되므로 모든 Node에 SSH 설정 변경해야 한다.

cp -v /etc/mcp/templates/sshd_config /ifs/.ifsvar/sshd_config.old

vi /etc/mcp/templates/sshd_config à 3des-cbc 삽입

cp -v /etc/mcp/templates/sshd_config /ifs/.ifsvar/sshd_config.new

isi_for_array "cp -vf /ifs/.ifsvar/sshd_config.new /etc/mcp/templates/sshd_config"

isi_for_array -s "md5 /etc/mcp/templates/sshd_config"

isi_for_array "killall -HUP sshd"

* 주의사항: ssh_config 이 아니다. sshd_config 을 수정해야 한다.

+++++++++++++++++++++<sshd_config 상세>+++++++++++++++++++++++

# X: ----------------

# X: This file is automatically generated and should not be

# X: edited directly. If you must make changes to the

# X: contents of this file it should be done via the

# X: template file located at /etc/mcp/templates/sshd_config

# X: ----------------

Port 22

HostKey /etc/ssh/ssh_host_key

HostDsaKey /etc/ssh/ssh_host_dsa_key

ServerKeyBits 768

LoginGraceTime 120

KeyRegenerationInterval 3600

PermitRootLogin yes

MaxStartups 10:30:60

IgnoreRhosts yes

StrictModes yes

X11Forwarding yes

X11DisplayOffset 10

PrintMotd yes

KeepAlive yes

SyslogFacility AUTH

LogLevel INFO

#RhostsAuthentication no

RhostsRSAAuthentication no

RSAAuthentication yes

PasswordAuthentication yes

PermitEmptyPasswords no

Subsystem sftp /usr/libexec/sftp-server

Protocol 2

Ciphers aes128-ctr,aes192-ctr,aes256-ctr,3des-cbc

MACs hmac-sha1

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++

2015년 12월 22일 화요일

[ ISILON ] local admin 계정을 sudo 명령어 사용 가능하게 하는법-2 isi_visudo

#sudo 사용을 위해서는 각 Node의 /usr/local/etc/sudoers 파일을 수정해야 한다.

#명령어 isi_visudo 를 이용하면 클러스터의 각 노드에 파일을 자동 수정해 준다.

1. # isi_visudo

2. 항목 추가 'op ALL=(ALL) NOPASSWD:/usr/bin/isi'

(이 경우는 op라는 계정을 만들고 해당 계정에 권한을 부여하는 경우임)

Window resize interrupted text input mode

tes/sudoers. Do not

## edit /etc/mcp/templates/sudoers.

## To add additional command permissions, enter the app

ropriate configuration

## lines below. To remove a command provided by default

, enter a negation line

## below.

## Example:

## To prevent admin from running SyncIQ, uncomment the

line below:

## admin ALL=(ALL) !/usr/bin/isi sync*

op ALL=(ALL) NOPASSWD:/usr/bin/isi

3. 끝

참조링크: https://community.emc.com/thread/219358?start=0&tstart=0

[ ISILON ] local admin 계정을 sudo 명령어 사용 가능하게 하는법

'sudoers' 파일에 'admin ALL=(ALL) ALL' 추가

1. #vi /user/local/etc/sudoers --> 위 항목 추가
2. 각 노드마다 진행해야 한다.

---------------------------------------------------------------
Isilon-1# cd /usr/local/etc/
Isilon-1# vi sudoers
## This file is auto-generated by /etc/mcp/scripts/sudoers.py.

User_Alias ADMINS = compadmin
User_Alias REMOTESUPPORT = remotesupport

Cmnd_Alias PROCESSES = /bin/kill, /usr/bin/renice, \
/usr/bin/pkill, /usr/bin/top, /usr/bin/killall
Cmnd_Alias SYSADMIN = /bin/date, /sbin/sysctl, /sbin/shutdown, \
/bin/ps, /usr/sbin/ntpdate, /sbin/ifconfig, \
/usr/sbin/nfsstat, /usr/sbin/pciconf, \
/usr/sbin/tcpdump
Cmnd_Alias ISI = /usr/bin/isi
Cmnd_Alias ISI_ADMIN = /usr/bin/isi_classic, /usr/bin/isi_for_array, \
/usr/bin/isi_gconfig, /usr/bin/isi_job_d, \
/usr/bin/isi_vol_copy
Cmnd_Alias ISI_SUPPORT = /usr/bin/gcore, /usr/bin/isi_bootdisk_status, \
/usr/bin/isi_cache_stats, \
/usr/bin/isi_client_stats, /usr/bin/isi_cpr, \
/usr/bin/isi_dmi_info, /usr/bin/isi_dmilog, \
/usr/bin/isi_dongle_sync, \
/usr/bin/isi_drivenum, \
/usr/bin/isi_dsp_install, \
/usr/bin/isi_dumpjournal, \
/usr/bin/isi_eth_mixer_d, \
/usr/bin/isi_evaluate_provision_drive, \
/usr/bin/isi_flush, \
/usr/bin/isi_gather_auth_info, \
/usr/bin/isi_gather_cluster_info, \
/usr/bin/isi_gather_info, \
/usr/bin/isi_get_itrace, \
/usr/bin/isi_get_profile, \
/usr/bin/isi_hangdump, /usr/bin/isi_hw_status, \
/usr/bin/isi_ib_bug_info, /usr/bin/isi_ilog, \
/usr/bin/isi_imdd_status, \
/usr/bin/isi_inventory_tool, \
/usr/bin/isi_km_diag, \
/usr/bin/isi_linmap_mod, \
/usr/bin/isi_make_abr, /usr/bin/isi_nodes, \
/usr/bin/isi_ntp_config, \
/usr/bin/isi_ovt_check, /usr/bin/isi_patch_d, \
/usr/bin/isi_promptsupport, \
/usr/bin/isi_radish, \
/usr/bin/isi_repstate_mod, \
/usr/bin/isi_restill, \
/usr/bin/isi_save_itrace, \
/usr/bin/isi_send_abr, \
/usr/bin/isi_smbios, \
/usr/bin/isi_stats_tool, \
/usr/bin/isi_transform_tool, \
/usr/bin/isi_ufp, \
/usr/bin/isi_update_cto, \
/usr/bin/isi_vol_copy, \
/usr/bin/isi_vol_copy_vnx, \
/usr/bin/nfsstat, \
/usr/sbin/isi_bootdisk_finish, \
/usr/sbin/isi_bootdisk_provider_dev, \
/usr/sbin/isi_bootdisk_unlock, \
/usr/sbin/isi_checkjournal, \
/usr/sbin/isi_disk_firmware_reboot, \
/usr/sbin/isi_clean_idmap, \
/usr/sbin/isi_fsa_tool, \
/usr/sbin/isi_kill_busy, \
/usr/sbin/isi_lid_d, \
/usr/sbin/isi_logstore, \
/usr/sbin/isi_mcp, \
/usr/sbin/isi_netlogger, \
/usr/sbin/isi_rbm_ping, \
/usr/sbin/isi_savecore, \
/usr/sbin/isi_umount_ifs

Cmnd_Alias ISI_HWTOOLS = /usr/bin/isi_hwtools/isi_cto_convert, \
/usr/bin/isi_hwtools/isi_cto_update, \
/usr/bin/isi_hwtools/isi_cto_upgrade, \
/usr/bin/isi_hwtools/isi_fcb_vpd_tool, \
/usr/bin/isi_hwtools/isi_fputil, \
/usr/bin/isi_hwtools/isi_flexnet_info, \
/usr/bin/isi_hwtools/isi_hw_check, \
/usr/bin/isi_hwtools/isi_ib_fw, \
/usr/bin/isi_hwtools/isi_ib_info, \
/usr/bin/isi_hwtools/isi_ipmicmc, \
/usr/bin/isi_hwtools/isi_lsiexputil, \
/usr/bin/isi_hwtools/isi_mpsutil, \
/usr/bin/isi_hwtools/isi_mps_fw_status, \
/usr/bin/isi_hwtools/isi_mpt_fw_status, \
/usr/bin/isi_hwtools/isi_ovt_check, \
/usr/bin/isi_hwtools/isi_rnvutil, \
/usr/bin/isi_hwtools/isi_sasphymon, \
/usr/bin/isi_hwtools/isi_sed, \
/usr/bin/isi_hwtools/isi_update_serialno, \
/usr/bin/isi_hwtools/isi_vitutil, \
/sbin/nvmecontrol

Cmnd_Alias REMOTE_SUPPORT_CMDS = /usr/bin/remotesupport/GetATAControlAndGMirror, \
/usr/bin/remotesupport/GetClusterEvents, \
/usr/bin/remotesupport/GetClusterStatus, \
/usr/bin/remotesupport/GetContactInfo, \
/usr/bin/remotesupport/GetContents-isi_client_stats, \
/usr/bin/remotesupport/GetNfsClients, \
/usr/bin/remotesupport/GetReadonlyStatus, \
/usr/bin/remotesupport/IsiGatherInfo, \
/usr/bin/remotesupport/IsiGatherInfo-Incremental, \
/usr/bin/remotesupport/IsiGatherInfo-Incremental-SingleNode, \
/usr/bin/remotesupport/IsiGatherInfo-SingleNode, \
/usr/bin/remotesupport/clean_watch_folders, \
/usr/bin/remotesupport/generate_dashboard_file_daily, \
/usr/bin/remotesupport/generate_dashboard_file_interval, \
/usr/bin/remotesupport/get_contents_var_crash, \
/usr/bin/remotesupport/get_data_abr, \
/usr/bin/remotesupport/get_data_application, \
/usr/bin/remotesupport/get_data_cluster, \
/usr/bin/remotesupport/get_data_domain, \
/usr/bin/remotesupport/get_data_fs, \
/usr/bin/remotesupport/get_data_ib, \
/usr/bin/remotesupport/get_data_logs, \
/usr/bin/remotesupport/get_data_messages, \
/usr/bin/remotesupport/get_data_network, \
/usr/bin/remotesupport/get_data_node, \
/usr/bin/remotesupport/get_data_protocol, \
/usr/bin/remotesupport/get_data_usage, \
/usr/bin/remotesupport/get_job_status, \
/usr/bin/remotesupport/restart_node, \
/usr/bin/remotesupport/tail_messages, \
/usr/bin/remotesupport/upload_messages

Cmnd_Alias ISI_PRIV_ANTIVIRUS = /usr/bin/isi avscan*
Cmnd_Alias ISI_PRIV_CLUSTER = /usr/bin/isi email*, \
/usr/bin/isi version*
Cmnd_Alias ISI_PRIV_EVENT = /usr/bin/isi alert*, \
/usr/bin/isi events*
Cmnd_Alias ISI_PRIV_DEVICES = /usr/bin/isi devices*, \
/usr/bin/isi servicelight*
Cmnd_Alias ISI_PRIV_FTP = /usr/bin/isi ftp*
Cmnd_Alias ISI_PRIV_ISCSI = /usr/bin/isi iscsi*, \
/usr/bin/isi lun*, \
/usr/bin/isi target*
Cmnd_Alias ISI_PRIV_LICENSE = /usr/bin/isi license*
Cmnd_Alias ISI_PRIV_NDMP = /usr/bin/isi ndmp*, \
/usr/bin/isi tape*
Cmnd_Alias ISI_PRIV_NETWORK = /usr/bin/isi networks*
Cmnd_Alias ISI_PRIV_SMARTPOOLS = /usr/bin/isi filepool*
Cmnd_Alias ISI_PRIV_SNMP = /usr/bin/isi snmp*
Cmnd_Alias ISI_PRIV_STATISTICS = /usr/bin/isi batterystatus*, \
/usr/bin/isi perfstat*, \
/usr/bin/isi stat*, \
/usr/bin/isi statistics*, \
/usr/bin/isi status*

Defaults syslog=auth

root ALL=(ALL) ALL
admin ALL=(ALL) ALL

ADMINS ALL=(ALL) NOPASSWD: PROCESSES, SYSADMIN, ISI, ISI_ADMIN, \
ISI_SUPPORT, ISI_HWTOOLS

REMOTESUPPORT ALL=(ALL) NOPASSWD: REMOTE_SUPPORT_CMDS

## begin auto-generated RBAC entries
User_Alias SECURITYADMIN = #10
User_Alias SYSTEMADMIN = #10
sudoers: 171 lines, 7258 characters.
Isilon-1#

---------------------------------------------------------------

[ ISILON ] Isilon의 Local admin 유저를 사용하여 SMB Share를 구성한 경우, GUI를 통한 Network Drive는 연결되는데, CLI를 통한 Network Drive는 에러 발생

# Isilon의 Local admin 유저를 사용하여 SMB Share를 구성한 경우,
GUI를 통한 Network Drive는 연결되는데, CLI를 통한 Network Drive는 에러 발생

증상:
1. GUI를 통한 접속은 되는데,
2. CLI를 통한 접속은 되지 않음
cli에서 사용한 명령어는 net use k: \\192.168.190.11\test$ /user:admin

3. /var/log/lsassd.log 로그 에러 상황
[lsass] Failed to authenticate user (name = 'admin') -> error = 40022, symbol = LW_ERROR_PASSWORD_MISMATCH, client pid = 4294967295

해결 방안: /user:UNIX_USERS\admin 으로 로그인 하면됨

개선된 명령줄
net use k: \\192.168.190.11\test$ /user:UNIX_USERS\admin