Showing posts with label AIX. Show all posts
Showing posts with label AIX. Show all posts

Monday, June 20, 2011

Copying files between servers with compression

You may have to copy very big files or file systems between servers sometimes, and the more you optimize this copy the faster you will transfer the files with less computer resources usage.

You might think that scp alone would do a good job, but without compression you may be wasting bandwith and time (talking about copying file systems of hundreds of gigabytes between servers far away one of the other). But if you have a lot of disc space available, why not to compress the file system first and then transfer it? Well, because compressing big file systems takes a lot of time and your disc I/O is severely degraded (the server reads a file and tries to write it compressed at the same time); you might be render your server practically unusable while compressing big files, at least in AIX.

But you can make a backup, compress it and send it to the other server, uncompressing it and writing just the files you want to transfer (not writing compressed files on disk at any time), compressing it on the fly with pipes. To this end, you may write this little script and put it in an appropriate path in the source server:

tar -cf - $1 | compress -c

Then, if you named this script mycompressedtar for example, you can transfer files or directories to your target server this way:

otheruser@targetsrv> ssh -l myuser sourcesrv /path/mycompressedtar /some/dir | uncompress -c | tar -xf -

If you issue this command at your target server, you will copy /some/dir from source server to the path where you stand at execution time; of course you have to have access to /some/dir with the myuser account.

But some times you have few but very big files (like Oracle datafiles), and you cannot transfer them with tar without getting an error in AIX; in that case you can just transfer file by file with this little script:

cat $1 | compress -c

Then, if you named this script mycompress for example, you can copy one file to your target server this way:

otheruser@targetsrv> ssh -l myuser sourcesrv /path/mycompress /some/file | uncompress -c > somefile

If you issue this command at your target server, you will copy file from source server to the path where you stand at execution time and name it somefile; you can use the same name as the source file or a different one.

But we're talking about different servers, and the files are so valuable we cannot afford to change a single byte. How can we be sure the files are exactly the same? In AIX, you can use the cksum command:

myuser@sourcesrv> cksum myfile.txt
1656934735 2229 myfile.txt

===================================

otheruser@targetsrv> cksum myfile.txt
1656934735 2229 myfile.txt

This way you'll be sure both files are equal.

More information:

cksum Command

Tuesday, May 3, 2011

Getting AIX basic system info

Sometimes you might need to know basic information about your AIX server like time up (uptime), operating system version (oslevel), firmware version (lsmcode), operating system parameters (lsattr), hardware (lscfg) or even storage errors (errpt), therefore I put this transcript as a reference. Some parameters are easy to understand and others are just for experienced sysadmins, but this information might be helpful even for an Oracle DBA working with an AIX server.

myserver> uptime
04:34PM up 79 days, 5:52, 1 user, load average: 1.20, 1.13, 1.11

myserver> oslevel -r
5300-11

myserver> lsmcode -c
The current permanent system firmware image is SF240_320
The current temporary system firmware image is SF240_332
The system is currently booted from the temporary firmware image.

myserver> lsattr -E -l sys0
SW_dist_intr false Enable SW distribution of interrupts True
autorestart true Automatically REBOOT system after a crash True
boottype disk N/A False
capacity_inc 1.00 Processor capacity increment False
capped true Partition is capped False
conslogin enable System Console Login False
cpuguard enable CPU Guard True
dedicated true Partition is dedicated False
ent_capacity 2.00 Entitled processor capacity False
frequency 1056000000 System Bus Frequency False
fullcore false Enable full CORE dump True
fwversion IBM,SF240_320 Firmware version and revision levels False
id_to_partition 0X80000819C3F00009 Partition ID False
id_to_system 0X80000819C3F00000 System ID False
iostat false Continuously maintain DISK I/O history True
keylock normal State of system keylock at boot time False
log_pg_dealloc true Log predictive memory page deallocation events True
max_capacity 3.00 Maximum potential processor capacity False
max_logname 9 Maximum login name length at boot time True
maxbuf 20 Maximum number of pages in block I/O BUFFER CACHE True
maxmbuf 0 Maximum Kbytes of real memory allowed for MBUFS True
maxpout 0 HIGH water mark for pending write I/Os per file True
maxuproc 1024 Maximum number of PROCESSES allowed per user True
min_capacity 2.00 Minimum potential processor capacity False
minpout 0 LOW water mark for pending write I/Os per file True
modelname IBM,9119-590 Machine name False
ncargs 512 ARG/ENV list size in 4K byte blocks True
nfs4_acl_compat secure NFS4 ACL Compatibility Mode True
pre430core false Use pre-430 style CORE dump True
pre520tune disable Pre-520 tuning compatibility mode True
realmem 18874368 Amount of usable physical memory in Kbytes False
rtasversion 1 Open Firmware RTAS version False
sed_config select Stack Execution Disable (SED) Mode True
systemid IBM,02025224F Hardware system identifier False
variable_weight 0 Variable processor capacity weight False

myserver> lscfg
INSTALLED RESOURCE LIST

The following resources are installed on the machine.
+/- = Added or deleted from Resource List.
* = Diagnostic support not available.

Model Architecture: chrp
Model Implementation: Multiple Processor, PCI bus

+ sys0 System Object
+ sysplanar0 System Planar
* pci45 U5791.001.9920F70-P1 PCI Bus
* pci46 U5791.001.9920F70-P1 PCI Bus
+ fcs2 U5791.001.9920F70-P1-C08-T1 FC Adapter
* fcnet2 U5791.001.9920F70-P1-C08-T1 Fibre Channel Network Protocol Device
+ fscsi2 U5791.001.9920F70-P1-C08-T1 FC SCSI I/O Controller Protocol Device
* hdisk5 U5791.001.9920F70-P1-C08-T1-W5006048ACC36E4E3-LF0000000000000 EMC Symmetrix FCP Raid1
* hdiskpower0 U5791.001.9920F70-P1-C08-T1-L40 PowerPath Device
* hdiskpower1 U5791.001.9920F70-P1-C08-T1-L41 PowerPath Device
* hdiskpower2 U5791.001.9920F70-P1-C08-T1-L42 PowerPath Device
* hdiskpower3 U5791.001.9920F70-P1-C08-T1-L43 PowerPath Device
* hdiskpower4 U5791.001.9920F70-P1-C08-T1-L44 PowerPath Device
* hdiskpower5 U5791.001.9920F70-P1-C08-T1-L45 PowerPath Device
* hdiskpower6 U5791.001.9920F70-P1-C08-T1-L46 PowerPath Device
* hdiskpower7 U5791.001.9920F70-P1-C08-T1-L47 PowerPath Device
* hdiskpower8 U5791.001.9920F70-P1-C08-T1-L48 PowerPath Device
* hdiskpower9 U5791.001.9920F70-P1-C08-T1-L49 PowerPath Device
* hdiskpower10 U5791.001.9920F70-P1-C08-T1-L50 PowerPath Device
* hdiskpower11 U5791.001.9920F70-P1-C08-T1-L51 PowerPath Device
* hdiskpower12 U5791.001.9920F70-P1-C08-T1-L52 PowerPath Device
* hdiskpower13 U5791.001.9920F70-P1-C08-T1-L53 PowerPath Device
* hdiskpower14 U5791.001.9920F70-P1-C08-T1-L54 PowerPath Device
* hdiskpower15 U5791.001.9920F70-P1-C08-T1-L55 PowerPath Device
* hdiskpower16 U5791.001.9920F70-P1-C08-T1-L56 PowerPath Device
* hdiskpower17 U5791.001.9920F70-P1-C08-T1-L57 PowerPath Device
* hdiskpower18 U5791.001.9920F70-P1-C08-T1-L58 PowerPath Device
* hdiskpower19 U5791.001.9920F70-P1-C08-T1-L60 PowerPath Device
* hdiskpower20 U5791.001.9920F70-P1-C08-T1-L61 PowerPath Device
* hdiskpower21 U5791.001.9920F70-P1-C08-T1-L62 PowerPath Device
* hdiskpower22 U5791.001.9920F70-P1-C08-T1-L63 PowerPath Device
* hdiskpower23 U5791.001.9920F70-P1-C08-T1-L64 PowerPath Device
* hdiskpower24 U5791.001.9920F70-P1-C08-T1-L65 PowerPath Device
* hdiskpower25 U5791.001.9920F70-P1-C08-T1-L66 PowerPath Device
* hdiskpower26 U5791.001.9920F70-P1-C08-T1-L59 PowerPath Device
* hdisk67 U5791.001.9920F70-P1-C08-T1-W5006048ACC36E4E3-L11000000000000 EMC Symmetrix FCP Raid5
* hdisk68 U5791.001.9920F70-P1-C08-T1-W5006048ACC36E4E3-L12000000000000 EMC Symmetrix FCP Raid5
* hdisk69 U5791.001.9920F70-P1-C08-T1-W5006048ACC36E4E3-L13000000000000 EMC Symmetrix FCP Raid5
* hdisk70 U5791.001.9920F70-P1-C08-T1-W5006048ACC36E4E3-L14000000000000 EMC Symmetrix FCP Raid5
* hdisk71 U5791.001.9920F70-P1-C08-T1-W5006048ACC36E4E3-L15000000000000 EMC Symmetrix FCP Raid5
* hdisk72 U5791.001.9920F70-P1-C08-T1-W5006048ACC36E4E3-L16000000000000 EMC Symmetrix FCP Raid5
* hdisk73 U5791.001.9920F70-P1-C08-T1-W5006048ACC36E4E3-L17000000000000 EMC Symmetrix FCP Raid5
* hdisk74 U5791.001.9920F70-P1-C08-T1-W5006048ACC36E4E3-L18000000000000 EMC Symmetrix FCP Raid5
* hdisk75 U5791.001.9920F70-P1-C08-T1-W5006048ACC36E4E3-L10A000000000000 EMC Symmetrix FCP Raid5
* hdisk78 U5791.001.9920F70-P1-C08-T1-W5006048ACC36E4E3-LF5000000000000 EMC Symmetrix FCP Raid5
* hdisk79 U5791.001.9920F70-P1-C08-T1-W5006048ACC36E4E3-LF6000000000000 EMC Symmetrix FCP Raid5
* hdisk107 U5791.001.9920F70-P1-C08-T1-W5006048ACC36E4E3-L2000000000000 EMC Symmetrix FCP Raid5
* hdisk108 U5791.001.9920F70-P1-C08-T1-W5006048ACC36E4E3-L3000000000000 EMC Symmetrix FCP Raid5
* hdisk109 U5791.001.9920F70-P1-C08-T1-W5006048ACC36E4E3-L4000000000000 EMC Symmetrix FCP Raid5
* hdisk110 U5791.001.9920F70-P1-C08-T1-W5006048ACC36E4E3-L5000000000000 EMC Symmetrix FCP Raid5
* hdisk111 U5791.001.9920F70-P1-C08-T1-W5006048ACC36E4E3-L6000000000000 EMC Symmetrix FCP Raid5
* hdisk112 U5791.001.9920F70-P1-C08-T1-W5006048ACC36E4E3-L7000000000000 EMC Symmetrix FCP Raid5
* hdisk113 U5791.001.9920F70-P1-C08-T1-W5006048ACC36E4E3-L8000000000000 EMC Symmetrix FCP Raid5
* hdisk114 U5791.001.9920F70-P1-C08-T1-W5006048ACC36E4E3-L9000000000000 EMC Symmetrix FCP Raid5
* hdisk115 U5791.001.9920F70-P1-C08-T1-W5006048ACC36E4E3-LA000000000000 EMC Symmetrix FCP Raid5
* hdisk116 U5791.001.9920F70-P1-C08-T1-W5006048ACC36E4E3-LB000000000000 EMC Symmetrix FCP Raid5
* hdisk117 U5791.001.9920F70-P1-C08-T1-W5006048ACC36E4E3-LC000000000000 EMC Symmetrix FCP Raid5
* hdisk118 U5791.001.9920F70-P1-C08-T1-W5006048ACC36E4E3-LD000000000000 EMC Symmetrix FCP Raid5
* hdisk119 U5791.001.9920F70-P1-C08-T1-W5006048ACC36E4E3-LE000000000000 EMC Symmetrix FCP Raid5
* hdisk120 U5791.001.9920F70-P1-C08-T1-W5006048ACC36E4E3-LF000000000000 EMC Symmetrix FCP Raid5
* hdisk121 U5791.001.9920F70-P1-C08-T1-W5006048ACC36E4E3-L10000000000000 EMC Symmetrix FCP Raid5
* hdisk122 U5791.001.9920F70-P1-C08-T1-W5006048ACC36E4E3-L100000000000000 EMC Symmetrix FCP Raid5
* hdisk123 U5791.001.9920F70-P1-C08-T1-W5006048ACC36E4E3-L101000000000000 EMC Symmetrix FCP Raid5
* hdisk124 U5791.001.9920F70-P1-C08-T1-W5006048ACC36E4E3-L102000000000000 EMC Symmetrix FCP Raid5
* hdisk125 U5791.001.9920F70-P1-C08-T1-W5006048ACC36E4E3-L103000000000000 EMC Symmetrix FCP Raid5
* hdisk126 U5791.001.9920F70-P1-C08-T1-W5006048ACC36E4E3-L104000000000000 EMC Symmetrix FCP Raid5
* hdisk127 U5791.001.9920F70-P1-C08-T1-W5006048ACC36E4E3-L105000000000000 EMC Symmetrix FCP Raid5
* hdisk128 U5791.001.9920F70-P1-C08-T1-W5006048ACC36E4E3-L106000000000000 EMC Symmetrix FCP Raid5
* hdisk129 U5791.001.9920F70-P1-C08-T1-W5006048ACC36E4E3-L107000000000000 EMC Symmetrix FCP Raid5
* hdisk130 U5791.001.9920F70-P1-C08-T1-W5006048ACC36E4E3-L109000000000000 EMC Symmetrix FCP Raid5
* hdisk131 U5791.001.9920F70-P1-C08-T1-W5006048ACC36E4E3-L110000000000000 EMC Symmetrix FCP Raid5
* hdisk132 U5791.001.9920F70-P1-C08-T1-W5006048ACC36E4E3-L111000000000000 EMC Symmetrix FCP Raid5
* hdisk133 U5791.001.9920F70-P1-C08-T1-W5006048ACC36E4E3-L112000000000000 EMC Symmetrix FCP Raid5
* hdiskpower27 U5791.001.9920F70-P1-C08-T1-L29 PowerPath Device
* hdiskpower28 U5791.001.9920F70-P1-C08-T1-L30 PowerPath Device
* hdiskpower29 U5791.001.9920F70-P1-C08-T1-L31 PowerPath Device
* hdiskpower30 U5791.001.9920F70-P1-C08-T1-L32 PowerPath Device
* hdiskpower31 U5791.001.9920F70-P1-C08-T1-L33 PowerPath Device
* hdiskpower32 U5791.001.9920F70-P1-C08-T1-L34 PowerPath Device
* hdiskpower33 U5791.001.9920F70-P1-C08-T1-L35 PowerPath Device
* hdiskpower34 U5791.001.9920F70-P1-C08-T1-L36 PowerPath Device
* hdiskpower35 U5791.001.9920F70-P1-C08-T1-L37 PowerPath Device
* hdiskpower36 U5791.001.9920F70-P1-C08-T1-L38 PowerPath Device
* hdiskpower37 U5791.001.9920F70-P1-C08-T1-L39 PowerPath Device
* pci15 U5791.001.9920F6T-P2 PCI Bus
* pci25 U5791.001.9920F6T-P2 PCI Bus
+ scsi12 U5791.001.9920F6T-P2-T6 Wide/Ultra-3 SCSI I/O Controller
+ hdisk18 U5791.001.9920F6T-P2-T6-L8-L0 16 Bit LVD SCSI Disk Drive (73400 MB)
+ hdisk19 U5791.001.9920F6T-P2-T6-L9-L0 16 Bit LVD SCSI Disk Drive (73400 MB)
+ ses10 U5791.001.9920F6T-P2-T6-L15-L0 SCSI Enclosure Services Device
* pci4 U5791.001.9920F6H-P1 PCI Bus
* pci5 U5791.001.9920F6H-P1 PCI Bus
+ ent0 U5791.001.9920F6H-P1-C09-T1 2-Port 10/100/1000 Base-TX PCI-X Adapter (14108902)
+ ent1 U5791.001.9920F6H-P1-C09-T2 2-Port 10/100/1000 Base-TX PCI-X Adapter (14108902)
* vio0 Virtual I/O Bus
* ent3 U9119.590.025224F-V9-C2-T1 Virtual I/O Ethernet Adapter (l-lan)
* vsa0 U9119.590.025224F-V9-C0 LPAR Virtual Serial Adapter
* vty0 U9119.590.025224F-V9-C0-L0 Asynchronous Terminal
* pci0 U5791.001.9920F6W-P1 PCI Bus
* pci44 U5791.001.9920F6W-P1 PCI Bus
+ fcs0 U5791.001.9920F6W-P1-C09-T1 FC Adapter
* fcnet0 U5791.001.9920F6W-P1-C09-T1 Fibre Channel Network Protocol Device
+ fscsi0 U5791.001.9920F6W-P1-C09-T1 FC SCSI I/O Controller Protocol Device
* hdisk2 U5791.001.9920F6W-P1-C09-T1-W5006048ACC36E4EC-LF0000000000000 EMC Symmetrix FCP Raid1
* hdisk58 U5791.001.9920F6W-P1-C09-T1-W5006048ACC36E4EC-L11000000000000 EMC Symmetrix FCP Raid5
* hdisk59 U5791.001.9920F6W-P1-C09-T1-W5006048ACC36E4EC-L12000000000000 EMC Symmetrix FCP Raid5
* hdisk60 U5791.001.9920F6W-P1-C09-T1-W5006048ACC36E4EC-L13000000000000 EMC Symmetrix FCP Raid5
* hdisk61 U5791.001.9920F6W-P1-C09-T1-W5006048ACC36E4EC-L14000000000000 EMC Symmetrix FCP Raid5
* hdisk62 U5791.001.9920F6W-P1-C09-T1-W5006048ACC36E4EC-L15000000000000 EMC Symmetrix FCP Raid5
* hdisk63 U5791.001.9920F6W-P1-C09-T1-W5006048ACC36E4EC-L16000000000000 EMC Symmetrix FCP Raid5
* hdisk64 U5791.001.9920F6W-P1-C09-T1-W5006048ACC36E4EC-L17000000000000 EMC Symmetrix FCP Raid5
* hdisk65 U5791.001.9920F6W-P1-C09-T1-W5006048ACC36E4EC-L18000000000000 EMC Symmetrix FCP Raid5
* hdisk66 U5791.001.9920F6W-P1-C09-T1-W5006048ACC36E4EC-L10A000000000000 EMC Symmetrix FCP Raid5
* hdisk76 U5791.001.9920F6W-P1-C09-T1-W5006048ACC36E4EC-LF5000000000000 EMC Symmetrix FCP Raid5
* hdisk77 U5791.001.9920F6W-P1-C09-T1-W5006048ACC36E4EC-LF6000000000000 EMC Symmetrix FCP Raid5
* hdisk80 U5791.001.9920F6W-P1-C09-T1-W5006048ACC36E4EC-L2000000000000 EMC Symmetrix FCP Raid5
* hdisk81 U5791.001.9920F6W-P1-C09-T1-W5006048ACC36E4EC-L3000000000000 EMC Symmetrix FCP Raid5
* hdisk82 U5791.001.9920F6W-P1-C09-T1-W5006048ACC36E4EC-L4000000000000 EMC Symmetrix FCP Raid5
* hdisk83 U5791.001.9920F6W-P1-C09-T1-W5006048ACC36E4EC-L5000000000000 EMC Symmetrix FCP Raid5
* hdisk84 U5791.001.9920F6W-P1-C09-T1-W5006048ACC36E4EC-L6000000000000 EMC Symmetrix FCP Raid5
* hdisk85 U5791.001.9920F6W-P1-C09-T1-W5006048ACC36E4EC-L7000000000000 EMC Symmetrix FCP Raid5
* hdisk86 U5791.001.9920F6W-P1-C09-T1-W5006048ACC36E4EC-L8000000000000 EMC Symmetrix FCP Raid5
* hdisk87 U5791.001.9920F6W-P1-C09-T1-W5006048ACC36E4EC-L9000000000000 EMC Symmetrix FCP Raid5
* hdisk88 U5791.001.9920F6W-P1-C09-T1-W5006048ACC36E4EC-LA000000000000 EMC Symmetrix FCP Raid5
* hdisk89 U5791.001.9920F6W-P1-C09-T1-W5006048ACC36E4EC-LB000000000000 EMC Symmetrix FCP Raid5
* hdisk90 U5791.001.9920F6W-P1-C09-T1-W5006048ACC36E4EC-LC000000000000 EMC Symmetrix FCP Raid5
* hdisk91 U5791.001.9920F6W-P1-C09-T1-W5006048ACC36E4EC-LD000000000000 EMC Symmetrix FCP Raid5
* hdisk92 U5791.001.9920F6W-P1-C09-T1-W5006048ACC36E4EC-LE000000000000 EMC Symmetrix FCP Raid5
* hdisk93 U5791.001.9920F6W-P1-C09-T1-W5006048ACC36E4EC-LF000000000000 EMC Symmetrix FCP Raid5
* hdisk94 U5791.001.9920F6W-P1-C09-T1-W5006048ACC36E4EC-L10000000000000 EMC Symmetrix FCP Raid5
* hdisk95 U5791.001.9920F6W-P1-C09-T1-W5006048ACC36E4EC-L100000000000000 EMC Symmetrix FCP Raid5
* hdisk96 U5791.001.9920F6W-P1-C09-T1-W5006048ACC36E4EC-L101000000000000 EMC Symmetrix FCP Raid5
* hdisk97 U5791.001.9920F6W-P1-C09-T1-W5006048ACC36E4EC-L102000000000000 EMC Symmetrix FCP Raid5
* hdisk98 U5791.001.9920F6W-P1-C09-T1-W5006048ACC36E4EC-L103000000000000 EMC Symmetrix FCP Raid5
* hdisk99 U5791.001.9920F6W-P1-C09-T1-W5006048ACC36E4EC-L105000000000000 EMC Symmetrix FCP Raid5
* hdisk100 U5791.001.9920F6W-P1-C09-T1-W5006048ACC36E4EC-L106000000000000 EMC Symmetrix FCP Raid5
* hdisk101 U5791.001.9920F6W-P1-C09-T1-W5006048ACC36E4EC-L107000000000000 EMC Symmetrix FCP Raid5
* hdisk102 U5791.001.9920F6W-P1-C09-T1-W5006048ACC36E4EC-L109000000000000 EMC Symmetrix FCP Raid5
* hdisk103 U5791.001.9920F6W-P1-C09-T1-W5006048ACC36E4EC-L110000000000000 EMC Symmetrix FCP Raid5
* hdisk104 U5791.001.9920F6W-P1-C09-T1-W5006048ACC36E4EC-L111000000000000 EMC Symmetrix FCP Raid5
* hdisk105 U5791.001.9920F6W-P1-C09-T1-W5006048ACC36E4EC-L112000000000000 EMC Symmetrix FCP Raid5
* hdisk106 U5791.001.9920F6W-P1-C09-T1-W5006048ACC36E4EC-L113000000000000 EMC Symmetrix FCP Raid5
+ fcs1 U5791.001.9920F6W-P1-C09-T2 FC Adapter
* fcnet1 U5791.001.9920F6W-P1-C09-T2 Fibre Channel Network Protocol Device
+ fscsi1 U5791.001.9920F6W-P1-C09-T2 FC SCSI I/O Controller Protocol Device
* rmt1 U5791.001.9920F6W-P1-C09-T2-W100000E00223B4FE-L1000000000000 Other FC SCSI Tape Drive
* rmt2 U5791.001.9920F6W-P1-C09-T2-W100000E00223B4FE-L2000000000000 Other FC SCSI Tape Drive
* rmt3 U5791.001.9920F6W-P1-C09-T2-W100000E00223D755-L0 Other FC SCSI Tape Drive
* rmt4 U5791.001.9920F6W-P1-C09-T2-W100000E00223D755-L1000000000000 Other FC SCSI Tape Drive
* rmt5 U5791.001.9920F6W-P1-C09-T2-W100000E00223FF7A-L0 Other FC SCSI Tape Drive
* rmt6 U5791.001.9920F6W-P1-C09-T2-W100000E002240263-L0 Other FC SCSI Tape Drive
* rmt0 U5791.001.9920F6W-P1-C09-T2-W100000E00223FF7A-L1000000000000 Other FC SCSI Tape Drive
* rmt7 U5791.001.9920F6W-P1-C09-T2-W100000E002240263-L1000000000000 Other FC SCSI Tape Drive
+ L2cache0 L2 Cache
+ mem0 Memory
+ proc0 Processor
+ proc2 Processor

myserver> errpt -a|more
---------------------------------------------------------------------------
LABEL: SC_DISK_ERR2
IDENTIFIER: B6267342

Date/Time: Mon Dec 13 06:53:34 CST 2010
Sequence Number: 000007
Machine Id: 001122334400
Node Id: myserver
Class: H
Type: PERM
Resource Name: hdisk111
Resource Class: disk
Resource Type: SYMM_RAID5
Location: U5791.001.9920F70-P1-C08-T1-W5006048ACC36E4E3-L6000000000000
VPD:
Manufacturer................EMC
Machine Type and Model......SYMMETRIX
ROS Level and ID............5671
Serial Number...............12345678
Part Number.................000000000000510023010187
EC Level....................751315
LIC Node VPD................0AA3
Device Specific.(Z0)........04
Device Specific.(Z1)........51
Device Specific.(Z2)........567100750000000000092508
Device Specific.(Z3)........12000000
Device Specific.(Z4)........54130008
Device Specific.(Z5)........BF80
Device Specific.(Z6)........4D

Description
DISK OPERATION ERROR

Probable Causes
DASD DEVICE

Failure Causes
DISK DRIVE
DISK DRIVE ELECTRONICS

Recommended Actions
PERFORM PROBLEM DETERMINATION PROCEDURES

Detail Data
PATH ID
0
SENSE DATA
0600 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0118 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0083 0000
0000 003D 0019
---------------------------------------------------------------------------
LABEL: SC_DISK_ERR2
IDENTIFIER: B6267342

Date/Time: Mon Dec 13 06:53:34 CST 2010
Sequence Number: 000006
Machine Id: 001122334400

Tuesday, April 26, 2011

Checking processes memory usage

Sometimes your server would have a heavy load because there is one or many processes consuming a lot of memory, and when there is no more physical memory (RAM) available the system begins using virtual memory (swap), and as long as RAM is thousands times faster than normal hard disks you'll notice for sure when your system begins to swap memory. Therefore, if you're a sysadmin or an Oracle DBA running instances in your own AIX server that doesn't run anything else, it would be good to pinpoint the top processes consuming the system's memory in an easy and quick way like this:

myserver> ps auwx|sort -k 5b,5nr|more
USER PID %CPU %MEM SZ RSS TTY STAT STIME TIME COMMAND
oracle 5669038 0.1 2.0 1536176 743136 - A Jan 21 18:43 oraclemyoradb (LOCAL=NO)
oracle 4755538 0.0 2.0 1472752 719680 - A Jan 15 16:44 oraclemyoradb (LOCAL=NO)
oracle 5922916 0.0 1.0 1463404 518768 - A Jan 21 9:05 oraclemyoradb (LOCAL=NO)
oracle 4575476 0.0 2.0 1419100 692160 - A 16:05:55 2:25 oraclemyoradb (LOCAL=NO)
oracle 2752638 0.0 2.0 1405344 692960 - A Jan 20 11:35 oraclemyoradb (LOCAL=NO)
oracle 6070354 0.1 2.0 1338124 736024 - A 16:03:06 5:23 oraclemyoradb (LOCAL=NO)
oracle 3518588 0.0 2.0 1273420 704220 - A Jan 20 10:56 oraclemyoradb (LOCAL=NO)
oracle 5390404 0.1 2.0 1272972 814148 - A Jan 22 13:47 oraclemyoradb (LOCAL=NO)
oracle 2371620 0.0 2.0 1259332 685920 - A Jan 20 7:40 oraclemyoradb (LOCAL=NO)
oracle 4784328 0.0 2.0 1259024 774984 - A Jan 20 11:02 oraclemyoradb (LOCAL=NO)
oracle 5877918 0.1 2.0 1200360 730492 - A Jan 24 5:12 oraclemyoradb (LOCAL=NO)
oracle 4374602 0.0 2.0 1193800 771632 - A Jan 21 9:35 oraclemyoradb (LOCAL=NO)
oracle 5804038 0.0 2.0 1178616 681944 - A Jan 20 19:32 oraclemyoradb (LOCAL=NO)
oracle 4931756 0.3 2.0 1156880 786928 - A 10:58:49 1:45 oraclemyoradb (LOCAL=NO)
oracle 3100862 0.0 2.0 1146228 664368 - A Jan 18 6:54 oraclemyoradb (LOCAL=NO)
oracle 1880168 0.0 2.0 1043492 709820 - A Jan 20 16:53 oraclemyoradb (LOCAL=NO)
oracle 3858452 0.0 2.0 1035528 723960 - A Jan 20 12:24 oraclemyoradb (LOCAL=NO)
oracle 4050954 0.0 2.0 1027828 678236 - A Jan 22 10:00 oraclemyoradb (LOCAL=NO)
oracle 3510290 0.0 2.0 1010724 610948 - A Jan 14 11:22 oraclemyoradb (LOCAL=NO)


As you can see, there is a lot of oracle processes related to myoradb database that are consuming too much memory and chocking the system.

Wednesday, April 20, 2011

Checking AIX cache

The cache area is a memory area where the operating system places blocks of files used often, therefore reducing costly I/O to slower devices like hard disks. It is a good concept and very useful for boosting performance of file servers (NFS, samba), but is not so good for database software like Oracle because an instance manages its own cache area.

Therefore, if you are an Oracle administrator of databases in AIX servers it's a good practice to be aware of memory configuration and usage, and maybe help a bit your fellow sysadmin just in case. The important parameters to check are the minimum and maximum percentage of RAM for file caching purposes (minperm% and maxperm%):

root:myserver> vmstat -v
2621440 memory pages
2525761 lruable pages
180000 free pages
4 memory pools
647419 pinned pages
80.0 maxpin percentage
20.0 minperm percentage
80.0 maxperm percentage
24.8 numperm percentage
627654 file pages
0.0 compressed percentage
0 compressed pages
24.8 numclient percentage
25.0 maxclient percentage
627654 client pages
0 remote pageouts scheduled
58246 pending disk I/Os blocked with no pbuf
10544297 paging space I/Os blocked with no psbuf
2228 filesystem I/Os blocked with no fsbuf
805 client filesystem I/Os blocked with no fsbuf
2215609 external pager filesystem I/Os blocked with no fsbuf
0 Virtualized Partition Memory Page Faults
0.00 Time resolving virtualized partition memory page faults

As you can see in this example, there is a lot of minimum memory allotted for file caching (20%) and wasted because Oracle caching is better for Oracle databases than operating system caching, but you can't assign that 20% RAM space to Oracle unless you change the virtual memory AIX parameters.

Moreover, the maximum memory reserved for file caching is limited by the maxclient percentage (25%) so in this system you can use up to 25% of RAM for file caching of non-JFS filesystem files and the other 55% for caching files of JFS filesystems. You can check JFS memory pages usage with this command:

root:myserver> svmon -G
size inuse free pin virtual
memory 2621440 2442500 178940 647452 2186997
pg space 5373952 925157

work pers clnt other
pin 562525 0 0 84927
in use 1734676 0 707824

PageSize PoolSize inuse pgsp pin virtual
s 4 KB - 2326020 925157 571068 2070517
m 64 KB - 7280 0 4774 7280

You might notice that there are 0 pages used for JFS file caching (pers), and that might be because there are no JFS file systems in this server:

root:myserver> mount
node mounted mounted over vfs date options
-------- --------------- --------------- ------ ------------ ---------------
/dev/hd1 / jfs2 Jun 05 19:09 rw,log=/dev/hd8
/dev/hd2 /usr jfs2 Jun 05 19:09 rw,log=/dev/hd8
/dev/hd5 /var jfs2 Jun 05 19:10 rw,log=/dev/hd8
/dev/hd3 /tmp jfs2 Jun 05 19:10 rw,log=/dev/hd8
/dev/hd4 /home jfs2 Jun 05 19:11 rw,log=/dev/hd8
/proc /proc procfs Jun 05 19:11 rw
/dev/hd6 /opt jfs2 Jun 05 19:11 rw,log=/dev/hd8
/dev/fslv00 /myfs jfs2 Jun 05 19:11 rw,log=/dev/loglv00

IBM recommends setting the parameters as follows:

maxperm = 90%
maxclient = 90%
minperm = 3%

More information:

Tuning the AIX file caches
Overview of AIX page replacement