SuperMicro

Hardware notes

SuperMicro manufactures its own motherboards. Find the models at www.supermicro.com
SuperMicro will assemble systems with various components from other manufacturers as required.
* LSI MegaRaid

ILOM notes

• See ClusterToolsIPMITOOL for the most current documentation of IPMIview, ipmitool and ipmicfg

IPMI Intelligent Platform Management Interface

• The IPMI card is found in Supermicro motherboards. It is Supermicro's LOM card. It is either built into the motherboard or comes as an add-on BMC Base Management Card.
• A built-in IPMI shares its RJ45 connection with the first ethernet port, called eth0 in Linux, but referred to as ETH1 in the IPMI documentation.
• The add-on card has a special connector on the motherboard, it does not occupy a PCI slot. The add-on card uses a 15" long USB cable to connect an RJ45 jack on a card slot bracket. So it will occupy a PCI slot bracket on the back of the PC. The add-on card has its own RJ45 connection, it doesn't share with eth0. On the bottom of the card is the serial number.

Hardware

• The card will come with an install CD. Put the IPMI CLI application on the system, preferably in an admin account, such as /home/cscf-adm/cli. For a 64 bit Linux system it is ipmicfg-linux.x86_64. Before continuing with the next step get a name and IP address for the IPMI.
• See ClusterToolsIPMITOOL

Installing Firmware

Get flash software from CD

• /media/cdrom0/IPMI_Solution/Utility/linux_flash_utility_64bit/*.*
• /media/cdrom0/Firmware/SIMxx/ugsim157.bin
• $ sudo sh load_ipmi_driver.sh

There may be two versions of the firmware on the CD. Determine which is for your card. Note the version number is in the software name e.g. ugsim157 is version 1.57.

• $ sudo ./updatefw.x86_64 -f ugsim157.bin

This requires several minutes. It will show a percentage of the progress.

INSTALL Linux IPMIView Remote Application

• See ClusterToolsIPMITOOL

If the system has Java 32 bit version you may need to install 32 bit libraries before installing IPMIView.

• $ apt-cache search -n ia32
• $ apt-get install ia32-libs

IPMI Card"> Connect to the IPMI Card

• Web:
  • Connect via a web interface: lom-machinename.cs.uwaterloo.ca
  • IpmiWeb access from Firefox on Ubuntu
• IPMIview
  • See ClusterToolsIPMITOOL

Adding a Machine to IPMIView

• See ClusterToolsIPMITOOL

<nsb;>Search for available IPMI cards in an IP range

• In the IPMIView applciation top menu File > Discover IPMI Device
  • This opens the "IPMI Device Discovering" dialog.
• Enter the "Network IP From" IP address of the IPMI card and press Start button.
• Select the name of the card from the list and press Save button.
• The main screen should show the card under the "IPMI Domain" column. Press Exit to leave the discovery dialog.
• Top menu File > Save Configuration

<nsb;>Alternatively add a system to the "IPMI Domain" column.

• Top menu File > New > System
• Enter IP and name of the IPMI card
  • The name should appear in the system list

<nsb;>Save the configuration

• Top menu File > Save Configuration

IPMI Card"> Using the IPMI Card

Double-click the machine name in the "IPMI Domain" column in the IPMIView interface to connect to a machine IPMI card or use a browser.
First Login

• Use the default factory name and password ADMIN, ADMIN
  • In the web interface change the password under the heading "Management and Change Password" links
    • The ADMIN user is the super user. In the web interface it can create new users, including those with admin rights. Other users, including admins, cannot create or manage users.
  • In the IPMIView double-click the card under the "IPMI Domain" column.
    • In the main dialog it will ask for Login. Use the factory default ADMIN, ADMIN for ID and Password.
    • It will confirm if the card is connected and several tabs will show at the bottom of the main dialog.
    • Select the "Users" tab to change the default password for ADMIN. Also add in the CSCF LOM ID and Password and set privileges.
    • Also enter the serial number for the device found on the card in the FRU information section.
• ADMIN is the superuser
  • In the web interface it is the only user that can create new users
  • In the IPMIView any user with admin privileges may create or administer users

IPMIView tabs

• BMC Settings
  • IP, MAC, DHCP settings can all be changed here if user has admin privileges
• IPMI Device
  • The computer can be powered up/down/cycled from here
• Users
  • Users may be created or deleted and given privileges
  • IPMI card information is displayed. Enter the Serial Number here.
• KVM Console
  • A connection to the server will be made with this tab. To view it in full screen mode click the Computer icon. Use Alt+Enter to exit full screen mode.
• Other tabs include Sensors, Event Log, Text Console, and Virtual Media

Web Interface

• Remote Control
  • KVM Console can be accessed here
• System Health
  • Control the computer power on/off/cycle
  • Monitor sensors and view logs
• User Management
  • Change passwords and manage users and groups
• Device Settings
  • Network settings
    • Set DHCP and turn off HTTP port so only HTTPS is on
  • Security
    • Force HTTPS check
    • KVM Encryption try - if KVM can't make a secure connection then default to normal HTTP
  • Certificate
    • Enter the information to create a certificate
    • Common name: the fully qualified domain name
    • Organizational Unit: David R. Cheriton School of Computer Science
    • Organization: University of Waterloo
    • Locality/City: Waterloo
    • State/Province: Ontario
    • Country: CA
    • Email: username@domainname (see note)
    • Challenge Password: must enter at least four characters
    • Confirm Challenge Password: must enter at least four characters
    • Key Length (bits): 2048
      • After clicking the Create key it will reply with a CSR Pending page
      • Download the csr.txt file (cert. signing request)
      • Use the csr.txt file to apply for an SSL certificate file and use the SSL cert file to Upload to the IPMI card
      • Obtaining a certificate via IST or GlobalSign may be require day. It is OK to exit the IPMI login. When returning to the IPMI web page the certification page will still be waiting for the cert.
      • When a cert has been obtained use the Upload button to apply the cert.
      • Browsers with have to accept the cert if it is UW/IST-CA assigned (grad image may include UW/IST-CA as recognized)
    • General certificate information IST
    • Create a GlobalSign certificate instructions
  • Date/Time
    • Choose synchronize with NTP server.

For the email address username@domainname, you should use No permission to view CFPrivate.EMailAddressCscfCerts

Using ipmitool Command Line Interface

• See ClusterToolsIPMITOOL

Event log commands

GET the event log
-----------------
cscf-adm@riemann:~ # sudo /usr/bin/ipmitool sel list
SEL has no entries

TEST the event log
------------------
cscf-adm@riemann:~> sudo ipmitool event 3
Sending SAMPLE event: Memory - Correctable ECC
   0 | Pre-Init Time-stamp   | Memory #0x53 | Correctable ECC | Asserted
cscf-adm@riemann:~> sudo ipmitool sel list
   1 | 04/16/2010 | 09:36:08 | Memory #0x53 | Correctable ECC | Asserted

CLEAR the event log
-------------------
cscf-adm@riemann:~> sudo ipmitool sel clear
Clearing SEL.  Please allow a few seconds to erase.
cscf-adm@riemann:~> sudo ipmitool sel list
SEL has no entries

Get sensor readings

# sudo /usr/bin/ipmitool sdr list
CPU1 Temp        | 35 degrees C      | ok
CPU2 Temp        | 35 degrees C      | ok
System Temp      | 26 degrees C      | ok
CPU1 Vcore       | 0.93 Volts        | ok
CPU2 Vcore       | 0.93 Volts        | ok
+5V              | 5.12 Volts        | ok
+12V             | 12 Volts          | ok
CPU1DIMM         | 1.54 Volts        | ok
CPU2DIMM         | 1.54 Volts        | ok
+1.5V            | 1.51 Volts        | ok
+3.3V            | 3.24 Volts        | ok
+3.3VSB          | 3.36 Volts        | ok
VBAT             | 3.31 Volts        | ok
Fan1             | 14040 RPM         | ok
Fan2             | 14040 RPM         | ok
Fan3             | 14040 RPM         | ok
Fan4             | no reading        | ns
PS Status        | 0 unspecified     | ok

Chassis power commands

# sudo /usr/bin/ipmitool chassis power help
chassis power Commands: status, on, off, cycle, reset, diag, soft
# sudo /usr/bin/ipmitool chassis power status
Chassis Power is on
# sudo /usr/bin/ipmitool chassis restart_cause
System restart cause: chassis power control command

BMC card information

# sudo ipmitool bmc info
# sudo ipmitool bmc reset cold
# sudo ipmitool bmc reset warm

Since ipmitool runs on the machine with the IPMI card, what if the machine is inaccessible?
Use ipmitool from a remote machine.

In ssh riemann17 terminal, could not get ipmitool to report any information on riemann17. It wouldn't actually connect...

riemann17:/opt/SUPERMICRO/IPMIView # ipmitool channel info
Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory
Unable to Get Channel Info

Exited riemann17 sub-node ssh session.
In riemann head-node ssh session:

Note riemann17 IP is 10.20.0.117
riemann:/opt/SUPERMICRO/IPMIView # ipmitool -H 10.20.0.117 -U admin lan print 1
Password:
Set in Progress         : Set Complete
Auth Type Support       : MD2 MD5 OEM
Auth Type Enable        : Callback : MD2 MD5 OEM
                        : User     : MD2 MD5 OEM
                        : Operator : MD2 MD5 OEM
                        : Admin    : MD2 MD5 OEM
                        : OEM      :
IP Address Source       : Static Address
IP Address              : 10.20.0.117
Subnet Mask             : 255.255.0.0
MAC Address             : 00:30:48:c9:8f:eb

riemann:/opt/SUPERMICRO/IPMIView # ipmitool -H 10.20.0.117 -U admin sensor list
Password:
CPU1 Temp  na   degrees C   na  0.000   0.000   0.000  81.000 82.000 83.000
CPU2 Temp  na   degrees C   na  0.000   0.000   0.000  81.000 82.000 83.000

riemann:/opt/SUPERMICRO/IPMIView # ipmitool -H 10.20.0.117 -U admin session info active
Password:
session handle                : 4
slot count                    : 30
active sessions               : 1
user id                       : 2
privilege level               : ADMINISTRATOR
session type                  : IPMIv1.5
channel number                : 0x01
console ip                    : 0.0.0.0
console mac                   : 00:00:00:00:00:00
console port                  : 0

riemann:/opt/SUPERMICRO/IPMIView # ipmitool -H 10.20.0.117 -U admin bmc reset cold
Password:
Sent cold reset command to MC

SSH to the IPMI card"> Using SSH to the IPMI card

The BMC card can be accessed by SSH. The card may run a BusyBox linux. This can be handy if the web interface port has changed to an unknown number.
Once connected run netstat to list the ports (there may be more than one) used by the web service called "webgo".

# netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:16416           0.0.0.0:*               LISTEN     29308/webgo
tcp        0      0 0.0.0.0:5120            0.0.0.0:*               LISTEN     484/cdserver
tcp        0      0 0.0.0.0:5123            0.0.0.0:*               LISTEN     496/fdserver
tcp        0      0 0.0.0.0:5988            0.0.0.0:*               LISTEN     375/sfcbd
tcp        0      0 0.0.0.0:555             0.0.0.0:*               LISTEN     29310/webgo
tcp        0      0 0.0.0.0:5900            0.0.0.0:*               LISTEN     29313/stunnel4
tcp        0      0 0.0.0.0:5901            0.0.0.0:*               LISTEN     29293/adviserd
tcp        0      0 0.0.0.0:623             0.0.0.0:*               LISTEN     325/
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN     29308/webgo
tcp        0      0 0.0.0.0:50000           0.0.0.0:*               LISTEN     448/discovery
tcp        0      0 127.0.0.1:8765          0.0.0.0:*               LISTEN     29293/adviserd
tcp6       0      0 :::22                   :::*                    LISTEN     418/dropbear
udp        0      0 172.19.96.37:1900       0.0.0.0:*                          448/discovery
udp        0      0 0.0.0.0:1900            0.0.0.0:*                          448/discovery
udp        0      0 0.0.0.0:623             0.0.0.0:*                          325/
udp        0      0 0.0.0.0:623             0.0.0.0:*                          325/

Then try each port on a browser e.g. http://ipaddress:16416 until the login web page is displayed. Once logged into the IPMI go to the configure tab and change the http port to 80.