flak rss random

OpenBSD on a Sun T5120

I’ve been looking for a sparc64 system for a while and noticed the Sun Enterprise T5120 models have become very affordable. They’re interesting machines and great for testing due to the built in virtualization support.

T5120

There are two console ports on the back, serial and net. If you’re lucky, the network port will come in a usable configuration; otherwise you’ll need a “Cisco” RJ45 serial cable. I was lucky.

Plug in network and power, wait a bit, watch dhcpd logs for a new request to come in. There it is. ssh root@10.10.10.10 and use the default passwrod, changeme. This lands at the ilom prompt, which is a little weird, but not too ridiculous. Let’s get the rest of this thing fired up. I recommend donning noise canceling headphones at this point. If the CS24 is a hairdryer, the T5120 is a vacuum cleaner.

start /SYS
start /SP/console

This should eventually (wait for it, wait for it) land at the classic ok prompt. I installed OpenBSD by booting a CD. It is theoretically possible to netboot, but if you haven’t done so with a Sun before, I’d steer clear. I started down that path, but gave up somewhere after rarpd and /etc/ethers but before rpc.bootparamd and /etc/bootparams. pxeboot4life.

T2

The T2 CPU is hyperhyperthreaded. OpenBSD doesn’t yet scale all that well to 64 CPUs, so you are well advised to chop this thing up into manageable pieces. Each core consists of 8 threads, but has two integer ALUs, so dividing into units of 8 seems reasonable.

Perhaps my understanding is flawed, but in practice this doesn’t work as well as theorized. I haven’t been able to map out exactly how ALUs are divided among ldoms, but it’s not as straightforward as I’d like to believe. Running CPU benchmarks in parallel in various ldoms led to some inexplicable results. Play around with it.

ldomize

The man page for ldomctl is unfortunately rather barren; this is based on Mark’s undeadly article.

First, don’t forget to enable ldomd in rc.conf. Start things off by copying the factory config.

mkdir default
cd default
ldomctl dump
cd ..
cp -r default myconfig
cd myconfig

Create some disks. Instead of using a second disk for the installer, I overwrite the front of the main disk instead.

dd if=/dev/zero of=vdisk1 bs=1m count=8000
dd if=miniroot55.fs of=vdisk1 bs=64k conv=notrunc

Now we create an ldom.conf. I created an explicit primary domain. This would normally be created automatically using whatever resources are left, but if you’re unlucky, ilom won’t like generated config and you’ll end up booting the factory-default again. I was unlucky.

domain primary {
        vcpu 8
        memory 8G
}

domain puffy {
        vcpu 8
        memory 4G
        vdisk "/home/tedu/vms/v1/vdisk1"
        vnet
}

domain huffy {
        vcpu 8
        memory 4G
        vdisk "/home/tedu/vms/v1/vdisk2"
        vnet
}

Install and reboot.

ldomctl init-system ldom.conf
cd ..
ldcomctl download myconfig
halt
(back at ok prompt)
#.
(back at ilom prompt)
reset /sys

To later switch configs in ilom:

set /HOST/bootmode config=configname

You can finish installing OpenBSD in the ldoms by running cu -l ttyV0 (ttyV1, etc.). In order to get networking working, we probably want to bridge the ldoms together.

ifconfig vnet0 -inet6 up
ifconfig vnet1 -inet6 up
ifconfig bridge0 up add vnet0 add vnet1

At this point it’s possible to bridge the interfaces to a physical em device, but I did something a little different. My primary ldom has IP address 10.10.10.76. I’m going to left shift that and create a 10.10.76.0 network inside the T5120. All the little ldoms are plugged into our bridge (switch), but we need to create another virtual interface to plug the primary domain in.

ifconfig vether0 up 10.10.76.1
ifconfig bridge0 add vether0

Then I started dhcpd listening on vether0, and configured puffy and huffy to use dhcp.

All in all, the ldom support is pretty solid, thanks to Mark Kettenis. I did all this without ever touching Solaris, or even installing any pkgs.

dmesg

From before carving into ldoms.

console is /virtual-devices@100/console@1
Copyright (c) 1982, 1986, 1989, 1991, 1993
	The Regents of the University of California.  All rights reserved.
Copyright (c) 1995-2014 OpenBSD. All rights reserved.  http://www.OpenBSD.org

OpenBSD 5.5 (GENERIC.MP) #173: Tue Mar  4 14:47:47 MST 2014
    deraadt@sparc64.openbsd.org:/usr/src/sys/arch/sparc64/compile/GENERIC.MP
real mem = 34225520640 (32640MB)
avail mem = 33670987776 (32111MB)
mainbus0 at root: SPARC Enterprise T5120
cpu0 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu1 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu2 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu3 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu4 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu5 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu6 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu7 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu8 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu9 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu10 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu11 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu12 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu13 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu14 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu15 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu16 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu17 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu18 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu19 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu20 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu21 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu22 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu23 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu24 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu25 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu26 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu27 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu28 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu29 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu30 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu31 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu32 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu33 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu34 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu35 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu36 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu37 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu38 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu39 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu40 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu41 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu42 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu43 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu44 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu45 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu46 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu47 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu48 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu49 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu50 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu51 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu52 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu53 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu54 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu55 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu56 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu57 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu58 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu59 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu60 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu61 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu62 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu63 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
vbus0 at mainbus0
"flashprom" at vbus0 not configured
"tpm" at vbus0 not configured
cbus0 at vbus0
vldc0 at cbus0
vldcp0 at vldc0 chan 0x14: ivec 0x228, 0x229 channel "spds"
"sunvts" at vldc0 chan 0x6 not configured
"sunmc" at vldc0 chan 0x7 not configured
"explorer" at vldc0 chan 0x8 not configured
"led" at vldc0 chan 0x9 not configured
"flashupdate" at vldc0 chan 0xa not configured
"ipmi" at vldc0 chan 0xc not configured
"system-management" at vldc0 chan 0xd not configured
vldc1 at cbus0
"spfma" at vldc1 chan 0x5 not configured
vldc2 at cbus0
vldcp1 at vldc2 chan 0x0: ivec 0x200, 0x201 channel "hvctl"
"ldom-primary" at vldc2 chan 0x1 not configured
"fmactl" at vldc2 chan 0x3 not configured
vldc3 at cbus0
"ldmfma" at vldc3 chan 0x4 not configured
"n2cp" at vbus0 not configured
"ncp" at vbus0 not configured
vrng0 at vbus0
vcons0 at vbus0: ivec 0x111, console
vrtc0 at vbus0
"niu" at mainbus0 not configured
vpci0 at mainbus0: bus 2 to 18, dvma map 80000000-ffffffff
pci0 at vpci0
ppb0 at pci0 dev 0 function 0 "PLX PEX 8533" rev 0xaa
pci1 at ppb0 bus 3
ppb1 at pci1 dev 1 function 0 "PLX PEX 8533" rev 0xaa
pci2 at ppb1 bus 4
ppb2 at pci2 dev 0 function 0 "PLX PEX 8517" rev 0xad
pci3 at ppb2 bus 5
ppb3 at pci3 dev 1 function 0 "PLX PEX 8517" rev 0xad
pci4 at ppb3 bus 6
ppb4 at pci4 dev 0 function 0 vendor "PLX", unknown product 0x8111 rev 0x21
pci5 at ppb4 bus 7
ohci0 at pci5 dev 0 function 0 "NEC USB" rev 0x43: ivec 0x16, version 1.0
ohci1 at pci5 dev 0 function 1 "NEC USB" rev 0x43: ivec 0x17, version 1.0
ehci0 at pci5 dev 0 function 2 "NEC USB" rev 0x04: ivec 0x14
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 "NEC EHCI root hub" rev 2.00/1.00 addr 1
usb1 at ohci0: USB revision 1.0
uhub1 at usb1 "NEC OHCI root hub" rev 1.00/1.00 addr 1
usb2 at ohci1: USB revision 1.0
uhub2 at usb2 "NEC OHCI root hub" rev 1.00/1.00 addr 1
ppb5 at pci3 dev 2 function 0 "PLX PEX 8517" rev 0xad
pci6 at ppb5 bus 8
em0 at pci6 dev 0 function 0 "Intel 82571EB" rev 0x06: ivec 0x17, address 00:14:4f:97:cd:a4
em1 at pci6 dev 0 function 1 "Intel 82571EB" rev 0x06: ivec 0x14, address 00:14:4f:97:cd:a5
ppb6 at pci3 dev 3 function 0 "PLX PEX 8517" rev 0xad
pci7 at ppb6 bus 9
em2 at pci7 dev 0 function 0 "Intel 82571EB" rev 0x06: ivec 0x14, address 00:14:4f:97:cd:a6
em3 at pci7 dev 0 function 1 "Intel 82571EB" rev 0x06: ivec 0x15, address 00:14:4f:97:cd:a7
ppb7 at pci1 dev 2 function 0 "PLX PEX 8533" rev 0xaa
pci8 at ppb7 bus 10
mpi0 at pci8 dev 0 function 0 "Symbios Logic SAS1068E" rev 0x02: msi
scsibus0 at mpi0: 112 targets
sd0 at scsibus0 targ 0 lun 0: <SEAGATE, ST914602SSUN146G, 0400> SCSI3 0/direct fixed naa.5000c5000615a07f
sd0: 140009MB, 512 bytes/sector, 286739329 sectors
sd1 at scsibus0 targ 1 lun 0: <ATA, OCZ-AGILITY2, 1.25> SCSI3 0/direct fixed naa.5e83a97f7a590cc2
sd1: 85857MB, 512 bytes/sector, 175836528 sectors
ppb8 at pci1 dev 8 function 0 "PLX PEX 8533" rev 0xaa
pci9 at ppb8 bus 11
ppb9 at pci9 dev 0 function 0 "PLX PEX 8533" rev 0xaa
pci10 at ppb9 bus 12
ppb10 at pci10 dev 1 function 0 "PLX PEX 8533" rev 0xaa: msi
pci11 at ppb10 bus 13
ppb11 at pci10 dev 2 function 0 "PLX PEX 8533" rev 0xaa: msi
pci12 at ppb11 bus 14
ppb12 at pci10 dev 8 function 0 "PLX PEX 8533" rev 0xaa: msi
pci13 at ppb12 bus 15
ppb13 at pci10 dev 9 function 0 "PLX PEX 8533" rev 0xaa: msi
pci14 at ppb13 bus 16
ppb14 at pci10 dev 10 function 0 "PLX PEX 8533" rev 0xaa: msi
pci15 at ppb14 bus 17
ppb15 at pci1 dev 9 function 0 "PLX PEX 8533" rev 0xaa: msi
pci16 at ppb15 bus 18
"pci-performance-counters" at mainbus0 not configured
ebus0 at mainbus0
com0 at ebus0 addr ca0000-ca0007 ivec 0x13: ns16550a, 16 byte fifo
umass0 at uhub0 port 2 configuration 1 interface 0 "Texas Instruments Incorporated 6250 FW: 03.03.02.22.562AD423" rev 2.00/2.00 addr 2
umass0: using SCSI over Bulk-Only
scsibus1 at umass0: 2 targets, initiator 0
cd0 at scsibus1 targ 1 lun 0: <TSSTcorp, CD/DVDW TS-T632A, SR03> SCSI0 5/cdrom removable serial.0451625fACCC3275122F
uhub3 at uhub0 port 4 "Cypress Semiconductor USB2 Hub" rev 2.00/0.0b addr 3
vscsi0 at root
scsibus2 at vscsi0: 256 targets
softraid0 at root
scsibus3 at softraid0: 256 targets
bootpath: /pci@0,0/pci@0,0/pci@2,0/scsi@0,0/disk@0,0
root on sd0a (ccfe4bf26855e98c.a) swap on sd0b dump on sd0b

Posted 18 Mar 2014 17:00 by tedu Updated: 08 Jan 2015 18:05
Tagged: computers openbsd