Remote Management of ZFS servers with Puppet and RAD

A few months ago I had the chance to test an Oracle ZFS Storage Appliance (ZFS SA) and the appliance made a very good impression on many areas. It especially brought to my mind again, that ZFS shines even more if you use it as NAS (Network Attached Storage), as central fileserver which shares its storage capacity for example via NFS.

But I did not really like the distributed storage configuration. E.g. a database server needs the correct ZFS properties set on the ZFS storage appliance via the web-interface or the custom CLI and also the corresponding NFS mount options in /etc/vfstab on the database server. Maybe this sounds like no big issue to you, for example, if you are also the admin responsible for the storage appliance, or if you have a perfect collaboration with the storage team. But especially if you want to automate the storage configuration, this distribution adds a significant complexity.

Of course I wanted to manage the configuration with Puppet like a local ZFS filesystem.
I don’t yet have a ZFS SA at work to deal with, but the availability of the new RAD REST interface in Solaris 11.3 motivated me to experiment with an own Puppet resource type to manage the remote ZFS filesystems directly on the client server.

Read More

Puppet with Solaris RAD

Solaris 11.3 beta ships with a REST-API for the Solaris Remote Administration Daemon (RAD), which makes RAD finally easy to use with Ruby and Puppet. The following is a small experiment to test its capabilities.

Puppet provides a nice abstraction layer of the configuration of a system. A Puppet manifest is usually easier and faster to understand than a documentation with many CLI commands. But there is no magic involved, Puppet usually executes also the same CLI commands in the background. With the debug option you can observe this command executions.

Read More

Zfsdir: Simple ZFS management with Puppet

ZFS is a great filesystem, with many, many features. But for all that it is still easy to manage, in my opinion easier than other filesystems. Managing storage is usually a high risk task, which makes automation harder. Would you change the size for a critical filesystem with an automated method? If it is an ext3 filesystem on LVM and software-raid, maybe not. If it is on ZFS, a low risk modification of the quota could be enough, e.g. zfs set quota=800g rpool/criticalfilesystem. That’s easy to automate. Nowadays automation becomes even necessary because, the amount of ZFS filesystems is growing. And if you like to use more features you likely need to set more ZFS properties.

Read More

End of year 2014 summary

The year 2014 is almost finished. It was a great year! I relaunched this blog this year to “archive” some of my technical work and motivate myself to finalize my side projects in a sharable form. And of course to connect to people with similar interests.
Also this year my side projects were primary motivated by making my main job as Solaris systems engineer easier and of course less busier. So my main topics this year were clearly Solaris and Puppet.
Before I start with my plans for 2015 I like to sum up this year to see what worked well and what not. Justin Jackson wrote a really nice report on this achievements, my numbers are a different dimension but I will use his nice structure. To have a nice baseline for 2015.

Read More

See me at Oracle Openworld and PuppetConf

I will be at Oracle Openworld 2014, finally as speaker. Together with Glynn Foster and Eric Saxe I will talk about Puppet on Solaris. There are still some free seats, so register if you are interested:

Session: Best Practice Configuration Management with Puppet [CON7849]

A few days before I will also attend the Automation event of the year. This time I will finally take part on my first PuppetConf in person. The line-up is amazing with speakers like Gene Kim the author of “The Phoenix Project” and many other early drivers of the DevOps community.

I am in San Francisco from 23rd September to 2nd October, if you like to meet in person just ping me.

Known issues in Puppet on Solaris 11.2

Update 25th Dec 2014: With the latest Solaris SRU ( the important bugs which are listened here, are fixed for me. Also Puppet, Facter and Hiera are now shipped in a modern version.

With 11.2, Puppet is shipped for the first time directly with Solaris. I think it is a quite good first release, but of course not yet perfect. Puppet itself is open source software and Oracle also published all patches and Puppet types under the CDDL, so you would have the option to maintain your own build.

The huge open source community is an important part of Puppet, maybe the most important. But with Solaris you have the additional advantage that you have this open source software also covered under your usual Oracle Premier Support. With no additional cost!

Read More

Solaris 11.2 released

After a short public beta phase, Oracle made Solaris 11.2 generally available last week. You can download it from in various forms like the usual install ISO, Unified Archives and as Virtual box appliance.

There are many, many new features, but also a lot of small incremental improvements.

Some of the new key features are:

  • Openstack
  • Kernel zones
  • Puppet
  • Unified Archives
  • Elastic Virtual Switch

    Read More

Control the size of the ZFS ARC cache dynamically

Last updated on: 25th Dec 2014

Solaris 11.2 deprecates the zfs_arc_max kernel parameter in favor of user_reserve_hint_pct and that’s cool.

ZFS has a very smart cache, the so called ARC (Adaptive replacement cache). In general the ARC consumes as much memory as it is available, it also takes care that it frees up memory if other applications need more.

In theory, this works very good, ZFS just uses available memory to speed up slow disk I/O. But it also has some side effects, if the ARC consumed almost all unused memory. Applications which request more memory need to wait, until the ARC frees up memory. For example, if you restart a big database, the startup is maybe significantly delayed, because the ARC could have used the free memory from the database shutdown in the meantime already. Additionally this database would likely request large memory pages, if ARC uses just some free segments, the memory gets easily fragmented.

Read More