Automation of operational tasks with Remote Execution

This post is part of a blog series which tries to simplify the adoption of configuration management frameworks.

Getting a fully automated data center in a bigger company which does not yet have an “automation friendly culture” is very hard. Setting up a company wide configuration management framework like Puppet, Chef or Cfengine can take ages. Additionally I experienced some engineers, who are not involved in the setup of configuration management framework itself, think they should wait until the framework is 100% established. Earliest after that they plan to start thinking how they can automate their repetitive tasks.

I think this “waiting” is a huge waste of time. Everybody can start understanding and automating their repetitive tasks already without the availability of a central company wide automation framework. In this post I will cover how the Remote Execution framework Fabric can be used to basically automate your SSH sessions.

Configuration Management Frameworks vs Remote Execution

In an ideal world, it is preferred to use state oriented configuration management frameworks like Puppet, Chef or Cfengine to change the configuration of the operating system or an application. And only use Remote Execution for ad hoc tasks which are not repetitive enough to be implemented in this frameworks. But of course you can use Remote Execution for tasks which are just not yet implemented in the state oriented frameworks. For example if you like to change the configuration of an apache server you can do the following remote execution with SSH:

$ scp new-apache2.conf root@server:/etc/apache2/apache2.conf
$ ssh root@server "/etc/init.d/apache2 restart"
 * Restarting web server apache2            [ OK ]

This is a simplified example, in a production environment you have to deal with many other issues, like permissions, password prompts etc. So even if you collect your remote executions in a shell script it gets complicated.

But luckily there are some awesome Remote Executions frameworks for your preferred scripting language like the Python based Fabric or the Ruby based Capistrano.

Fabric

As I am Python fan, Fabric was my first choice. Basically Fabric is a SSH client which can be scripted with Python, with additional handy features like:

  • Transferring Files
  • Sudo support
  • Fetching password prompts and answering them
  • Addressing whole destination server groups
  • Parallel mode

That means if you have access to servers with SSH and a workstation with a Python interpreter you can start using Fabric.
Installation is simple, use the package of your OS distribution or the pip installer:

# pip install fabric

The Remote Execution tasks need to be defined as Python function in the file called fabfile.py. In my opinion you do not need extended Python skills to start using Fabric, because you can start with a simple subset of functions, like run(),sudo() and put(). E.g.:

fabfile.py
1
2
3
4
5
6
7
from fabric.api import run,put,sudo

def deploy_apache_cfg():
    run('/etc/init.d/apache2 status')
    put('new-apache2.conf','/tmp/new-apache2.conf')
    sudo('cp /tmp/new-apache2.conf /etc/apache2/apache2.conf')
    sudo('/etc/init.d/apache2 restart')

If you have defined the task you can execute it for example on three servers:

$ fab --hosts webserver1,webserver2,webserver3 deploy_apache_cfg
[webserver1] Executing task 'deploy_apache_cfg'
[webserver1] run: /etc/init.d/apache2 status
[webserver1] Login password for 'mzach': 
[webserver1] out:  * apache2 is running
[webserver1] put: new-apache2.conf -> /tmp/new-apache2.conf
[webserver1] sudo: cp /tmp/new-apache2.conf /etc/apache2/apache2.conf
[webserver1] out: sudo password:
[webserver1] sudo: /etc/init.d/apache2 restart
[webserver1] out: sudo password:
[webserver1] out:  * Restarting web server apache2
[webserver1] out:    ...done.

[webserver2] Executing task 'deploy_apache_cfg'
...

[webserver3] Executing task 'deploy_apache_cfg'
...

Done.
Disconnecting from webserver1... done.
Disconnecting from webserver2... done.
Disconnecting from webserver3... done.

This is a very basic example, with Fabric you can also use the full power of Python. For example you can use the output of remote commands and search within Python with RegEx for a specific string and maybe write this string in a local text file. You can even completely avoid using the fab command line tool and just use Fabric directly in your own Python application. For more details check the Fabric documentation.

Benefits: Speeding up the creation of an automation friendly culture

Using Remote Execution can of course decrease the workload of engineers, but I see also big benefits for environments with many engineers which are “waiting” for teams who own the central automation system, as mentioned at the beginning. If people start to implement their tasks for example with Fabric, they very likely start thinking about the following questions:

  • Which commands do I really want to execute on the production server (increased planning)
  • How do I verify whether the remote system is in the expected state, after the Remote Execution. Usually this leads to the definition of a verification task. If you have such a task you can for example first execute the verification task, then the modification task and then the verification task again. So you check quickly which servers are broken after remote execution. Or the need for proper monitoring and central remote logging becomes obvious.
  • “OMG this task is too dangerous to automate!” - In some situations it makes sense to change the design or the internal configuration standards in order to simplify the automation and of course also to minimize the risk of a manual execution by a human.
  • Can I execute the task on all servers at the same time in parallel. Maybe it makes more sense to group the servers in specific groups and ensure e.g. with load balancing that there is no service interruption for the end users (better orchestration understanding).

For some tasks it will definitely turn out that they should be implemented a state oriented framework like Puppet, but I am very convinced that the important lessons for using automation can be also learned with tools like Fabric.

If you have an automation friendly culture in the company the move between different automation frameworks is easy.

Please leave a comment if you like to share your feedback, experience or horror story where you have e.g. shutdown a whole data center with Remote Execution.

Share Comments