CloudMaker

Configuration Format

CloudMaker uses YAML as a configuration file format. It supports all properties of Ubuntu CloudInit but extends it to support composability of configurations, setting of environment variables, reading values from shell scripts, and launching EC2 instances.

Composability

Using DRY as a guiding principle CloudMaker allows you to import config files into each other. This functionality is implemented as a deep merge with the last configuration winning. Array merging is done via concatenation with duplicates removed. An example should serve to illustrate this.

cloud-maker:
  foo: bar
  another_value: tom
  numbers:
  - 1
  - 2
  - 3

cloud-maker:
  foo: baz
  another_value: bob
  numbers:
  - 1
  - 4
  - 5

cloud-maker:
  foo: qux
  numbers:
  - 6

import:
- example1.cm.yaml
- example2.cm.yaml

Given example1.cm.yaml and example2.cm.yaml then import-example.cm.yaml will be identical to the following:

cloud-maker:
  foo: qux
  another_value: bob
  numbers:
  - 1
  - 2
  - 3
  - 4
  - 5
  - 6

Import URIs

You can specify the location of CloudMaker configs in multiple ways:

• github://user/repo/path/to/file.cm.yaml which will use the GitHub API to fetch the file.
  Note If it is a private repository you will be prompted for your credentials the first time you access it. This will generate an OAuth token that will be saved and used in the future. You can revoke access to the token at anytime from the GitHub website.
• http(s)://www.yourdomain.com/path/to/file.cm.yaml which will fetch the file via HTTP.
• /local/path/to/file.cm.yaml Will access the path from your local filesystem.
  Note If you use a relative path it will be relative to the directory you run cloud-maker from.

Custom Properties

Custom properties are passed to the instance via the cloud-maker property. cloud-maker is a hash of values. By default each of these values will be set as an environment variable on the launched instance. Custom properties can either be set as key: value pairs, or as key: configuration_hash as described below. All configuration hash properties are optional and will have their default values used if not specified.

• default: Set a default value for the property. (default: blank)
• environment: Should the property be available as an environment variable? (default: true)
• required: This property must be set for the instance to launch. (default: false)
• description: Shown by the CLI when prompting for a value. (default: blank)
• value: The value of the property (if you just do key: value you are actually setting this). (default: blank)

Shell Execution

One of the guiding principlals of CloudMaker is that it should be able to work with existing tools. To facilitate that it provides a custom node type that executes a local shell command and is replaced by the value. To use it prefix your shell command with !shell-script as in the example below. This example sets an environment variable on the target instance with the local username of the person who launched the instance.

cloud-maker:
  launched_by: !shell-script whoami

EC2 Properties

EC2 properties are all passed in via the cloud-maker hash. These properties will be used locally by CloudMaker to launch the instance.

Properties List

• ami The Amazon AMI ID for the instance. (required)
• instance_type The Amazon instance type, eg. m1.small. (required)
• availability_zone The Amazon availability zone, eg. us-east-1a (required)
• s3_archive_bucket The S3 bucket where information about the instance should be stored. (required)
• key_pair The name of an Amazon key pair, so you can actually login to the instance.(default: "")
• elastic_ip An elastic IP address you control that you would like to associate to the instance. (default: "")
• security_group The Amazon EC2 security group to launch the instance with. (default: "default")
• iam_role The IAM instance profile name or ARN you would like to use. (default: "")
• cname A dns entry you would like to CNAME to this instance. (default: "")

Caveats

• Elastic IPs need to have been created in advance.
• CNAMEs must not already exist in Route 53 and will not be removed when the instance is terminated.

Command Line Interface

The command line interface cloud-maker allows you to launch instances, retrieve information about them, and to terminate them. Data about an instance is stored in an S3 bucket (set using the s3_archive_bucket property). All interactions with the CLI take the form of cloud-maker <command> [arguments] [options].

Launch

Launch is used to create a new EC2 instance.

Usage

cloud-maker launch <config_file.yaml>

Options

• --aws-access-key-id=AWS_ACCESS_KEY_ID Your AWS access key id
• --aws-secret-access-key=AWS_SECRET_ACCESS_KEY Your AWS secret access key
• [--set=key:value [key:value]...] Set parameters in the CloudMaker config
• [--tags=key:value [key:value]...] Set tags for EC2, merges with and overrides the tags property in the CloudMaker config
• [--yes] Answer yes, or accepts default values to any confirmation prompts automatically.

Info

Info retrieves the archived information that was stored when an instance was launched.

Usage

cloud-maker info <AWS instance ID>

Options

• --aws-access-key-id=AWS_ACCESS_KEY_ID Your AWS access key id
• --aws-secret-access-key=AWS_SECRET_ACCESS_KEY Your AWS secret access key

User Data

User Data generates CloudInit user data for an instance. It can be used as a dry run or if you prefer to actually launch the instance using the Amazon console or another tool.

Usage

cloud-maker user_data <config_file.yaml>

Options

• [--set=key:value [key:value]...] Set parameters in the CloudMaker config
• [--tags=key:value [key:value]...] Merges with and overrides the tags property in the CloudMaker config. Since there is no actual EC2 instance created this is just a convenience setter for the tags custom property.

Terminate

Terminate will terminate an AWS instance after first displaying the info about how it was launched.

Usage

cloud-maker terminate <AWS instance ID>

Options

• --aws-access-key-id=AWS_ACCESS_KEY_ID Your AWS access key id
• --aws-secret-access-key=AWS_SECRET_ACCESS_KEY Your AWS secret access key

Notes

The guiding principle behind CloudMaker is to keep things simple and modular. The small piece of the puzzle that it tries to solve is launching an instance and pushing an initial configuration to it. If you see something missing suggestions on improvements are welcome, however where such improvements could be better made by providing a hook for integration with another tool that will be the preferred solution.

Suggestions

• Ubuntu CloudInit provides the ability to pull shell scripts via HTTP and execute them at instance launch. This provides an excellent mechanism for doing customized build and configuration.
• There is a tool named ARX that allows an executable script to be combined with a tarball of files making it easy to distribute packages of files with a built-in build script. These work very well as included scripts.

Troubleshooting

By default it can be difficult to get good information about what may have gone wrong with a build process. Add the following to your CloudMaker configuration file generate a build log at /var/log/cloud-init-output.log.

output:
  # generate a detailed log of the cloud init
  all: '| tee -a /var/log/cloud-init-output.log'