Synapse

A transparent service discovery framework for connecting an SOA
2,034
By Igor Serebryany

Synapse is Airbnb's new system for service discovery. Synapse solves the problem of automated fail-over in the cloud, where failover via network re-configuration is impossible. The end result is the ability to connect internal services together in a scalable, fault-tolerant way.

Motivation

Synapse emerged from the need to maintain high-availability applications in the cloud. Traditional high-availability techniques, which involve using a CRM like pacemaker, do not work in environments where the end-user has no control over the networking. In an environment like Amazon's EC2, all of the available workarounds are suboptimal:

  • Round-robin DNS: Slow to converge, and doesn't work when applications cache DNS lookups (which is frequent)
  • Elastic IPs: slow to converge, limited in number, public-facing-only, which makes them less useful for internal services
  • ELB: Again, public-facing only, and only useful for HTTP

One solution to this problem is a discovery service, like Apache Zookeeper. However, Zookeeper and similar services have their own problems:

  • Service discovery is embedded in all of your apps; often, integration is not simple
  • The discovery layer itself is subject to failure
  • Requires additional servers/instances

Synapse solves these difficulties in a simple and fault-tolerant way.

How Synapse Works

Synapse runs on your application servers; here at Airbnb, we just run it on every box we deploy. The heart of synapse is actually HAProxy, a stable and proven routing component. For every external service that your application talks to, we assign a synapse local port on localhost. Synapse creates a proxy from the local port to the service, and you reconfigure your application to talk to the proxy.

Synapse comes with a number of watchers, which are responsible for service discovery. The synapse watchers take care of re-configuring the proxy so that it always points at available servers. We've included a number of default watchers, including ones that query zookeeper and ones using the AWS API. It is easy to write your own watchers for your use case, and we encourage submitting them back to the project.

Example Migration

Let's suppose your rails application depends on a Postgres database instance. The database.yaml file has the DB host and port hardcoded:

production:
  database: mydb
  host: mydb.example.com
  port: 5432

You would like to be able to fail over to a different database in case the original dies. Let's suppose your instance is running in AWS and you're using the tag 'proddb' set to 'true' to indicate the prod DB. You set up synapse to proxy the DB connection on localhost:3219 in the synapse.conf.yaml file. Add a hash under services that looks like this:

---
 services:
  proddb:
   default_servers:
    -
     name: "default-db"
     host: "mydb.example.com"
     port: 5432
   discovery:
    method: "awstag"
    tag_name: "proddb"
    tag_value: "true"
   haproxy:
    port: 3219
    server_options: "check inter 2000 rise 3 fall 2"
    frontend: mode tcp
    backend: mode tcp

And then change your database.yaml file to look like this:

production:
  database: mydb
  host: localhost
  port: 3219

Start up synapse. It will configure HAProxy with a proxy from localhost:3219 to your DB. It will attempt to find the DB using the AWS API; if that does not work, it will default to the DB given in default_servers. In the worst case, if AWS API is down and you need to change which DB your application talks to, simply edit the synapse.conf.json file, update the default_servers and restart synapse. HAProxy will be transparently reloaded, and your application will keep running without a hiccup.

Links