Synapse is Airbnb’s new system for service discovery. Synapse solves the problem of automated fail-over in the cloud, where failover via network re-configuration is impossible. The end result is the ability to connect internal services together in a scalable, fault-tolerant way.
Synapse emerged from the need to maintain high-availability applications in the cloud. Traditional high-availability techniques, which involve using a CRM like pacemaker, do not work in environments where the end-user has no control over the networking. In an environment like Amazon’s EC2, all of the available workarounds are suboptimal:
One solution to this problem is a discovery service, like Apache Zookeeper. However, Zookeeper and similar services have their own problems:
Synapse solves these difficulties in a simple and fault-tolerant way.
Synapse runs on your application servers; here at Airbnb, we just run it on every box we deploy. The heart of synapse is actually HAProxy, a stable and proven routing component. For every external service that your application talks to, we assign a synapse local port on localhost. Synapse creates a proxy from the local port to the service, and you reconfigure your application to talk to the proxy.
Synapse comes with a number of
watchers, which are responsible for service discovery. The synapse watchers take care of re-configuring the proxy so that it always points at available servers. We’ve included a number of default watchers, including ones that query zookeeper and ones using the AWS API. It is easy to write your own watchers for your use case, and we encourage submitting them back to the project.
Let’s suppose your rails application depends on a Postgres database instance. The database.yaml file has the DB host and port hardcoded:
production: database: mydb host: mydb.example.com port: 5432
You would like to be able to fail over to a different database in case the original dies. Let’s suppose your instance is running in AWS and you’re using the tag ‘proddb’ set to ‘true’ to indicate the prod DB. You set up synapse to proxy the DB connection on
localhost:3219 in the
synapse.conf.yaml file. Add a hash under
services that looks like this:
--- services: proddb: default_servers: - name: "default-db" host: "mydb.example.com" port: 5432 discovery: method: "awstag" tag_name: "proddb" tag_value: "true" haproxy: port: 3219 server_options: "check inter 2000 rise 3 fall 2" frontend: mode tcp backend: mode tcp
And then change your database.yaml file to look like this:
production: database: mydb host: localhost port: 3219
Start up synapse. It will configure HAProxy with a proxy from
localhost:3219 to your DB. It will attempt to find the DB using the AWS API; if that does not work, it will default to the DB given in
default_servers. In the worst case, if AWS API is down and you need to change which DB your application talks to, simply edit the
synapse.conf.json file, update the
default_servers and restart synapse. HAProxy will be transparently reloaded, and your application will keep running without a hiccup.