First things first.
What is EGF2? It is a scalable graph oriented back-end API framework.
Intro and a Bit of History
I feel that a short explanation on the motivation for EGF2 is necessary. Feel free to skip the introduction if in hurry or not interested to get to the details of the framework.
3 years ago we started on a big and ambitious project in the US healthcare domain. The primary goal of the project was to be able to collect data about a patient from various sources (hospitals, labs, insurers, etc) and present a unified health record. In addition to that, all participants of the healthcare process were supposed to have an app or several to make the platform more social. For example, we did an app (iOS and Android) for providers (doctors) allowing them to follow patients of interest, see patients’ care teams, see updates about patients in a timeline, communicate between each other using secure IM feature, etc, etc.
As a basis for the data model we took FHIR standard. Practically all entities from the FHIR were ported, additional fields and relations were added. It was clear from the very beginning that to accommodate such complex and rapidly changing data model we need to have proper data abstraction in place for all parts of the system. We decided that graphs give us what we need. Facebook Graph API has served as a primary source of inspiration for our exposed APIs.
From the very start back-end requirements were as tough as they get. The system should be able to process giga / tera bytes of data fast, should support hundreds of thousands active users, should be fault tolerant and scalable.
The system we have created is now powering Operating System for Precision Medicine for our affiliate Flow Health.
A couple of years ago we realized that there is a framework bit we have done with the Flow Health project so we took the framework part and did it a bit differently using another tech stack. Since then we’ve been using this framework (we call it EGF, which stands for EigenGraph Framework) for all of our projects. Our own utilization of EGF has shown that it is really convenient and allows us to have large part of back-end in place with very little effort.
A couple of months ago we decided to intensify our efforts related to EGF further and we started working on EGF2 - a version of framework that we plan to make open source.
EGF2 Architecture is presented on the diagram below. Yellowish area represents services and tools that are not exposed to the Internet. Arrows go from services that make requests or otherwise push data to other participants.
EGF2 is comprised of a bunch of services (I could probably call them micro-services even though while working on them we were not consciously following the micro-services paradigm). I will talk about each of them in turn below, but before we get there I would like to clarify a couple of things.
EGF2 provides graph oriented API. The way API endpoints are structured is very similar to Facebook Graph API. We felt strongly that graph API is beneficial for projects with broad and non trivial data models, heavy on relations between entities. Graph API allows us to declare a bunch of endpoints in the beginning and never really add new ones, no matter what we do with the data model (expand and modify it, that is). And we liked what Facebook was doing with their Graph API a lot, have to admit that.
EGF2 is not a web framework, like Ruby on Rails or Python Django or whatever other web framework out there. It is a pure API back-end framework built with scalability and fault tolerance in mind. We provide client side libraries for iOS, Android and web though. In other words, there is no template generation and client side code within EGF2 deployments.
That’s where client requests come in and where graph API endpoints are exposed. It is a stateless service with the following primary responsibilities:
- Get client requests
- Control who the request is coming from, e.g. authorize requests. In order to understand who has made the request client-api calls auth service with an auth token that was passed along with request.
- Perform access control for a request. ACL component of the client-api is capable of checking whether a user with a particular role is allowed to do a particular graph action, i.e. creating an object or an edge, modifying an object, removing an object or an edge. Info on who can do what is gathered in graph config which is reused by all services.
- Perform object field validations. Field level restrictions are specified in graph config as well.
- If all is well and request is valid, authorized and allowed client-api talks to client-data service to perform necessary data manipulations.
Due to the fact that client-api service is fully stateless it can be scaled (or better still autoscaled with AWS ASG, for example) easily by adding more instances, if necessary. All public endpoints exposed by the EGF2 are configured and channelled via a single entry point, usually deployed with NGINX. Having a single entry point allows the framework to fully close access from Internet to all services by default. Hidden behind a firewall are not only services but other parts of infrastructure, like DB, ElasticSearch, caches, etc, to be precise.
This service provides access to the data of the system. It exposes almost the same graph API as client-api does (it doesn’t provide expand feature though). client-data is not available from Internet, it is visible for other framework services.
client-data is the only way data can be manipulated and / or retrieved on demand in the system. All services use it, nobody has direct access to a DB. client-data uses DB and (optionally) caching service to serve data needs.
client-data is fully stateless and can be scaled easily by adding more instances running the service. client-data hides details about what DB is being used and provides unified data access experience for other services. We currently support RethinkDB and are working on supporting Cassandra (will be ready in 1-2 months) as a data storage mechanism.
There is a small library that simplifies access to client-data for all services.
A service that is responsible for user authentication / registration features. It is a fully stateless service and can be scaled easily. auth provides a set of endpoints related to user registration and sign in. It also provides internal endpoints for other services, in particular client-api.
This is where asynchronous business logic lives. This service is fully stateless and can be scaled easily. It listens to the system queue, gets Event objects and reacts to them in different ways. Complex business logic lives here. client-api may also contain some business logic processing in case it is important to make sure that changes were completed when a request returns OK to the client. If that is not a requirement processing is performed in logic.
logic has a way of easily add particular business rules implementation, it simplifies and organizes business logic in the system. When a system is built with EGF2 logic is the place where good part of custom code lives.
A fully stateless scalable service that is responsible for synchronizing ElasticSearch cluster with the changes in the data. Listens to the changes queue. In addition to the graph oriented API EGF2 exposes search endpoint that allows to leverage full power of ES cluster to do various kinds of searches, filterings, sortings, etc on the data. More info on the way we use ES will be provided in the following posts.
Stateless, scalable service that is responsible for sending out notifications, be it email, SMS, WebSockets or mobile platform pushes. Template based, listens to the changes queue.
It is getting repetitive, but file is a stateless scalable service that handles file upload / download tasks. It provides some external and internal endpoints.
The same words about stateless, this service is for handling long hard tasks that may tax the system. Things like reports, exports, imports, etc are handled with this service.
These are the most important services that we have in EGF2.
|DB||RethinkDB or Cassandra|
|System event bus||RethinkDB changes feeds (only for small deployments, not really scalable) or Kafka or AWS Kinesys|
|File Storage||AWS S3|
|Web framework used in services||Restify|
As you can see, we list AWS Kinesys as an option for event bus. With Kinesys it would be a shame not to have AWS Lambda deployment option for services that listen to the event bus and does not provide internal or external endpoints. Some services can even have internal endpoints and still deployed as AWS Lambda functions. EGF2 supports service deployment as AWS Lambda functions.
Here goes the list:
- Every service in the framework is scalable separately, all of them are fully stateless.
- No need to bother with DB level storage details. Objects and edges are added to the system via configuration, no need for migrations unless they are really needed.
- We expose graph oriented API which makes life easier in most cases, especially for non-trivial systems.
- Business logic is separated from user request processing.
- All changes in the system are wrapped in Event objects and pushed through the system event bus. Events are preserved in the DB layer thus guaranteeing zero data loss.
- The last, but not the least, using EGF2 it is possible to build scalable or big data systems with much less effort than without it
Coming Next - more details on graph API and other endpoints provided by EGF2.