EGF2 File Operations

file subsystem is optional for the framework, but, in our experience, every project needs some form of file upload / download functionality. Getting file service up and running only requires a small bit of configuration. The following endpoints can be used for file operations.

In order to upload a file to EGF S3 first call GET /v1/new_image orGET /v1/new_file passing mime_type and title in query. Request will return File object JSON with a link that should be used for uploading image in File.upload_url field.

file service makes a distinction between a generic file and an image. In addition to upload and download it is possible to request resizes for images.

Client / server interactions:

  • Client sends GET request
  • Server creates new File object, sets mime_type and title. It prepares a short lived S3 link for file uploading and sets it to File.upload_url and also sets File.url link.
  • Server sets File.resizes for image based on kind parameter, returns response to a client.
  • Client uploads file using File.upload_url link
  • Client sets File.uploaded = true and updates the object using regular graph PUT.
  • Server listens to the File.uploaded = true event and
    • sets File.url to a permanent S3 GET link, saves the File object
    • Schedules resizing jobs for all File.resizes entries. Jobs will upload resizing results to S3 and update File object

In addition to mime_type and title specified for GET /v1/new_image request you can send kind parameter with a value, defined in file service config. In config it is possible to define any number of image “kinds” that are necessary for the system. For example, there can be an “avatar” kind with resize set to 40x40 and “product” kind with resize set to 200x200. When a new image creation is requested kind parameter will define what resizes should be applied to the new image.

Note: File.url link is a long lived public S3 URL. Anybody who has access to this URL will be able to copy and share it with anybody else, thus making image publicly available. File URLs are formed using URIs, making them hard to guess though. It is possible to modify file service to control access to S3 URLs as well as access to File objects, which is controlled via ACL now. Let us know if you are interested in this use case.

When a File object is deleted file service picks up deletion event and removes the physical file from S3.

file service also supports a use case when File objects can be created, physical files uploaded to S3 but these files are not connected to any edge or object field. This can happen in a client app UI that allows a bunch of files to be added at once and only after the files are uploaded it allows them to be added to a collection of some sort. Files can be created and uploaded but then user can close the browser tab, thus leaving files in a state that we call “standalone”. Standalone files will be removed from the system automatically within one day from the moment of creation unless they are connected to an edge or an object field. All File objects are initially created as standalone and converted to regular as described above.

To sum up, with file service you get extremely robust file upload / download functionality with ACL access control based on File object. Robustness bit is provided by AWS S3 and the fact that file service is stateless and can be autoscaled easily :-)

comments powered by Disqus