Build API's You Won't Hate: Book Summary

By passing on some best practices and general good advice you can hit the ground running with API development, combined with some horror stories and how they were overcome/avoided/averted. This book will discuss the theory of designing and building APIs in any language or framework, with this theory applied in PHP-based examples.

Chapter 1: Useful Database Seeding

  • The process of populating a database is known as "seeding";
  • Instead of entering your real data, it is far easier to use “dummy data” to test if the schema is appropriate for your API application.
  • Break your database seeders down into logical groupings. This does not need to be “one seeder-per-table”, but it can be.
  • It can be difficult to wipe a database when foreign keys constraints are enforced, so in that scenario your seeder should run DB::statement('SET FOREIGNKEYCHECKS=0;'); before the truncation of the tables and DB::statement('SET FOREIGNKEY- CHECKS=1;'); afterwards to enable the checks again.

Chapter 2: Planning and Creating Endpoints

  • Try thinking of everything your API would need to handle.
  • Not every single action requires its own endpoint. e.g If you need to “favorite” a Place, just send is_favorite to that endpoint and you’ve favorited it.
  • Use subresources for getting data e.g
    • GET /places/X/checkins - Find all the checkins for a specific place.
    • GET /users/X/checkins - Find all the checkins for a specific user.
    • GET /users/X/checkins/Y - Find a specific checkin for a specific user.
  • Use unique identifiers instead of incrementing id's when building the URL for a resource eg /users/da12axwe instead of /users/2
  • PUT is used if you know the entire URL beforehand and the action is idempotent. Idempotent is a fancy word for “can do it over and over again without causing different results”.
  • Pick plural for API endpoints as it is the most obvious:
  • To follow the REST Approach, create API endpoints with nouns eg /users/1/messages instead of /users/1/send-message

Chapter 3: Input and Output Theory.

  • Input is an HTTP request and output is a response
  • Don't mix JSON and Form Data in an HTTP request
  • Most modern API's support JSON
  • In XML, everything is considered to be a string
  • A response should either contain a chunk of data or an error.

Chapter 4: Status Codes, Errors and Messages.

  • If something goes wrong, however, you want to let people know what is wrong using two simultaneous approaches:
    1. HTTP status codes
    2. Custom error codes and messages
  • Http Status Codes
    • 2xx is all about success
    • 3xx is all about redirection
    • 4xx is all about client errors
    • 5xx is all about server errors.
  • It can be tempting to try and squeeze as many error codes in as you can, but I would advise you to try and keep it simple. You won’t unlock any achievement badges for using them all.
  • Error codes are usually strings or integers that act as a unique index to a corresponding human-readable error message with more information about what is going wrong.
  • Some folks will try to use HTTP status codes exclusively and skip using error codes because they do not like the idea of making their own error codes or having to document them, but this is not a scalable approach.
  • You can use error codes to make an application respond intelligently to failure of something as basic as a posted Twitter status.
  • Your API should return multiple errors rather than just returning the first error.
  • An error object MAY have the following members:
    • "id" - A unique identifier for this particular occurrence of the problem.
    • "href" - A URI that MAY yield further details about this particular occurrence of the problem.
    • "status" - The HTTP status code applicable to this problem, expressed as a string value.
    • "code" - An application-specific error code, expressed as a string value.
    • "title" - A short, human-readable summary of the problem. It SHOULD NOT change from occurrence to occurrence of the problem, except for purposes of localization.
    • "detail" - A human-readable explanation specific to this occurrence of the problem.
    • "links" - Associated resources, which can be dereferenced from the requested document.
    • "path" - The relative path to the relevant attribute within the associated resource(s). Only appropriate for problems that apply to a single resource or type of resource.

Chapter 5: Endpoint Testing

  • You have to set up your tests as early as possible so you actually bother using them, otherwise, they become the next thing that just never gets done.
  • With an API, there are a few things to test, but the most basic idea is, “when I request this URL, I want to see a foo resource”, and “when I throw this JSON at the API, it should a) accept it or b) freak out.”
  • API Testing with PHP is done using a BDD framework called Behat
  • It is strongly recommended that you try to test your own API (brand new or existing) too, as this is the most value you could get from the book.
  • Writing tests first before development is a great way to go. (TDD).

Chapter 6: Outputting Data

  • Your controller should definitely not have ORM/Query Builder logic scattered around the methods.
  • Use a trasformer to change your Model data into the format which is ready to be converted into a JSON output
  • Using Pagination is a performance boost when outputting a resource which has many items.

Chapter 7: Data Relationships.

  • Relationships for your API output do not need to be directly mapped to database relationships.
  • One very simplistic way to approach related data is to offer up new URLs for your API consumers to digest.
  • An API needs the flexibility, and making subresources the only way to load related data is restrictive for the API consumer.
  • Another approach to related data is to provide an array of foreign keys in the output.
  • Instead of just putting the foreign keys into the resource, you can take things a step further and sideload the data, which is also recommended by JSON-API.
  • Instead of flattening the entire response to top-level collections and losing the obvious context of the data, embedding data leaves it in the structure a client would expect.

Chapter 8: Debugging

  • Working with an API, you are mostly just working with requests and responses, but you need to initiate these requests in a repeatable way, often with full control over all of the HTTP headers, body content, etc.

  • Command Line Debugging

    • Debugging via the command-line by using tools like curl is a great option for some. They tout the benefits of being able to do it from inside a network firewall.
    • The CLI is a pain in the backside when you have a lot of endpoints with lots of potential values. Please, if you take yourself, your API, or your job as a developer seriously, do not do this.
  • Browser Debugging
    • Working in the browser is a great way to do things, and developers are fairly used to it. Sadly, most browsers can only really handle GET and POST requests by default, and a RESTful API requires PUT, DELETE, PATCH, etc., too.
    • HTTP clients like Postman allow you to format your HTTP request through a convenient GUI, choosing the HTTP verb, adding headers, entering a body, etc., then present the HTTP response to you with formatting or in source view if you prefer.
    • Anything to do with a slow page return, silent fails, unexpected results, etc., needs more information, and to do that you probably need another extension. Clockwork in chrome is good for this scenario
  • Network Debugging
    • Sometimes you need to debug network activity to find out what is actually happening by spying on the request and getting the response.

Chapter 9: Authentication

  • Authentication allows APIs to track users, give endpoints user context (“find all of my posts”), limit users’ access to various endpoints, filter data, or even throttle and deactivate accounts
  • One concern with just leaving all the security up to the network is that, if the network is breached, then hackers would be able to do rather a lot of damage.
  • Different approaches to authentication.
    • Approach 1: Basic Authentication
      • HTTP Basic authentication (BA) implementation is the simplest technique for enforcing access controls to web resources because it doesn’t require cookies, session identifier and login pages. Rather, HTTP Basic authentication uses static, standard HTTP headers which means that no handshakes have to be done in anticipation.
      • This is easy to implement and it works on any browser
      • The main con is that Passwords can be stored by the browser, meaning a honeypot of user data is sitting around waiting to be gobbled up
      • HTTP Basic Auth may be a good fit for a relatively unimportant internal API, which needs some basic protection and needs to be implemented quickly but certainly is not any good for anything that handles money, air traffic, or nuclear weapons.
    • Approach 2: Digest Authentication
      • Instead of transmitting passwords in plain text, it will calculate an MD5 hash and send that. Unlike the Base64-based passwords used in the basic auth, MD5 is a one-way hash meaning you cannot simply take the hash and calculate the original password without trying out a lot of different combinations.
      • The nonce is a unique number, which can contain (but should not be only) a timestamp. This helps to avoid replay attacks as the same hash will not be usable later on.
      • The main advantage of this is The use of nonce helps negate rainbow table attacks
      • Just like basic auth, It's still insecure over HTTP and passwords can be stored in the browser
    • Approach #3: OAuth 1.0a
      • OAuth provides a method for clients to access server resources on behalf of a resource owner (such as a different client or an end-user). It also provides a process for end-users to authorize third-party access to their server resources without sharing their credentials (typically, a username and password pair), using user-agent redirections.
      • For example, when Twitter swapped from HTTP Basic integration to OAuth 1.0 it meant that instead of third-parties (iPhone apps, other websites, CMSs, whatever) asking end-users to enter their username and password (which would be saved somewhere in plain text), the third party could redirect the user to the Twitter website, get them to log in, and have them come back to their service to save a special token, instead of saving a password. OAuth 1.0a called these tokens an ‘OAuth Token’ and an ‘OAuth Token Secret’.
      • There was also xAuth (which is still OAuth 1.0a), designed for mobile and desktop applications that do not have easy access to a browser.
      • In the end, if you got the OAuth Token and Secret, you would place the OAuth Token in the request as a header and use the secret to sign the signature, which would encrypt the request and make the whole thing nice and secure.
      • This is very secure even without SSL but given that tokens never change, a hacker could still guess the token if he sniffed enough packets.
    • Approach #4: OAuth 2.0
      • OAuth 2 dropped the secret token, so users are simply getting an access token now.
      • It requires SSL to handle the encryption of the request.
      • You should always try to use the Authorization header to send your tokens whenever possible. The query-string is secured when using SSL, but unless they are intentionally blocked then access tokens could start turning up in server logs and various other places. Also, browsers will store the full URL (including query-string) in history. This could easily compromise the integrity of users security if their computer is stolen or if a sibling decides to play a prank.
      • OAuth 2.0’s access tokens will (can) expire after an arbitrary period of time, which is defined by the OAuth server.
      • When you request an access token, you will usually be provided with a refresh token and an expiry offset, which is the number of seconds until the token expires.
      • One further massive benefit OAuth 2.0 provides over OAuth 1.0a is the ability to have multiple (even custom) grant types.
    • Other Approaches
  • Many assume that the OAuth 2.0 server should be part of their API server. While it certainly could, it definitely does not need to be.
  • If you can think of it, you can make a custom grant type for Auth 2.0

Chapter 10: Pagination

  • Pagination is a sequence of numbers assigned to pages in a book or periodical.
  • In terms of APIs, pagination is Any way you want to go about splitting up your data into multiple HTTP requests, for the sake of limiting HTTP Response size.
  • When you take the limit/number parameter from the client, you absolutely have to set an upper bound on that number, make sure it is over 0 and depending on the data source you might want to make sure it is an integer as decimal places could have some interesting effects.
  • Another tricky issue with the count-everything-then-pick-which-page- number approach is that if a new item is added between HTTP requests, the same content can show up twice.
  • The main trouble with this method is the SELECT count(*) that is required to find out the total, which can be a very expensive request.
  • Another common pagination method the use of “cursors”, sometimes called “markers”. A cursor is usually a unique identifier, or an offset so that the API can just request more data.
  • Every pagination system needs to respond to an empty collection.
  • Some argue that pagination is metadata, and metadata should be kept out of the response.
  • Inserting pagination data into the API response in a 'pagination' namespace is very common and has been my go-to solution for years. I would slot it next to the 'data' namespace, and that makes it very easy for clients who a) cannot read those HTTP headers and b) do not know to look there.
  • Every API should choose its approach to pagination itself. Using this specific header does not make it “more RESTful” regardless of how many people seem to think that is the case. It just makes it more “HTTPish” than defining your own pagination metadata.

Chapter 11: Documentation

  • Regardless of whether you decide to keep an API private or release it to the general public, documentation is incredibly important.
  • Even if the API is in use internally, without a single source of regularly updated documentation for your API, you will be answering nonstop questions from anyone using it.
  • Types of Documentation:
    • API Reference
      • The “API Reference” is sometimes referred to as “Endpoint Reference”. This is essentially a list of all endpoints (and their associated HTTP Methods), descriptions of what they do and a list of all arguments that can be passed, with descriptions about what values work and in what format those values could be.
    • Sample Code
      • “Sample Code” is generally just a case of building one or two libraries or code packages in different languages, documenting their API with tools like phpDocumentor, and showing lots of common scenarios covering the basics of how that code works.
      • Despite your own personal preferences, please, for the love of every god in the world, make your sample code look as good as you can in each language.
    • Guides or Tutorials
      • Take a subject like “Authentication” and talk through it like a blog post. Images, diagrams, code examples of the libraries handling various situations in one or multiple languages using tabs, etc.
  • API Blueprint has a very easy to understand set of Getting Started instructions, which has a series of approaches to creating your documentation with various languages and tool combinations.

Chapter 12: HATEEOAS

  • HATEOAS stands for Hypermedia as the Engine of Application State, and is pronounced as either hat-ee-os, hate O-A-S or hate-ee-ohs.
  • However, you want to try and say it, it basically means two things for your API:
    1. Content negotiation
      • A good API would simply have /statuses/210462857140252672. This has the dual benefit of letting the API respond with a default content type or respecting the Accept header and either outputting the request content type or spitting out a 415 status code if the API does not support it.
      • URIs are not supposed to be a bunch of folders and file names and an API is not a list of JSON files or XML files. They are a list of resources that can be represented in different formats depending on the Accept header, and nothing else.
    2. Hypermedia controls
      • Without hypermedia controls, an API is not RESTful
      • Hypermedia controls are just links to other content, relationships, and further actions. These allow a consumer to browse around the API, discovering actions as it goes.
      • The general underlying theme of hypermedia is that an API should be able to make perfect sense to an API client application and the human looking at the responses, entirely without having to hunt through documentation to work out what is going on.
      • The acronym “URI” is often used to refer to only content after the protocol, hostname, and port (meaning URI is the path, extension and query string), whilst “URL” is used to describe the full address.
      • This is literally a case of shoving some links into your data output. However you chose to do that, it can be part of your “transformation” or “presentation” layer.

Chapter 13: API Versioning

  • Once you have built your wonderful new API, at some point it will need to be replaced or have new features added. Sadly, there is no real consensus on what approach is the best, but instead, there are pros and cons to each approach.
  • Different approaches to API Versioning
    • Approach #1: URI
      • Throwing a version number in the URI is a very common practice amongst popular public APIs.
      • Essentially, all you do here is put a ‘v1’ or ‘1’ in the URL, so that the next version can be easily changed. e.g https://api.example.com/v1/places
      • Consider making each version its own code base. This means the code is totally separate, executed separately, with different web server vhosts or maybe even on different servers.
      • Pros
        • Incredibly simple for API developers
        • Incredibly simple for API consumers
        • Copy-and-pasteable URLs
      • Cons
        • Not technically RESTful
        • Tricky to separate onto different servers
        • Forces API consumers to do weird stuff to keep links up-to-date
    • Approach #2: Hostname
      • Some API developers try to avoid the issues with server setup found with putting the version in the URI and simply put the version number in the hostname (or subdomain) instead: e.g https://api-v1.example.com/places
      • Pros
        • Incredibly simple for API developers
        • Incredibly simple for API consumers
        • “Copy-and-paste-able” URLs
        • Easy to use DNS to split versions over multiple servers
      • Cons
        • Not technically RESTful
        • Forces API consumers to do weird stuff to keep links up-to-date
    • Approach #3: Body and Query Params
      • If you are going to take the URI version out of the URL, then one of the two other places to put it is the HTTP body itself: POST /places HTTP/1.1 Host: api.example.com Content-Type: application/json { "version" : "1.0" }
      • Another approach is to move the parameter to the query string, but now the API version is in the URL again! e.g api.example.com/places?version=1.0
      • Pros
        • Simple for API developers
        • Simple for API consumers
        • Keeps URLs the same when param is in the body
        • Technically a bit more RESTful than putting version in the URI
      • Cons
        • Different content types require different params, and some (like CSV) just do not fit.
        • Forces API consumers to do weird stuff to keep links up-to-date when the param is in the query string.
    • Approach #4: Custom Request Header
      • So if the URL and the HTTP body is a bad place to put API version information, where else is left? Well, headers of course! GET /places HTTP/1.1 Host: api.example.com BadApiVersion: 1.0
      • Pros
        • Simple for API consumers (if they know about headers)
        • Keeps URLs the same
        • Technically a bit more RESTful than putting version in the URI
      • Cons
        • Cache systems can get confused
        • API developers can get confused (if they do not know about headers)
    • Approach #5: Content Negotiation
      • The Accept header is designed to ask the server to respond with a specific resource in a different format. e.g Accept: application/vnd.github.v3+json
      • Pros
        • Simple for API consumers (if they know about headers)
        • Keeps URLs the same
        • HATEOAS-friendly
        • Cache-friendly
        • Sturgeon-approved
      • Cons
        • API developers can get confused (if they do not know about headers)
        • Versioning the WHOLE thing can confuse users (but this is the same as all previous approaches)
    • Approach #6: Content Negotiation for Resources
      • Generally accepted to be the proper HATEOAS approach, content negotiation for specific resources using media types is one of the most complex solutions, but is a very scalable way to approach things e.g Accept: application/vnd.github.user+json; version=4.0
      • Pros
        • HATEOAS-friendly
        • Cache-friendly
        • Keeps URLs the same
        • Easier upgrades for API consumers
        • Can be one code base or multiple
      • Cons
        • API consumers need to pay attention to versions
        • Splitting across multiple code bases is not impossible, but it is hard
        • Putting it in the same code base leads to accidental breakage if transformers are not versioned
    • Approach #7: Feature Flagging
      • Facebook does not version their entire API with simple numbers like anybody else does. They do not version their resources, and they do not allow you to request different versions with headers, parameters or anything else.
      • They essentially make a custom version for each single client application. The way this works is there are various feature flags, which they call “migrations”. They put out a migration every few months, write a blog, email API developers about it, and ask those developers to log into the developer area on the Facebook platform to manage their application.
      • Generally speaking, the Feature Flag solution is the easiest for API consumers if the changes happen to hit a part of the API they do not care about. They do not need to be scared of changing to an entirely new version of the API, they know their code will work, and things seem safer. If they do require changes then... well a few if statements never really hurt anyone.
  • The real truth is that all of the approaches are annoying in some ways, or technically ‘unRESTful’ in some respects, or difficult, or a combination of it all. You have to pick what is realistic for your project in both the difficulty of the implementation and the skill/knowledge level of your target audience.