Protect your API from three dimensions
API has become a foundational element of today’s app-driven world. Many companies including Amazon have adopted the API-first approach. It views the role of APIs as independent products, rather than integration solutions with other systems.
By design, APIs expose valuable information to the world such as public weather data and private sensitive Personally Identifiable Information (PII) data, and because of this, APIs have become new targets for hackers. Without secure APIs, building successful products would be impossible.
This article is supposed to be a wake-up call if you haven’t applied any security strategy to your API yet. As usual, I will be providing Python implementation for some of the best practices. Securing your API can be viewed as a non-functional requirement (NFR). In common with other NFRs like performance and reliability, it’s hard to define when exactly the API is secure enough.
That’s why many companies would hire a third party to execute an API penetration test to have a fair judgment. In short, the pen tester will try to hack against your API to check for any exploitable vulnerability.
Context of API Security
First things first. Why do we even need to care about API security? API defines a set of operations that the caller can use. If I don’t want this operation to be called, then I can simply exclude it or put it behind a firewall.
To answer this question, we need to know what exactly we need to secure. API security involves several security disciplines: information security, network security, and application security.
Information security is concerned with the protection of sensitive data. Each endpoint is an operation applied to one or more objects. The operation of an object can be accessible to only a group of users with a special role.
People with different roles can sit in the same room sharing the same wifi, thus simply blocking the network is not an optimal solution. A sort of authentication and authorization needs to be done when API receives requests from the client.
The difference between authentication and authorization. Simply put, authentication is the process of verifying who you are. For example, there is an internal endpoint within a company that exposes employees’ salaries.
Authentication verifies you are the employee of this company. Authorization is the process of verifying what specific objects you have access to. In this case, if your identify check passes, it verifies if you have permission to check company PII data.
Network security protects the network and data from breaches and other threats. There are a whole lot of things around this topic, but in the context of API, the most applicable component is TLS protocol (Transport Layer Security) which is to provide communication security.
That’s basically the ‘S’ part of HTTPS protocol. During the TLS handshake, the two parties exchange messages to acknowledge each other and agree on the encryption algorithm and session key they will use. From then on, the communication will be encrypted using the session key.
This is to avoid the man-in-the-middle attack.
It’s worth noting that TLS is used to encrypt data in transit not data at rest. That’s beyond the scope of this article.
Application security ensures that the software is designed to withstand attacks and misuse. For example, hackers could inject script or code into the request and the malicious data can trick the software into executing unintended commands. Applying input validation becomes crucial.
In the rest of the article, I will list a few practical security tips in Python to safeguard the application.
After knowing what we want to secure in the context of API, let’s jump into some examples which are inspired by this OWASP API Security Top 10 list.
Authentication and Authorization
This is probably the most important principle that we need to follow when it comes to API security. There are several authentication and authorization methods and they vary in complexity and security. But they reach a consensus — the client must provide some kind of credentials related to its identity and the server will compare what was sent to what has been stored.
If the credentials match, the server will create a user session and issue a cookie to the client.
Basic authentication is, well, very basic. The client simply includes their username and password in the request header. It could be in plaintext or encoded into base 64 to save space.
requests.get('https://httpbin.org/basic-auth/foo/bar', auth=('foo', 'bar'))
Basic authentication is the least authentication method because if the traffic is hijacked by a hacker, the hacker can easily get credentials from the request header or perform a brute-force attack where they try different usernames and passwords until they find something works.
An API key is a unique string generated by the server to authenticate clients. It is different from the base64 string in basic authentication in which the string is generated by the client themselves. Once the API client gets the key from the server, they can include it somewhere in the request specified by the provider, which can be query string parameters, request header, or request body.
The process of key generation depends on the provider. For example, NASA Open API requires clients to provide their name and email. And the key is supposed to be passed as a URL parameter.
An API key is more secure than basic authentication. First, it’s long, complex, and randomly generated which makes it harder for hackers to brute-force attack. Besides, the key can contain more information such as expiration time. Even if the key is somehow disclosed, it will soon become invalid after the expiration time.
However, API keys are still not considered secure. Usually, the software uses one algorithm to generate the key and the key might contain user information. Hackers may just guess the algorithm by learning the API clients.
According to the recommendation from Google, it makes sense to require API keys if you want to understand usage patterns in your API traffic, block anonymous traffic, or control the number of calls, but don’t use API keys to authenticate or authorize users.
JWT (JSON Web Token)
JWT is a token-based authentication. The client first authenticates to the API provider with a username and password. The provider generates a JWT and sends it back to the client. Then the client adds the token to the authorization header in the API request.
The token itself has three parts: the
Header section contains the algorithm used for the signature and as well as the token type. The
Payload section contains the token data like the username, token generation date, and expiration date. The
Signature section is the result of
Payload, concatenated and encrypted with a private key.
When the server receives the token, it will decrypt the signature with the private key and compare it with the header and payload. JWT is more secure than API key because 1) the token doesn’t contain sensitive data like passwords, and 2) it’s way harder to decrypt the signature which is signed using HMAC-SHA256.
PyJWT is a Python library that allows you to encode and decode JSON Web Tokens (JWT). It’s mostly used on the server side.
However, there are still some drawbacks to JWT. If a user wants to change their password and if authentication has been performed beforehand, then the token with the previous password will still be valid until expiry. Besides, the user needs to reauthenticate after the token gets expired.
To deal with these challenges, some JWT libraries allow the refresh token mechanisms or force a user to reauthenticate in some cases.
OAuth 2.0 is an authorization standard that allows different services to access each other’s data. A use case is that my application can log in to a user’s Twitter account and retrieve the user’s tweets without knowing the user’s Twitter credential.
Before OAuth, an application would just retrieve your Twitter credentials through the form and do the login on your behalf, which is generally risky because the application can store all the user’s Twitter credentials. With OAuth, the communication between services is token-based.
Here is the abstract Oauth2.0 flow. In general, there are six steps. The application first requests authorization to access Twitter data from the user. If the user authorizes the request, meaning successfully login into Twitter, the application will receive an authorization grant.
Later, the application requests the access token from Twitter by presenting authentication of its own identity and authorization grant. If everything is verified, the application retrieves user resources by presenting an access token.
The actual implementation varies from provider to provider. I wrote an article about building a Twitter login component using NextJS and Flask where I explained the Twitter Oauth flow in detail.
A rate limit is the number of API calls a user can make within a given time period. If this limit is exceeded, the user may be throttled. Any public API is subject to a rate limit because it protects API from excessive use to avoid things like DDoS attacks.
Here is the vulnerability from Zoom that allows an attacker to attempt 1 million Zoom passwords in a matter of minutes due to a lack of a rate limit. In a nutshell, rate-limiting limits access for users to access API based on the threshold set by the API’s provider.
Besides protecting resource usage, rate limiting can be implemented to control data flow. In a distributed system, API loads might not be evenly distributed across processors. Having a rate limit avoids the situation where one processor is overloaded while the other is idle.
Rate limits can also be implemented to control costs. Every request will always generate a cost, and the more requests an API gets, the more costs it will accumulate. Rate limits can play an extremely important role to save costs. And the same goes for the API users.
Many modern API services provide different subscription models based on the rate limit. In that case, different rate limiting can be applied to individual users based on their needs and it will ensure fair use without disrupting others’ access.
One of the Python libraries that implements rate limiting is
ratelimiter. In fact, this library applies a rate limit on any function, not just specific to APIs. You can choose to use it as a decorator or context manager.
from ratelimiter import RateLimiter
@RateLimiter(max_calls=10, period=1)rate_limiter = RateLimiter(max_calls=10, period=1)
for i in range(100):
When a rate limit is in place and triggered, your API should return the following HTTP status code:
429 Too Many Requests . Although there is no standard name for the HTTP headers to inform users of the rate limit, many modern API services including GitHub and Twitter use the following headers.
The description of each header can vary from provider to provider. But in general, they mean:
X-RateLimit-Limit: The maximum number of calls allowed per limit window (can be 1 hour/minute).
X-RateLimit-Remaning: The number of requests remaining in the current limit window.
X-RateLimit-Reset: The time at which the current rate limit window resets in UTC.
Some APIs also use headers like
Retry-After to tell users when they can call it again. So, please find this information in the API documentation.
As a user, what should we do when receiving a 429 status code? We shouldn’t dodge it, but instead, handle it accordingly. There are several options: 1) The response usually contains one of the above headers with the number of seconds you should wait until the next request. Keep in mind that waiting for too long in a task queue is not considered a good practice.
You should instead retry the request at a later time to free up the queue for other things. 2) If the server doesn’t tell you how long to wait, you can use exponential backoff. The client periodically retries a failed request with increasing delays between requests. The delay increases from one, two, four, eight seconds, and longer. Google provides a Retry class in google-api-core which is a helper for retrying functions with exponential back-off.
Another aspect of securing your API is to have full visibility on “who did what, where, and when”? Many enterprises like Google and Slack provide Audit Log API which provides analysis of how your resources are being accessed.
The idea is to give resource owners the ability to query user action. With this API, the resource owner could feed resource access data into an auditing tool. The owner can also monitor for potential security issues or malicious access attempts.
So, if you are a client, think about leveraging Audit log API to build monitoring and alerting systems. If you are the service provider, think about providing such API to your clients and help them be aware of any malicious user patterns.
Input validation is part of the application security. It is to ensure only properly formed data is entering the system, preventing malformed data from persisting in the database or triggering malicious scripts. Input validation should happen as early as possible in the data flow. In the context of API, it should happen as soon as the data is received from the client.
It’s generally a good practice to apply automatic schema validation as soon as the data arrives. Many API frameworks like connexion support OpenAPI specifications. The specification file includes all kinds of validations that will be automatically applied to the request body by the framework.
The returned status code is normally
400: Bad Request with a descriptive error message like
The value is expected to be a positive integer. It helps the client to understand what goes wrong and change their request accordingly.
When an error occurs on either the client’s side or the server’s side, the API response should give some hints regarding the error reason. But API should never leak too much information about the internal server error. For example, status 500 means an internal server error. And this is enough.
Don’t put the actual error reason in the response body, for instance, a specific database error message or memory error. Unlike
400 Bad Request error, this type of message won’t help the client because they can’t do anything with it. The rule of thumb is to ensure that APIs only return as much information as is necessary to fulfill their function.
I hope you find this article useful! The list can definitely go on and on.
Feel free to comment below if you know other API security items everybody needs to be aware of.