What is Instagram?

Instagram is a social networking service that enables its users to upload and share their photos and videos with other users. Instagram users can choose to share information either publicly or privately. Anything shared publicly can be seen by any other user, whereas privately shared content can only be accessed by the specified set of people. Instagram also enables its users to share through many other social networking platforms, such as Facebook, Twitter, Flickr, and Tumblr.

We plan to design a simpler version of Instagram for this design problem, where a user can share photos and follow other users. The “News Feed” for each user will consist of top photos of all the people the user follows.

Requirements and Goals of the System

We’ll focus on the following set of requirements while designing Instagram:
Functional Requirements:

1. Users should be able to upload/download/view photos;
2. Users can perform searches based on photo/video titles;
3. Users can follow other users;
4. The system should generate and display a user’s News Feed consisting of top photos from all the people the user follows;

Non-Functional Requirements:

1. Our service needs to be highly available;
2. The acceptable latency of the system is 200ms for News Feed generation;
3. Consistency can take a hit (in the interest of availability) if a user doesn’t see a photo for a while; it should be fine;
4. The system should be highly reliable; any uploaded photo or video should never be lost;

Not in Scope: Adding tags to photos, searching photos on tags, commenting on photos, tagging users to photos, who to follow, etc.

What is Pastebin?

Pastebin like services enable users to store plain text or images over the network (typically the Internet) and generate unique URLs to access the uploaded data. Such services are also used to share data over the network quickly, as users would just need to pass the URL to let other users see it.

If you haven’t used pastebin.com before, please try creating a new “Paste” there and spend some time going through the different options their service offers. This will help you a lot in understanding this chapter.

Requirements and Goals of the System

Our Pastebin service should meet the following requirements:
Functional Requirements:

1. Users should be able to upload or “paste” their data and get a unique URL to access it;
2. Users will only be able to upload text;
3. Data and links will expire after a specific time span automatically; users should also be able to specify expiration time;
4. Users should optionally be able to pick a custom alias for their paste;

Non-Functional Requirements:

1. The system should be highly reliable, any data uploaded should not be lost;
2. The system should be highly available. This is required because if our service is down, users will not be able to access their pastes;
3. Users should be able to access their pastes in real-time with minimum latency;
4. Paste links should not be guessable (not predictable);

Extended Requirements:

1. Analytics, e.g., how many times a paste was accessed?
2. Our service should also be accessible through REST APIs by other services;

Why do We Need URL Shortening?

URL shortening is used to create shorter aliases for long URLs. We call these shortened aliases “short links”. Users are redirected to the original URL when they hit these short links. Short links save a lot of space when displayed, printed, messaged, or tweeted. Additionally, users are less likely to mistype shorter URLs.

For example, if we shorten this page through TinyURL:

https://www.educative.io/collection/page/5668639101419520/5649050225344512/5668600916475904/

We would get:

http://tinyurl.com/jlg8zpc

The shortened URL is nearly one-third the size of the actual URL.
URL shortening is used to optimize links across devices, track individual links to analyze audience, measure ad campaigns’ performance, or hide affiliated original URLs.

If you haven’t used tinyurl.com before, please try creating a new shortened URL and spend some time going through the various options their service offers. This will help you a lot in understanding this chapter.

A lot of software engineers struggle with System Design Interviews (SDIs) primarily because of three reasons:

1. The unstructured nature of SDIs, where the candidates are asked to work on an open-ended design problem that doesn’t have a standard answer;
2. Candidates lack experience in developing complex and large scale systems;
3. Candidates did not spend enough time to prepare for SDIs;

Like coding interviews, candidates who haven’t put a deliberate effort to prepare for SDIs, mostly perform poorly, especially at top companies like Google, Facebook, Amazon, Microsoft, etc. In these companies, candidates who do not perform above average have a limited chance to get an offer. On the other hand, a good performance always results in a better offer (higher position and salary) since it shows the candidate’s ability to handle a complex system.

Polling, WebSocket, and Server-Sent Events are popular communication protocols between a client like a web browser and a web server. First, let’s start with understanding what a standard HTTP web request looks like. Following are a sequence of events for regular HTTP request:

1. The client opens a connection and requests data from the server;
2. The server calculates the response;
3. The server sends the response back to the client on the opened request;

Ajax Polling

Polling is a standard technique used by the vast majority of Ajax applications. The basic idea is that the client repeatedly polls (or requests) a server for data. The client makes a request and waits for the server to respond with data. If no data is available, an empty response is returned:

1. The client opens a connection and requests data from the server using regular HTTP;
2. The requested webpage sends requests to the server at regular intervals (e.g., 0.5 seconds);
3. The server calculates the response and sends it back, just like regular HTTP traffic;
4. The client repeats the above three steps periodically to get updates from the server;

The problem with Polling is that the client has to keep asking the server for any new data. As a result, a lot of responses are empty, creating HTTP overhead.