Why Yelp or Proximity Server?

Proximity servers are used to discover nearby attractions like places, events, etc. If you haven’t used yelp.com before, please try it before proceeding (you can search for nearby restaurants, theaters, etc.) and spend some time understanding different options that the website offers. This will help you a lot in understanding this chapter better.

Requirements and Goals of the System

What do we wish to achieve from a Yelp like service? Our service will be storing information about different places so that users can perform a search on them. Upon querying, our service will return a list of places around the user.
Our Yelp-like service should meet the following requirements:
Functional Requirements:

1. Users should be able to add/delete/update places;
2. Given their location (longitude/latitude), users should be able to find all nearby places within a given radius;
3. Users should be able to add feedback/review about a place. The feedback can have pictures, text, and a rating;

Non-Functional Requirements:

1. Users should have a real-time search experience with minimum latency;
2. Our service should support a heavy search load. There will be a lot of search requests compared to adding a new place;

What is Facebook’s Newsfeed?

A Newsfeed is the constantly updating list of stories in the middle of Facebook’s homepage. It includes status updates, photos, videos, links, app activity, and “likes” from people, pages, and groups that a user follows on Facebook. In other words, it is a compilation of a complete scrollable version of your friends’ and your life story from photos, videos, locations, status updates, and other activities.
For any social media site you design - Twitter, Instagram, or Facebook - you will need a newsfeed system to display updates from friends and followers.

Requirements and Goals of the System

Let’s design a newsfeed for Facebook with the following requirements:
Functional Requirements:

1. Newsfeed will be generated based on the posts from the people, pages, and groups that a user follows;
2. A user may have many friends and follow a large number of pages/groups;
3. Feeds may contain images, videos, or just text;
4. Our service should support appending new posts as they arrive to the newsfeed for all active users;

Non-Functional Requirements:

1. Our system should be able to generate any user’s newsfeed in real-time - maximum latency seen by the end user would be 2s;
2. A post shouldn’t take more than 5s to make it to a use’s feed assuming a new newsfeed request comes in;

What is a Web Crawler?

A web crawler is a software program which browses the World Wide Web in a methodical and automated manner. It collects documents by recursively fetching links from a set of starting pages. Many sites, particularly search engines, use web crawling as a means of providing up-to-date data. Search engines download all the pages to create an index on them to perform faster searches.
Some other uses of web crawlers are:

• To test web pages and links for valid syntax and structure;
• To monitor sites to see when their structure or contents change;
• To maintain mirror sites for popular Web sites;
• To build a special-purpose index, e.g., one that has some understanding of the content stored in multimedia files on the Web;

Requirements and Goals of the System

Let’s assume we need to crawl all the web:

• Scalability: Our service needs to be scalable such that it can crawl the entire Web and can be used to fetch hundreds of millions of Web documents;
• Extensibility: Our service should be designed in a modular way with the expectation that new functionality will be added to it. There could be newer document types that need to be downloaded and processed in the future;

What is Twitter Search?

Twitter users can update their status whenever they like. Each status (called tweet) consists of plain text and our goal is to design a system that allows searching over all the user tweets.

Requirements and Goals of the System

We need to design a system that can efficiently store and query tweets:

• Let’s assume Twitter has 1.5 billion total users with 800 million daily active users;
• On average Twitter gets 400 million tweets every day;
• The average size of a tweet is 300 bytes;
• Let’s assume there will be 500M searches every day;
• The search query will consist of multiple words combined with AND/OR;

What is a Rate Limiter?

Imagine we have a service which is receiving a huge number of requests, but it can only serve a limited number of requests per second. To handle this problem we would need some kind of throttling or rate limiting mechanism that would allow only a certain number of requests so our service can respond to all of them. A rate limiter, at a high level, limits the number of events an entity (user, device, IP, etc.) can perform in a particular time window. For example:

• A user can send only one message per second;
• A user is allowed only three failed credit card transactions per day;
• A single IP can only create twenty accounts per day;

In general, a rate limiter caps how many requests a sender can issue in a specific time window. It then blocks requests once the cap is reached.