Opportunities of Yelp and Google Places APIs

As the access to the USA company registry is quite limited, you can get your own company database using such means like Yelp API and Google Places API. Of course, every respective company would have an account on Google Places, and most of them on Yelp. Fortunately, we have a solution which will help you not only to gather the list of such companies, but also have them filtered by categories, location, as well as get their description, logo, reviews and other valuable information.

Read more about what data can be provided by Google Places and Yelp here https://www.linkedin.com/pulse/yelp-vs-google-reviews-which-one-best-darnell-montalvo.

Issues

I’ll start with problems we faced. If we had had it when we first worked with Yelp API, it’d have saved us weeks of work. And we wouldn’t have become more stress resistant:) We hope that it’ll help you to save your energy and time.

  1. Yelp API limits.
    Yelp API has limitation of 25 000 requests. But we haven’t found any remarks in the documentation that it is prohibited to use several API keys.
    Yelp API doesn’t provide some data.
    How to get Yelp reviews, company website….? Use Google Places:).

  2. Yelp doesn’t provide some data through API. For example, you can’t get reviews and authors photos, company website and etc. Parsing/scraping reviews from Yelp can lead to getting blocked on the service, so it’s not an option. But these data can be added through Google Places API. We’ll talk about detailed solution in this article.

  3. Minor bugs in Yelp API.
    Before you plan some manupulation with Yelp API, be ready that sometimes it doesn’t work as it’s supposed to according to the documentation. For instance, it doesn’t give adequate results if filtered by postcode. In most cases, it can be handled with other means, but if you do not put some additional time into a plan, it can cause problems.

  4. Pagination.
    Pagination on Yelp is quite tricky. When you exceed the limits and want to start off from the place you left, you’ll need to implement a solution I describe later on.

Yelp API has a problems. For example, they do not give some data. Parsing/scraping reviews from yelp can be blocked by Yelp. But we can fill this data from Google Places

Solution

A few words about Yelp API

I will not dive deep into details of the Yelp API, as it's got a great documentation. I'll show only a few examples and nuances, which are useful in this simple project.

First of all, to work with Yelp API you have to get API keys from this link.

Next, you are to install python-yelp-v2 package via pip package manager:

pip install python-yelp-v2

Now, let's create a function that returns an instance of Yelp object. We'll use this function throughout this article.

import yelp

YELP_API_KEY = {
    'consumer_key': '',
    'consumer_secret': '',
    'access_token_key': '',
    'access_token_secret': '',
}

def get_yelp_instance(yelp_key):
    yelp_instance = yelp.Api(**yelp_key)
    return yelp_instance

yelp_instance = get_yelp_instance(YELP_API_KEY)

"Search" method has only one required parameter - location:

yelp_instance.Search(location='US')

Result of this method will be a response from Yelp, with all companies in US.

To get the next page you can set "offset" parameter
Another very important parameter is limit, which indicates the number of companies returned (with offset subtracted). The maximum number of companies per page is 20, and maximum offset is 50. If offset is exceeded, you get an error message. To calculate the offset you can multiply an overall limit by the number of page you want to receive. Offset calculation starts from zero, by default if you do not specify this option it is always zero.

limit = 20
page = 1
offset = page * limit
yelp_instance.Search(location='US', limit=limit, offset=offset)

Additionally, Yelp API supports search by keyword via "term" parameter and filter by category via category_filter. Category name must be a slug, you can find a slug of your category here

yelp_instance.Search(location='US', term="google", category_filter="gokarts")

This was a quick introduction to the Yelp api. Now, let's move to the Google Places API.

A few words about Google Places API

We will use Google Places API to obtain more detailed information about the companies:

  • user reviews and photos
  • companies websites
  • rating on a five-point scale
  • working hours
  • other

To start working with this API, you need to get an application token by clicking on the following link https://console.developers.google.com/.

Create an API token and attach it to your google application.

Please install python-google-places package:

pip install python-google-places

As well as the Yelp API, let's create a function that will return an instance of GooglePlaces class:

import googleplaces

GOOGLE_API_KEY = ""

def get_google_instance(google_key):
    google_instance = googleplaces.GooglePlaces(google_key)
    return google_instance

google_instance = get_google_instance(GOOGLE_API_KEY)

To obtain more information about the company via Google Places, we'll use the latitude and longitude of the company, as well as the name from the Yelp, to improve the accuracy of search:

yelp_results = yelp_instance.Search(location='US', term="google", category_filter="gokarts")

data = []

for business in yelp_results.businesses:
    places = google_instance.text_search(
        business.name,
        lat_lng={
            'lat': business.location.coordinate['latitude'],
            'lng': business.location.coordinate['longitude']
        }
    )

    company = {}
    for place in places.places:
        place.get_details()
        company.update(place.details)

    data.append(company)

Btw, usage quota os Google Places API is limited to 2000 requests per day that is mostly unacceptable to bear living in such a hasty world:) Fortunately, you can increase the number of requests to 250 000 if you add a credit card to your google account (api usage is still free here, you will not be charged).

Creating a simple Django Project

Now we will create a small Django project. I will show you an example of how to use Google Places API with Yelp API with it.

We will use django-skeleton package to decrease time spent on Django project setup.

django-admin startproject yelp-api-example --template=https://github.com/aliev/django-skeleton/archive/master.zip
cd yelp-api-example

Lets create a virtual environment and activate it, add the required packages to work with google places and yelp api in requirements.txt:

virtualenv .env
source .env/bin/activate
echo "python-yelp-v2==0.5.7" >> requirements.txt
echo "python-google-places==1.1.0" >> requirements.txt
pip install -r requirements.txt

We should get project structure like this

├── AUTHORS.md
├── Procfile
├── README.md
├── log
│   ├── README.md
│   └── error.log
├── requirements.txt
└── src
    ├── __init__.py
    ├── apps
    │   ├── __init__.py
    │   └── core
    │       ├── __init__.py
    │       ├── apps.py
    │       ├── forms.py
    │       ├── models.py
    │       ├── templates
    │       │   └── index.html
    │       ├── urls.py
    │       ├── utils.py
    │       └── views.py
    ├── conf
    │   ├── __init__.py
    │   ├── base.py
    │   ├── local.example.py
    │   └── local.py
    ├── db
    │   ├── README.md
    │   └── development.db
    ├── manage.py
    ├── static
    │   └── js
    │       └── app.js
    ├── templates
    │   └── base.html
    ├── urls.py
    └── wsgi.py

Our starting Django project is almost ready. To be sure that everything works properly, let's deploy this project to Heroku. django-skeleton already has Heroku settings set.

git init .
heroku create
git add .
git commit -m 'first commit'
git push heroku master

The last adjustment that we need to do is to set an API key for Google Places and Yelp in Heroku virtual environment. As to store API keys or passwords in the git project root is not the very best idea, let's keep them in the server's environment. To do this, go to the Dashboard of our account, and select the application that created the heroku command, open the Settings and click on the Reveral config vars button. We set the value for the Config Vars as shown in the picture:

Our Django application should know about our API keys. Add next lines in src/conf/base.py file:

YELP_API_KEY = {
    'consumer_key': os.getenv('consumer_key'),
    'consumer_secret': os.getenv('consumer_secret'),
    'access_token_key': os.getenv('access_token_key'),
    'access_token_secret': os.getenv('access_token_secret'),
}

GOOGLE_API_KEY = os.getenv('GOOGLE_API_KEY')

To make setting env variables in the production easier, create a ~/.env.secret file in your home directory with the following lines:

export access_token_key = 'your yelp access token'
export access_token_secret = 'your yelp token secret'
export consumer_key = 'your yelp consumer key'
export consumer_secret = 'your yelp consumer secret'
export GOOGLE_API_KEY = 'your google api key'

Now make source ~/.env.secret before starting the project.

First of all let's describe a form in src/core/forms.py

from django import forms


class ApiListForm(forms.Form):
    category_filter = forms.CharField(required=False)
    location = forms.CharField(required=True, max_length=2)
    term = forms.CharField(required=False)
    offset = forms.IntegerField(required=False)
    limit = forms.IntegerField(required=False)

We also need a function that returns instances of Google Places and yelp.Api classes, the description of which we made earlier. Let's place them in src/core/utils.py file:

import yelp
import googleplaces

def get_yelp_instance(yelp_key):
    yelp_instance = yelp.Api(**yelp_key)
    return yelp_instance


def get_google_instance(google_key):
    google_instance = googleplaces.GooglePlaces(google_key)
    return google_instance

Let's implement ApiFormView class. This view implements get method, in which the form is rendered. If the form data submitted is not validated, server returns a response with status 400 and JSON with a detailed validation errors. If the validation is successful, there will be returned a result of get_data method, which we will implement in our child class.

import json
import decimal
from django.views.generic import View
from django.http import HttpResponse


class DecimalEncoder(json.JSONEncoder):
    """
        Render decimal as string
        Encoder for json.dumps
    """
    def default(self, obj):
        if isinstance(obj, decimal.Decimal):
            return str(obj)
        return json.JSONEncoder.default(self, obj)


class ApiFormView(View):
    """
        The base class for rendering form
        Returns form and API errors as JSON response
    """
    def get_data(self, cleaned_data):
        """ Method should return a list of companies in the dictionary format """
        return cleaned_data

    def get(self, request, *args, **kwargs):
        data = {}
        form = self.form_class(self.request.GET)
        if form.is_valid():
            status_code = 200
            data['success'] = True
            try:
                data['data'] = self.get_data(form.cleaned_data)
            except Exception as e:
                data['errors'] = e.message
                data['success'] = False
                status_code = 400
        else:
            status_code = 400
            data['errors'] = form._errors
            data['success'] = False


        return HttpResponse(json.dumps(data, cls=DecimalEncoder),
                            status=status_code,
                            content_type='application/json')

Example of ApiListView class with get_data specified:

from django.conf import settings
from .forms import ApiListForm
from .utils import get_yelp_instance, get_google_instance


class ApiListView(ApiFormView):
    """ The base companies list view class """
    form_class = ApiListForm

    def get_data(self, cleaned_data):
        """ Method should return a list of companies in the dictionary format """
        cleaned_data = super(ApiListView, self).get_data(cleaned_data)

        category_filter = cleaned_data.get('category_filter')
        location = cleaned_data.get('location')
        term = cleaned_data.get('term')
        page = cleaned_data.get('offset') or 0
        limit = cleaned_data.get('limit') or 20

        data = []

        _offset_end = 50
        _page_size = limit

        google_instance = get_google_instance(settings.GOOGLE_API_KEY)
        yelp_instance = get_yelp_instance(settings.YELP_API_KEY)

        kwargs = {
            'location': cleaned_data.get('location'),
            'limit': _page_size,
            'offset': _page_size * page
        }

        if category_filter:
            kwargs.update({'category_filter': category_filter})

        if term:
            kwargs.update({'term': term})

        yelp_response = yelp_instance.Search(**kwargs)

        for business in yelp_response.businesses:
            places = google_instance.text_search(
                business.name,
                lat_lng={
                    'lat': business.location.coordinate['latitude'],
                    'lng': business.location.coordinate['longitude']
                }
            )

            company = {}
            for place in places.places:
                place.get_details()
                company.update(place.details)

            data.append(company)

        return data

Now, let's deploy our changes to heroku:

git add .
git commit -m 'Project deployment'
git push heroku master

Usage Example

To build http requests I use httpie - wonderful replacement of curl. Now is an occasion to test this great utility :) Here, I query http://googleplaces.herokuapp.com, while you will have a different domain to deploy.

pip install httpie

http http://googleplaces.herokuapp.com/api/list/?location=US&limit=1
http http://googleplaces.herokuapp.com/api/list/?location=US&limit=1&term=fitness
http http://googleplaces.herokuapp.com/api/list/?location=US&limit=1&category_filter=fitness

HTTP/1.1 200 OK
Connection: keep-alive
Content-Type: application/json
Date: Sat, 26 Mar 2016 23:12:36 GMT
Server: gunicorn/19.4.5
Transfer-Encoding: chunked
Via: 1.1 vegur
X-Frame-Options: SAMEORIGIN

{
    "data": [
        {
            "address_components": [
                {
                    "long_name": "592",
                    "short_name": "592",
                    "types": [
                        "street_number"
                    ]
                },
                {
                    "long_name": "3rd Street",
                    "short_name": "3rd St",
                    "types": [
                        "route"
                    ]
                },
                {
                    "long_name": "San Francisco",
                    "short_name": "SF",
                    "types": [
                        "locality",
                        "political"
                    ]
                },
                {
...googleplaces.herokuapp.com

Project repository is available here https://github.com/7WebPages/yelp_google_places
Demo project is available here http://googleplaces.herokuapp.com/api/list/. I am open to discussion. Should you have any ideas to improve, write to me as well.

2016-04-19