Python3: Getting Weather Conditions Through API

For a project, I wanted the current outside temperature for my local area. For different reasons, I decided on an external choice but mainly because it would be more accurate then dangling a temperature sensor out of the window!

I will walk you through the steps of building your own API calls in Python3.

The API

There are many different API’s out there in the wild, some premium and some entirely free. I stumbled upon a site called https://openweathermap.org/ which has free limited access or the option for a more powerful premium service.
Currently, the accounts have a limit of 60 per minute for their Current Weather API which is well within my needs (1 call per 15 minutes), so I chose this one.

After signing up for a free account, (you do not need to supply a payment method), you are able to create an API key for your app. There should be one already made by default, which I just changed the name of to my current project.

Requests

Once you have an API key for your project, you might need to wait for it to activate but it should be ready to use fairly quickly. You are now ready to start building your request!

To make the API request over http, I used the powerful requests library which makes this job more “pythonic” and easier to work with then the standard urllib.

Installing requests is easy with pip:

pip3 install requests

Now we can start building our API call. In a new .py file, add the following.

# requests docs : http://docs.python-requests.org/en/master/user/quickstart/
import requests

def main():

    # define the GET string for the Http query
    payload = {
               'q': 'London,UK',  # q is the place name for your query
               'units': 'metric',  # units is the unit of measurement
               'APPID': 'YOUR_API_KEY_HERE',  # APPID is for your app's API key (keep secret)
              }

    # The URL for the current weather API call
    # (full docs here: https://openweathermap.org/api)
    api_url = 'https://api.openweathermap.org/data/2.5/weather'

    # make the API call with defined parameters
    query = requests.get(api_url, params=payload)

    # convert raw json response to a python dictionary with json()
    data = query.json()

    print(data)

if __name__ == "__main__":
    main()

In the payload dictionary, change the “APPID” to your newly created API key.
Requests will use the supplied payload dictionary to form a complete GET string at the end of the api_url and will automatically build websafe escaping for special characters where required!

Response

If everything is succesful, the data from your API call will be parsed from a raw JSON object into a Python3 dictionary object. Your output will be like so:

{
  'coord': {
            'lon': -0.13,
            'lat': 51.51
           },
  'weather': [
              {
                'id': 741,
                'main': 'Fog',
                'description': 'fog',
                'icon': '50n'
              },
              {
                'id': 500,
                'main': 'Rain',
                'description': 'light rain',
                'icon': '10n'
              },
              {
                'id': 701,
                'main': 'Mist',
                'description': 'mist',
                'icon': '50n'
              }
             ],
  'base': 'stations',
  'main': {
            'temp': -0.01,
            'pressure': 997,
            'humidity': 100,
            'temp_min': -1,
            'temp_max': 1
          },
  'visibility': 8000,
  'wind': {
            'speed': 3.6,
            'deg': 300
          },
  'clouds': {
              'all': 90
            },
  'dt': 1548199200,
  'sys': {
           'type': 1,
           'id': 1414,
           'message': 0.0041,
           'country': 'GB',
           'sunrise': 1548143498,
           'sunset': 1548174807
         },
  'id': 2643743,
  'name': 'London',
  'cod': 200
}

As you can see, it is mainly made up by a parent dictionary object containing inner lists and child dictionary objects.

You can now navigate the data through the usual way in Python. For example:

data['wind']['speed']

will return the wind speed value of “3.6” (m/s) in this example.

And to retreive the current recorded temperature, you will use they key values:

data['main']['temp']

which will return a chilly value of “-0.01” (degrees)!

Conclusion

In just a few lines of Python code you have an endless pit of on-demand data at your fingertips. This is incredibly useful in many different situations, and not limited to the example seen here.

Though this code is simple, it was designed to show a working illustration of using APIs in Python. In a real project, it would be necessary to check the response status to make sure the data has been delivered correctly. Without this, a program can crash by another of exception errors.

Further Implementation

In order to properly use this code into a working application, you may need to think of corner cases to catch exeptions and stop it from crashing in the event of an unexpected circumstance.
For example:

  • What happens if the current device loses internet connection or the URL is unreachable?
  • What happens with a bad API request?
  • What happens if the API key expires or gets blocked?

These 2 cases are infact quite similar, but can lead to many different errors further down the line.

In my case, I will:

  • Check the requests status code first. If this fails, I will record the information as “NULL” and skip everything else.
  • If the status is good, I will use a Try Except clause to access the data through dictionary keys. If the data is somehow not there due to a bad request, I will avoid a ValueError exception and record the data as “NULL” instead.

There might be a few more cases that I havn’t mentioned, but that is down mainly to what YOU decide to do with the data and how important it is for your application.

Questions? Have I missed something?
Comment below!

tools.jsephler.co.uk

tools.jephler.co.uk is a project site that I have set up which primarily houses different tools ranging from convertors and calculators to more generally links to reference sites. It was built with ease of use and portability in mind and is fully compatible with mobile devices too!

I think of the site as my coding playground and allows me to create and share tools which I not only find interesting, but more importantly useful. Thanks to Django, this does not limit the site to what you see today and will hopefully be expanded upon through time.

I have some existing projects that currently live in a myriad of Python files which I am hopeful to convert and eventually host here.

Suggestions or comments welcome! Please get involved.

tools.jsephler.co.uk

Compile and install Python’s MOD_WSGI for Apache2 in Ubuntu 18.04

Having installed mod_wsgi a couple of times on different machines, I decided to note down the steps I took and put together a guide. There are also some solutions to some issues that you may encounter, which left me clueless for a while. If you are looking for a guide to deploying a Flask/Django project, you will have to look elsewhere (for the time being).

This is for use with python3 using Ubuntu 18.04’s shipped version. You may need to tweak this guide to point to non-standard versions of python3 that you specifically want to use.

I believe some Ubuntu versions have pre-packaged mod_wsgi installations available. However, noted in a previous blog post, apache2 may flood error logs pointing to a mismatch with either differing python3 or apache2 versions and in my view, a clean compilation ensures a tailored fit to your current system without the worry of logging and runtime errors.

Things to note:

  • I prefer to compile inside the “/opt” directory. Please amend the guide to suite your preference
  • I will refer to the current latest version of mod_wsgi (4.6.4 at time of writing). This is for illustrative purposes only, and some directory paths will be different with different versions
  • The guide is built around CLi, and assumes you are comfortable with the basics

Pre-Requisites

You will need to have installed some additional packages:

  • python3-dev  –  mainly for python3 header files for compilation
  • apache2
  • apache2-dev
  • gcc  –  *or an equivalent C compiler*

You may use apt install for all the above packages

Next, you will need to locate the latest mod_wsgi source files (found here)

Downloading

Navigate to /opt and download the latest version of mod_wsgi:

$ cd /opt
$ wget https://github.com/GrahamDumpleton/mod_wsgi/archive/4.6.4.tar.gz

Unpack the tar:

$ tar -xzf 4.6.4.tar.gz mod_wsgi-4.6.4/

Configure Make file

Change into the unpacked directory:

$ cd mod_wsgi-4.6.4/

Complete a “test” run of the configure script to so you can debug any errors:

$ ./configure

 

  • Error: “configure: error: no acceptable C compiler found in $PATH
    Solution: install a C compiler:

    $ sudo apt install gcc

 

  • Error: “Checking Apache Version.. ./configure: line 2765: apsx: command not found
    Solution: install apache2-dev

    $ sudo apt install apache2-dev

 

  • Error: “checking for python… no
    Solution: add argument to ./configure to point to python3 (global) or full path

    $ ./configure --with-python=python3

If you run configure without any errors, you’re ready to compile!

Compiling mod_wsgi

The make file is ready. Run Make:

$ make

This will now compile mod_wsgi. It should complete without error.

We finish the process by installing mod_wsgi to our apache2 installation:

$ make install

Configure Apache2

Finally, we need to configure apache2 correctly to load the mod_wsgi module. There are 2 ways we can do this:

  1. The “lazy way”:
    Add the following to the end of the apache2.conf file:
    (located: “/etc/apache2/apache2.conf“)

    LoadModule wsgi_module /usr/lib/apache2/modules/mod_wsgi.so

    Restart Apache2:

    $ sudo systemctl restart apache2

    If apache2 restarts without errors, you have successfully installed and loaded mod_wsgi!

  2. The “proper” way:
    Navigate to “/etc/apache2/mods-available“Create “mod_wsgi.conf”:

    $ sudo touch mod_wsgi.conf

     

    Add the following to” mod_wsgi.conf”:

    <IfModule mod_wsgi.c>
    </IfModule>

    For further information about the configuration options that can be included in “mod_wsgi.conf”, refer to docs : https://modwsgi.readthedocs.io/en/develop/configuration.html

    Create “mod_wsgi.load”:

    $ sudo touch mod_wsgi.load

    Add the following to “mod_wsgi.load”:

    LoadModule wsgi_module /usr/lib/apache2/modules/mod_wsgi.so

    Activate module in Apache and follow onscreen instructions:

    $ sudo a2enmod mod_wsgi

    If apache2 restarts without errors, you have successfully installed and loaded mod_wsgi!

SQLite3 and Python3 : Generating Statements

I’ve been working on a project that interacts with a database, and happened upon a some interesting problems.

The data I want to input into the database is initially stored in a dict variable.

  • If you weren’t already aware, the order of a Python dict changes, even if you use a blueprint or template. This means that a pre-prepared database statement wouldn’t necessarily align with the values of the dict, deeming the data to be parsed into the wrong fields in a database.

The dict may contain some or all of the tables’ fields

  • My program will acquire as many data values from input as possible to fully populate the dict, however I have allowed this to be dynamic in a sense that if the data is irretrievable, or doesn’t match a regex for the field, it will ignore said field and move on, deeming the dict value to default to “None”.

I want to make correct use of the API’s escape mechanism

  • This is most important. Not only to make fully sure that I’m not inserting “unclean and potentially harmful” data, but also to follow the line of best practice. I’m really sure (like 99%) that the data will be clean. However, mitigating the risk further will bullet proof the commit.

I don’t want to bore you with many lines of code, so I will explain my methodology of my trials instead.

Many Loops

At first, I kind of penned down the rough idea of the function in long hand.
I started with a loop to store the key/value pairs in 2 separate lists. I had to include a couple of cases; to ignore “None” values and some keys which weren’t destined for the intended table.

For each list, I wrote a loop that builds a string that had to be properly formatted for the final statement.

INSERT INTO x (f1, f2) VALUES(v1, v2)

This worked well but the code really was too verbose. 30~50 lines in fact. I thought about it and lessened the load.

Less loops, messy code

Trying to lessen the loops, I went with the same loop to extract the dict key/values as above.

For reference:

_ = [[],[]]

ignore = ["ig1", "ig2"]

for key, value in dict:
    if key in ignore:
        continue
    _[0].append(key)
    _[1].append(value)

I decided to use the string .join() method to create the raw statements from each list, which was concatenated to make the final statement.

It worked and lessened the loop burden. However I had forgotten an extremely important step, the SQLite3 escape mechanism.

Success in less words

It suddenly dawned on me that through these statement creating methods, I’m joining raw data to this string for input, without properly parsing the data. I had also tried:

"{v0}".format(v0=data[0])

This of course gave me the same result.

The fix was fairly simple, however.
The values in the statement needed to be changed from the “data values” to “?”.
The execute method also required the second argument as a tuple to allow for parsing and “?” substitution. I simply converted the list to a tuple.

The code is now down from around 50 to 15 lines which I’m happy about.

ignore = ["ig1", "ig2"]  # list of dict keys to ignore.
col = []  # list of col headings
val = []  # list of col values

# loop dict, append key/value to lists respectively
for key, value in dict.items():
    if key not in ignore:
        if value:
            col.append(key)
            val.append(value)

val = tuple(val)  # convert list to tuple

f = ", ".join(col)  # make string of fields
v = ",".join(["?" for i in range(len(col))])  # make string of "?"'s
statement = "INSERT INTO x (" + f + ") VALUES(" + v + ")"  # finalise statement

dbc.execute(statement, val)

This code sits right with me. It has taken a few attempts, but the final result gets the job done correctly:

INSERT INTO x (f1, f2, f3) VALUES(?,?,?)

It goes to show that even something fairly trivial as chucking information into a database can require some careful planning, especially when taking limitations into account (in this case datatypes), and the nature of task and has taught me some important lessons.