Python3: Getting Weather Conditions Through API

For a project, I wanted the current outside temperature for my local area. For different reasons, I decided on an external choice but mainly because it would be more accurate then dangling a temperature sensor out of the window!

I will walk you through the steps of building your own API calls in Python3.

The API

There are many different API’s out there in the wild, some premium and some entirely free. I stumbled upon a site called https://openweathermap.org/ which has free limited access or the option for a more powerful premium service.
Currently, the accounts have a limit of 60 per minute for their Current Weather API which is well within my needs (1 call per 15 minutes), so I chose this one.

After signing up for a free account, (you do not need to supply a payment method), you are able to create an API key for your app. There should be one already made by default, which I just changed the name of to my current project.

Requests

Once you have an API key for your project, you might need to wait for it to activate but it should be ready to use fairly quickly. You are now ready to start building your request!

To make the API request over http, I used the powerful requests library which makes this job more “pythonic” and easier to work with then the standard urllib.

Installing requests is easy with pip:

pip3 install requests

Now we can start building our API call. In a new .py file, add the following.

# requests docs : http://docs.python-requests.org/en/master/user/quickstart/
import requests

def main():

    # define the GET string for the Http query
    payload = {
               'q': 'London,UK',  # q is the place name for your query
               'units': 'metric',  # units is the unit of measurement
               'APPID': 'YOUR_API_KEY_HERE',  # APPID is for your app's API key (keep secret)
              }

    # The URL for the current weather API call
    # (full docs here: https://openweathermap.org/api)
    api_url = 'https://api.openweathermap.org/data/2.5/weather'

    # make the API call with defined parameters
    query = requests.get(api_url, params=payload)

    # convert raw json response to a python dictionary with json()
    data = query.json()

    print(data)

if __name__ == "__main__":
    main()

In the payload dictionary, change the “APPID” to your newly created API key.
Requests will use the supplied payload dictionary to form a complete GET string at the end of the api_url and will automatically build websafe escaping for special characters where required!

Response

If everything is succesful, the data from your API call will be parsed from a raw JSON object into a Python3 dictionary object. Your output will be like so:

{
  'coord': {
            'lon': -0.13,
            'lat': 51.51
           },
  'weather': [
              {
                'id': 741,
                'main': 'Fog',
                'description': 'fog',
                'icon': '50n'
              },
              {
                'id': 500,
                'main': 'Rain',
                'description': 'light rain',
                'icon': '10n'
              },
              {
                'id': 701,
                'main': 'Mist',
                'description': 'mist',
                'icon': '50n'
              }
             ],
  'base': 'stations',
  'main': {
            'temp': -0.01,
            'pressure': 997,
            'humidity': 100,
            'temp_min': -1,
            'temp_max': 1
          },
  'visibility': 8000,
  'wind': {
            'speed': 3.6,
            'deg': 300
          },
  'clouds': {
              'all': 90
            },
  'dt': 1548199200,
  'sys': {
           'type': 1,
           'id': 1414,
           'message': 0.0041,
           'country': 'GB',
           'sunrise': 1548143498,
           'sunset': 1548174807
         },
  'id': 2643743,
  'name': 'London',
  'cod': 200
}

As you can see, it is mainly made up by a parent dictionary object containing inner lists and child dictionary objects.

You can now navigate the data through the usual way in Python. For example:

data['wind']['speed']

will return the wind speed value of “3.6” (m/s) in this example.

And to retreive the current recorded temperature, you will use they key values:

data['main']['temp']

which will return a chilly value of “-0.01” (degrees)!

Conclusion

In just a few lines of Python code you have an endless pit of on-demand data at your fingertips. This is incredibly useful in many different situations, and not limited to the example seen here.

Though this code is simple, it was designed to show a working illustration of using APIs in Python. In a real project, it would be necessary to check the response status to make sure the data has been delivered correctly. Without this, a program can crash by another of exception errors.

Further Implementation

In order to properly use this code into a working application, you may need to think of corner cases to catch exeptions and stop it from crashing in the event of an unexpected circumstance.
For example:

  • What happens if the current device loses internet connection or the URL is unreachable?
  • What happens with a bad API request?
  • What happens if the API key expires or gets blocked?

These 2 cases are infact quite similar, but can lead to many different errors further down the line.

In my case, I will:

  • Check the requests status code first. If this fails, I will record the information as “NULL” and skip everything else.
  • If the status is good, I will use a Try Except clause to access the data through dictionary keys. If the data is somehow not there due to a bad request, I will avoid a ValueError exception and record the data as “NULL” instead.

There might be a few more cases that I havn’t mentioned, but that is down mainly to what YOU decide to do with the data and how important it is for your application.

Questions? Have I missed something?
Comment below!

First look at building a configuration file parser – Python3

Intro and context

The project that I’m working on is actually based on a previous (now defunct) project that had to be re-written. I was in the middle of creating a scrape tool to pull data from a website.

The original (I’ll refer to as MK1) worked really well, until the site was completely re-designed. I always knew of implications around this, but continued with it regardless. Looking back, I could have mitigated to lessen the impact of unexpected changes. This post is less about the details of the project and more about future-proofing expected changes, and creating an easier way in order to do so.

Anyone who has worked with an HTML parser knows that they can only work with what they are given, and if the HTML changes, so does the way the rest of the script behaves. I thought long at hard whilst rethinking the program… I thought about the 3 main objectives I wanted to achieve.

  1. Get data (input)
  2. Extract and order data (process)
  3. Save data (output)

I wanted this to be an automated, unsupervised process. There are (will be) many test cases if things go wrong… but still “want/need to” store the bread crumbs of “broken” data records for completeness.

Being a cup half full kinda guy, I broke MK1 down bit by bit looking for worst case scenarios and weaknesses.


Input

The webpage is the input, it can’t be changed after its received. It’s fairly simple to programmatically grab HTML from from the internet, but what if I needed multiple pages? URLs change all the time, how to speed the process of changing a list of hard coded sites in a script? What if I wanted to add a new site entirely?

Ideally, I needed a simpler way, with as less hard-coding as possible, to pull raw data and push it onto process. If things change, this impact will be minimal. I also needed an accessible list of URLs to queue, which can be changed whenever needed.

Process

From the HTML, I want to focus only on the elements of usefulness. Things I need. I look for similarities in lines of text. I find many different words, phrases, numbers expressing the same things differently. I could dedicate an entire function to do this for each group of data I want to extract. Adding to an ever growing list of if statements or switch cases, some may stretch for literally hundreds of lines of code for 25 different cases. (Like an exaggerated MK1). What if these 25 cases suddenly change… It could mean 100 lines+ of code needs to be reworked. What if the phrases that I were originally looking for, also changes?

I opted for files to hold these rules. They can all be read, loaded and used within a single loop statement without the need to build these in to the script.

Output

I know what the output should be and how it should be stored. What if I wanted to add more data to the data set? Maybe I have fragments of data that processing has missed? Shall I just discard of it?

Here, I decided to include a list of values inside some of the config files used in the input stage. These will correspond to database columns and can be added/changed whenever needed.

Eventually I was able to group the problems together and create a logical solution for them all.

If you notice, there’s alot of “daisy chaining” going on. I don’t mind this as config files are a lot easier to manipulate then creating a database and a front end to manipulate it, and easier still then hardcoding the majority of variables that are needed.

Creating a parser in python

Essentially, the txt configuration files will contain your own little language and syntax in order to sort and use the data appropriately.

Let’s take this simple configuration:

urls.txt

#this is a configuration file
#syntax: sitename=url

#url1
jsephler=https://jsephler.co.uk

#url2
time50=https://time50.jsephler.co.uk

In the example, we have some familiar sights of a typical config file.

  1. “#” for block comments. We must tell the parser to ignore these
  2. Empty lines (or new line characters) to make the file more human readable. We must tell the parser to ignore.
  3. Finally, the configurations. “jsephler=https://jsephler.co.uk”

In python, we need to first open up the configuration file. Lets assume that “urls.txt” is in the same directory as our script.

openconfig.py

def main():
    urllist = []  # a list for config data
    filename = "urls.txt"  # path to file
    with open(filename, "r") as urlfile:  # open file
        for line in urlfile:  # iterate through file
            if line[0] != "#" and line.startswith("\n") is False:  # ignore "#" and "newline" characters
                tmp = line.strip().split("=")  # strip line of "whitespace" and split string by "=" character
                urllist.append(tmp)  # append to list

    print(urllist)  # print list to console

if __name__ == "__main__":  # initiate "main()" first
    main()

If permissions will allow, the script will open our file and refer to it as “urlfile”.

The loop will iterate through every line in the file, while the if statements check for any lines that start with “#” or “\n” new line characters.

Before we store our data, we remove whitespace (strip) and seperate (split) the string by the “=” character.

Only after this, we can append it to our urllist array.

Output should look like this:

[['jsephler', 'https://jsephler.co.uk'], ['time50', 'https://time50.jsephler.co.uk']]

An array of arrays, where each member of urllist:

[0] is the sitename, and [1] is the url.

Breaking this down further, you could have a configuration like this:

jsephler=https://jsephler.co.uk|copy

After the first “=” split, you could split the second member of the array a second time by using the “|”  character to end up with another 2 pieces of data. Copy could call a function to do just that, copy!
Obviously plan ahead and use the characters wisely. You do not want to use a character that could be included in a URL as you may need to use it the future.

By doing this, you can create a config file that’s not only simple but powerful too.

Conclusion

There is a Python config parser library, however I preferred to create my own. My reasons for doing so:

  1. I didn’t actually know it existed until I started writing this post.
  2. It is fairly simple logic and I can tailor the syntax and uses.
  3. You could potentially save on overheads instead of loading and using a separate module.
  4. It’s alot of fun to experiment with!

For reference, here is the documentation for the standard parser library: https://docs.python.org/3/library/configparser.html