Python3: Time-critical code

Whilst thinking about how to tackle my Raspberry Pi temperature project, I found and issue that needed resolving.
In short, I want to record the temperature of my room every 15 minutes. I wanted to record every 15th minute of every hour for a couple of reasons:

  1. To keep the data fair: simply rounding to the nearest quarter hour could mean a difference of up to 15 minutes should the Pi turn off and start running again.
  2. Uniformity: I didn’t want the time to be every 15 minutes after the program starts, for example 10:21 and then 10:36. I want to make a clear comparison of the temperature exactly at the same time for any time I start or stop the program.

Now there are a couple of ways to start this automation, specifically in Linux:

  • cronjob: can run code on a specified time/interva
  • init.d startup: can run code once when startup commences

The cronjob method would be good, but let’s think about this in some more detail:
Every 15 minutes, the system will fire up the code from scratch. Which means all imports will take place, load functions into memory, assign variables etc. which costs time (yes I am being THAT pedantic) and would not always be accurate. Also, it takes extra setting up (so if someone else uses my project, it can be more complicated to set up)

Init.d will simply run the code once on startup. This will allow you to manage time better, although if the code crashes (for some reason), you will have to manually start it or restart the machine.

There a pros and cons to both methods, but ultimately I wanted full control over time and as you may have gathered, opted for init.d.

Now, the interesting bit! Getting the time right, every time.
Again there were a couple of ways that can accomplish this. I needed to offset the start time (start of the script) to take measurement at exactly every quarter hour.
Obviously, this can be any time between a 15 minute window.
I could have arbitrarily looped over the current UNIX time until it divided by 900 (seconds in 15 minutes) but this takes processing time and if the difference to run the loop (in MS) pushes it over a second, I could potentially miss a record of data.

Ideally, I wanted to calculate the time difference and wait it out. Then I could take a record of data and process it, before calculating the difference again for the next time. Doing things this way means I have a 14 minute 59 seconds window to run whatever code I wanted to, until the next time. Obviously, 14 minutes 59 seconds is a long time in terms of running code, but obviously depends on application.
For example, processing dynamic data could take you different times to complete.

Below is my algorithm to calculate the time needed to wait until the next quarter hour. Immediately after time.sleep(), the time critical code, then after that, you have almost a full 15 minutes to do what you need to catch the next one.

# Calculate the time required to wait until the next quart hour

from datetime import datetime
import time

# minute, second, millisecond to a string and split into list
min_sec = datetime.now().strftime("%M %S .%f").split(" ")

# calculate minutes to wait for the next whole quart hour
# explanation of m calculation: e.g. time = 12:33 : min = 33
# 33 % 15 = 3 : find out minutes since last quart
# 15 - 3 = 12 : whole minutes to wait for next quart
# 12 * 60 = 720 : whole seconds to wait with time.sleep()
m = (15-(int(min_sec[0])%15))*60  # convert string to int and calculate seconds

s = int(min_sec[1])  # convert string to int
ms = float(min_sec[2])  # convert ms string to float

# time to sleep until next whole quart
# calculation:
# minutes in seconds minus seconds over whole minute minus ms over whole second
# sleep for required time
time.sleep(float(m-s) - ms)

# ..
# whole quart of hour reached....
# time should be X:(00/15/30/45):00.000
# run time sensitive code here

“The War of the Worlds” : a 21st Century project (Pt. 2)

H. G. Wells – “The War of the Worlds” Map

Map Theory

With the full list of place names successfully extracted, it was time to think about tackling the map itself.
Initially, I thought of using the Googlemaps API, however it required the longlitude and lattitude for each place name. In order to do this, I would need to lookup the places with Google’s Geolocation API (or similar) but this is a premium service. It seemed that the API routes were quickly dwindling; I had to find a different way of producing a map.

Still using Google maps, it offers the a way of sharing a map that can be shared on the internet. In hindsight, this was the best option for the project. Google maps allowed me to upload a CSV spreadsheet (comma seperated values) which were headed Title, Place name.

Running the lists through a program to output the CSV, I capitalised the title which was the place name. I then uploaded this data to the map.

The first time that I saw all the places as points on a map was incredible! However I quickly realised that there were anomolies in the points of data.
Such as a place called “Wellington Street”, where Google decided the best location was in New Zealand. Instantly knowing that NZ isn’t a place that is mentioned in the book, this was the start of a manual refining process… although most of the locations were correct

I spent some time almost triangulating the exact street or place that I could identify was in the wrong place. After some time, the map slowly became a concentrated spread of dots from Surrey through to West London and finally becoming more sparse on the East Coast. Unfortunatley, there are some places that don’t appear in the final map as they have since been built over such as the “inkerman Barracks”.

Conclusion

This was an interesting project that became alot more time consuming then I first thought.

Context

The way that Wells’ casually describes or talks about places meant that not all the places were extracted straight from the text. This often meant cross checking against the text itself to understand that exactly the place that he implies.

Exact location

This was also difficult, as I ideally wanted a comprehensive map although I couldn’t quite decide whether to keep it historically accurate or keep it within the confines of the current modern map.
There were also some locations that weren’t in the right place after importing the data which again had to be checked, moved and deleted.

Overall, I am happy with the outcome, although it was a little more tricky than I first expected. It took some manual intervention to get it correct, but it has highlighted some interesting problems.

“The War of the Worlds” : a 21st Century project (Pt. 1)

Addlestone, Addlestone golf links, Albany street, Albert hall, Albert road, Albert terrace, Aldershot, Asia, Astronomical exchange, Atlantic, Baker street, Banstead, Barnes, Barnet, Belsize road, Berkshire, Berlin, Birmingham, Bishopsgate street, Blackfriars bridge, Blackwater, British museum, Broadstairs, Brompton, Brompton road, Bushey park, Byfleet, Byfleet golf links, Byfleet road, Cannon street, Cardigan, Castle hill, Chalk farm, English Channel, Chatham, Chelmsford, Chertsey, Chipping barnet, Chipping ongar, Chobham, Chobham road, Clacton, Clapham, Clapham junction, Colchester, College arms – pub, Coombe, Crewe, Crouch, Crystal palace, Deal, Derby, Ditton, Dublin, Ealing, East barnet, East end, East ham, Edgware, Edgware road, Edinburgh, Earth, England, Epping, Epsom, Epsom downs, Epsom high street, Esher, Essex, Euston, Exhibition road, Eybridge, Fleet street, Foulness, Foundling hospital, France, Fulham, Fulham road, Germany, Great north road, Guildford, Hadley, Haggerston, Halliford, Hamburg, Hammersmith, Hampstead, Hampsted, Hampton, Hampton court, Hanwell, Harrow road, Harwich, Haverstock hill, High barnet, Highbury, Highgate, Horse guards, Horsell, Horsell bridge, Horsell common, Hounslow, Hoxton, Hyde park, Imperial institute, Inkerman, Inkerman barracks, Irish sea, Isleworth, Kensington, Kensington gardens, Kew, Kew lodge, Kilburn, Kingston, Kingston hill, Knaphill, Laleham, Lambeth, Langham, Langham hotel, Leatherhead, Lick observatory, Limehouse, Lisbon, Liverpool street, London, Malden, Maldon, Manchester, Marble arch, Marylebone, Marylebone road, Mauritius, Mars, Maybury, Maybury bridge, Maybury hill, Maybury inn, Merrow, Middlesex, Midland railway company, Mole, Molesey, Mortlake, Moscow, Natural history museum, Naze, Neasden, New barnet, New malden, Newhaven, Nice, Ockham, Old woking, Oriental college, Oriental terrace, Ostend, Ottershaw, Oxford street, Painshill, Painshill park, Paris, Park road, Park terraces, Parliament, Perrotin, Petersham, Pinner, Pompeii, Pool, Portland place, Portman square, Portsmouth, Primrose hill, Putney, Putney bridge, Putney common, Putney hill, Pyrford, Pyrford church, Regent street, Regent’s canal, Regent’s park, Richmond, Richmond bridge, Richmond hill, Richmond park, Ripley, Ripley street, Roehampton, Royal academy of arts, Send, Serpentine, Sheen, Shepperton, Shepperton church, Shepperton lock, Shoebury, Shoeburyness, Shoreditch, Sodom and gomorrah, South kensington, Southampton, Southend, Spotted dog – pub, St. albans, St. edmund’s terrace, St. george’s hill, St. haggerston, St. john’s wood, St. martin’s-le-grand, St. pancras, St. paul’s cathedral, Staines, Stanmore, Strand, Street cobham, Sunbury, Surrey, Thames, Thames valley, Thames-side, The mole, The wandle, Tillingham, Tower bridge, Trafalgar square, Twickenham, Upper halliford, Venus, Victoria, Virginia water, Walham green, Waltham abbey, Waltham abbey powder mills, Walton, Wandsworth, Waterloo, Waterloo bridge, Waterloo road, Wellington street, West hill, West surrey, Westbourne park, Westminster, Westminster bridge, Weybridge, Wimbledon, Wimbledon common, Winchester, Windsor, Woking, Woolwich, Zoological gardens.

You may be confused at first. Please read on.

Background

As a kid, I was first brought to the attention of “The War of the Worlds” through Jeff Wayne’s musical adaption, and naturally it freaked me out! (Seriously it gave me nightmares and my fear of the dark)
I have since watched the film 2005 film, and again more recently.
Although the two adaptations had scenes that crossed over, there were differences between them and so I set upon finding which version was more true.
(In short we are lucky to have so many great versions of the story, including the novel itself)

The original story was written by H. G. Wells which was first published in 1897. I was surprised at the incredible depth and detail Wells takes the trouble to recount as the lonesome protagonist during the martians’ escapade.

The thing that struck me most about the novel are the endless places that Wells mentions in the book to really help the reader feel like they were in amongst the chaos. Not much thought was given to the distance between two places until you remind yourself that 1897 was a car-less time and infact travelling by foot or carriage, there were sometimes imense distances involved. I wanted to create something that would be more visual to a reader in modern times, a map like the sort that J. R. R. Tolkein included in his fiction novels.

The Map Idea

Wouldn’t it be cool to piece together a map based on the places that feature in the book? Luckily we have a great platform in the form of Google Maps that will allow us to place pins on places of interest. Before this stage however there is work to be done.

The age of the novel allows anyone to gain access to the text in many different formats for either free or for a small fee. I downloaded my copy at Gutenberg.org.
I downloaded the most basic format (Plaintext UTF-8) which will allow me to manipulate the novel without the need for an advanced parser.

Wells capitalizes all the places that he mentions in the book, which helps greatly when trying to identify them. I split the entire book into a list of words which would have been seperated by the space character.
From this, I created a “shortlist” of words that were capitalized. During this process, I made sure to keep consecutive capitalized words together incase they made up a place name.

The shortlist was rather large, so I got hold of Ordnance Survey data and created a list of all unique places and roads that appear throughout the UK. I then ran the shortlist through the OS data, yet again creating two new lists;

  • Shortlist strings that match with the OS data
  • Shortlist strings that didn’t match the OS data

Both the lists had duplicate items in them. The problem stemmed from the fact that in order to keep place names like “St. Paul’s” together, I had to ignore all punctuation. So we might have 2 similar entries within the list, such like “St. Paul’s,” and “St. Paul’s.”.
This became a bigger issue then I first thought, with sometimes 5 duplicates for the same place. However, after having a total of 1500 strings to sort through, I wasn’t too troubled with refining it further by hand.

I went through the list manually, checking the “more probable” list first against the novel. I’m glad that I did this method; yes it seems a little long winded, but there are some corner cases that this resolves.

For example, Wells writes about St. Pauls Cathedral, but simply refers to it as “St. Paul’s”. There isn’t a place simply called St. Paul’s however in context, you can understand that the Cathedral was exactly what he was talking about.
There is another word he uses such as “Naze”. In the text, he refers to it as “the Naze”, but in context, he might be refering to either Walton-on-the Naze or a place just north of Walton called The Naze. I am unsure as to the exact location of this place as of yet, but I have left it in the final list. (I have a feeling it might be the latter, although Walton-on-the-Naze railway station was opened in 1867, fairly recent in relation to the publish date)

Amongst the “less probable” list, I found some interesting places such as planets, continents, and altogether different countries. He even refers to an observatory in America which now seems to be a historic site of interest.

The final list is mostly a direct copy of places from the book, shown exactly as found from the text. As stated above, there are changes that I have made, that might not be directly found in the novel, but rather (with 100% certainty) of the place being refered to in the book.

Conclusion

So to round this off; although this endeavor didn’t exactly perform as planned, I am happier with the final list then I might have been simply allowing a simple algorithm choose what it thinks to be a place and discard the rest. Some manual work has gone into this to make a nice comprehensive list to work with when deciding where to pin these places to a map.

Afterthoughts

After refining the list to the one that is mostly complete, I may now have something to work with. I could pick out the whole sentences that contain the place names and try and create a weighing algorithm to more closeley identify places. I wonder how many similar words are used in sentences where other place names reside.
Also I wonder how many place names contain similar sequences of letters that may also be an identifier in trying to decide whether a word is a noun or not.

multi-search: tool for searching keywords on multiple websites

Intro

multi-search is a small GUI tool designed to search for a term or keyword on multiple websites.
It’s built in Python using the appJar GUI suite.
It’s intended as a tool for buyers or sellers to compare the price of an item, where comparison websites may fail for different reasons.
It can will save time, as well as passively suggesting different sites to search.
Additionally, the multi-search tool doesn’t attempt track or collect any user information.

The program is simple by design, which tries hard to be as quick and intuitive as possible.

multi-search main
multi-search program

After the “Sites to search” are selected in the dropdown, the URL links are presented for use.

multi-search results

There’s an additional option to automatically open the links in the web-browser.

How it works

It’s really simple. The program loads the site and search data from a “.txt” file.

It builds a URL string from the data and the search term provided.

It shows or opens the URL’s for the user!

Project State

Up to now in it’s current form, the program is completely useable, although still in its infancy.

Project Direction

  • Bundle the project for Windows and Linux, making it widely available

Where to download

The project is currently available at github.com


Changes:

  • 21/02/2018: Major Changes
    Added “multi-search.ini” to set user configurations
    Improved naming convention for site URLs – allowing for international support
  • 20/02/2018: Major Changes
    Added JSON support for REST API’s

Python: A visual look at randomly changing values

I wanted to visually see what was happening whilst randomly filling a Python list or dict with a value. For these examples, I used (mainly) identical code to produce the visuals, but changed the main loop method.

I created the the first program with what I thought was the best method, but was quickly disappointed

The First Attempt

I started by creating an empty dict of 100 key/value pairs; this will be my working space. I felt 100 was a nice round number to use, and it easily fits within the confines of a terminal.
Using a loop, I randomly choose a number between 0 – 99 and change the corresponding dict value to “x” resembling a change.
If the dict value had already been changed, it loops over until a new unchanged value is picked. The program stops when all values have been changed.

The whole process took little time to execute, so decided to slow down the program with the sleep method to clearly see the changes.

(Source: Python_dict_from_random)

In the video, the end result was a total of 415 conflicting random picks and 100 successful picks. This really surprised me as I have used this method in some programs before, but never realised how inefficient it actually was.
This approach essentially uses brute-force to change the values, without any control. I needed control.

The Second Attempt

Thinking about the problem, control was key. Infact, the key is control. With every filled dict value, the probability of getting an unchanged value goes down.
Ideally, the probability of getting an unchanged value should be (or as close to) 100% to be efficient.

I decided to create a new list, based on the dicts’ keys. From there, I could randomly select a value from the list, change the value of the corresponding dict key, and then remove the key from the list. I will be left with a list of dict keys which are yet to be populated. This means that every random pick will always be an empty value. I gave it a go and it worked really well!

(Source: python_dict_from_random_reduction)

As you can see, 0 conflicts; there can’t be a conflict. The only downside to this might arguably be the overhead of manipulating the list and storing the list, especially if the dict key is a string rather then an integer. However I am certain that the likelihood of the first example picking 100/100 keys on the first 100 loop iterations is really low.

Notes

In the examples above, I used dicts as the data medium. Obviously with some tweaking, you could use lists. Infact, if you use the first method especially, you SHOULD use a list/index (unless the dict key is indexed by an integer)
Secondly, you will need to pre-allocate the desired length in the list in order to prevent list index exceptions!