2019 Week 40: Charts and Code

Short version: Intuitions about how we look at data. Taking small steps with code. Art at the Tate Modern.

Long version:

Sneaky Spreadsheets

This week I have been processing some experimental data and realised how the same data can be plotted in seemingly different ways. Often looking at the numbers themselves can be challenging, particularly if there are hundreds of points with multiple values at each point. For this example we take some simple data about income distribution in the UK. The numbers alone are accurate but making inferences takes consideration and a little mental arithmetic.

A linear column chart is the simplest and fastest way to visualise the data in a spreadsheet program (excel, google sheets, open office calc). It shows the 99th percentile as much larger than the other values, and the increases seem bigger between the higher percentiles.

Taking a logarithmic plot makes each step along the distribution seem more even, but the 99th percentile still seems noticeably larger.

But by changing the minimum axis value on a logarithmic plot the distribution seems much flatter overall.

And we can go to the opposite extreme, by plotting a pie chart the high income earners (top 1%) seem to take a massive share of the pie.

Code

I am not proficient in code, and find that embarrassing. That is an odd statement, I’m not proficient at many things (skateboarding, Portuguese speaking, cello playing) but few of those things cause me to feel embarrassment. Technological proficiency, unlike skating or playing a string instrument, is a part of my identity, and I feel that being able to code (or at least write basic scripts) is a part of being good with tech. When I go to write code, I am confronted by the gap between what I feel my skill should be and where my skills actually are, which is uncomfortable. Moreover it causes me to hide, refraining from asking for help, and I end up doing things in a slow and repetitive way. To push back against that urge, here is an attempt to write a simple python script to extract folder information from a directory tree.

import os
import re
print "This removes files from saved directory trees"
path = "./"
print "I will run in the local folder"
linecount = 0
for filename in os.listdir(path):
        if filename[len(filename)-4:]==".txt":
            with open(os.path.join(path, filename), "r") as file:
                output= open(filename[0:len(filename)-4]+"_folders.txt","w+")
                for line in  file:
                    if len(line) > 6:
                        if line[len(line)-6]!=".":
                            if line[len(line)-5]!=".":
                                if line[len(line)-6:]!="Cloud\n":
                                    output.write(line)
                                    linecount = linecount +1
            print("I found "+str(linecount)+" lines and put them into "+filename[0:len(filename)-4]+"_folders.txt")
            linecount = 0
        else:
            print("I did not process "+filename)
            linecount = 0

Subscribe to email alerts!

Photos from the week

From the Tate Modern exhibition by Olafur Eliasson

2019 Week 35: Tears, Triathlons

Short version: Back to work. Jocko’s letter made me cry and that’s OK. First triathlon, went fine. Sriracha sauce as a spaceship.

Long version:

Back to work

I returned to work this week after taking nearly two weeks of holiday. I continue to find it odd that when I am busier I also get more done, but I also suspect that is the combination of routines and inspiring colleagues. The ONI band is practicing more often, and live music provides a wonderful reminder of my family home. Hearing from others about their successes both in work and outside it creates some envy, but also motivates me to strive to make more out of my time.

Something that made me cry

At the end of my holiday I took some time to reflect on the past year, including when I experienced particularly strong emotions. Often in a professional context it is appropriate to suppress emotions, and it is also common that men feel the need to project a stoic facade at the world. In the past I have taken this too far and lost awareness of my emotions, which is detrimental. I still feel generally we should act out of reason rather than emotion, but we should also be aware of our emotions and use their patterns to guide our understanding of ourselves. To this end, I wanted to share this letter written by a Navy Seal to his daughter. It left me feeling a powerful mix emotions, seeing aspects of myself in both the father and the daughter. It left me in tears.

First Triathlon

I took part in my first triathlon on Monday. It was fun. I’ve written a race report you can read here.

Instagram Art

Social media is often an echochamber, but there are opportunities to find new ideas. This week I came across spacegooose who imagines everyday items as spaceships, e.g. Sriracha sauce.

Photos from the Week

Email Notification Sign Up