In this tutorial, I am going to show you some basics about processing text files (or log files) in Python.
The text files can be roughly classified into two categories: structured or non-structured. The structured text files is relatively more straightforward to process as they all follow certain conventions. For the non-structured text files, it is a bit complex as you need to find out certain structures yourself so as to do it automatically. Let's start with the non-structured one first.
Non-structured text file:
There are always some queer fishes......
-----------snippet from a simulation ------
[5/15/2013 2:17:26 PM] Session Start
[5/15/2013 2:17:26 PM] Leaving sequence: loadXML, moving forward.
[5/15/2013 2:17:30 PM] Player submitted name: Carl
[5/15/2013 2:17:30 PM] Leaving sequence: InputNameScreen, moving forward.
[5/15/2013 2:17:31 PM] Player submitted name: Carl
[5/15/2013 2:17:31 PM] Leaving sequence: startScreen, moving forward.
[5/15/2013 2:17:50 PM] Player submitted name: Carl
[5/15/2013 2:17:50 PM] Leaving sequence: slide2, moving forward.
[5/15/2013 2:17:55 PM] Player submitted name: Carl
[5/15/2013 2:17:55 PM] Leaving sequence: slide2b, moving forward.
[5/15/2013 2:18:34 PM] Player submitted name: Carl
[5/15/2013 2:18:34 PM] Leaving sequence: slide2c, moving forward.
[5/15/2013 2:20:09 PM] Player submitted name: Carl
[5/15/2013 2:20:09 PM] Leaving sequence: slide3, moving forward.
[5/15/2013 2:20:13 PM] Player submitted name: Carl
[5/15/2013 2:20:13 PM] Leaving sequence: slide4, moving forward.
How to approach it? Python is your friend!
1. Choose the necessary packages
import pandas as pd
import numpy as np
from datetime import datetime
2. Read the file
with open('unstructured_example_log.txt') as f:
txt = f.readlines()
3. Check the length and contents of the txt
len(txt)
4. Clean the list
n = len(txt)
for i in range(n):
txt[i] = txt[i].strip()
or in a more elegant way
txt = [t.strip() for t in txt]
5. Chucking
txt[0].split(']')
txt[0].split(']')[1]
txt[0].split(']')[1].upper()
6. Datetime
s = txt[0].split(']')[0].strip('[')
dtfmt ='%m/%d/%Y %I:%M:%S %p' # %H -> 24 hours, %I-> 12 hours, for ISO 8601 format, use: %Y-%m-%dT%H:%M:%S.%f%z
dt = datetime.strptime(s, dtfmt)
7. Put into a data frame
first, we need to determine columns
col1 = []
col2 = []
col3 = []
then, we fill the columns
for line in txt:
s1=line.split(']')[0].strip('[')
dt = datetime.strptime(s1, dtfmt)
col1.append(dt)
s= line.split(']')[1].strip().split(':')
col2.append(s[0])
if len(s) == 2:
col3.append(s[1])
else:
col3.append(np.nan)
finally, we create the dataframe
df = pd.DataFrame([col1,col2,col3])
df = df.T
df.columns=['datetime','event_name', 'event_result']
8. Normalize datetime to seconds
df['delta_t'] =df.datetime - df.datetime[0]
convert that to seconds
df['delta_t_seconds'] = 0
for i in range(df.shape[0]):
df.ix[i,'delta_t_seconds'] = df.delta_t.iloc[i].seconds
9. Save to csv file
df.to_csv('test_log.csv', index=False)