Introduction
Python is a high-level, general-purpose programming language. Created by Guido van Rossum in 1989 as a scripting language for administrative tasks; it was named after a television show. First released in 1991. There has been a growing community of Python developers since then, evolving into a well-supported programming language. Python 3.0, released in 2008, was a significant revision of the language. Python has become a common programming language for machine learning and the primary language for TensorFlow. The Python Software Foundation (PSF), a non-profit organization, manages for Python development. See Python website at www.python.org
Installing Python¶
Anaconda distribution of Python is recommended since it already contains many data science packages. Anaconda is free (the download is large and take time) and can be installed on work or school or personal computers. Anaconda comes bundled with about pre-installed 600 packages. Download the latest version of Python 3 (As of August 2020, 3.8) from Anaconda.com/downloads and install it on your computer (only press next).
Installing Jupyter and Required Modules¶
The Jupyter notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text. Jupyter Notebooks Allow Python and Markdown to coexist. We can write LaTeX equation in Jupyter notebook:
$ f'(x) = \lim_{a\to0} \frac{f(x+a) - f(x)}{a}. $
Jupyter notebooks will be frequently used in this course. It should be installed into your base Python environment. Go to Windows Start menu, type Anaconda Prompt (Anaconda3), right click on that and select Run as administrator to launch the anaconda console (see Figure below). Check to see if pip (pip is a package manager for Python packages, or modules) is installed by typing this command in the anaconda console and press enter: pip3 --version
Make sure that you have a recent version of pip installed. To upgrade the pip module, type this command in the anaconda console and press enter: pip3 install --upgrade pip
Now install and upgrade all required modules and their dependencies by typing this command in the anaconda console: pip3 install --upgrade jupyter matplotlib numpy pandas scipy scikit-learn
It may take a few minutes to complete. Try to import every module to check installation:
python -c "import jupyter, matplotlib, numpy, pandas, scipy, sklearn"
There should be no error and no output in console after pressing enter.
Launch Jupyter Notebook¶
Install jupyter_nbextensions_configurator by typing this command in the anaconda console and press enter:
pip install jupyter_nbextensions_configurator
Type jupyter notebook in the anaconda console and press enter to pop up empty workspace. Now create a new Python notebook by clicking on the New button (and selecting the appropriate Python version) (see numbers 1 and 2 in Figure blow).
A new notebook file called Untitled.ipynb is created in your workspace; it also starts a Jupyter Python kernel to run this notebook; and, this notebook opens in a new tab. This notebook can be renamed to “Script” (this automatically rename the file to Script.ipynb) by clicking Untitled and typing the new name (see number 3 in Figure blow). A notebook has a list of cells. Each cell can include executable code or formatted text. The notebook contains only one empty code cell when it is launched first (see Figure below for labeled In [1]:. Now type print("Hello world!") in the cell (see number 4 in Figure blow), and click on the play button (see number 5 in Figure blow) or press Shift-Enter. This sends the current cell to this notebook’s Python kernel, which runs it and returns the output. The result is displayed below the cell, a new cell is automatically created since reaching the end of the notebook Aurélien Géron, 2019.
Launch Jupyter Notebook in a Specific Folder¶
If you want to launch a new or previously saved notebook in a specific directory, go to Windows Start menu and type Anaconda Prompt (Anaconda3) to launch the anaconda console: you can pin Anaconda Prompt to taskbar (see Figure below).
Then, if you want to change drive type this command: drive name:. For example, if you want to go from drive C to F, type F: (see Figure below). Then, if you want to a folder, type cd folder's name. For example, Figure below shows how to launch a new notebook in drive F, folder Python.
Introduction to Python¶
Print Function, Srings and Variables¶
We start with printing "Hello World" similar to most programming tutorial. The code below passes a constant string, containing the text "Hello World" to the print function: a string is a series of characters, surrounded by single or double quotes.
In [1]:
print('Hello World')
To elaborate what you are coding for readers of your code, you can have comments in your code. In Python, any line of instructions containing the # symbol means the start of a comment. The rest of the line will be ignored when the program is run.
In [2]:
# When the program is run, this part has no effect.
print("Hello World") # print the string "Hello World"
Python also allows numbers as literal constants. An integer or a floating-point number can be differentiated by a decimal point. For example, the value 88 is an integer; 88.0 is a floating-point number. The value 88.0 is a floating-point number, but without fractional part. The value 88.6 is a floating-point number with fractional part. See the codes below.
In [3]:
print(88)
print(88.0)
print(88.6)
We have seen string values and literal numeric. When your program is run, the constants do not change. We can have variables that change as the program runs: variables are used to store values. The variables have names referencing their values. The code below assigns a string value to a variable "x" and an integer value to another variable "y".
In [4]:
x = 3
y = "three"
print(x)
print(y)
You can rewrite the code above to reduce length of your code using semicolon ;.
In [5]:
x = 3; y = "three"; print(x); print(y) # using ";" to reduce numebr of lines in your code
The variables can change as you run your code.
In [6]:
x = 10; print(x)
x = x + 20; print(x) # add 20 to the variable with the assigned value 10
Srings and variables can be concatenated (combining strings) in python. The variables with number must be converted to string with "str" function.
In [7]:
x = 'This course is '
y = 'Machine Learning – Energy Systems'
print(x+y)
In [8]:
x = 'The value of y variable is '
y = 8
print(x+str(y))
In [9]:
x = 7; y = 10
print('x+y= '+str(x+y))
Although Python has many other ways to print numbers, we will use str() function for this course. The following code presents other ways of printing numbers:
In [10]:
x = 25
print('The value of x is ' + str(x)) # This is preferred in this course.
print(f'The value of x is {x}')
print('The value of x is {}'.format(x))
print('The value of x is %d' % (x))
Lists and Tuples, Dictionaries¶
Similar to most modern programming languages, Python contains built-in data structures. In this section, each of these collection types will be discussed in more detail.
Lists and Tuples¶
A Python list stores a series of items in a particular order. Many programming languages includes a data collection called an array. There is no built-in array type in Python. However, a list can be easily used in place of an array in Python. Arrays in most programming languages requires to assign maximum length ahead of time. This restriction leads to array-overrun (overflow) bugs. The Python list does not require to assign length and the program can dynamically alter the size of a list. Lists allow programmers to store sets of information in one place, does not matter a few items or millions of items. Lists are one of Python's most powerful features.
Tuples are similar to lists. The only difference is the items in a tuple can't be modified: a tuple is immutable, which means the program cannot change it. It is possible to get by as a programmer using only lists and ignoring tuples. One advantage of tuples over lists is tuples may be a little faster for iteration over than lists.
For making a list, use square brackets [], and use commas , to separate individual items in the list.
In [11]:
Lists = ['a', 'b', 'c', 'd']
print(Lists)
Individual elements in a list are called the index. The index of the first element is 0, the index of the second element is 1, and so forth. Negative indices denote to the items at the end of the list. To get a specific element, write the name of the list and then the index of the element in square brackets.
$List= \,\,\,\,[\,\,\,\,'a'\,\,\,,\,\,\,'b' \,\,\,,\,\,\, 'c'\,\,\,\,,\,\,\,\,'d'\,\,\,\,]\\ Index:\,\,\,\,\,\,\,\,\,0\,\,\,\,\,\,\,\,\,\,\,\,1\,\,\,\,\,\,\,\,\,\,\,\,\,2\,\,\,\,\,\,\,\,\,\,\,\,\,3 \\ Index:\,\,\,\,\,-4\,\,\,\,\,\,\,-3\,\,\,\,\,-2\,\,\,\,\,\,\,-1$
In [12]:
Lists[-1] # Get last element
Out[12]:
In [13]:
Lists[0] # Get first element
Out[13]:
You can change individual elements in a list by referring to the index of the item you want to change.
In [14]:
Lists[0] = 'a_changed' # Change first element to a_changed
Lists[2] = 'c_changed' # Change third element to c_changed
print(Lists)
The items in a tuple can't be modified and it gives TypeError. Tuples are enclosed with parenthesis ().
In [15]:
Tuples = ('a', 'b', 'c', 'd')
print(Tuples)
In [16]:
#Tuples[0] = 'a_changed' # Try to change first element of tuples but it gives TypeError
Tuples do not allow the program to add any element after definition. This is possible with list; you can add elements to the end (using append() function) or wherever (using insert() function) you like in a list.
In [17]:
# Adding an element to the end of the list
Lists.append('added_element')
print(Lists)
In [18]:
# Making a list with an empty list
Lists = []
Lists.append('First')
Lists.append('Second')
Lists.append('Third')
print(Lists)
The programmer must specify an index for the insert function.
In [19]:
# Insert
Lists = ['a', 'b', 'c',]
Lists.insert(2, 'a2')
print(Lists)
Elements can be removed by their position in a list (using remove() function) , or by the value of the item (using del() function). Python removes only the first item if you remove an item by its value.
In [20]:
Lists.remove('a2') # Remove by value
print(Lists)
In [21]:
del Lists[0] # Remove at index
print(Lists)
If a list contains numerical data, there are a number of simple statistics you can be applied.
In [22]:
ages_years = [92, 51, 78, 12, 63, 82, 65, 51, 4, 35]
youngest = min(ages_years) # Find the minimum value
print('The youngest age is '+str(youngest))
#
oldest = max(ages_years) # Find the maximum value
print('The oldest age is '+str(oldest))
#
total_years = sum(ages_years)
print('The total years are '+str(total_years))
You can work on a portion of a list that is called a slice. To slice a list, start with the index of the first item you want, then add a colon and the index after the last item you want. To start at the beginning of the list, leave off the first index. To slice through the end of the list, leave off the last index.
In [23]:
# Get the three items from index 1 to 3
lists = ['a', 'b', 'c', 'd', 'e', 'f']
middle_three = lists[1:4]
print(middle_three)
In [24]:
# Get the first three items
first_three = lists[:4]
print(first_three)
In [25]:
# Get the last three items
last_three = lists[-3:]
print(last_three)
You can find the index of a value in a list using index() function. Index() returns the lowest index in the list:
In [26]:
index=lists.index('a')
print(index)
If the value is not in the list, it returns a ValueError.
In [27]:
#index=lists.index('k')
In [28]:
# What lowest index means
lists_1 = ['a', 'b', 'a', 'd']
index=lists_1.index('a')
print(index)
Dictionaries¶
Python provides a dictionary that allow you to connect pieces of related information. Each piece of information in a dictionary is stored as a key-value pair. Python returns the value associated with that key when a key is provided.
For making a dictionary, use curly braces {} to define a dictionary. Use colons ":" for connecting keys and values, and commas "," for separating individual key-value pairs.
In [29]:
Dic = {'name': 'Tom', 'midterm': 87,'Final':95 ,'grade':'A'}
print(Dic['name'])
print(Dic['Final'])
In [30]:
# Get the value of a key with get()
print(Dic.get('name'))
print(Dic.get('Final'))
In [31]:
# All of the keys of dictionary
print(Dic.keys())
In [32]:
# All of the values of dictionary
print(Dic.values())
We can have a list for each key in dictionary. The following code shows a list of five values for each key.
In [33]:
name=['Tom', 'Liam', 'Jared', 'Alex', 'John']
grade=['A','A+','A-','B+','A']
Dic_l = {'name': name,'grade':grade}
i=0
print('Name= ',Dic_l['name'][i],', grade= ',Dic_l['grade'][i])
Arrays¶
As it was mentioned, in Python we have lists that serve the purpose of arrays. However, lists can be very slow to process for large data set. Arrays are very frequently applied in data science, where speed and resources are very important. NumPy is a Python library used for working with arrays. The reason why NumPy is faster than lists is NumPy arrays are stored at one continuous place in memory unlike lists, so it can be very efficiently processed and manipulated.
First you need to install NumPy and import it in your applications by adding the import keyword. You can create an alias with the as keyword while importing:
In [34]:
import numpy as np
list_=[1, 2, 3, 4, 5, 6, 7, 8]
arr = np.array(list_)
print(arr)
Dimensions¶
0-D Arrays (Scalars)
0-D Arrays are Scalars. Each value in an array is a 0-D array.
In [35]:
arr = np.array(82)
print(arr)
1-D Arrays (Vectors)
Uni-dimensional or 1-D array is an array that has 0-D arrays (values) as its elements.
In [36]:
arr = np.array([1, 2, 3, 4, 5, 6, 7, 8])
print(arr)
2-D Arrays (Matrices)
A 2-D array is an array that has 1-D arrays as its elements. 2-D arrays often used to represent matrix or second order tensors.
In [37]:
arr = np.array([[1, 2, 3, 4, 5, 6], [7, 8, 8, 10, 11, 12]]) # This is 2-D matrix that has two rows and 6 columns
print(arr)
3-D Arrays
3-D array is an array that has 2-D arrays (matrices) as its elements. 3-D array are often used to show third order tensor.
In [38]:
import numpy as np
arr = np.array([[[1, 2], [3, 4]], [[5, 6], [7, 8]]])
print(arr)
ndim attribute tells how many dimensions the array have.
In [39]:
arr.ndim
Out[39]:
Shape of an array can be achieved by shape attribute that returns a tuple with each index having the number of corresponding elements.
In [40]:
arr.shape
Out[40]:
Slicing 1D-arrays are similar to list:
In [41]:
arr = np.array([1, 2, 3, 4, 5, 6, 7])
print(arr[-1])
Slicing 2-D arrays:
In [42]:
arr = np.array([[1, 2, 3, 4, 5], [6, 7, 8, 9, 10]])
print(arr[0, :2])
We can also reshape an array from one dimension to another dimension using reshape function:
In [43]:
arr = np.array([1, 2, 3, 4, 5, 6, 7, 8])
print(arr.shape)
new_arr = arr.reshape(4, 2)
print(new_arr.shape)
Searching a Value in Arrays
you can find all indices of a value in an array by where():
In [44]:
arr = np.array([3, 5, 3, 2, 3, 8, 7, 9])
idx = np.where(arr==3)
print(idx)
Generate Random Number¶
Random means that the data can not be predicted logically. The code below generates random integer from 10 to 50 with NumPy. First you need to import random.
In [45]:
from numpy import random
random.seed(89)
rand = random.randint(10,50)
print(rand)
The random module's rand() method returns a random float between 0 and 1.
In [46]:
#random.seed(45)
rand = random.rand()
print(rand)
The following code generates a 1-D array containing 5 random integers from 10 to 50:
In [47]:
rand = random.randint(10,50, size=(100))
print(rand)
The following code generates a 2-D array with 2 rows each has 4 random integer from 10 to 50:
In [48]:
rand = random.randint(10,50, size=(2,4))
print(rand)
The code below generates a 1-D array containing 5 random floats:
In [49]:
rand = random.rand(5)
print(rand)
2-D array of float numbers can be generated:
In [50]:
rand = random.rand(2, 3)*20
print(rand)
If Statements and Loops¶
If statements are applied to test for particular conditions and respond properly. Conditional tests are:
not equal $\;\;\;\;\;\;\;\;\;$ x != 82
equals $\;\;\;\;\;\;\;\;\;\;\;\;$ x == 82
greater than $\;\;\;\;\;$ x > 82
or equal to $\;\;\;\;\;\;\;$ x >= 82
less than $\;\;\;\;\;\;\;\;\;$ x < 82
or equal to $\;\;\;\;\;\;\;$ x <= 82
If-statements require appropriate indents to define blocks of code to execute together. A block usually begins after a colon and includes any lines at the same level of indent. Python uses whitespace to define blocks of code, unlike many other programming languages. One of main annoyance for new Python programmers is defining this whitespace."
In [51]:
# If-elif-else statements
val = 20
if val<20:
print('The variable is less than 20')
elif val>20:
print('The variable is greater than 20')
else:
print('The variable is equal to 20')
To loop over a range of numbers in Python, the range function can be used. The following code shows a for loop and a range function to loop between 1 and 5.
In [52]:
for number in range(6):
print(number)
In [53]:
for number in range(1, 6):
print('Iteration '+str(number)) # looped printing of string and number.
In [54]:
# Use loop, range and if functions to generate a list of square numbers for even number
sqs= []
for x in range(1, 11):
if (x%2==0): # A number is even if division by 2 gives a remainder of 0
sq= x**2
sqs.append(sq)
print(sqs)
In [55]:
# Looping over a list
add=0
for x in sqs:
add+=x
print("Add "+str(x)+", suming number so far= "+str(add) )
We can use python comprehension to generate a list; it is the same as for loop.
In [56]:
number = [x for x in range(1, 6)]
print(number)
In [57]:
# Use comprehension to generate a list of square numbers for even number
sqs = [x**2 for x in range(1, 11) if (x%2==0)]
print(sqs)
Zip and Enumerate¶
Two lists can be combined into a single list by the zip command.
To see the results of the zip function, we convert the returned zip object into a list. As you can see, the zip function returns a list of tuples. Each tuple represents a pair of items that the function zipped together. The order in the two lists was maintained.
In [58]:
x = [12,18,25,33,37]
y = [1,2,3,4,5]
for x,y in zip(x,y):
print('Print x - y= ',x-y)
You can use the enumerate command to define the index location for an element.
In [59]:
x = [7, 50, 75, 99, 32, 49, 67, 58]
for i, j in enumerate(x):
print('index= '+str(i)+', value= '+str(j))
The following program changes every element of the list less than 50 to 0; and filter the list. The enumerate command makes it possible to know which element index it is currently on, to change the value of that.
In [60]:
x = [7, 50, 75, 99, 32, 49, 67, 58]
new_list=[]
for i, j in enumerate(x):
if j<50:
x[i] = 0
else:
new_list.append(x[i])
print('List for less than 50 to 0= ',x)
print('Filter list for less than 50= ',new_list)
The enumerate command can be used to build a dictionary. This approach is commonly used to build an index to column names.
In [61]:
colmn = ['col1','col2', 'col3', 'col4']
Dic = {key:value+1 for (value,key) in enumerate(colmn)}
print(Dic)
we saw that index() function only return the lowest, but how we can get all indices of a value in list? we can use enumerate and comprehension:
In [62]:
list_=[1,3,1,9,45,85,1,69,3,4,2,1,9,3,7,2]
# Get lowest index
idx=list_.index(1)
print('Lowest index= ',idx)
# Get all indices using enumerate and comprehension
inds = [idx for idx, n in enumerate(list_) if n == 1]
print('All indices= ',inds)
Reading and Writing a File¶
To read from a file, your code should open the file and then read the contents. It is possible to read the entire contents of the file at once, or read the file line by line.
In [63]:
# Reading an entire file at once
filename = './Data/Wells.txt'
with open(filename) as f_cont:
contents = f_cont.read()
print(contents)
For reading line by line, each line has a newline character at the end of the line. The rstrip() function eliminates the the extra blank lines.
In [64]:
# Reading line by line
with open(filename) as f_cont:
for line in f_cont:
print(line.rstrip())
The advantage of reading line by line is that you can filter or manipulate the data while reading. This is very efficient for reading very large dataset. For example, the following code skip two lines.
In [65]:
# Reading line by line
with open(filename) as f_cont:
for i in range(2):
next(f_cont) # Skip two lines
for line in f_cont:
print(line.rstrip())
To get a specific column, you should split the line with function split. The following code shows how to get depths of the wells (column 3). Notice, python reads data as string, if you want to do any math on the data, you should convert it to number by float() function.
In [66]:
# Reading line by line
Depth_km=[]
with open(filename) as f_cont:
for i in range(2):next(f_cont) # Skip two lines
for line in f_cont:
p = line.split() # Split line
Depth_m = float(p[2]) # Get depth value which is index 2 (column 3) and convert to number
Depth_km.append(Depth_m/1000)
print('Depth in km',Depth_km)
Now if you want to write a list to a file, you should pass the 'w' argument to open() that tells Python to write to the file. Be careful; this will erase the contents of the file if it already exists. Passing the 'a' argument tells Python you want to append to the end of an existing file.
In [67]:
filename = 'Output.txt'
with open(filename, 'w') as f:
for i in range(len(Depth_km)):
if(i==0):
f.write('Depth of wells in km\n') # Make a header for your output file
f.write(str(Depth_km[i])+'\n')
You can import excel file in python. But, first you should import xlrd. The following code shows how to read data from sheet2 of th excel file "Wells.xlsx"
In [68]:
import xlrd
book = xlrd.open_workbook('./Data/Wells.xlsx')
# Select excel sheet starts with 0
sheet_num=0
sheet = book.sheet_by_index(sheet_num)
# Get number of rows and columns
rows = sheet.nrows
columns = sheet.ncols
print('Rows=',rows,'Columns=',columns)
# Read row and column starts with 0
row=2
column=0
val=sheet.cell_value(row, column)
print('The value of row '+str(row+1)+ ' and column '+str(column+1)+' is',val)
In [69]:
# Read the entire selected column 2
column=1
for i in range(rows-1):
print(sheet.cell_value(i+1, column) )
In [70]:
# Read 4th row of sheet 2
current_row=3
for i in range(sheet.ncols):
if(i==0): print('Reading row '+str(current_row+1))
print(sheet.cell_value(current_row,i))
Reading and writing data, data analysis, data manupulation and processing is easier with Pandas open source library for Python. We will discuss about it in the coming classes.
Functions and Lambdas¶
Functions and lambdas are advanced methods to process. Here, these techniques are introduced with some examples and will be discussed more in next classes.
Functions¶
Functions are designed to do one specific job, and named blocks of code. Function only runs when it is called. They are used in all programming language. The main aim of writing functions is to avoid a long code (avoid repetition) and have a clean and well-organized code. It starts with def followed by function name of function, parentheses () and colon :.
In [71]:
# A simple function
def greet():
"""Greeting.""" # Give a little information about the fucntion
print("Hello World!")
greet()
In [72]:
# Passing an argument
def greet(text):
"""Display greeting."""
print(text)
mytext= 'Hi, how are you doing!'
greet(mytext)
In [73]:
# Making default values for parameters
def pizza_order(type_='chicken',topping='Mushroom',size='Large'):
"""Order pizza."""
print("Type of pizza: " + type_)
print("Topping : " + topping)
print("Size: " + size)
#pizza_order()
pizza_order(type_='beef',topping='olives')
In [74]:
# Returning a value
def multipy_numbers(x, y):
""" Multipy two input parameters."""
value=x*y
return value
multipy_numbers(20,10)
Out[74]:
Filter¶
This is a function to filter data based on some rules.
In [75]:
def positive(x):
return x>0
mylist=[-10,4,-2,36,6,-8,12]
filtered=list(filter(positive, mylist))
print(filtered)
Lambda¶
It is tedious to have a function for a simple task of finding values greater than 0. A lambda function is very efficient which an unnamed function.
In [76]:
mylist=[-10,4,-2,36,6,-8,12]
filtered=list(filter(lambda i: i>0, mylist))
print(filtered)
Data visualization¶
One advantage of Python compared with other programming language is its capability of excellent data visualization. The matplotlib package of Python is extremely flexible to make interesting representations of the data. You should install matplotlib first (see previous lectures) then import it before use. Here are a few examples of data visualization.
In [77]:
# Make a graph with Plot() function
import matplotlib.pyplot as plt
x = [-30,-20, -10, 0, 10, 20, 30]
x_squared = [900, 400, 100, 0, 100, 400, 900]
plt.plot(x, x_squared)
plt.show()
In [78]:
# Make a graph with Plot() function
x = list(range(-30,30))
x_squared = [i**2 for i in x] # use comprehension to square your values
plt.plot(x, x_squared)
plt.show()
In [79]:
# Make a graph with Scatter() function
x = list(range(-30,30))
x_squared = [i**2 for i in x]
plt.scatter(x, x_squared)
plt.title("Plot for Squared Numbers", fontsize=15) # Title of your plot
plt.xlabel("Value", fontsize=12) # Label of your x axis
plt.ylabel("Squared Value", fontsize=12) # Label of your y axis
plt.show()
In [80]:
# Make a location map with scatter()
x_ = [50,220,986,830,680,357,773,935,400]
y_ = [269,789,30,99,698,947,875,624,400]
value_ = [0.368,0.051,0.224,0.198,0.287,0.329,0.094,0.187,0.33]
plt.scatter(x_, y_,c=value_,s=105,cmap='jet', edgecolor='k')
plt.colorbar()
plt.xlim((0, 1000))
plt.ylim((0, 1000))
plt.title("Location Map", fontsize=18) # Title of your plot
plt.xlabel("Easting (m)", fontsize=14) # Label of your x axis
plt.ylabel("Northing (m)", fontsize=14) # Label of your y axis
plt.savefig('Figure.png', dpi=200, bbox_inches='tight') # SavE a plot
plt.show()
In [81]:
# Make histogram
plt.hist(x_squared, bins=8,ec='black')
plt.title("Histogram ", fontsize=15) # Title of your plot
plt.xlabel("Squared Value", fontsize=12) # Label of your x axis
plt.ylabel("Frequency", fontsize=12) # Label of your y axis
plt.show()
You can include as many individual graphs in one figure.
In [82]:
fig = plt.subplots(figsize=(10,3), dpi= 200, facecolor='w')
ax1=plt.subplot(1,2,1)
plt.scatter(x_, y_,c=value_,s=105,cmap='jet', edgecolor='k')
plt.colorbar()
plt.title("Location Map", fontsize=15)
plt.xlabel("Easting (m)", fontsize=12)
plt.ylabel("Northing (m)", fontsize=12)
plt.xlim((0, 1000))
plt.ylim((0, 1000))
ax2=plt.subplot(1,2,2)
plt.hist(x_squared, bins=15,ec='black')
plt.title("Histogram ", fontsize=15)
plt.xlabel("Squared Value", fontsize=12)
plt.ylabel("Frequency", fontsize=12)
plt.savefig('Subplot.png', dpi=200, bbox_inches='tight') # SavE a plot
plt.show()
Introduction to Pandas¶
Pandas is an open source Python package for data manipulation and analysis with high-performance and easy-to-use methods. It is based on the dataframe concept in the R programming language. Pandas can be used as the primary tools for data processing, data manipulation and data cleaning.
We will use a dataset for oil production of 1000 hydrocarbon wells in Alberta (WPD.csv). The target is oil production prediction using well properties including location, depth, deviation, porosity, permeability and so on. The well properties are retrieved from geoSCOUT; but location of the wells are changed and some key properties are manipulated for confidentially reason and for not being used for other reasons. This data is retrieved and generated for this course to learn data processing and apply Machine Learning. The prediction should NOT be used for any Publication because the original data is modified data.
Look at Data¶
In [83]:
# Reading data in Pandas
import pandas as pd # import Pandas
df = pd.read_csv("./Data/WPD.csv") # Read data file
#df.head()
df[0:5] # Display data
Out[83]:
The columns function gives the name of the columns in your data file.
In [84]:
column=list(df.columns)
print(column)
The info() function gives some information about the data type: string float or null (missing values).
In [85]:
df.info()
The describe function gives some statistical analysis for each column if the data type is not string.
In [86]:
df.describe()
Out[86]:
Data Cleaning¶
Missing (NaN) Values¶
It is ideal to have valid value for every rows and columns. In reality, missing values is one of the main challenges faced by machine learning. There are some approaches in practice to deal with missing values.
Option 1: Drop Column
One naive approach is to remove columns that have missing values with drop() function.
In [87]:
Select_column=column[3:8] # Select columns that have missing
Select_column
Out[87]:
In [88]:
df_drop_c=df.copy() # Copy to avoid modifying the original data
df_drop_c.drop(Select_column,axis=1, inplace=True) # Drop selected 5 columns
df_drop_c
Out[88]:
In [89]:
aa=df_drop_c.drop(1,axis=0)
aa
Out[89]:
In [90]:
df_drop_c
Out[90]:
In [91]:
df_drop_c.info()
This approach is not reliable since the information of 5 columns without missing values have been removed from data set.
Option 2: Drop Row
Another approach is to remove rows that have missing values with dropna() function.
In [92]:
df_drop_=df.copy() # Copy to avoid modifying the original data
df_drop_r=df_drop_.dropna() # Drop rows with missing values
df_drop_r
Out[92]:
In [93]:
df_drop_r.info()
This approach is also not recommend because number of data (rows) have decreased from 1000 to 426.
Option 3: Replace with Median (mean)
A common practice is to replace missing values with the median (mean) value for that column. The median is the middle value of a list that 50% are less and 50% are bigger. See Median. The following code replaces all missing values in 5 columns with the median value of each column values.
In [94]:
df_im=df.copy()
# Get the median of each variable and replace the missing values with median
for i in range(len(Select_column)):
P50 = df[Select_column[i]].median() # Calculate median of each column
df_im[Select_column[i]] = df_im[Select_column[i]].fillna(P50) # replace NaN with median(P50)
df_im
Out[94]:
In [95]:
df_im.info()
Remove Outliers¶
Outliers are usually extreme high or low values that are different from pattern of majority data. Outliers should be removed from dataset (we should be careful not to consider novelty as outlier). A straightforward approach to detect outliers are several (n) standard deviations from the mean $m\pm n\sigma$ ($m$=mean, $\sigma$=standard deviation). For example, an outlier can bigger than $m+ 3\sigma $ or less than can bigger than $m- 3\sigma $. The following function can be used to apply this approach.
In [96]:
def outlier_remove(df, n,name):
"""Delete rows for a specified column where values are out of +/- n*sd standard deviations
df : Pandas dataframe
n : n in the equation □±□□
name: Column name
"""
mean=df[name].mean() # Calclute mean of column
sd=df[name].std() # Calclute standard deviation of column
drop_r = df.index[(mean -n * sd> df[name]) | (mean+n * sd< df[name])] # Find data that are not within □±□□
df.drop(drop_r, axis=0, inplace=True) # Drop data
df.reset_index(inplace=True, drop=True) # Reset index
In [97]:
# Drop outliers in last column 'OIL Prod. (e3m3/month)'
df_im_out=df_im.copy()
outlier_remove(df_im_out,n=3,name='OIL Prod. (e3m3/month)')
df_im_out
Out[97]:
Concatenation¶
A new data frames can be built by concatenating rows and columns. A new dataframe is created by concatenating two columns together 'Measured Depth (m)' and 'OIL Prod. (e3m3/month)' using the concat() function.
In [98]:
col_1= df_im['Measured Depth (m)']
col_2 = df_im['OIL Prod. (e3m3/month)']
result = pd.concat([col_1, col_2], axis=1)
result
Out[98]:
Two rows can be concatenated together using concat() function. The code below concatenates the first 4 rows and the last 3 rows of the previous dataset after imputation.
In [99]:
row_1= df_im[0:4] # Retrieve first 4 rows
row_2 = df_im[-4:-1] # Retrieve last 3 rows
result = pd.concat([row_1, row_2], axis=0) # Concatenate rows
result
Out[99]:
Saving a Dataframe as CSV¶
You can simply save a dataframe as CSV by the following code. The following code performs a shuffle and then saves a new copy.
In [100]:
filename="myfile.csv"
df_im.to_csv(filename, index=False) # index = False not writing row numbers
Shuffling, Grouping and Sorting¶
Shuffling Dataset
An example of shuffling is playing the game of cards. All the cards are collected at the end of each round of play, shuffled to ensure that cards are distributed randomly and each player receives cards based on chance. In Machine Learning, the data set should be split into training and validation datasets. So, shuffling is very important to avoid any element of bias/patterns in the split datasets.
If consistent shuffling of the data set is required for several times of running the code, a random seed can be used. See code below.
In [101]:
import numpy as np
#np.random.seed(82)
df_shfl = df_im.reindex(np.random.permutation(df.index))
df_shfl.reset_index(inplace=True, drop=True) # Reset index
df_shfl[0:5]
Out[101]:
Grouping Data Set
Grouping is applied to summarize data. The following codes performs grouping; generating the mean and sum of each category for feature 'Prod. Formation' in target variable 'OIL Prod. (e3m3/month)'.
In [102]:
gb_mean=df_im.groupby('Prod. Formation')['OIL Prod. (e3m3/month)'].mean()
gb_mean
Out[102]:
In [103]:
gb_sum=df_im.groupby('Prod. Formation')['OIL Prod. (e3m3/month)'].sum()
gb_sum
Out[103]:
Sorting Data Set
In [104]:
df_sort = df_im.sort_values(by='Measured Depth (m)', ascending=True)
#df_sort.reset_index(inplace=True, drop=True) # Reset index
df_sort
Out[104]:
Feature Engineering¶
Feature engineering is the process of extracting new feature from raw data via data mining techniques. The new features can be used to enhance the performance of machine learning algorithms. New features could also be calculated from the other fields. For the data set WPD.csv, we can add "Surface-Casing Weight (kg/m)" and "Production-Casing Weight (kg/m)" as a new data set.
In [105]:
df_im_c=df_im.copy()
df_im.insert(1, 'New Column', (df_im['Surface-Casing Weight (kg/m)']*df_im['Production-Casing Weight (kg/m)']).astype(int))
df_im
Out[105]:
Standardize (Normalize) Values¶
Most Machine Learning algorithms require to standardize the input to have a reliable comparison between two values of different variables. For example, if you have variable one with a range of 3 to 1000 and variable two with range 1 to 3; we cannot have a good comparison since the highest value of variable two (3) is the lowest value in variable two. This inconsistency in data values leads to low performance in Machine Learning algorithms. So, it is highly recommended to standardize (normalize) your data to have the same the mean($\mu$) and the standard deviation ($\sigma$) before feeding Machine Learning algorithms. One very common Machine learning standardization (normalization) is the Z-Score:
$\Large z = \frac{x - \mu}{\sigma} $
where $ \mu = \frac{1}{n} \sum_{i=1}^n x_i$, $\sigma = \sqrt{\frac{1}{n} \sum_{i=1}^n (x_i - \mu)^2}$, $n$ is number of data. This ensures that each variable have $\mu=0$ and $\sigma=1$
This standardization should be applied for all features except the target (label). For WPD.csv data set, the target is "OIL Prod. (e3m3/month)" that should not be changed.
The following Python code replaces the 'Measured Depth (m)' with a z-score. This can be done for all variables.
In [106]:
from scipy.stats import zscore
mean=df_im['Measured Depth (m)'].mean()
variance=df_im['Measured Depth (m)'].var()
print("Before standardization: Mean= ",int(mean), ", variance= ",int(variance))
df_im['Measured Depth (m)'] = zscore(df_im['Measured Depth (m)'])
mean=df_im['Measured Depth (m)'].mean()
variance=df_im['Measured Depth (m)'].var()
print("After standardization: Mean= ",int(mean), ", variance= ",int(variance))
In [107]:
df_im[0:5]
Out[107]:
Categorical Values¶
The input data for Machine Learning should be completely numeric which means any text should be converted to number. In the data set WPD.csv, the columns 'Deviation (True/False)' and 'Prod. Formation' have categorical values. Use value_counts() function to find out what categories exist and how many data belong to each category.
In [108]:
df_c=df_im.copy()
df_c['Deviation (True/False)'].value_counts()
Out[108]:
In [109]:
df_c['Prod. Formation'].value_counts()
Out[109]:
We can replace each test with numbers starting from 0
In [110]:
df_c['Deviation (True/False)']=df_c['Deviation (True/False)'].replace(False, 0) # Replace False with 0
df_c['Deviation (True/False)']=df_c['Deviation (True/False)'].replace(True, 1) # Replace True with 1
df_c[0:5]
Out[110]:
In [111]:
df_c['Prod. Formation']=df_c['Prod. Formation'].replace('Shale', 0) # Replace Shale with 0
df_c['Prod. Formation']=df_c['Prod. Formation'].replace('Sand', 1) # Replace Sand with 1
df_c['Prod. Formation']=df_c['Prod. Formation'].replace('Shale-Sand', 2) # Replace Shale-Sand with 2
df_c[0:5]
Out[111]:
One challenge with this representation is that Machine Learning algorithms consider two nearby values are more similar than two distant values. This may be fine for some cases (order of categories “bad”, “average”, “good”), but this is not always true. A common solution is to have one binary attribute per category. This is called one-hot encoding, because we only have one attribute equal to 1 (hot), while the others are 0 (cold).
In [112]:
One_hot = pd.get_dummies(['a','b'],prefix='Deviation')
print(One_hot)
These dummies should be merged back into the data frame.
In [113]:
One_hot = pd.get_dummies(df_im['Deviation (True/False)'],prefix='Deviation')
print(One_hot[0:10]) # Just show the first 10
In [114]:
df_one_hot = pd.concat([df_im,One_hot],axis=1)
df_one_hot[0:5]
Out[114]:
This should be done for 'Prod. Formation' column.
In [115]:
One_hot = pd.get_dummies(df_im['Prod. Formation'],prefix='Formation')
df_one_hot = pd.concat([df_one_hot,One_hot],axis=1)
df_one_hot[0:5]
Out[115]:
Finally, the original columns 'Deviation (True/False)' and 'Prod. Formation' should be removed.
In [116]:
df_one_hot.drop(['Deviation (True/False)','Prod. Formation'], axis=1, inplace=True)
#df_one_hot
Split Data to Training and Validation¶
Machine Learning models should be evaluated based on prediction for never-before-seen data set. The models usually have high performance for the data that are trained by. However, the main aim of training is to have high performance on completely new data set. So, data are split into training and validation set. The machine learning models learn from the training data, but based on the validation data are evaluated: the data should be divided into training and validation set according to some ratio. The common ratios are 80% training and 20% validation. The image below shows how a model is trained on 80% of the data and then validated against the remaining 20%. Based on feedback performance on evaluation set, Machine learning algorithms are modified. This process is repeated until the model's performance is satisfied. This repetition also could lead to information leak. The model could perform well on the validation set but not on a new data set. So, sometimes, a portion of data are hold-out for testing. We will talk about this on the next lectures
The following code splits the data into 80% training and 20% validation set.
In [117]:
import numpy as np
df = df.reindex(np.random.permutation(df.index)) # Shuffle the data
df.reset_index(inplace=True, drop=True) # Reset index
MASK = np.random.rand(len(df)) < 0.8
df_train = pd.DataFrame(df[MASK])
df_validation = pd.DataFrame(df[~MASK])
print("Number of Training: "+ str(len(df_train)))
print("Number of Validation: "+ str(len(df_validation)))