6.3 Reading and Writing Data

DataFrame Basics

Note

As always, read the Python documentation for more information on reading and writing files: Python open().

Python provides a built-in module for working with files, including reading and writing functions. These are generally built around the concept of a file object, which is an abstraction of a file on disk. You can open a file, read from it, write to it, and close it when you’re done.

Understanding File Paths

When working with files, it’s important to understand file paths. Paths are the series of folders (or directories) that lead from the root of your file system (in windows, this is typically the C: drive; in Unix-based systems like macOS and Linux, it’s simply /) to the file you want to access. Each folder name is separated by a / (forward slash) on Unix-based systems or a \ (backslash) on Windows systems. It is standard procedure to use forward slashes (/) in Python file paths, even on Windows systesms, as Python automatically converts them to backslashes when necessary. So a typical file path might look like this: /home/user/documents/data.txt (Unix-based) or C:/Users/User/Documents/data.txt (Windows).

There are two types of file paths:

Absolute Patsh: The path that starts at the root of the file system.
Relative Paths: The path that starts from the current working directory of your program.

Most frequently, you will use relative paths. When specifying the file path, we can use short hand indicators to represent moving through directries in relative paths. For instance a . (dot) represent the curren directory. A .. (double dot) represents the parent directory (one level up). For example, if your current working directory is /home/user/documents/ and you want to access a file in the parent directory called data.txt, you would use the relative path ../data.txt. If you are in the same directory as the file, you can simply use data.txt. I prefer to be more explicit and use ./data.txt to indicate that the file is in the current directory. But both work.

Opening Files

The open() function is used to open a file and returns a file object. The first argument is the file name, and the second argument is the mode in which you want to open the file. You can specify the mode in which you want to open the file, such as read ('r'), write ('w'), or append ('a'). Reading (r) a file means that you are opening it for reading its contents. You cannot write to a file opened in read mode. Writing (w) a file means that you are opening it for writing, and if the file already exists, it will be truncated (emptied) before writing. Appending (a) a file means that you are opening it for writing, but the existing contents of the file will not be truncated; instead, new data will be added to the end of the file. These are the basic modes for working with files in Python, but there are more. See the following table:

Mode	Description
`'r'`	Read mode (default) - opens a file for reading.
`'w'`	Write mode - opens a file for writing, truncating the file to zero length if it exists. If the file does not exist, it creates a new file.
`'a'`	Append mode - opens a file for writing, appending to the end of the file if it exists. If the file does not exist, it creates a new file.
`'b'`	Binary mode - opens a file in binary mode, which is used for non-text files (e.g., images, audio). This mode can be combined with other modes (e.g., `'rb'` for reading a binary file, `'wb'` for writing a binary file).
`'x'`	Exclusive creation mode - opens a file for writing, but fails if the file already exists. This is useful for creating new files without overwriting existing ones.
`'t'`	Text mode (default) - opens a file in text mode, which is used for text files. This mode can be combined with other modes (e.g., `'rt'` for reading a text file, `'wt'` for writing a text file).
`'+'`	Update mode - opens a file for both reading and writing.

Note

When combining modes, remember that "w+" and "w+b" will both open and truncate the file, while "r+" and "r+b" will open the file for reading and writing without truncating it. The 'b' mode is used for binary files, while the 't' mode is used for text files (default).

When you open a file, you can read its contents using methods like read(), readline(), or readlines(). The read() method reads the entire file into a single string, while readline() reads a single line at a time, and readlines() reads all lines into a list. Similarly, in writing mode, you can use methods like write() to write data to the file.

It is important when working with file objects to ensure that you close the file after you’re done with it to free up system resources. This can be done using the close() method or by using a context manager (with statement) which automatically closes the file when the block is exited.

Based on syntax we are familiar with so far, we would declare a variable, let’s call it file, and assign it the result of the open() function. For example, if we wanted to read a file called data.txt, we would do:

file = open('data.txt', 'r')

But when we are done with the file, we need to close it:

file.close()

However, following this pattern can lead to errors if we forget to close the file or if an error occurs before we close it. A better approach is to use a context manager, which ensures that the file is closed properly even if an error occurs. The context manager is a construct that sets up and tears down resources automatically using the with statement. It ensures that setup actions (like opening a file) and cleanup actions (like closing the file) are handled properly, even if errors occur. To use the context manager, we simply wrap the open() function in a with statement that creates a block of code that will automatically close the file when the block is exited. The with statement follows the pattern of with <context_manager> as <variable>:. In this case, the context manager is the file object returned by open(), and the variable file is used to access the file within the block. When the block is exited, whether normally or due to an error, the file is automatically closed. This is the safest way to handle files in Python. See the following example:

with open('filename.txt', 'r') as file:
    content = file.read()

Once the file is opened, you can read its contents using methods like read(), readline(), or readlines(). The read() method reads the entire file into a single string, while readline() reads a single line at a time, and readlines() reads all lines into a list. Similarly, in writing mode, you can use methods like write() and writelines() to write data to the file. writelines() is used to write multiple lines at once, taking a list of strings as input.

Reading and Writing with Pandas

Note

As always, for more information on reading and writing files, read the Pandas documentation on IO tools.

It is important to know how to work with files in Python, but since we are primarily working with data that will be structured in tabular form, we will mostly use Pandas to read and write data. Pandas provides convenient functions for reading and writing data in various formats, including CSV, Excel, JSON, and more. The most commonly used functions for reading and writing data are read_csv(), to_csv(), read_excel(), to_excel(), read_json(), and to_json(). These functions abstract away the complexities of file handling and provide a simple interface for working with data.

We will focus on the read_csv(), read_html(), and to_csv() functions, which are commonly used for reading and writing CSV files. The read_csv() function reads a CSV file into a DataFrame, while the to_csv() function writes a DataFrame to a CSV file. The read_html() function reads HTML tables into a DataFrame. It will automatically parse the HTML and extract the table data into a DataFrame. If you understand how to read and write CSV files, you will be able to work with other formats as well, as the syntax is similar.

import pandas as pd

# Reading a CSV file into a DataFrame
df = pd.read_csv('./data.csv') # data.csv exists in the current directory

Throughout this textbook, we will work almost exclusively with CSV files, so this is all you need to know at the moment.

When we are done working with the file, we can write it, using the to_csv() function. CSVs however do not understand or support indexes, so we need to specify index=False to prevent Pandas from writing the index to the CSV file.:

df.to_csv("./output.csv", index=False)