How to Create Log File in Python ? Logging for Data Scientist

As a Data Scientist Programmer, you might have to build a project based on machine learning, deep learning, and other related projects. It’s easy to compile and see the output of a project according to your requirement. But what does the compiler doing (Debugging) we ignored that. In the developing phase, it does not matter much. But in the production phase, it’s very important to keep track of logs in a file while the program is running. Otherwise, you will not be able to know which modules or lines of code have got errors and problems. This leads to high time taking to fix the errors if the program crashes. In this post, you will know how to use logging in python to keep records of the occurred events within a program.

Before coding the custom logging let’s understand the basic logging

Basic Logging in Python

For the logging, you use the logging module that is already available in the python standard library. You can import using the import logging statement. In python logging, there are 5 ways you can record the log messages. These are :

logging.debug()
logging.error()
logging.info()
logging.warning()
logging.critical()

Each message tells you something about the application status. Each message tells us the following things.

logging.debug() : – Gives you the diagnostic information useful for debugging.
logging.error(): – It tells you that a certain operations cannot be performed due to some problem.
logging.info(): – General information that contains the results of program execution.
logging.warning(): It warns that there can be a future problem if you have not solved it.
logging.critical() : – Serious error. The program cannot continue or run.

Coding the Basic Logging in Python

By default logging module only outputs the warning messages as the other levels are high from loggin.warning(). You can see I have written all the messages but in the output, I am not getting the info and debug messages.

basic logging part 1 — Without the Debug Level

You can set the minimum level in the logging.basicConfig() method to output all messages. Pass the level argument with the logging.DEBUG and you are able to see all the messages. The best practice is to create a separate log file inside it. To do so you have to specify the filename parameter inside the basicConfig() method.

basic logging part with the debug level — With the debug level

basic logging wit the output file — Log File Creation

Custom Logging in Python

Basic logging simply writes the message of the level to the log file. But you can also add some other things like function name, line number data, etc to know from where these messages are coming. For this, you will add the format and datefmt values inside the basicConfig() method. These are the group of formatted strings. You can look out for all these on the official Python Logging tutorial. For example, I have added the following.

fmtstr = " Name: %(user_name)s : %(asctime)s: (%(filename)s): %(levelname)s: %(funcName)s Line: %(lineno)d - %(message)s"

By default, the logging module takes other date and time formats. The datefmt parameters take the format of the date and time you define. Like in this example I will take the following DateTime format.

06/20/2019 03:56:16 PM

Then I will define the datefmt as:

datestr = "%m/%d/%Y %I:%M:%S %p "

After defining all the required formats you will use inside the basicConfig() method.


   #basic logging config
    logging.basicConfig(
        filename="custom_log_output.log",
        level=logging.DEBUG,
        filemode="w",
        format=fmtstr,
        datefmt=datestr,
    )

Output of the Custom Log — The output of the Custom Log

Now Whenever you think that there can be issues in class, function, or any lines of code. Then use these logging messages with all the above configurations. And when something happens inside the application, these log messages will be written inside the log file. By doing this you can quickly find the problem and solve it easily. For example lets, I create a function that adds two integers. I will output the function name in the log file.

def sum():
    logging.debug("This is the debug message")
    num1 = 5
    num2 = 10
    sum = num1 + num2
    print(f'Sum:{sum}')

You can see the log file has added the debug message with the function name and the line number and other messages where the message has come.

Conclusion

Creating a log file is a must for every data scientist or other program in the production phase. It not only is the best practice but also the best way to solve the problems occurred in the application quickly. Whenever there is a problem just go to the log file find the line number or function name and solve that problem. This tutorial is only the basic part of logging. If you want to learn more about it then you can read the official Python Logging module.

Full Code Available on Github

Hope you have enjoyed the tutorial. If you want to know anything and have suggestions then you can contact us or message us on the data science learner official page for instant support.