Adding A Error Log Message Row In Pandas Dataframe

Based on this answer, Avoiding KeyError in dataframe, I am able to do my validations. But I need to keep a track as to which row is failing due to which validation condition. Is t

Solution 1:

With df1 = df1[m1 & m2 & m3 & m4 & m5 & m6 & m7 & m8 & m9 & m10 & m11].drop(d1.keys(), axis=1) you are selecting the rows where none of your conditions failed. So clearly you will not have what you would like here, and that is ok, as this is the validated part which should not have errors.

You can get the errors by doing another selection before dropping the failed lines:

df_error = df1.copy()
df_error['error_message'] = ~m1
...

If the column had an error, you could define some error text to be displayed in the table:

df_error['failed_on_name'] = pd.where(m1, your_message_here)

If you want to display the error to a log you can loop over your error table and output your message (considering the first version with boolean values in the columns):

for _, row in df_error.iterrows():
    print (error_message(dict(row)))

So you will be able to process the rows with a function like this:

def error_message(row):
    row_desc = []
    error_msg = []
    for k, v in row.items():
        if isinstance(v, bool):
            if v:
                error_msg.append(k)
        else:
            row_desc.append(v)
    return 'Row ' + ' '.join(row_desc) + ' failed with errors: ' + ' '.join(error_msg)