docs/source/working-notes/troyraen/SuperNNova/snn_validation.md
SuperNNova validation: Classify known Ia’s
Download csv from TNS website. (Search for recent Ia’s)
Query database to see what SNN classified these as
The docs indicate that “usually” class0 indicates a Ia and class1 indicates non-Ia, but this will depend on how the model was trained.
I checked by classifying a objects recently reported to TNS as Ias.
Results indicate the opposite of expected. Will check with Anais.
import matplotlib.pyplot as plt
import os
import pandas as pd
from broker_utils import gcp_utils
# get objectIds from TNS csv
f = 'tns_search_SNIa.csv'
df = pd.read_csv(f)
ztfdf = df.loc[df['Disc. Internal Name'].str.startswith("ZTF")]
objectIds = list(ztfdf['Disc. Internal Name'].unique())
# query bigquery
project_id = os.getenv('GOOGLE_CLOUD_PROJECT')
dataset = 'ztf_alerts'
table = 'SuperNNova'
query = f"""
SELECT *
FROM `{project_id}.{dataset}.{table}`
WHERE objectId IN ('{"','".join(objectIds)}')
"""
bqdf = gcp_utils.query_bigquery(query).to_dataframe()
cleandf = bqdf.sort_values('candid', ascending=False).drop_duplicates(subset='objectId', keep='first')
date = "20211017"
fout = f"snn_results_{date}.png"
cleandf.hist(['prob_class0', 'prob_class1'], sharex=True)
plt.xlabel("SuperNNova result")
plt.suptitle("recent SNIa's reported to TNS")
plt.savefig(fout)
plt.show(block=False)
Classify known Ia - Original Test
The docs indicate that “usually” class0 indicates a Ia and class1 indicates non-Ia, but this will depend on how the model was trained.
I (Troy) checked by classifying a recent observation of the known Ia that I recently pulled from file storage to send to Ella (SN_2021rhu aka ZTF21abiuvdk, ZTF21abiuvdk_lightcurve.png and compare with Alerce).
Results indicate the opposite of expected… SN_2021rhu is assigned to class1 with high confidence.
Adding this to my list of questions for Anais about the trained model.
import main
from broker_utils import data_utils, gcp_utils
snIa = 'ZTF21abiuvdk'
fname = '/Users/troyraen/Documents/broker/ella/avros/ZTF21abiuvdk.1664460940815015004.ztf_20210723_programid1.avro'
alert_dict = data_utils.decode_alert(fname)
snn_dict = main.classify_with_snn(alert_dict)
snn_dict
# output is:
{'objectId': 'ZTF21abiuvdk',
'candid': 1664460940815015004,
'prob_class0': 0.04458457976579666,
'prob_class1': 0.9554154872894287,
'pred_class': 1}
Again on Sept 5 with all observations
import main
import os
import pandas as pd
from broker_utils import data_utils, gcp_utils, schema_maps
schema_map = schema_maps.load_schema_map('ztf', False)
snIa = 'ZTF21abiuvdk'
bucketid = 'ztf-alert_avros'
localdir = f'/Users/troyraen/Documents/broker/troy/troy/SNN/{snIa}'
gcp_utils.cs_download_file(snIa, bucketid, localdir)
dfs = []
for f in os.listdir(localdir):
dfs.append(data_utils.decode_alert(f'{localdir}/{f}', 'df', schema_map))
df = pd.concat(dfs)
model_path = '/Users/troyraen/Documents/broker/snn/broker/cloud_functions/classify_snn/ZTF_DMAM_V19_NoC_SNIa_vs_CC_forFink/vanilla_S_0_CLF_2_R_none_photometry_DF_1.0_N_global_lstm_32x2_0.05_128_True_mean.pt'
# step through the formatting and classification from cloud fnc main.py
snn_dict
# output is:
{'objectId': 'ZTF21abiuvdk',
'candid': 1707409520815015012,
'prob_class0': 0.010303422808647156,
'prob_class1': 0.9896966218948364,
'predicted_class': 1}