convert_masterfiles_to_csv module

convert_masterfiles_to_csv module#

Created on Wed Jan 16 17:10:10 2019

This script extracts all the data from FTT excel sheets in the “/In/FTTAssumptions/[model]” folders and saves them in separate csv files.

The user can select one or more scenarios to convert from the excel sheet:

  1. If running this code as a script, set the input files in the final lines of the code

2. If the script is run at the start of the model run, it will automatically detect which files to convert, based on available csv files and whether the masterfile is newer than the csv files.

@author: MM and Femke Nijsse

convert_masterfiles_to_csv.are_csvs_older_than_masterfiles(vars_to_convert, out_dir, models, model, scen, dir_masterfiles)#

Find which files were last modified. If masterfile is newest then newest csv files, overwrite all csv files.

Returns true if the csvs need updating and are older

convert_masterfiles_to_csv.convert_1D_var_to_timeline(data, var, row_title, out_dir, timeline_dict)#

Some variables (e.g. TEWW, MEWW), are 1D in the excel sheets. However, in the model, these variables change over time, and we store this data. Therefore, the csv files should have a time dimension

convert_masterfiles_to_csv.convert_masterfiles_to_csv(models, ask_user_input=False, overwrite_existing_csvs=False)#

The main function to convert masterfiles to csv files. Depending on how you run it, it can have three types of behaviour:

a) If you run it as a script, it will overwrite files. It will ask for confirmation before overwriting the gamma values b) If you call it from another function

b1) It will generate csvs if files don’t exist yet b2) It will overwrite files if the masterfiles are newer than the csv files

This will include gamma values.

convert_masterfiles_to_csv.costs_to_gam(data, var, reg, timeline_dict, dims, out_dir)#

In P, Tr, H and Fr, gamma values are not saved separately, but instead part of the cost variable. Here, those values are extracted to ensure the gamma values are defined for each year.

convert_masterfiles_to_csv.csv_exists(file_path)#

Check if a CSV file already exists.

convert_masterfiles_to_csv.directories_setup()#

Set up directory masterfile and the general input directory

convert_masterfiles_to_csv.extract_data(raw_data, sheet_name, row_start, rdim, ci, cf)#

Extract a slice of data and convert it to a numpy array.

convert_masterfiles_to_csv.gamma_input_on_overwrite(out_dir, var, gamma_options)#

When the script is run as a script, and the gamma csv files already exist, confirm with the user if you want to overwrite, given that the user may not want to lose their calibrated gamma values

convert_masterfiles_to_csv.get_model_classification(variables_df)#

Get the dimensions for the classifications

convert_masterfiles_to_csv.get_remaining_variables(vars_to_convert, out_dir, model, var_dict, gamma_options, overwrite_existing_csvs)#
Remove variables from the to-convert list if:

  1. We do not overwrite existing csv and

  2. They already exist

Returns:

  • the remaining variables to convert

  • the gamma user input (unchanged if not applicable)

convert_masterfiles_to_csv.get_sheets_to_convert(var_dict, model, scen)#

Get all the variables to convert to CSV files There are two options:

  1. Convert all variables from excel sheet (for S0)

  2. Convert only policy variables (for other scenarios).

You can change which variables are in all scenarios in the FTT_variables file, column Scenario

convert_masterfiles_to_csv.read_data(models, model, dir_masterfiles, scen, sheets)#

Read the masterfiles

Returns:

the dataframe with all the excel data

Return type:

raw_data

convert_masterfiles_to_csv.set_up_cols(model, var, var_dict, dims, timeline_dict)#

Set up the size and name of a column.

convert_masterfiles_to_csv.set_up_rows(model, var, var_dict, dims)#

Setting up the size of the rows, and the name of the rows

convert_masterfiles_to_csv.variable_setup(dir_masterfiles, models)#

Set up the various containers and metadata for variables: variables_df_dict, var_dict, vars_to_convert, scenarios, timeline_dict

convert_masterfiles_to_csv.write_to_csv(data, row_title, col_title, var, out_dir, reg=None)#

Write the variables to a csv file, or to multiple csv files for 3D vars