src.digitaltwin.data_to_db

This script fetches geospatial data from various providers using the ‘geoapis’ library and stores it in the database. It also saves user log information in the database.

Attributes

log

Exceptions

NoNonIntersectionError

Exception raised when no non-intersecting area is found.

Functions

get_nz_geospatial_layers(→ pandas.DataFrame)

Retrieve geospatial layers from the database that have a coverage area of New Zealand.

get_non_nz_geospatial_layers(→ pandas.DataFrame)

Retrieve geospatial layers from the database that do not have a coverage area of New Zealand.

get_geospatial_layer_info(→ Tuple[str, int, str, str])

Extracts geospatial layer information from a single layer entry.

get_vector_data_id_not_in_db(→ Set[int])

Get the IDs from the fetched vector_data that are not present in the specified database table

nz_geospatial_layers_data_to_db(→ None)

Fetches New Zealand geospatial layers data using 'geoapis' and stores it into the database.

get_non_intersection_area_from_db(→ geopandas.GeoDataFrame)

Get the non-intersecting area from the catchment area and user log information table in the database

process_new_non_nz_geospatial_layers(→ None)

Fetches new non-NZ geospatial layers data using 'geoapis' and stores it into the database.

process_existing_non_nz_geospatial_layers(→ None)

Fetches existing non-NZ geospatial layers data using 'geoapis' and stores it into the database.

non_nz_geospatial_layers_data_to_db(→ None)

Fetches non-NZ geospatial layers data using 'geoapis' and stores it into the database.

store_geospatial_layers_data_to_db(→ None)

Fetches geospatial layers data using 'geoapis' and stores it into the database.

user_log_info_to_db(→ None)

Store user log information to the database.

Module Contents

src.digitaltwin.data_to_db.log
exception src.digitaltwin.data_to_db.NoNonIntersectionError

Bases: Exception

Exception raised when no non-intersecting area is found.

src.digitaltwin.data_to_db.get_nz_geospatial_layers(engine: sqlalchemy.engine.Engine) → pandas.DataFrame

Retrieve geospatial layers from the database that have a coverage area of New Zealand.

Parameters:

engine (Engine) – The engine used to connect to the database.

Returns:

Data frame containing geospatial layers that have a coverage area of New Zealand.

Return type:

pd.DataFrame

src.digitaltwin.data_to_db.get_non_nz_geospatial_layers(engine: sqlalchemy.engine.Engine) → pandas.DataFrame

Retrieve geospatial layers from the database that do not have a coverage area of New Zealand.

Parameters:

engine (Engine) – The engine used to connect to the database.

Returns:

Data frame containing geospatial layers that do not have a coverage area of New Zealand.

Return type:

pd.DataFrame

src.digitaltwin.data_to_db.get_geospatial_layer_info(layer_row: pandas.Series) → Tuple[str, int, str, str]

Extracts geospatial layer information from a single layer entry.

Parameters:

layer_row (pd.Series) – A geospatial layer row that represents a single geospatial layer along with its associated information.

Returns:

A tuple containing the values for data_provider, layer_id, table_name, and unique_column_name.

Return type:

Tuple[str, int, str, str]

src.digitaltwin.data_to_db.get_vector_data_id_not_in_db(engine: sqlalchemy.engine.Engine, vector_data: geopandas.GeoDataFrame, table_name: str, unique_column_name: str, area_of_interest: geopandas.GeoDataFrame) → Set[int]

Get the IDs from the fetched vector_data that are not present in the specified database table for the area of interest.

Parameters:

  • engine (Engine) – The engine used to connect to the database.

  • vector_data (gpd.GeoDataFrame) – A GeoDataFrame containing the fetched vector data.

  • table_name (str) – The name of the table in the database.

  • unique_column_name (str) – The name of the unique column in the table.

  • area_of_interest (gpd.GeoDataFrame) – A GeoDataFrame representing the area of interest.

Returns:

The set of IDs from the fetched vector_data that are not present in the specified table in the database.

Return type:

Set[int]

src.digitaltwin.data_to_db.nz_geospatial_layers_data_to_db(engine: sqlalchemy.engine.Engine, crs: int = 2193, verbose: bool = False) → None

Fetches New Zealand geospatial layers data using ‘geoapis’ and stores it into the database.

Parameters:

  • engine (Engine) – The engine used to connect to the database.

  • crs (int = 2193) – The coordinate reference system (CRS) code to use. Default is 2193.

  • verbose (bool = False) – Whether to print messages. Default is False.

Returns:

This function does not return any value.

Return type:

None

src.digitaltwin.data_to_db.get_non_intersection_area_from_db(engine: sqlalchemy.engine.Engine, catchment_area: geopandas.GeoDataFrame, table_name: str) → geopandas.GeoDataFrame

Get the non-intersecting area from the catchment area and user log information table in the database for the specified table.

Parameters:

  • engine (Engine) – The engine used to connect to the database.

  • catchment_area (gpd.GeoDataFrame) – A GeoDataFrame representing the catchment area.

  • table_name (str) – The name of the table in the database.

Returns:

The non-intersecting area, or the original catchment area if no intersections are found.

Return type:

gpd.GeoDataFrame

Raises:

NoNonIntersectionError – If the non-intersecting area is empty, it suggests that the catchment area is already fully covered.

src.digitaltwin.data_to_db.process_new_non_nz_geospatial_layers(engine: sqlalchemy.engine.Engine, data_provider: str, layer_id: int, table_name: str, area_of_interest: geopandas.GeoDataFrame, crs: int = 2193, verbose: bool = False) → None

Fetches new non-NZ geospatial layers data using ‘geoapis’ and stores it into the database.

Parameters:

  • engine (Engine) – The engine used to connect to the database.

  • data_provider (str) – The data provider of the geospatial layer.

  • layer_id (int) – The ID of the geospatial layer.

  • table_name (str) – The database table name of the geospatial layer.

  • area_of_interest (gpd.GeoDataFrame) – A GeoDataFrame representing the area of interest.

  • crs (int = 2193) – The coordinate reference system (CRS) code to use. Default is 2193.

  • verbose (bool = False) – Whether to print messages. Default is False.

Returns:

This function does not return any value.

Return type:

None

src.digitaltwin.data_to_db.process_existing_non_nz_geospatial_layers(engine: sqlalchemy.engine.Engine, data_provider: str, layer_id: int, table_name: str, unique_column_name: str, area_of_interest: geopandas.GeoDataFrame, crs: int = 2193, verbose: bool = False) → None

Fetches existing non-NZ geospatial layers data using ‘geoapis’ and stores it into the database.

Parameters:

  • engine (Engine) – The engine used to connect to the database.

  • data_provider (str) – The data provider of the geospatial layer.

  • layer_id (int) – The ID of the geospatial layer.

  • table_name (str) – The database table name of the geospatial layer.

  • unique_column_name (str) – The unique column name used for record identification in the database table.

  • area_of_interest (gpd.GeoDataFrame) – A GeoDataFrame representing the area of interest.

  • crs (int = 2193) – The coordinate reference system (CRS) code to use. Default is 2193.

  • verbose (bool = False) – Whether to print messages. Default is False.

Returns:

This function does not return any value.

Return type:

None

src.digitaltwin.data_to_db.non_nz_geospatial_layers_data_to_db(engine: sqlalchemy.engine.Engine, catchment_area: geopandas.GeoDataFrame, crs: int = 2193, verbose: bool = False) → None

Fetches non-NZ geospatial layers data using ‘geoapis’ and stores it into the database.

Parameters:

  • engine (Engine) – The engine used to connect to the database.

  • catchment_area (gpd.GeoDataFrame) – A GeoDataFrame representing the catchment area.

  • crs (int = 2193) – The coordinate reference system (CRS) code to use. Default is 2193.

  • verbose (bool = False) – Whether to print messages. Default is False.

Returns:

This function does not return any value.

Return type:

None

src.digitaltwin.data_to_db.store_geospatial_layers_data_to_db(engine: sqlalchemy.engine.Engine, catchment_area: geopandas.GeoDataFrame, crs: int = 2193, verbose: bool = False) → None

Fetches geospatial layers data using ‘geoapis’ and stores it into the database.

Parameters:

  • engine (Engine) – The engine used to connect to the database.

  • catchment_area (gpd.GeoDataFrame) – A GeoDataFrame representing the catchment area.

  • crs (int = 2193) – The coordinate reference system (CRS) code to use. Default is 2193.

  • verbose (bool = False) – Whether to print messages. Default is False.

Returns:

This function does not return any value.

Return type:

None

src.digitaltwin.data_to_db.user_log_info_to_db(engine: sqlalchemy.engine.Engine, catchment_area: geopandas.GeoDataFrame) → None

Store user log information to the database.

Parameters:

  • engine (Engine) – The engine used to connect to the database.

  • catchment_area (gpd.GeoDataFrame) – A GeoDataFrame representing the catchment area.

Returns:

This function does not return any value.

Return type:

None