Skip to content

Geoconda

Geoconda is a collection of python packages that facilitate the development of python scripts for geoinformatics applications. It includes following python packages:

Additionally geoconda includes:

  • spyder - Scientific Python Development Environment with graphical interface (similar to RStudio for R).
  • GDAL/OGR commandline tools
  • GMT The Generic Mapping Tools
  • landsatlinks - for creating download URLs for Landsat Collection 2 Level 1 product bundles using the USGS/EROS Machine-to-Machine API. Use python3.10 -m landsatlinks.
  • PDAL - Point Data Abstraction Library
  • ncview for visualizing netcdf files

Python has multiple packages for parallel computing, for example multiprocessing, joblib and dask. In our Puhti Python examples there are examples how to utilize these different parallelisation libraries.

If you think that some important GIS package for Python is missing from here, you can ask for installation from CSC Service Desk.

Available

The geoconda module is available:

  • 3.11.9 (Python 3.11.9, PDAL 2.7.2, GDAL 3.9.1, created August 2024), in Puhti and Mahti.
  • 3.10.9 (Python 3.10.9, PDAL 2.5.2, GDAL 3.6.2, created March 2023), in Puhti.
  • 3.10.6 (Python 3.10.6, PDAL 2.4.1, GDAL 3.5.0, created September 2022), in Puhti and Mahti.

Version number is the same as the Python version.

Usage

For using Python packages and other tools listed above, you can initialize them with:

module load geoconda

By default the latest geoconda module is loaded. If you want a specific version you can specify the version number of geoconda:

module load geoconda/[VERSION]

To check the exact packages and versions included in the loaded module:

list-packages

You can add more Python packages to geoconda by following the instructions in our Python usage guide.

You can edit your Python code in Puhti with:

To open Spyder in Puhti web interface with remote desktop:

  1. Log in to Puhti web interface.
  2. Open Remote desktop: Apps -> Desktop.
  3. After launching the remote desktop open Terminal (Desktop icon) and start Spyder:
module load geoconda
spyder

r5py memory settings

r5py by default does not correctly understand how much memory it has available in a supercomputer so, it has to be defined manually. It is using Java in the background, so add environmental variable to set maximum memory available for Java:

  • export _JAVA_OPTIONS="-Xmx4g" from command-line before starting Python OR
  • os.environ["_JAVA_OPTIONS"] = "-Xmx4g" in the beginning of your Python code.

Google Earth Engine authentication set up

For using Google Earth Engine (GEE) API with earthengine-api package, one needs to have an account in GEE. Before first usage, also set up GEE authentication in Puhti:

  1. Open Puhti web interface
  2. Start Desktop app
  3. In the Desktop, open:
    • Web Browser under Applications menu and
    • Terminal from shortcuts
  4. In the Terminal:
module load geoconda
python

import os
os.environ['PATH'] = '/appl/opt/csc-cli-utils/google-cloud-sdk/bin:' + os.environ['PATH']

import ee
ee.Authenticate()

This prints out a long link and asks for a code. Copy the link to the Web Browser and open the Google log-in page. Log-in and copy the created code back to Python.

Using Allas from Python

There are two Python libraries installed in Geoconda that can interact with Allas. Swiftclient uses the swift protocol and boto3 uses S3 protocol. You can find CSC examples how to use both here.

It is also possible to read and write files from and to Allas or other cloud object storage directly with GDAL-based packages such as geopandas and rasterio. Please check our Using geospatial files directly from cloud, inc Allas tutorial for instructions and examples.

With large quantities of data in Allas, consider using virtual rasters.

License

All packages are licensed under various free and open source licenses (FOSS), see the linked pages above for exact details.

Citation

Please see the above linked package pages for citation information per package.

Acknowledgement

Please acknowledge CSC and Geoportti in your publications, it is important for project continuation and funding reports. As an example, you can write "The authors wish to thank CSC - IT Center for Science, Finland (urn:nbn:fi:research-infras-2016072531) and the Open Geospatial Information Infrastructure for Research (Geoportti, urn:nbn:fi:research-infras-2016072513) for computational resources and support".

Installation

Geoconda was installed to Puhti and Mahti using Tykkys conda-containerize functionality. The WhiteboxTools conda package installs only WhiteboxTools installer, therefore for proper installation of Whiteboxtools required additional post installation command and folder to wrap commandline tools.

conda-containerize new --mamba --prefix install_dir --post download_wbt -w miniconda/envs/env1/lib/python3.10/site-packages/whitebox/WBT/whitebox_tools geoconda_3.10.9.yml

Geoconda conda environment files and download_wbt and start_wbt.py needed for WhiteboxTools are available in CSCs geocomputing repository. Note that for reproducibility, you'll need to define the package versions in the environment file, which can be checked on Puhti and Mahti using list-packages command after loading the geoconda module.

References