Skip to content

Import

Importer flow

sequenceDiagram
    User->>+DatasetService: call reimport hook
    DatasetService->>+Queue: enqueue job (dataset id to update
    DatasetWorker->>Queue: poll queue
    DatasetWorker->>DatasetWorker: download file
    DatasetWorker->>DatasetWorker: process features
    DatasetWorker->>Database: store import result

    User->>+DatasetService: get dataset status
    DatasetService->>Database: get status
    Database ->> DatasetService: status
    DatasetService ->> User: status

Upon Dataset creation a reimport key is generated, and the reimport url for the hook is returned.

Downloading the archive

Input restrictions

Zipped shapefiles that do not exceed 55mb in size. Currently only the EPSG:4326 projection is supported.

The importer issues a HEAD request against the url.

The response must return a 200 status code Content-Type: application/zip The webserver hosting the file to import must answer:

HEAD url_to_import
content-type: application/zip
content-length: 51184186

if Content-Type is not application/zip or the Content-Length exceed the maximum allowed size the import will report a DatasetStatus with error.

If the conditions are met, a GET request will be issued (only one redirect is allowed) to download the archive.

Importing

Find a file in the zip archive that looks like a ShapeFile, use gdal Virtual file system (vsizip, vsimem) to open the dataset.

Get the spatial reference system for the dataset. Get the layer attributes schema. Import feature geometries and attributes in the database.

Exisiting features will be removed

If features are present when the import is triggered they will be removed before importing the new ones.