ZenSVI

Primary Author: Koichi Ito (National University of Singapore)

Besides this documentation, we have published a comprehensive paper with detailed information and demonstration use cases. The paper provides in-depth insights into the package's architecture, features, and real-world applications.

ZenSVI is a comprehensive Python package for downloading, cleaning, and analyzing street view imagery. For more information about the package or to discuss potential collaborations, please visit my website at koichiito.com. The source code is available on GitHub.

This package is a one-stop solution for downloading, cleaning, and analyzing street view imagery, with comprehensive API documentation available at zensvi.readthedocs.io.

Table of Contents

• ZenSVI
  • Table of Contents
  • Installation of zensvi
  • Installation of pytorch and torchvision
  • Usage
    • Downloading Street View Imagery
    • Analyzing Metadata of Mapillary Images
    • Running Segmentation
    • Running Places365
    • Running PlacePulse 2.0 Prediction
    • Running Global Streetscapes Prediction
    • Running Grounding Object Detection
    • Running Depth Estimation
    • Running Embeddings
    • Running Low-Level Feature Extraction
    • Transforming Images
    • Creating Point Clouds from Images
    • Visualizing Results
  • Contributing
  • License
  • Credits

Installation of zensvi

$ pip install zensvi

Installation of pytorch and torchvision

Since zensvi uses pytorch and torchvision, you may need to install them separately. Please refer to the official website for installation instructions.

Usage

Downloading Street View Imagery

Mapillary

For downloading images from Mapillary, utilize the MLYDownloader. Ensure you have a Mapillary client ID:

from zensvi.download import MLYDownloader

mly_api_key = "YOUR_OWN_MLY_API_KEY"  # Please register your own Mapillary API key
downloader = MLYDownloader(mly_api_key=mly_api_key)
# with lat and lon:
downloader.download_svi("path/to/output_directory", lat=1.290270, lon=103.851959)
# with a csv file with lat and lon:
downloader.download_svi("path/to/output_directory", input_csv_file="path/to/csv_file.csv")
# with a shapefile:
downloader.download_svi("path/to/output_directory", input_shp_file="path/to/shapefile.shp")
# with a place name that works on OpenStreetMap:
downloader.download_svi("path/to/output_directory", input_place_name="Singapore")

KartaView

For downloading images from KartaView, utilize the KVDownloader:

from zensvi.download import KVDownloader

downloader = KVDownloader()
# with lat and lon:
downloader.download_svi("path/to/output_directory", lat=1.290270, lon=103.851959)
# with a csv file with lat and lon:
downloader.download_svi("path/to/output_directory", input_csv_file="path/to/csv_file.csv")
# with a shapefile:
downloader.download_svi("path/to/output_directory", input_shp_file="path/to/shapefile.shp")
# with a place name that works on OpenStreetMap:
downloader.download_svi("path/to/output_directory", input_place_name="Singapore")

Amsterdam

For downloading images from Amsterdam, utilize the AMSDownloader:

from zensvi.download import AMSDownloader

downloader = AMSDownloader()
# with lat and lon:
downloader.download_svi("path/to/output_directory", lat=4.899431, lon=52.379189)
# with a csv file with lat and lon:
downloader.download_svi("path/to/output_directory", input_csv_file="path/to/csv_file.csv")
# with a shapefile:
downloader.download_svi("path/to/output_directory", input_shp_file="path/to/shapefile.shp")
# with a place name that works on OpenStreetMap:
downloader.download_svi("path/to/output_directory", input_place_name="Amsterdam")

Global Streetscapes

For downloading the NUS Global Streetscapes dataset, utilize the GSDownloader:

from zensvi.download import GSDownloader

downloader = GSDownloader()
# Download all data
downloader.download_all_data(local_dir="data/")
# Or download specific subsets
downloader.download_manual_labels(local_dir="manual_labels/")
downloader.download_train(local_dir="manual_labels/train/")
downloader.download_test(local_dir="manual_labels/test/")
downloader.download_img_tar(local_dir="manual_labels/img/")

Analyzing Metadata of Mapillary Images

To analyze the metadata of Mapillary images, use the MLYMetadata:

from zensvi.metadata import MLYMetadata

path_input = "path/to/input"
mly_metadata = MLYMetadata(path_input)
mly_metadata.compute_metadata(
    unit="image", # unit of the metadata. Other options are "street" and "grid"
    indicator_list="all", # list of indicators to compute. You can specify a list of indicators in space-separated format, e.g., "year month day" or "all" to compute all indicators
    path_output="path/to/output" # path to the output file
)

Running Segmentation

To perform image segmentation, use the Segmenter:

from zensvi.cv import Segmenter

segmenter = Segmenter(dataset="cityscapes", # or "mapillary"
                      task="semantic" # or "panoptic"
                      )
segmenter.segment("path/to/input_directory", 
                  dir_image_output = "path/to/image_output_directory",
                  dir_summary_output = "path/to/segmentation_summary_output"
                  )

Running Places365

To perform scene classification, use the ClassifierPlaces365:

from zensvi.cv import ClassifierPlaces365

# initialize the classifier
classifier = ClassifierPlaces365(
    device="cpu",  # device to use (either "cpu", "cuda", or "mps)
)

# set arguments
classifier = ClassifierPlaces365()
classifier.classify(
    "path/to/input_directory",
    dir_image_output="path/to/image_output_directory",
    dir_summary_output="path/to/classification_summary_output"
)

Running PlacePulse 2.0 Prediction

To predict the PlacePulse 2.0 score, use the ClassifierPerception:

from zensvi.cv import ClassifierPerception

classifier = ClassifierPerception(
    perception_study="safer", # Other options are "livelier", "wealthier", "more beautiful", "more boring", "more depressing"
)
dir_input = "path/to/input"
dir_summary_output = "path/to/summary_output"
classifier.classify(
    dir_input,
    dir_summary_output=dir_summary_output
)

You can also use the ViT version for perception classification:

from zensvi.cv import ClassifierPerceptionViT

classifier = ClassifierPerceptionViT(
    perception_study="safer", # Other options are "livelier", "wealthier", "more beautiful", "more boring", "more depressing"
)
dir_input = "path/to/input"
dir_summary_output = "path/to/summary_output"
classifier.classify(
    dir_input,
    dir_summary_output=dir_summary_output
)

Running Global Streetscapes Prediction

To predict the Global Streetscapes indicators, use:

• ClassifierGlare: Whether the image contains glare
• ClassifierLighting: The lighting condition of the image
• ClassifierPanorama: Whether the image is a panorama
• ClassifierPlatform: Platform of the image
• ClassifierQuality: Quality of the image
• ClassifierReflection: Whether the image contains reflection
• ClassifierViewDirection: View direction of the image
• ClassifierWeather: Weather condition of the image

from zensvi.cv import ClassifierGlare

classifier = ClassifierGlare()
dir_input = "path/to/input"
dir_summary_output = "path/to/summary_output"
classifier.classify(
    dir_input,
    dir_summary_output=dir_summary_output,
)

Running Grounding Object Detection

To run grounding object detection on the images, use the ObjectDetector:

from zensvi.cv import ObjectDetector

detector = ObjectDetector(
    text_prompt="tree",  # specify the object(s) (e.g., single type: "building", multi-type: "car . tree")
    box_threshold=0.35,  # confidence threshold for box detection
    text_threshold=0.25  # confidence threshold for text
)

detector.detect_objects(
    dir_input="path/to/image_input_directory",
    dir_output="path/to/image_output_directory",
    dir_summary_output="path/to/detection_summary_output",
    save_format="json" # or "csv"
)

Running Depth Estimation

To estimate the depth of the images, use the DepthEstimator:

from zensvi.cv import DepthEstimator

depth_estimator = DepthEstimator(
    task="relative" # task to perform (either "relative" or "absolute")
)

dir_input = "path/to/input"
dir_image_output = "path/to/image_output" # estimated depth map
depth_estimator.estimate_depth(
    dir_input,
    dir_image_output
)

Running Embeddings

To generate embeddings and search for similar images, use the Embeddings:

from zensvi.cv import Embeddings

emb = Embeddings(model_name="resnet-1", cuda=True)
emb.generate_embedding(
    "path/to/image_directory",
    "path/to/output_directory",
    batch_size=1000,
)
results = emb.search_similar_images("path/to/target_image_file", "path/to/embeddings_directory", 20)

Running Low-Level Feature Extraction

To extract low-level features, use the get_low_level_features:

from zensvi.cv import get_low_level_features

get_low_level_features(
    "path/to/input_directory",
    dir_image_output="path/to/image_output_directory",
    dir_summary_output="path/to/low_level_feature_summary_output"
)

Transforming Images

Transform images from panoramic to perspective or fisheye views using the ImageTransformer:

from zensvi.transform import ImageTransformer

dir_input = "path/to/input"
dir_output = "path/to/output"
image_transformer = ImageTransformer(
    dir_input="path/to/input", 
    dir_output="path/to/output"
)
image_transformer.transform_images(
    style_list="perspective equidistant_fisheye orthographic_fisheye stereographic_fisheye equisolid_fisheye",  # list of projection styles in the form of a string separated by a space
    FOV=90,  # field of view
    theta=120,  # angle of view (horizontal)
    phi=0,  # angle of view (vertical)
    aspects=(9, 16),  # aspect ratio
    show_size=100,  # size of the image to show (i.e. scale factor)
    use_upper_half=True,  # use the upper half of the image for sky view factor calculation
)

Creating Point Clouds from Images

To create a point cloud from images with depth information, use the PointCloudProcessor:

from zensvi.transform import PointCloudProcessor
import pandas as pd

processor = PointCloudProcessor(
    image_folder="path/to/image_directory",
    depth_folder="path/to/depth_maps_directory",
    output_coordinate_scale=45,  # scaling factor for output coordinates
    depth_max=255  # maximum depth value for normalization
)

# Create a DataFrame with image information
# The DataFrame should have columns similar to this structure:
data = pd.DataFrame({
    "id": ["Y2y7An1aRCeA5Y4nW7ITrg", "VSsVjWlr4orKerabFRy-dQ"],  # image identifiers
    "heading": [3.627108491916069, 5.209303414492613],           # heading in radians
    "lat": [40.77363963371641, 40.7757528007],                   # latitude
    "lon": [-73.95482278589579, -73.95668603003708],             # longitude
    "x_proj": [4979010.676803163, 4979321.30902424],             # projected x coordinate
    "y_proj": [-8232613.214232705, -8232820.629621736]           # projected y coordinate
})

# Process images and save point clouds
processor.process_multiple_images(
    data=data,
    output_dir="path/to/output_directory",
    save_format="ply"  # output format, can be "pcd", "ply", "npz", or "csv"
)

Visualizing Results

To visualize the results, use the plot_map, plot_image, plot_hist, and plot_kde functions:

from zensvi.visualization import plot_map, plot_image, plot_hist, plot_kde

# Plotting a map
plot_map(
    path_pid="path/to/pid_file.csv",  # path to the file containing latitudes and longitudes
    variable_name="vegetation", 
    plot_type="point"  # this can be either "point", "line", or "hexagon"
)

# Plotting images in a grid
plot_image(
    dir_image_input="path/to/image_directory", 
    n_row=4,  # number of rows
    n_col=5   # number of columns
)

# Plotting a histogram
plot_hist(
    dir_input="path/to/data.csv",
    columns=["vegetation"],  # list of column names to plot histograms for
    title="Vegetation Distribution by Neighborhood"
)

# Plotting a kernel density estimate
plot_kde(
    dir_input="path/to/data.csv",
    columns=["vegetation"],  # list of column names to plot KDEs for
    title="Vegetation Density by Neighborhood"
)

Contributing

Interested in contributing? Check out the contributing guidelines. Please note that this project is released with a Code of Conduct. By contributing to this project, you agree to abide by its terms.

License

zensvi was created by Koichi Ito. It is licensed under the terms of the MIT License.

Please cite the following paper if you use zensvi in a scientific publication:

@article{ito2024zensvi,
    title={ZenSVI: An Open-Source Software for the Integrated Acquisition, Processing and Analysis of Street View Imagery Towards Scalable Urban Science},
    author={Ito, Koichi and Zhu, Yihan and Abdelrahman, Mahmoud and Liang, Xiucheng and Fan, Zicheng and Hou, Yujun and Zhao, Tianhong and Ma, Rui and Fujiwara, Kunihiko and Ouyang, Jiani and Quintana, Matias and Biljecki, Filip},
    journal={arXiv preprint arXiv:2412.18641},
    year={2024}
}

Credits

• Logo design by Kunihiko Fujiwara
• All the packages used in this package: requirements.txt

---