owslib.wmts.WebMapTileService cannot load World Imagery Wayback wmts layers

[Fanchengyan]

Hi all,

Thanks for your work on owslib.

I can load the wmts layer of ESRI World Imagery Wayback in QGIS, but I can't load it in python using owslib， when I run the following code:

from owslib.wmts import WebMapTileService
url = "https://wayback.maptiles.arcgis.com/arcgis/rest/services/World_Imagery/MapServer/WMTS/1.0.0/WMTSCapabilities.xml"
wmts = WebMapTileService(url)

I get an error:

{
	"name": "AttributeError",
	"message": "'NoneType' object has no attribute 'findall'",
	"stack": "---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
File Untitled-1:4
      2 from owslib.wmts import WebMapTileService
      3 url = \"https://wayback.maptiles.arcgis.com/arcgis/rest/services/World_Imagery/MapServer/WMTS/1.0.0/WMTSCapabilities.xml\"
----> 4 wmts = WebMapTileService(url)

File /opt/miniconda3/envs/qgis/lib/python3.12/site-packages/owslib/wmts.py:197, in WebMapTileService.__init__(self, url, version, xml, username, password, parse_remote_metadata, vendor_kwargs, headers, auth, timeout)
    194     raise ServiceException(err_message, xml)
    196 # build metadata objects
--> 197 self._buildMetadata(parse_remote_metadata)

File /opt/miniconda3/envs/qgis/lib/python3.12/site-packages/owslib/wmts.py:249, in WebMapTileService._buildMetadata(self, parse_remote_metadata)
    247             self.contents[cm.id] = cm
    248         gather_layers(elem, cm)
--> 249 gather_layers(caps, None)
    251 self.tilematrixsets = {}
    252 for elem in caps.findall(_TILE_MATRIX_SET_TAG):

File /opt/miniconda3/envs/qgis/lib/python3.12/site-packages/owslib/wmts.py:238, in WebMapTileService._buildMetadata.<locals>.gather_layers(parent_elem, parent_metadata)
    237 def gather_layers(parent_elem, parent_metadata):
--> 238     for index, elem in enumerate(parent_elem.findall(_LAYER_TAG)):
    239         cm = ContentMetadata(
    240             elem, parent=parent_metadata, index=index + 1,
    241             parse_remote_metadata=parse_remote_metadata)
    242         if cm.id:

AttributeError: 'NoneType' object has no attribute 'findall'"
}

I have tried to use parse_remote_metadata=True and version="1.0.0", but it doesn't work.

Is there any way to load the wmts layer of ESRI World Imagery Wayback in python using owslib?

[CJGutz]

Looks like the error comes from the fact that _WMTS_NS checks for namespace with http and not https.
Your xml file uses the same protocol for its namespaces as the url.
As a temporary fix you can use http://wayback.maptiles.arcgis.com/arcgis/rest/services/World_Imagery/MapServer/WMTS/1.0.0/WMTSCapabilities.xml
But it would be best to support both namespaces.

[geographika]

@Fanchengyan - @CJGutz is correct - only http namespaces are supported in OWSLib. I'm not sure if http://wayback.maptiles.arcgis.com is an outlier in that it switches between http and https for xmlns values based on the protocol used. Other servers I've checked use http for their namespaces for both http and https requests.

Looking at the code it is not an easy change to support both, but if there are more severs/services that do this it could be considered.

[Fanchengyan]

Hi @CJGutz and @geographika

Thanks for the detailed explanation. Switching to HTTP namespaces seems appropriate for now. HTTPS is definitely the future trend for web protocols, and supporting it would be a significant improvement for security and modern web standards. Additionally, QGIS can already directly recognize and work with HTTPS protocols. Interestingly, since OWSLib is built into QGIS, I initially assumed QGIS was using OWSLib for this functionality, but it appears they've implemented their own solution.

[geographika]

@Fanchengyan - just to clarify - OWSLib works fine with reading over https. The issue here is the server wayback.maptiles.arcgis.com returns XML with xsi:schemaLocation="https://www.opengis.net/wmts/1.0 https://schemas.opengis.net/wmts/1.0/wmtsGetCapabilities_response.xsd" whereas it should be using xsi:schemaLocation="http://www.opengis.net/wmts/1.0 http://schemas.opengis.net/wmts/1.0/wmtsGetCapabilities_response.xsd".

The OWSLib WMTS code searches in the XML for the string http://www.opengis.net which it can't find.

This is either a server configuration error, or the WMTS spec allows for both URLs to be used in its XML in which case we should look at updating OWSLib to handle this (and WMTS isn't the only protocol which assumes http in the XML responses).

It seems this is still in issue in newer OGC services, see:

• provide schemas.opengis.net via HTTPS? opengeospatial/ogcapi-features#276
• http://schemas.opengis.net/ is not running in HTTPs pygeoapi#312

From the above link:

Once OGC completes HTTPS support on schemas.opengis.net, we would change the default location to https://schemas.opengis.net/

@tomkralidis - should we consider supporting both http and https namespaces throughout OWSLib?

[tomkralidis]

The canonical namespace URI for WMTS 1.0.0 is http://www.opengis.net/wmts/1.0 only. This means the WMTS server needs to be updated to support this namespace URI for XML responses. OWSLib is doing its job according to the standard (note that QGIS WMTS support is not using OWSLib but baked in C++, so behaviour may be different in QGIS itself).