The Poll an External Website for XML Input Connector can be used to retrieve and adapt event data, formatted as XML, from an external website. For examples illustrating how ArcGIS GeoEvent Server expects typical XML data records to be formatted, see the Esri Community blog XML Data Structures - Characteristics and Limitations.
When multiple event data records are returned as a batch they can be organized beneath a single root node as illustrated below. The XML Adapter will handle each object as a separate event record when the root node beneath which individual records are organized is specified.
Usage notes
- Use this input connector to query a web server or web service for data, formatted as XML, and adapt it to create event records for processing.
- This input connector pairs the XML Inbound Adapter with the HTTP Inbound Transport.
- A typical XML data structure, with event records organized using a hierarchy of XML tags, element, and attribute values, is illustrated above.
- The XML Object Name property is used to identify the name of a group element within the XML document structure.
- Data records organized beneath a specified group element will be handled as separate event records when the grouping element's name is specified.
- When XML Object Name is left unspecified the entire data structure beneath the XML document's root element is considered; the schema of event records (i.e. the GeoEvent Definition) will reflect the entire XML document structure.
- The HTTP Method specifies the type of HTTP request used to query a web service for data. The default is to use an HTTP/GET request.
- An XML record does not have to contain data which represents a geometry.
- The adapter supports the ability to construct a point geometry from X, Y, and Z attribute values.
- Query parameters can be specified using the Parameters property to help specify data of interest.
- Specifying query parameters as part of an HTTP/GET request can improve performance by narrowing the number of items returned.
- The name and value(s) of any specified query parameter depend on what requests are supported by the external web service.
- This input connector assumes the external web server or web service will accept HTTP/GET requests without requiring authentication. For external websites requiring authentication, consider creating a custom input connector that pairs the XML Inbound Adapter with an HTTP Inbound Transport that supports basic, certificate, or OAuth authentication.
- The Receive New Data Only property is used to request that a feed return only previously unobserved or newly updated data. This capability depends on the data feed correctly incorporating Last-Modified into the header of its response to an HTTP request. If the response header does not indicate a date/time the content was last updated, GeoEvent Server will assume that data in the response represents new or recently updated data. You can read more about the use of Last-Modified in response headers in the RFC 7232 HTTP specification.
- This input connector includes a Learning Mode which can be useful when you need to allow the input connector to modify a GeoEvent Definition it has constructed. The purpose of this property is to temporarily accept that event data received will have a variable schema or data structure. The input connector will use a sample of received data records to learn more about the variable data structure and append new, previously unobserved, attribute fields to an existing GeoEvent Definition.
Allowing a GeoEvent Definition to be changed on-the-fly can negatively impact the design of real-time analytics in a GeoEvent Service. As a best practice, it is recommended that if schema variance is expected in your inbound event data that you use Learning Mode for as brief a period of time as possible to produce a GeoEvent Definition which supports all expected variants of your inbound data. Learning mode can then be turned off and the auto-generated GeoEvent Definition copied and tailored for production deployment.
Parameters
| Parameter | Description |
|---|---|
Name | A descriptive name for the input connector used for reference in GeoEvent Manager. |
URL | The base URL of the external website hosting the data as XML. GeoEvent Server will issue a poll request to the URL to retrieve the XML data. |
Create GeoEvent Definition | Specifies whether a new or existing GeoEvent Definition should be used for the inbound event data. A GeoEvent Definition is required for GeoEvent Server to understand the inbound event data attribute fields and data types.
|
GeoEvent Definition Name (New) (Conditional) | The name assigned to a new GeoEvent Definition. If a GeoEvent Definition with the specified name already exists, the existing GeoEvent Definition will be used. The first data record received will be used to determine the expected schema of subsequent data records, a new GeoEvent Definition will be created based on that first data record's schema. Parameter is shown when Create GeoEvent Definition is set to Yes and is hidden when set to No. |
GeoEvent Definition Name (Existing) (Conditional) | The name of an existing GeoEvent Definition to use when adapting received data to create event data for processing by a GeoEvent Service. Parameter is shown when Create GeoEvent Definition is set to No and is hidden when set to Yes. |
HTTP Method | The HTTP request method to use when GeoEvent Server polls the external server providing the data feed. The default is to use an HTTP/GET request.
|
Post From Post Body Post body MIME Type Post Parameters (Conditional) | These properties are hidden, by default, when the HTTP Method is set to Get. For more information, see the Limitations below. |
Default Spatial Reference | The well-known identifier (WKID) of a default spatial reference to be used when constructing point geometry from coordinate values or the received data contains geometry strings that do not include a spatial reference. |
Receive New Data Only | Specifies whether the adapter will look for a Last-Modified value in the data feed response’s header and will attempt to use this value to avoid processing data already been received and not updated. The parameter’s name can be misleading. The transport will always receive whatever data a feed includes in its response to the connector’s query. When Receive New Data Only is specified, however, the transport will cache the date/time of its last request and will only relay data it receives to the adapter if the response from the data feed includes a Last-Modified value which is greater-than the transport’s cached value. Specifying a Last-Modified value is optional. In practice, not many data feeds include it in their responses. The default for this parameter is Yes, but if a Last-Modified value is not found, the parameter is basically ignored as the date/time needed to determine whether data in the response is new or updated is not available.
For more information about the use of Last-Modified in response headers, see the RFC 7232 HTTP specification. |
Frequency (in seconds) | The interval (in seconds) an external web server or web service will be polled for data. The default is 10 seconds (one poll request will be made every 10 seconds). |
Header Parameter Name:Value List | The parameters to include in a request’s HTTP header when an external web server or web service is polled. The header parameters should be specified using Name:Value pairs (colon included) separated by commas. |
Construct Geometry from Fields | Specifies whether the input connector should construct a point geometry using coordinate values received as attributes. The default is No.
|
X Geometry Field (Conditional) | The attribute field in the inbound event data containing the X coordinate part (for example horizontal or longitude) of a point location. Parameter is shown when Construct Geometry from Fields is set to Yes and is hidden when set to No. |
Y Geometry Field (Conditional) | The attribute field in the inbound event data containing the Y coordinate part (for example vertical or latitude) of a point location. Parameter is shown when Construct Geometry from Fields is set to Yes and is hidden when set to No. |
Z Geometry Field (Conditional) | The name of the field in the inbound event data containing the Z coordinate part (for example depth or altitude) of a point location. If left blank, the Z value will be omitted and a 2D point geometry will be constructed. Parameter is shown when Construct Geometry from Fields is set to Yes and is hidden when set to No. |
XML Object Name | The name of an XML element used to identify individual data records within a received XML data structure. When XML Object Name is used to specify an XML element by name, the adapter will search for nodes matching the specified name and will only consider those nodes as data records. When left unspecified, which is the default, all data beneath uppermost XML root element are considered. |
Parameters | Specifies a query parameter (or multiple query parameters) to append to the base URL. Query parameters are specific to each web service and are not supported by all services. When supported, query parameters help specify data of particular interest. This parameter has no default value. Query parameters are normally incorporated into the URL parameter. |
Use URL Proxy | Specifies whether GeoEvent Server will use a URL proxy when making its HTTP requests. The default is No.
|
URL Proxy (Conditional) | The URL proxy to use when making an HTTP request. This parameter has no default value. Parameter is shown when the Use URL Proxy is set to Yes and is hidden when set to No. |
Expected Date Format | The pattern used to match expected string representations of date/time values and convert them to Java Date values. The pattern's format follows the Java SimpleDateFormat class convention. This parameter has no default value. While GeoEvent Server prefers date/time values to be expressed in the ISO 8601 standard, several string representations of date/time values commonly recognized as date values can be converted to Java Date values without specifying an Expected Date Format pattern. These include:
If the date/time values received are expressed using a convention other than one of the five shown above, you will have to specify an Expected Date Format so GeoEvent Server knows how the date/time values should be adapted. |
Acceptable MIME Types (Client Mode) | Specifies the MIME type(s) this connector would like to see when receiving a response to its query. The external web service's response specifies the actual MIME type(s) for any data sent to GeoEvent Server. This connector understands the following MIME type (the default): application/xml For more information on official MIME types, see Internet Assigned Numbers Authority (IANA). For more information on the MIME types not used by ArcGIS Server, see MIME types not used by ArcGIS Server. This parameter is shown when the HTTP Method is set to Get and is hidden when set to either Post or Put. |
HTTP Timeout (in seconds) | Specifies the maximum time (in seconds) the input connector will wait for a response from the external web service it has queried. The value must be an integer between 1 and 2147483. The default value is 30 seconds (the HTTP request will timeout after 30 seconds if no response is received). |
Learning Mode | Specifies whether Learning Mode is active or disabled. When learning mode is set to Yes, the inbound adapter will append new fields to a GeoEvent Definition it has created and is maintaining.
Learning Mode can be useful when you need to allow the input connector to modify a GeoEvent Definition it has constructed. The purpose of this parameter is to temporarily accept that event data received will have a variable schema or data structure. The input connector will use a sample of received data records to learn more about the variable data structure and append new, previously unobserved, attribute fields to an existing GeoEvent Definition. |
Limitations
Technically, a polling input should always use an HTTP/GET request, which does not include a Content-Body, and any parameters should be query parameters used to specify data of interest. The parameters below are only applicable to output connectors which send data to an external web server or web service. Leave the HTTP Method parameter set to the default HTTP/GET and the parameters below will remain hidden.
| Parameter | Description |
|---|---|
Post From (Conditional) | Specifies whether this connector will use an HTTP/POST or HTTP/PUT request.
Parameter is shown when HTTP Method is set to either Post or Put and is hidden when set to Get. |
Post Body (Conditional) | Specifies data to include in the Content-Body of an HTTP/POST or HTTP/PUT request. Parameter is shown when Post From is set to Content Body and is hidden when set to Parameters. |
Post body MIME Type (Conditional) | Specifies the MIME type of data included in the HTTP request’s Content-Body. MIME types typically consist of a type/subtype structure and are used to specify the internet media type(s) used and accepted by an HTTP client/server. This parameter has no default value. Parameter is shown when Post From is set to Content Body and is hidden when set to Parameters. |
Post Parameters (Conditional) | Specifies parameters to include as part of the HTTP request. Parameters should be specified using Name:Value pairs (colon included) separated by commas. This parameter has no default value. Parameter is shown when Post From is set to Parameters and is hidden when set to Content Body. |