1.28.0 (Pending)

Incompatible behavior changes

Changes that are expected to cause an incompatibility if applicable; deployment changes are likely required

  • UHV: Introduced runtime flag envoy.reloadable_features.enable_universal_header_validator for toggling Universal Header Validator (UHV) on and off. The default value is off. This option is currently functional only when the ENVOY_ENABLE_UHV build flag is enabled. See https://github.com/envoyproxy/envoy/issues/10646 for more information about UHV.

  • eds: Introduced caching of EDS assignments when used with ADS. Prior to this change, Envoy required that EDS assignments were sent after an EDS cluster was updated. If no EDS assignment was received for the cluster, it ended up with an empty assignment. Following this change, after a cluster update, Envoy waits for an EDS assignment until initial_fetch_timeout times out, and will then apply the cached assignment and finish updating the warmed cluster. This change is disabled by default, and can be enabled by setting the runtime flag envoy.restart_features.use_eds_cache_for_ads to true.

  • http: Introduced a new runtime flag envoy.reloadable_features.no_downgrade_to_canonical_name to disable the name downgrading in the per filter config searching. See doc Http filter route specific config or issue https://github.com/envoyproxy/envoy/issues/29461 for more specific detail and examples.

  • http: Switch from http_parser to BalsaParser for handling HTTP/1.1 traffic. See https://github.com/envoyproxy/envoy/issues/21245 for details. This behavioral change can be reverted by setting runtime flag envoy.reloadable_features.http1_use_balsa_parser to false.

  • jwt: Changed behavior of the jwt extraction, passing entire token for validation, instead cut him in the non-Base64 character. This change can be reverted temporarily by setting the runtime guard envoy.reloadable_features.token_passed_entirely to false.

  • udp_proxy: When the UDP proxy has session filters, choosing the upstream host and creating a socket only happens after iterating all onNewSession() calls for all the filters in the chain. Upstream host health check for each downstream datagram does not apply when there are session filters, and per-packet load balancing can’t be used when there are session filters.

  • zone-aware routing: Zone-aware routing is now enabled even when the originating and upstream cluster have different numbers of zones. Previously, zone-aware routing was disabled in that case and the lb_zone_number_differs stat on the cluster was incremented. This behavioral change can be reverted by setting runtime guard envoy.reloadable_features.enable_zone_routing_different_zone_counts to false. Additionally, zone-aware routing now works correctly even when the originating and upstream cluster have different zone sets. Previously, zone-aware routing would not route fairly in this case. To revert the entire change, set the runtime flag envoy.reloadable_features.locality_routing_use_new_routing_logic to false to get the old behavior and well-tested codepaths, undoing both changes.

Minor behavior changes

Changes that may cause incompatibilities for some users, but should not for most

  • alternate_protocols_cache_filter: Changed the alternate protocols cache filter to get the cache from cluster config rather than filter config. This allows one downstream filter to be used with multiple clusters with different caches. This change can be reverted by setting runtime guard envoy.reloadable_features.use_cluster_cache_for_alt_protocols_filter to false.

  • ext_authz: Don’t append the local address to x-forwarded-for header when sending an http (not gRPC) auth request. This behavior can be reverted by setting runtime flag envoy.reloadable_features.ext_authz_http_send_original_xff to false.

  • ext_authz: removing any query parameter in the presence of repeated query parameter keys no longer drops the repeats.

  • ext_proc: Envoy will only take mode_override when waiting for the header responses. It will be ignored if it is in other processing states.

  • http oauth2 filter: Change HMAC cookie encoding to base64-encoded only. This change can be reverted temporarily by setting the runtime guard envoy.reloadable_features.hmac_base64_encoding_only to false.

  • local_rate_limit: Added new configuration field always_consume_default_token_bucket to allow for setting if default token bucket should be always consumed or only be consumed when there is no matching descriptor.

  • outlier detection: Outlier detection will always respect max_ejection_percent now. This behavioral change can be reverted by setting runtime guard envoy.reloadable_features.check_mep_on_first_eject to false.

  • quic: Enable QUICHE request and response headers validation. This behavior can be reverted by setting runtime flag envoy.reloadable_features.FLAGS_envoy_quic_reloadable_flag_quic_act_upon_invalid_header to false.

  • redis: The redis network filter connection_rate_limit_per_sec must be greater than 0. A config that sets this value to 0 will be rejected.

  • router: Enable copying response_code from the upstream stream_info onto the downstream stream_info. This behavior can be reverted by setting runtime guard envoy.reloadable_features.copy_response_code_to_downstream_stream_info to false.

  • xds: Set the lower bound of fill_rate to once per year. Values lower than once per year will automatically be set to that value.

Bug fixes

Changes expected to improve the state of the world and are unlikely to have negative effects

  • aws signer: fixed a bug where expiration timestamp on task roles failed to validate. This causes failure of credential caching which results in constant hits to the task role metadata URL.

  • connection limit: fixed a use-after-free bug in the connection limit filter.

  • dns: Fixed a bug where dns response was not always conforming [RFC 2181](https://datatracker.ietf.org/doc/html/rfc2181) for TTL values. Previously a malicious user could add a TTL greater than 2^31 - 1, and with c-ares library using 32 bit signed int data type would overflow and send a negative TTL.

  • dns: Fixed a bug where when respect_dns_ttl was set to true, c-ares dns resolver only considered address record for ttl calculation while ignoring CNAME records TTL. Now when respect_dns_ttl is set to true minimum of all TTL records is considered.

  • extension_discovery_service: Fixed a bug causing crash if ECDS is used with upstream HTTP filters.

  • healthcheck: The default behavior of unejecting outlier-detection-ejected host on successful active health checking can be disabled by setting outlier_detection.successful_active_health_check_uneject_host to false. This new configuration flag is a substitute for the removed runtime option envoy.reloadable_features_successful_active_health_check_uneject_host.

  • http1: Fixed a bug where HTTP/1.1 requests with “Connection: close” header is handled differently if the requested is internally redirected. Without internal redirect, the response will also have a “Connection: close” header and the connection will be closed after finishing that request. Requests with internal redirect should be handled in the same way. This behavior can be reverted by setting runtime envoy.reloadable_features.http1_connection_close_header_in_redirect to false.

  • redis: Fixed a bug where redis key with % in the key is failing with a validation error.

  • redis: fixed a bug where redis key formatter is using the closed stream because of life time issues.

  • router check tool: Fixed a bug where the route coverage is not correctly calculated when a route has weighted clusters.

  • subset load balancer: Fixed a bug where overprovisioning_factor and weighted_priority_health values were not respected when subset load balacing was enabled. The default values of 140 and false were always used.

  • tls: fixed a bug where handshake may fail when both private key provider and cert validation are set.

  • unix domain sockets: Fixed a crash on some versions of macOS when using a listener on a unix-domain socket.

Removed config or runtime

Normally occurs at the end of the deprecation period

  • dns: Removed envoy.restart_features.use_apple_api_for_dns_lookups and legacy code paths.

  • healthcheck: Removed envoy.reloadable_features_successful_active_health_check_uneject_host runtime option and substituted it with outlier_detection.successful_active_health_check_uneject_host outlier detection configuration flag.

  • http filters: Removed envoy_reloadable_features_http_filter_avoid_reentrant_local_reply runtime flag and legacy code paths.

  • listener: Removed envoy.reloadable_features.enable_update_listener_socket_options runtime flag and legacy code paths.

  • quic: Removed envoy.reloadable_features.reject_require_client_certificate_with_quic and legacy code paths.

  • runtime: Removed envoy.restart_features.remove_runtime_singleton and legacy code paths.

  • runtime: Removed envoy_reloadable_features_append_query_parameters_path_rewriter and legacy code paths.

  • tcp: Removed runtime key envoy.reloadable_features.tcp_pool_idle_timeout.

  • tcp_proxy: Removed envoy_reloadable_features_finish_reading_on_decode_trailers runtime flag and legacy code paths.

  • xDS: Removed envoy.restart_features.explicit_wildcard_resource and legacy code paths.

New features

  • access_log: added %RESPONSE_FLAGS_LONG% substitution string, that will output a pascal case string representing the resonse flags. The output response flags will correspond with %RESPONSE_FLAGS%, only with a long textual string representation.

  • access_log: added a field lookup to %FILTER_STATE% for objects that have reflection enabled.

  • access_logs: added json_format_options config option to support JSON output formatting and the sort_properties option to print the JSON output with sorted properties.

  • config: Added the capability to defer broadcasting of certain cluster (CDS, EDS) to worker threads from the main thread. This optimization can save significant amount of memory in cases where there are (1) a large number of workers and (2) a large amount of config, most of which is unused. This capability is guarded by enable_deferred_cluster_creation.

  • ext_proc: added disable_immediate_response config API to ignore the immediate_response message from the external processing server.

  • extension_discovery_service: added ECDS support for :ref:` downstream network filters<envoy_v3_api_field_config.listener.v3.Filter.config_discovery>`.

  • extension_discovery_service: added ECDS support for :ref:` upstream network filters<envoy_v3_api_field_config.cluster.v3.Filter.config_discovery>`.

  • extension_discovery_service: added metric listener.listener_stat.network_extension_config_missing to track closed connections due to missing config.

  • http: added Json-To-Metadata filter.

  • lua: added downstreamRemoteAddress() method to the Stream info object API.

  • original_dst: added support for the internal listener address recovery using the original destination listener filter.

  • otlp_stats_sink: added :ref:` stats prefix option<envoy_v3_api_field_extensions.stat_sinks.open_telemetry.v3.SinkConfig.stats_prefix>` to OTLP stats sink that enables adding a static prefix to all stats flushed by this sink.

  • quic: added support for QUIC listener filters with ECDS support reusing the same config API listener_filters as TCP does.

  • redis: Added new configuration field read_command_policy to specify Envoy should route read commands to another cluster.

  • redis: added support for lmove command.

  • redis: added support for time command (returns a local response).

  • tap: added custom_sink type to enable writing tap data out to a custom sink extension.

  • tap: added record_downstream_connection to control writing downstream connection address info in trace output.

  • tap: added record_headers_received_time to control writing request and response headers received time in trace output.

  • tcp: added the support to detect and send TCP RST for raw buffer socket based connections. This is currently supported on Linux only. It can be disabled by the runtime guard envoy_reloadable_features_detect_and_raise_rst_tcp_connection.

  • tls: added disable_stateful_session_resumption config option to disable stateful TLS session resumption.

  • tls: added fallback fallback to support private key provider to fallback to boringssl tls handshake. If the private key provider isn’t available (eg. the required hardware capability doesn’t existed), Envoy will fallback to the BoringSSL default implementation when the fallback is true. The default value is false.

  • tracing: Added spawn_upstream_span to control whether to create separate upstream span for upstream request.

  • udp_proxy: added http_capsule UDP session filter that can be used to encapsule or decapsulate UDP datagrams in HTTP, when used for UDP tunneling.

  • udp_proxy: added session_filters config to support optional filters that will run for each upstream UDP session. More information can be found in the UDP proxy documentation.

  • udp_proxy: added injectDatagramToFilterChain() callback to UDP session filters that allows session filters to inject datagrams downstream or upstream the filter chain during a filter chain iteration. This can be used, for example, by session filters that are required to buffer datagrams due to an asynchronous call.

  • upstream: Added the ability to specify a custom upstream local address selector using local_address_selector:.

  • upstream: added allow_redundant_keys to suppport redundant keys in request metadata for subset load balancing.

  • zookeeper: added support for emitting per opcode request bytes metrics via enable_per_opcode_request_bytes_metrics. added support for emitting per opcode response bytes metrics via enable_per_opcode_response_bytes_metrics.

Deprecated

  • tracing: start_child_span is deprecated by spawn_upstream_span. Please use the new field to control whether to create separate upstream span for upstream request.

  • tracing: OpenTracing is deprecated and will be removed at version 1.30, since the upstream project has been abandoned.

  • tracing: Opencensus is deprecated and will be removed at version 1.30, since the upstream project has been abandoned.