Cortex v1.20.0-rc.0

Cortex 1.20.0 Release Notes

This release contains 368 contributions from 38 contributors. We also have 14 new contributors. Thank you all for the contributions!

Some notable changes in this release are:

• Prometheus Remote Write 2.0 Support: Experimental support for Prometheus Remote Write 2.0 protocol.
• Parquet Format Support: Experimental Parquet based block storage. A new parquet converter service to convert TSDB blocks to parquet and querier to query parquet files.
• Query Federation with Regex Tenant Resolver: Introduce experimental regex tenant resolver allowing regex patterns in X-Scope-OrgID header via -tenant-federation.regex-matcher-enabled flag
• gRPC Stream Push between Distributor and Ingester: Experimental feature to use gRPC stream connections for push requests.
• More Native Histogram Support: Out-of-order native histogram ingestion support, per-tenant native histogram ingestion config, native histogram active series metrics and limits
• Resource-Based Monitor and Limiter: ResourceMonitor to collect CPU and Heap usage for Cortex and ResourceBasedLimiter in Ingesters and StoreGateways to protect the service from incoming requests when hitting limits
• UTF-8 Name: UTF-8 name support via -name-validation-scheme flag

What's Changed

• [CHANGE] StoreGateway/Alertmanager: Add default 5s connection timeout on client. #6603
• [CHANGE] Ingester: Remove EnableNativeHistograms config flag and instead gate keep through new per-tenant limit at ingestion. #6718
• [CHANGE] Validate a tenantID when to use a single tenant resolver. #6727
• [CHANGE] Ring: Add zone label to ring_members metric. #6900
• [FEATURE] Distributor: Add an experimental -distributor.otlp.enable-type-and-unit-labels flag to add __type__ and __unit__ labels for OTLP metrics. #6969
• [FEATURE] Distributor: Add an experimental -distributor.otlp.allow-delta-temporality flag to ingest delta temporality otlp metrics. #6934
• [FEATURE] Query Frontend: Add dynamic interval size for query splitting. This is enabled by configuring experimental flags querier.max-shards-per-query and/or querier.max-fetched-data-duration-per-query. The split interval size is dynamically increased to maintain a number of shards and total duration fetched below the configured values. #6458
• [FEATURE] Querier/Ruler: Add query_partial_data and rules_partial_data limits to allow queries/rules to be evaluated with data from a single zone, if other zones are not available. #6526
• [FEATURE] Update prometheus alertmanager version to v0.28.0 and add new integration msteamsv2, jira, and rocketchat. #6590
• [FEATURE] Ingester/StoreGateway: Add ResourceMonitor module in Cortex, and add ResourceBasedLimiter in Ingesters and StoreGateways. #6674
• [FEATURE] Support Prometheus remote write 2.0. #6330
• [FEATURE] Ingester: Support out-of-order native histogram ingestion. It is automatically enabled when -ingester.out-of-order-time-window > 0 and -blocks-storage.tsdb.enable-native-histograms=true. #6626 #6663
• [FEATURE] Ruler: Add support for percentage based sharding for rulers. #6680
• [FEATURE] Ruler: Add support for group labels. #6665
• [FEATURE] Query federation: Introduce a regex tenant resolver to allow regex in X-Scope-OrgID value. #6713

• Add an experimental tenant-federation.regex-matcher-enabled flag. If it enabled, user can input regex to X-Scope-OrgId, the matched tenantIDs are automatically involved. The user discovery is based on scanning block storage, so new users can get queries after uploading a block (generally 2h).
• Add an experimental tenant-federation.user-sync-interval flag, it specifies how frequently to scan users. The scanned users are used to calculate matched tenantIDs.

• [FEATURE] Experimental Support Parquet format: Implement parquet converter service to convert a TSDB block into Parquet and Parquet Queryable. #6716 #6743
• [FEATURE] Distributor/Ingester: Implemented experimental feature to use gRPC stream connection for push requests. This can be enabled by setting -distributor.use-stream-push=true. #6580
• [FEATURE] Compactor: Add support for percentage based sharding for compactors. #6738
• [FEATURE] Querier: Allow choosing PromQL engine via header X-PromQL-EngineType. #6777
• [FEATURE] Querier: Support for configuring query optimizers and enabling XFunctions in the Thanos engine. #6873
• [FEATURE] Query Frontend: Add support /api/v1/format_query API for formatting queries. #6893
• [FEATURE] Query Frontend: Add support for /api/v1/parse_query API (experimental) to parse a PromQL expression and return it as a JSON-formatted AST (abstract syntax tree). #6978
• [ENHANCEMENT] Upgrade the Prometheus version to 3.6.0 and add a -name-validation-scheme flag to support UTF-8. #7040 #7056
• [ENHANCEMENT] Distributor: Emit an error with a 400 status code when empty labels are found before the relabelling or label dropping process. #7052
• [ENHANCEMENT] Parquet Storage: Add support for additional sort columns during Parquet file generation #7003
• [ENHANCEMENT] Modernizes the entire codebase by using go modernize tool. #7005
• [ENHANCEMENT] Overrides Exporter: Expose all fields that can be converted to float64. Also, the label value max_local_series_per_metric got renamed to max_series_per_metric, and max_local_series_per_user got renamed to max_series_per_user. #6979
• [ENHANCEMENT] Ingester: Add cortex_ingester_tsdb_wal_replay_unknown_refs_total and cortex_ingester_tsdb_wbl_replay_unknown_refs_total metrics to track unknown series references during wal/wbl replaying. #6945
• [ENHANCEMENT] Distributor: Introduce a Protobuf model for Prometheus Remote Write 2.0 and a pool to improve performance. #6917
• [ENHANCEMENT] Ruler: Emit an error message when the rule synchronization fails. #6902
• [ENHANCEMENT] Querier: Support snappy and zstd response compression for -querier.response-compression flag. #6848
• [ENHANCEMENT] Tenant Federation: Add a # of query result limit logic when the -tenant-federation.regex-matcher-enabled is enabled. #6845
• [ENHANCEMENT] Query Frontend: Add a cortex_slow_queries_total metric to track # of slow queries per user. #6859
• [ENHANCEMENT] Query Frontend: Change to return 400 when the tenant resolving fail. #6715
• [ENHANCEMENT] Querier: Support query parameters to metadata api (/api/v1/metadata) to allow user to limit metadata to return. Add a -ingester.return-all-metadata flag to make the metadata API run when the deployment. Please set this flag to false to use the metadata API with the limits later. #6681 #6744
• [ENHANCEMENT] Ingester: Add a cortex_ingester_active_native_histogram_series metric to track # of active NH series. #6695
• [ENHANCEMENT] Query Frontend: Add new limit -frontend.max-query-response-size for total query response size after decompression in query frontend. #6607
• [ENHANCEMENT] Alertmanager: Add nflog and silences maintenance metrics. #6659
• [ENHANCEMENT] Querier: limit label APIs to query only ingesters if start param is not specified. #6618
• [ENHANCEMENT] Alertmanager: Add new limits -alertmanager.max-silences-count and -alertmanager.max-silences-size-bytes for limiting silences per tenant. #6605
• [ENHANCEMENT] Add compactor.auto-forget-delay for compactor to auto forget compactors after X minutes without heartbeat. #6533
• [ENHANCEMENT] StoreGateway: Emit more histogram buckets on the cortex_querier_storegateway_refetches_per_query metric. #6570
• [ENHANCEMENT] Querier: Apply bytes limiter to LabelNames and LabelValuesForLabelNames. #6568
• [ENHANCEMENT] Query Frontend: Add a too_many_tenants reason label value to cortex_rejected_queries_total metric to track the rejected query count due to the # of tenant limits. #6569
• [ENHANCEMENT] Alertmanager: Add receiver validations for msteamsv2 and rocketchat. #6606
• [ENHANCEMENT] Query Frontend: Add a -frontend.enabled-ruler-query-stats flag to configure whether to report the query stats log for queries coming from the Ruler. #6504
• [ENHANCEMENT] OTLP: Support otlp metadata ingestion. #6617
• [ENHANCEMENT] AlertManager: Add keep_instance_in_the_ring_on_shutdown and tokens_file_path configs for alertmanager ring. #6628
• [ENHANCEMENT] Querier: Add metric and enhanced logging for query partial data. #6676
• [ENHANCEMENT] Ingester: Push request should fail when label set is out of order #6746
• [ENHANCEMENT] Querier: Add querier.ingester-query-max-attempts to retry on partial data. #6714
• [ENHANCEMENT] Distributor: Add min/max schema validation for Native Histogram. #6766
• [ENHANCEMENT] Ingester: Handle runtime errors in query path #6769
• [ENHANCEMENT] Compactor: Support metadata caching bucket for Cleaner. Can be enabled via -compactor.cleaner-caching-bucket-enabled flag. #6778
• [ENHANCEMENT] Distributor: Add ingestion rate limit for Native Histogram. #6794 and #6994
• [ENHANCEMENT] Ingester: Add active series limit specifically for Native Histogram. #6796
• [ENHANCEMENT] Compactor, Store Gateway: Introduce user scanner strategy and user index. #6780
• [ENHANCEMENT] Querier: Support chunks cache for parquet queryable. #6805
• [ENHANCEMENT] Parquet Storage: Add some metrics for parquet blocks and converter. #6809 #6821
• [ENHANCEMENT] Compactor: Optimize cleaner run time. #6815
• [ENHANCEMENT] Parquet Storage: Allow percentage based dynamic shard size for Parquet Converter. #6817
• [ENHANCEMENT] Query Frontend: Enhance the performance of the JSON codec. #6816
• [ENHANCEMENT] Compactor: Emit partition metrics separate from cleaner job. #6827
• [ENHANCEMENT] Metadata Cache: Support inmemory and multi level cache backend. #6829
• [ENHANCEMENT] Store Gateway: Allow to ignore syncing blocks older than certain time using ignore_blocks_before. #6830
• [ENHANCEMENT] Distributor: Add native histograms max sample size bytes limit validation. #683...

v1.19.1

What's Changed

• Fix remote read snappy input due to empty request body #7025

Full Changelog: v1.19.0...v1.19.1

v1.19.0

What's Changed

• [CHANGE] Deprecate -blocks-storage.tsdb.wal-compression-enabled flag (use blocks-storage.tsdb.wal-compression-type instead). #6529
• [CHANGE] OTLP: Change OTLP handler to be consistent with the Prometheus OTLP handler. #6272

• target_info metric is enabled by default and can be disabled via -distributor.otlp.disable-target-info=true flag
• Convert all attributes to labels is disabled by default and can be enabled via -distributor.otlp.convert-all-attributes=true flag
• You can specify the attributes converted to labels via -distributor.promote-resource-attributes flag. Supported only if -distributor.otlp.convert-all-attributes=false

• [CHANGE] Change all max async concurrency default values 50 to 3 #6268
• [CHANGE] Change default value of -blocks-storage.bucket-store.index-cache.multilevel.max-async-concurrency from 50 to 3 #6265
• [CHANGE] Enable Compactor and Alertmanager in target all. #6204
• [CHANGE] Update the cortex_ingester_inflight_push_requests metric to represent the maximum number of inflight requests recorded in the last minute. #6437
• [CHANGE] gRPC Client: Expose connection timeout and set default to value to 5s. #6523
• [CHANGE] Enable analysis on Thanos PromQL engine #6472
• [CHANGE] Log grafana headers along with query request before query execution #6391
• [CHANGE] Log query requests in QFE before query execution #6390
• [CHANGE] Log when a request starts running in querier #6525
• [CHANGE] Remove openstack swfit experimental in blocks storage #6322
• [CHANGE] Remove openstack swift experimental #6316
• [CHANGE] Replace cespare/xxhash with cespare/xxhash/v2 #6467
• [CHANGE] Stop using global instant query codec #6328
• [CHANGE] Unify query frontend instant and range protos #6180
• [FEATURE] Ruler: Add an experimental flag -ruler.query-response-format to retrieve query response as a proto format. #6345
• [FEATURE] Ruler: Pagination support for List Rules API. #6299
• [FEATURE] Query Frontend/Querier: Add protobuf codec -api.querier-default-codec and the option to choose response compression type -querier.response-compression. #5527
• [FEATURE] Ruler: Experimental: Add ruler.frontend-address to allow query to query frontends instead of ingesters. #6151
• [FEATURE] Ruler: Minimize chances of missed rule group evaluations that can occur due to OOM kills, bad underlying nodes, or due to an unhealthy ruler that appears in the ring as healthy. This feature is enabled via -ruler.enable-ha-evaluation flag. #6129
• [FEATURE] Store Gateway: Add an in-memory chunk cache. #6245
• [FEATURE] Chunk Cache: Support multi level cache and add metrics. #6249
• [FEATURE] Distributor: Accept multiple HA Tracker pairs in the same request. #6256
• [FEATURE] Ruler: Add support for per-user external labels #6340
• [FEATURE] Query Frontend: Support a metadata federated query when -tenant-federation.enabled=true. #6461
• [FEATURE] Query Frontend: Support an exemplar federated query when -tenant-federation.enabled=true. #6455
• [FEATURE] Ingester/StoreGateway: Add support for cache regex query matchers via -ingester.matchers-cache-max-items and -blocks-storage.bucket-store.matchers-cache-max-items. #6477 #6491
• [FEATURE] Add reason why the key was evicted in the cortex_ingester_expanded_postings_cache_evicts metric #6318
• [FEATURE] Create feature flag to switch between current shuffle sharding group planner and partition compaction group planner #6141
• [FEATURE] Hook up partition compaction end to end implementation #6510
• [FEATURE] Implement partition compaction grouper #6172
• [FEATURE] Implement partition compaction planner #6469
• [FEATURE] Make LivenessCheck Timeout Configurable #6227
• [FEATURE] Querier: Add day range limit for LabelNames and LabelValues #6233
• [ENHANCEMENT] Query Frontend: Add more operation label values to the cortex_query_frontend_queries_total metric. #6519
• [ENHANCEMENT] Query Frontend: Add a source label to query stat metrics. #6470
• [ENHANCEMENT] Query Frontend: Add a flag -tenant-federation.max-tenant to limit the number of tenants for federated query. #6493
• [ENHANCEMENT] Querier: Add a -tenant-federation.max-concurrent flags to configure the number of worker processing federated query and add a cortex_querier_federated_tenants_per_query histogram to track the number of tenants per query. #6449
• [ENHANCEMENT] Query Frontend: Add a number of series in the query response to the query stat log. #6423
• [ENHANCEMENT] Store Gateway: Add a hedged request to reduce the tail latency. #6388
• [ENHANCEMENT] Ingester: Add metrics to track succeed/failed native histograms. #6370
• [ENHANCEMENT] Query Frontend/Querier: Add an experimental flag -querier.enable-promql-experimental-functions to enable experimental promQL functions. #6355
• [ENHANCEMENT] OTLP: Add -distributor.otlp-max-recv-msg-size flag to limit OTLP request size in bytes. #6333
• [ENHANCEMENT] S3 Bucket Client: Add a list objects version configs to configure list api object version. #6280
• [ENHANCEMENT] OpenStack Swift: Add application credential configs for Openstack swift object storage backend. #6255
• [ENHANCEMENT] Query Frontend: Add new query stats metrics cortex_query_samples_scanned_total and cortex_query_peak_samples to track scannedSamples and peakSample per user. #6228
• [ENHANCEMENT] Ingester: Add option ingester.disable-chunk-trimming to disable chunk trimming. #6300
• [ENHANCEMENT] Ingester: Add blocks-storage.tsdb.wal-compression-type to support zstd wal compression type. #6232
• [ENHANCEMENT] Query Frontend: Add info field to query response. #6207
• [ENHANCEMENT] Query Frontend: Add peakSample in query stats response. #6188
• [ENHANCEMENT] Ruler: Add new ruler metric cortex_ruler_rule_groups_in_store that is the total rule groups per tenant in store, which can be used to compare with cortex_prometheus_rule_group_rules to count the number of rule groups that are not loaded by a ruler. #5869
• [ENHANCEMENT] Ingester/Ring: New READONLY status on ring to be used by Ingester. New ingester API to change mode of ingester #6163
• [ENHANCEMENT] Ruler: Add query statistics metrics when --ruler.query-stats-enabled=true. #6173
• [ENHANCEMENT] Ingester: Add new API /ingester/all_user_stats which shows loaded blocks, active timeseries and ingestion rate for a specific ingester. #6178
• [ENHANCEMENT] Distributor: Add new cortex_reduced_resolution_histogram_samples_total metric to track the number of histogram samples which resolution was reduced. #6182
• [ENHANCEMENT] StoreGateway: Implement metadata API limit in queryable. #6195
• [ENHANCEMENT] Ingester: Add matchers to ingester LabelNames() and LabelNamesStream() RPC. #6209
• [ENHANCEMENT] KV: Add TLS configs to consul. #6374
• [ENHANCEMENT] Ingester/Store Gateway Clients: Introduce an experimental HealthCheck handler to quickly fail requests directed to unhealthy targets. #6225 #6257
• [ENHANCEMENT] Upgrade build image and Go version to 1.23.2. #6261 #6262
• [ENHANCEMENT] Ingester: Introduce a new experimental feature for caching expanded postings on the ingester. #6296
• [ENHANCEMENT] Querier/Ruler: Expose store_gateway_consistency_check_max_attempts for max retries when querying store gateway in consistency check. #6276
• [ENHANCEMENT] StoreGateway: Add new cortex_bucket_store_chunk_pool_inuse_bytes metric to track the usage in chunk pool. #6310
• [ENHANCEMENT] Distributor: Add new cortex_distributor_inflight_client_requests metric to track number of ingester client inflight requests. #6358
• [ENHANCEMENT] Distributor: Expose cortex_label_size_bytes native histogram metric. #6372
• [ENHANCEMENT] Add new option -server.grpc_server-num-stream-workers to configure the number of worker goroutines that should be used to process incoming streams. #6386
• [ENHANCEMENT] Distributor: Return HTTP 5XX instead of HTTP 4XX when instance limits are hit. #6358
• [ENHANCEMENT] Ingester: Make sure unregistered ingester joining the ring after WAL replay. #6277
• [ENHANCEMENT] Distributor: Add a new -distributor.num-push-workers flag to use a goroutine worker pool when sending data from distributor to ingesters. #6406
• [ENHANCEMENT] Ingester: If a limit per label set entry doesn't have any label, use it as the default partition to catch all series that doesn't match any other label sets entries. #6435
• [ENHANCEMENT] Querier: Add new cortex_querier_codec_response_size metric to track the size of the encoded query responses from queriers. #6444
• [ENHANCEMENT] Distributor: Added cortex_distributor_received_samples_per_labelset_total metric to calculate ingestion rate per label set. #6443
• [ENHANCEMENT] Added metric name in limiter per-metric exceeded errors. #6416
• [ENHANCEMENT] StoreGateway: Added cortex_bucket_store_indexheader_load_duration_seconds and cortex_bucket_store_indexheader_download_duration_seconds metrics for time of downloading and loading index header files. #6445
• [ENHANCEMENT] Add cleaner logic to clean partition compaction blocks and related files #6507
• [ENHANCEMENT] Add support for native histograms in querier protobuf codec #6368
• [ENHANCEMENT] Add timeout on lifecycler heartbeat #6212
• [ENHANCEMENT] Added UserReplicaGroupMetrics #6463
• [ENHANCEMENT] Blocks Storage: Allow use of non-dualstack endpoints for S3 blocks storage via -blocks-storage.s3.disable-dualstack. #6522
• [ENHANCEMENT] Discarded samples per labelset metrics for throttle by labelset #6492
• [ENHANCEMENT] Expanded Postings Cache can cache results without the nearly created series under high load. #6417
• [ENHANCEMENT] Ingester: Disable chunk trimming. #6270
• [ENHANCEMENT] Improve consistency check warn log #6366
• [ENHANCEMENT] Improve streaming on MetricsForLabelMatchersStream method #6436
• [ENHANCEMENT] Improve validation metrics for discarded samples and exemplars #6218
• [E...

v1.19.0-rc.2

What's Changed

• Prepare release 1.19 rc.2 by @CharlieTLe in #6567

Full Changelog: v1.19.0-rc.1...v1.19.0-rc.2

v1.17.2

What's Changed

• [BUGFIX] Fix single distributor latency #5921

Full Changelog: v1.17.1...v1.17.2

v1.19.0-rc.1

What's Changed

• [CHANGE] Deprecate -blocks-storage.tsdb.wal-compression-enabled flag (use blocks-storage.tsdb.wal-compression-type instead). #6529
• [CHANGE] OTLP: Change OTLP handler to be consistent with the Prometheus OTLP handler. #6272

• target_info metric is enabled by default and can be disabled via -distributor.otlp.disable-target-info=true flag
• Convert all attributes to labels is disabled by default and can be enabled via -distributor.otlp.convert-all-attributes=true flag
• You can specify the attributes converted to labels via -distributor.promote-resource-attributes flag. Supported only if -distributor.otlp.convert-all-attributes=false

• [CHANGE] Change all max async concurrency default values 50 to 3 #6268
• [CHANGE] Change default value of -blocks-storage.bucket-store.index-cache.multilevel.max-async-concurrency from 50 to 3 #6265
• [CHANGE] Enable Compactor and Alertmanager in target all. #6204
• [CHANGE] Update the cortex_ingester_inflight_push_requests metric to represent the maximum number of inflight requests recorded in the last minute. #6437
• [CHANGE] gRPC Client: Expose connection timeout and set default to value to 5s. #6523
• [CHANGE] Enable analysis on Thanos PromQL engine #6472
• [CHANGE] Log grafana headers along with query request before query execution #6391
• [CHANGE] Log query requests in QFE before query execution #6390
• [CHANGE] Log when a request starts running in querier #6525
• [CHANGE] Remove openstack swfit experimental in blocks storage #6322
• [CHANGE] Remove openstack swift experimental #6316
• [CHANGE] Replace cespare/xxhash with cespare/xxhash/v2 #6467
• [CHANGE] Stop using global instant query codec #6328
• [CHANGE] Unify query frontend instant and range protos #6180
• [FEATURE] Ruler: Add an experimental flag -ruler.query-response-format to retrieve query response as a proto format. #6345
• [FEATURE] Ruler: Pagination support for List Rules API. #6299
• [FEATURE] Query Frontend/Querier: Add protobuf codec -api.querier-default-codec and the option to choose response compression type -querier.response-compression. #5527
• [FEATURE] Ruler: Experimental: Add ruler.frontend-address to allow query to query frontends instead of ingesters. #6151
• [FEATURE] Ruler: Minimize chances of missed rule group evaluations that can occur due to OOM kills, bad underlying nodes, or due to an unhealthy ruler that appears in the ring as healthy. This feature is enabled via -ruler.enable-ha-evaluation flag. #6129
• [FEATURE] Store Gateway: Add an in-memory chunk cache. #6245
• [FEATURE] Chunk Cache: Support multi level cache and add metrics. #6249
• [FEATURE] Distributor: Accept multiple HA Tracker pairs in the same request. #6256
• [FEATURE] Ruler: Add support for per-user external labels #6340
• [FEATURE] Query Frontend: Support a metadata federated query when -tenant-federation.enabled=true. #6461
• [FEATURE] Query Frontend: Support an exemplar federated query when -tenant-federation.enabled=true. #6455
• [FEATURE] Ingester/StoreGateway: Add support for cache regex query matchers via -ingester.matchers-cache-max-items and -blocks-storage.bucket-store.matchers-cache-max-items. #6477 #6491
• [FEATURE] Add reason why the key was evicted in the cortex_ingester_expanded_postings_cache_evicts metric #6318
• [FEATURE] Create feature flag to switch between current shuffle sharding group planner and partition compaction group planner #6141
• [FEATURE] Hook up partition compaction end to end implementation #6510
• [FEATURE] Implement partition compaction grouper #6172
• [FEATURE] Implement partition compaction planner #6469
• [FEATURE] Make LivenessCheck Timeout Configurable #6227
• [FEATURE] Querier: Add day range limit for LabelNames and LabelValues #6233
• [ENHANCEMENT] Query Frontend: Add more operation label values to the cortex_query_frontend_queries_total metric. #6519
• [ENHANCEMENT] Query Frontend: Add a source label to query stat metrics. #6470
• [ENHANCEMENT] Query Frontend: Add a flag -tenant-federation.max-tenant to limit the number of tenants for federated query. #6493
• [ENHANCEMENT] Querier: Add a -tenant-federation.max-concurrent flags to configure the number of worker processing federated query and add a cortex_querier_federated_tenants_per_query histogram to track the number of tenants per query. #6449
• [ENHANCEMENT] Query Frontend: Add a number of series in the query response to the query stat log. #6423
• [ENHANCEMENT] Store Gateway: Add a hedged request to reduce the tail latency. #6388
• [ENHANCEMENT] Ingester: Add metrics to track succeed/failed native histograms. #6370
• [ENHANCEMENT] Query Frontend/Querier: Add an experimental flag -querier.enable-promql-experimental-functions to enable experimental promQL functions. #6355
• [ENHANCEMENT] OTLP: Add -distributor.otlp-max-recv-msg-size flag to limit OTLP request size in bytes. #6333
• [ENHANCEMENT] S3 Bucket Client: Add a list objects version configs to configure list api object version. #6280
• [ENHANCEMENT] OpenStack Swift: Add application credential configs for Openstack swift object storage backend. #6255
• [ENHANCEMENT] Query Frontend: Add new query stats metrics cortex_query_samples_scanned_total and cortex_query_peak_samples to track scannedSamples and peakSample per user. #6228
• [ENHANCEMENT] Ingester: Add option ingester.disable-chunk-trimming to disable chunk trimming. #6300
• [ENHANCEMENT] Ingester: Add blocks-storage.tsdb.wal-compression-type to support zstd wal compression type. #6232
• [ENHANCEMENT] Query Frontend: Add info field to query response. #6207
• [ENHANCEMENT] Query Frontend: Add peakSample in query stats response. #6188
• [ENHANCEMENT] Ruler: Add new ruler metric cortex_ruler_rule_groups_in_store that is the total rule groups per tenant in store, which can be used to compare with cortex_prometheus_rule_group_rules to count the number of rule groups that are not loaded by a ruler. #5869
• [ENHANCEMENT] Ingester/Ring: New READONLY status on ring to be used by Ingester. New ingester API to change mode of ingester #6163
• [ENHANCEMENT] Ruler: Add query statistics metrics when --ruler.query-stats-enabled=true. #6173
• [ENHANCEMENT] Ingester: Add new API /ingester/all_user_stats which shows loaded blocks, active timeseries and ingestion rate for a specific ingester. #6178
• [ENHANCEMENT] Distributor: Add new cortex_reduced_resolution_histogram_samples_total metric to track the number of histogram samples which resolution was reduced. #6182
• [ENHANCEMENT] StoreGateway: Implement metadata API limit in queryable. #6195
• [ENHANCEMENT] Ingester: Add matchers to ingester LabelNames() and LabelNamesStream() RPC. #6209
• [ENHANCEMENT] KV: Add TLS configs to consul. #6374
• [ENHANCEMENT] Ingester/Store Gateway Clients: Introduce an experimental HealthCheck handler to quickly fail requests directed to unhealthy targets. #6225 #6257
• [ENHANCEMENT] Upgrade build image and Go version to 1.23.2. #6261 #6262
• [ENHANCEMENT] Ingester: Introduce a new experimental feature for caching expanded postings on the ingester. #6296
• [ENHANCEMENT] Querier/Ruler: Expose store_gateway_consistency_check_max_attempts for max retries when querying store gateway in consistency check. #6276
• [ENHANCEMENT] StoreGateway: Add new cortex_bucket_store_chunk_pool_inuse_bytes metric to track the usage in chunk pool. #6310
• [ENHANCEMENT] Distributor: Add new cortex_distributor_inflight_client_requests metric to track number of ingester client inflight requests. #6358
• [ENHANCEMENT] Distributor: Expose cortex_label_size_bytes native histogram metric. #6372
• [ENHANCEMENT] Add new option -server.grpc_server-num-stream-workers to configure the number of worker goroutines that should be used to process incoming streams. #6386
• [ENHANCEMENT] Distributor: Return HTTP 5XX instead of HTTP 4XX when instance limits are hit. #6358
• [ENHANCEMENT] Ingester: Make sure unregistered ingester joining the ring after WAL replay. #6277
• [ENHANCEMENT] Distributor: Add a new -distributor.num-push-workers flag to use a goroutine worker pool when sending data from distributor to ingesters. #6406
• [ENHANCEMENT] Ingester: If a limit per label set entry doesn't have any label, use it as the default partition to catch all series that doesn't match any other label sets entries. #6435
• [ENHANCEMENT] Querier: Add new cortex_querier_codec_response_size metric to track the size of the encoded query responses from queriers. #6444
• [ENHANCEMENT] Distributor: Added cortex_distributor_received_samples_per_labelset_total metric to calculate ingestion rate per label set. #6443
• [ENHANCEMENT] Added metric name in limiter per-metric exceeded errors. #6416
• [ENHANCEMENT] StoreGateway: Added cortex_bucket_store_indexheader_load_duration_seconds and cortex_bucket_store_indexheader_download_duration_seconds metrics for time of downloading and loading index header files. #6445
• [ENHANCEMENT] Add cleaner logic to clean partition compaction blocks and related files #6507
• [ENHANCEMENT] Add support for native histograms in querier protobuf codec #6368
• [ENHANCEMENT] Add timeout on lifecycler heartbeat #6212
• [ENHANCEMENT] Added UserReplicaGroupMetrics #6463
• [ENHANCEMENT] Blocks Storage: Allow use of non-dualstack endpoints for S3 blocks storage via -blocks-storage.s3.disable-dualstack. #6522
• [ENHANCEMENT] Discarded samples per labelset metrics for throttle by labelset #6492
• [ENHANCEMENT] Expanded Postings Cache can cache results without the nearly created series under high load. #6417
• [ENHANCEMENT] Ingester: Disable chunk trimming. #6270
• [ENHANCEMENT] Improve consistency check warn log #6366
• [ENHANCEMENT] Improve streaming on MetricsForLabelMatchersStream method #6436
• [ENHANCEMENT] Improve validation metrics for discarded samples and exemplars #6218
• [E...

v1.19.0-rc.0

What's Changed

• [CHANGE] Deprecate -blocks-storage.tsdb.wal-compression-enabled flag (use blocks-storage.tsdb.wal-compression-type instead). #6529
• [CHANGE] OTLP: Change OTLP handler to be consistent with the Prometheus OTLP handler. #6272

• target_info metric is enabled by default and can be disabled via -distributor.otlp.disable-target-info=true flag
• Convert all attributes to labels is disabled by default and can be enabled via -distributor.otlp.convert-all-attributes=true flag
• You can specify the attributes converted to labels via -distributor.promote-resource-attributes flag. Supported only if -distributor.otlp.convert-all-attributes=false

• [CHANGE] Change all max async concurrency default values 50 to 3 #6268
• [CHANGE] Change default value of -blocks-storage.bucket-store.index-cache.multilevel.max-async-concurrency from 50 to 3 #6265
• [CHANGE] Enable Compactor and Alertmanager in target all. #6204
• [CHANGE] Update the cortex_ingester_inflight_push_requests metric to represent the maximum number of inflight requests recorded in the last minute. #6437
• [CHANGE] gRPC Client: Expose connection timeout and set default to value to 5s. #6523
• [CHANGE] Enable analysis on Thanos PromQL engine #6472
• [CHANGE] Log grafana headers along with query request before query execution #6391
• [CHANGE] Log query requests in QFE before query execution #6390
• [CHANGE] Log when a request starts running in querier #6525
• [CHANGE] Remove openstack swfit experimental in blocks storage #6322
• [CHANGE] Remove openstack swift experimental #6316
• [CHANGE] Replace cespare/xxhash with cespare/xxhash/v2 #6467
• [CHANGE] Stop using global instant query codec #6328
• [CHANGE] Unify query frontend instant and range protos #6180
• [FEATURE] Ruler: Add an experimental flag -ruler.query-response-format to retrieve query response as a proto format. #6345
• [FEATURE] Ruler: Pagination support for List Rules API. #6299
• [FEATURE] Query Frontend/Querier: Add protobuf codec -api.querier-default-codec and the option to choose response compression type -querier.response-compression. #5527
• [FEATURE] Ruler: Experimental: Add ruler.frontend-address to allow query to query frontends instead of ingesters. #6151
• [FEATURE] Ruler: Minimize chances of missed rule group evaluations that can occur due to OOM kills, bad underlying nodes, or due to an unhealthy ruler that appears in the ring as healthy. This feature is enabled via -ruler.enable-ha-evaluation flag. #6129
• [FEATURE] Store Gateway: Add an in-memory chunk cache. #6245
• [FEATURE] Chunk Cache: Support multi level cache and add metrics. #6249
• [FEATURE] Distributor: Accept multiple HA Tracker pairs in the same request. #6256
• [FEATURE] Ruler: Add support for per-user external labels #6340
• [FEATURE] Query Frontend: Support a metadata federated query when -tenant-federation.enabled=true. #6461
• [FEATURE] Query Frontend: Support an exemplar federated query when -tenant-federation.enabled=true. #6455
• [FEATURE] Ingester/StoreGateway: Add support for cache regex query matchers via -ingester.matchers-cache-max-items and -blocks-storage.bucket-store.matchers-cache-max-items. #6477 #6491
• [FEATURE] Add reason why the key was evicted in the cortex_ingester_expanded_postings_cache_evicts metric #6318
• [FEATURE] Create feature flag to switch between current shuffle sharding group planner and partition compaction group planner #6141
• [FEATURE] Hook up partition compaction end to end implementation #6510
• [FEATURE] Implement partition compaction grouper #6172
• [FEATURE] Implement partition compaction planner #6469
• [FEATURE] Make LivenessCheck Timeout Configurable #6227
• [FEATURE] Querier: Add day range limit for LabelNames and LabelValues #6233
• [ENHANCEMENT] Query Frontend: Add more operation label values to the cortex_query_frontend_queries_total metric. #6519
• [ENHANCEMENT] Query Frontend: Add a source label to query stat metrics. #6470
• [ENHANCEMENT] Query Frontend: Add a flag -tenant-federation.max-tenant to limit the number of tenants for federated query. #6493
• [ENHANCEMENT] Querier: Add a -tenant-federation.max-concurrent flags to configure the number of worker processing federated query and add a cortex_querier_federated_tenants_per_query histogram to track the number of tenants per query. #6449
• [ENHANCEMENT] Query Frontend: Add a number of series in the query response to the query stat log. #6423
• [ENHANCEMENT] Store Gateway: Add a hedged request to reduce the tail latency. #6388
• [ENHANCEMENT] Ingester: Add metrics to track succeed/failed native histograms. #6370
• [ENHANCEMENT] Query Frontend/Querier: Add an experimental flag -querier.enable-promql-experimental-functions to enable experimental promQL functions. #6355
• [ENHANCEMENT] OTLP: Add -distributor.otlp-max-recv-msg-size flag to limit OTLP request size in bytes. #6333
• [ENHANCEMENT] S3 Bucket Client: Add a list objects version configs to configure list api object version. #6280
• [ENHANCEMENT] OpenStack Swift: Add application credential configs for Openstack swift object storage backend. #6255
• [ENHANCEMENT] Query Frontend: Add new query stats metrics cortex_query_samples_scanned_total and cortex_query_peak_samples to track scannedSamples and peakSample per user. #6228
• [ENHANCEMENT] Ingester: Add option ingester.disable-chunk-trimming to disable chunk trimming. #6300
• [ENHANCEMENT] Ingester: Add blocks-storage.tsdb.wal-compression-type to support zstd wal compression type. #6232
• [ENHANCEMENT] Query Frontend: Add info field to query response. #6207
• [ENHANCEMENT] Query Frontend: Add peakSample in query stats response. #6188
• [ENHANCEMENT] Ruler: Add new ruler metric cortex_ruler_rule_groups_in_store that is the total rule groups per tenant in store, which can be used to compare with cortex_prometheus_rule_group_rules to count the number of rule groups that are not loaded by a ruler. #5869
• [ENHANCEMENT] Ingester/Ring: New READONLY status on ring to be used by Ingester. New ingester API to change mode of ingester #6163
• [ENHANCEMENT] Ruler: Add query statistics metrics when --ruler.query-stats-enabled=true. #6173
• [ENHANCEMENT] Ingester: Add new API /ingester/all_user_stats which shows loaded blocks, active timeseries and ingestion rate for a specific ingester. #6178
• [ENHANCEMENT] Distributor: Add new cortex_reduced_resolution_histogram_samples_total metric to track the number of histogram samples which resolution was reduced. #6182
• [ENHANCEMENT] StoreGateway: Implement metadata API limit in queryable. #6195
• [ENHANCEMENT] Ingester: Add matchers to ingester LabelNames() and LabelNamesStream() RPC. #6209
• [ENHANCEMENT] KV: Add TLS configs to consul. #6374
• [ENHANCEMENT] Ingester/Store Gateway Clients: Introduce an experimental HealthCheck handler to quickly fail requests directed to unhealthy targets. #6225 #6257
• [ENHANCEMENT] Upgrade build image and Go version to 1.23.2. #6261 #6262
• [ENHANCEMENT] Ingester: Introduce a new experimental feature for caching expanded postings on the ingester. #6296
• [ENHANCEMENT] Querier/Ruler: Expose store_gateway_consistency_check_max_attempts for max retries when querying store gateway in consistency check. #6276
• [ENHANCEMENT] StoreGateway: Add new cortex_bucket_store_chunk_pool_inuse_bytes metric to track the usage in chunk pool. #6310
• [ENHANCEMENT] Distributor: Add new cortex_distributor_inflight_client_requests metric to track number of ingester client inflight requests. #6358
• [ENHANCEMENT] Distributor: Expose cortex_label_size_bytes native histogram metric. #6372
• [ENHANCEMENT] Add new option -server.grpc_server-num-stream-workers to configure the number of worker goroutines that should be used to process incoming streams. #6386
• [ENHANCEMENT] Distributor: Return HTTP 5XX instead of HTTP 4XX when instance limits are hit. #6358
• [ENHANCEMENT] Ingester: Make sure unregistered ingester joining the ring after WAL replay. #6277
• [ENHANCEMENT] Distributor: Add a new -distributor.num-push-workers flag to use a goroutine worker pool when sending data from distributor to ingesters. #6406
• [ENHANCEMENT] Ingester: If a limit per label set entry doesn't have any label, use it as the default partition to catch all series that doesn't match any other label sets entries. #6435
• [ENHANCEMENT] Querier: Add new cortex_querier_codec_response_size metric to track the size of the encoded query responses from queriers. #6444
• [ENHANCEMENT] Distributor: Added cortex_distributor_received_samples_per_labelset_total metric to calculate ingestion rate per label set. #6443
• [ENHANCEMENT] Added metric name in limiter per-metric exceeded errors. #6416
• [ENHANCEMENT] StoreGateway: Added cortex_bucket_store_indexheader_load_duration_seconds and cortex_bucket_store_indexheader_download_duration_seconds metrics for time of downloading and loading index header files. #6445
• [ENHANCEMENT] Add cleaner logic to clean partition compaction blocks and related files #6507
• [ENHANCEMENT] Add support for native histograms in querier protobuf codec #6368
• [ENHANCEMENT] Add timeout on lifecycler heartbeat #6212
• [ENHANCEMENT] Added UserReplicaGroupMetrics #6463
• [ENHANCEMENT] Blocks Storage: Allow use of non-dualstack endpoints for S3 blocks storage via -blocks-storage.s3.disable-dualstack. #6522
• [ENHANCEMENT] Discarded samples per labelset metrics for throttle by labelset #6492
• [ENHANCEMENT] Expanded Postings Cache can cache results without the nearly created series under high load. #6417
• [ENHANCEMENT] Ingester: Disable chunk trimming. #6270
• [ENHANCEMENT] Improve consistency check warn log #6366
• [ENHANCEMENT] Improve streaming on MetricsForLabelMatchersStream method #6436
• [ENHANCEMENT] Improve validation metrics for discarded samples and exemplars #6218
• [E...

v1.18.1

This release includes security fixes. Thanks to @99-not-out and @jeanlouisboudart for reporting it

What's Changed

• [BUGFIX] Backporting upgrade to go 1.22.7 to patch CVE-2024-34155, CVE-2024-34156, CVE-2024-34158 #6217 #6264

Full Changelog: v1.18.0...v1.18.1

Cortex v1.18.0

This release contains 230 contributions from 29 contributors. We also have 9 new contributors. Thank you all for the contributions!

Some notable changes release are:

• Experimental native histogram ingestion via -blocks-storage.tsdb.enable-native-histograms flag
• Support for filtering alerts on ListRules API
• Add query rejection mechanism to protect queries
• Introduce token bucket limiter to protect store gateway
• Implement ingester metadata API limits
• Remove -querier.query-store-for-labels-enabled flag
• Remove -querier.at-modifier-enabled flag
• Remove oltp_endpoint config
• ruler.evaluation-delay-duration mark as deprecated. Use ruler.query-offset
• -querier.max-outstanding-requests-per-tenant and -query-scheduler.max-outstanding-requests-per-tenant mark as deprecated. Use frontend.max-outstanding-requests-per-tenant

What's Changed

• [CHANGE] Ingester: Remove -querier.query-store-for-labels-enabled flag. Querying long-term store for labels is always enabled. #5984
• [CHANGE] Server: Instrument cortex_request_duration_seconds metric with native histogram. If native-histograms feature is enabled in monitoring Prometheus then the metric name needs to be updated in your dashboards. #6056
• [CHANGE] Distributor/Ingester: Change cortex_distributor_ingester_appends_total, cortex_distributor_ingester_append_failures_total, cortex_distributor_ingester_queries_total, and cortex_distributor_ingester_query_failures_total metrics to use the ingester ID instead of its IP as the label value. #6078
• [CHANGE] OTLP: Set AddMetricSuffixes to true to always enable metric name normalization. #6136
• [CHANGE] Querier: Deprecate and enable by default querier.ingester-metadata-streaming flag. #6147
• [CHANGE] QueryFrontend/QueryScheduler: Deprecate -querier.max-outstanding-requests-per-tenant and -query-scheduler.max-outstanding-requests-per-tenant flags. Use frontend.max-outstanding-requests-per-tenant instead. #6146
• [CHANGE] Ingesters: Enable 'snappy-block' compression on ingester clients by default. #6148
• [CHANGE] Ruler: Scheduling ruler.evaluation-delay-duration to be deprecated. Ruler will use the highest value between ruler.evaluation-delay-duration and ruler.query-offset #6149
• [CHANGE] Querier: Remove -querier.at-modifier-enabled flag. #6157
• [CHANGE] Tracing: Remove deprecated oltp_endpoint config entirely. #6158
• [CHANGE] Store Gateway: Enable store gateway zone stable shuffle sharding by default. #6161
• [FEATURE] Ingester/Distributor: Experimental: Enable native histogram ingestion via -blocks-storage.tsdb.enable-native-histograms flag. #5986 #6010 #6020
• [FEATURE] Querier: Enable querying native histogram chunks. #5944 #6031
• [FEATURE] Query Frontend: Support native histogram in query frontend response. #5996 #6043
• [FEATURE] Ruler: Support sending native histogram samples to Ingester. #6029
• [FEATURE] Ruler: Add support for filtering out alerts in ListRules API. #6011
• [FEATURE] Query Frontend: Added a query rejection mechanism to block resource-intensive queries. #6005
• [FEATURE] OTLP: Support ingesting OTLP exponential metrics as native histograms. #6071 #6135
• [FEATURE] Ingester: Add ingester.instance-limits.max-inflight-query-requests to allow limiting ingester concurrent queries. #6081
• [FEATURE] Distributor: Add validation.max-native-histogram-buckets to limit max number of bucket count. Distributor will try to automatically reduce histogram resolution until it is within the bucket limit or resolution cannot be reduced anymore. #6104
• [FEATURE] Store Gateway: Introduce token bucket limiter to enhance store gateway throttling. #6016
• [FEATURE] Ruler: Add support for query_offset field on RuleGroup and new ruler_query_offset per-tenant limit. #6085
• [ENHANCEMENT] Ruler: Add support to persist tokens in rulers. #5987
• [ENHANCEMENT] Query Frontend/Querier: Added store gateway postings touched count and touched size in Querier stats and log in Query Frontend. #5892
• [ENHANCEMENT] Query Frontend/Querier: Returns warnings on prometheus query responses. #5916
• [ENHANCEMENT] Ingester: Allowing to configure -blocks-storage.tsdb.head-compaction-interval flag up to 30 min and add a jitter on the first head compaction. #5919 #5928
• [ENHANCEMENT] Distributor: Added max_inflight_push_requests config to ingester client to protect distributor from OOMKilled. #5917
• [ENHANCEMENT] Distributor/Querier: Clean stale per-ingester metrics after ingester restarts. #5930
• [ENHANCEMENT] Distributor/Ring: Allow disabling detailed ring metrics by ring member. #5931
• [ENHANCEMENT] KV: Etcd Added etcd.ping-without-stream-allowed parameter to disable/enable PermitWithoutStream #5933
• [ENHANCEMENT] Ingester: Add a new limits_per_label_set limit. This limit functions similarly to max_series_per_metric, but allowing users to define the maximum number of series per LabelSet. #5950 #5993
• [ENHANCEMENT] Store Gateway: Log gRPC requests together with headers configured in http_request_headers_to_log. #5958
• [ENHANCEMENT] Ingester: Add a new experimental -ingester.labels-string-interning-enabled flag to enable string interning for metrics labels. #6057
• [ENHANCEMENT] Ingester: Add link to renew 10% of the ingesters tokens in the admin page. #6063
• [ENHANCEMENT] Ruler: Add support for filtering by state and health field on Rules API. #6040
• [ENHANCEMENT] Ruler: Add support for filtering by match field on Rules API. #6083
• [ENHANCEMENT] Distributor: Reduce memory usage when error volume is high. #6095
• [ENHANCEMENT] Compactor: Centralize metrics used by compactor and add user label to compactor metrics. #6096
• [ENHANCEMENT] Compactor: Add unique execution ID for each compaction cycle in log for easy debugging. #6097
• [ENHANCEMENT] Compactor: Differentiate retry and halt error and retry failed compaction only on retriable error. #6111
• [ENHANCEMENT] Ruler: Add support for filtering by state and health field on Rules API. #6040
• [ENHANCEMENT] Compactor: Split cleaner cycle for active and deleted tenants. #6112
• [ENHANCEMENT] Compactor: Introduce cleaner visit marker. #6113
• [ENHANCEMENT] Query Frontend: Add cortex_query_samples_total metric. #6142
• [ENHANCEMENT] Ingester: Implement metadata API limit. #6128
• [BUGFIX] Configsdb: Fix endline issue in db password. #5920
• [BUGFIX] Ingester: Fix user and type labels for the cortex_ingester_tsdb_head_samples_appended_total TSDB metric. #5952
• [BUGFIX] Querier: Enforce max query length check for /api/v1/series API even though ignoreMaxQueryLength is set to true. #6018
• [BUGFIX] Ingester: Fix issue with the minimize token generator where it was not taking in consideration the current ownership of an instance when generating extra tokens. #6062
• [BUGFIX] Scheduler: Fix user queue in scheduler that was not thread-safe. #6077 #6160
• [BUGFIX] Ingester: Include out-of-order head compaction when compacting TSDB head. #6108
• [BUGFIX] Ingester: Fix cortex_ingester_tsdb_mmap_chunks_total metric. #6134
• [BUGFIX] Query Frontend: Fix query rejection bug for metadata queries. #6143

New Contributors

• @Yaxhveer made their first contribution in #5920
• @KrisBuytaert made their first contribution in #5933
• @wilguo made their first contribution in #5935
• @rapphil made their first contribution in #5987
• @harshitasao made their first contribution in #6061
• @deradiri made their first contribution in #6099
• @shekeriev made their first contribution in #6125
• @klingerf made their first contribution in #6131
• @SungJin1212 made their first contribution in #6142

Full Changelog: v1.17.1...v1.18.0

Cortex v1.18.0-rc.0

This release contains 230 contributions from 29 contributors. We also have 9 new contributors. Thank you all for the contributions!

Some notable changes release are:

• Experimental native histogram ingestion via -blocks-storage.tsdb.enable-native-histograms flag
• Support for filtering alerts on ListRules API
• Add query rejection mechanism to protect queries
• Introduce token bucket limiter to protect store gateway
• Implement ingester metadata API limits
• Remove -querier.query-store-for-labels-enabled flag
• Remove -querier.at-modifier-enabled flag
• Remove oltp_endpoint config
• ruler.evaluation-delay-duration mark as deprecated. Use ruler.query-offset
• -querier.max-outstanding-requests-per-tenant and -query-scheduler.max-outstanding-requests-per-tenant mark as deprecated. Use frontend.max-outstanding-requests-per-tenant

What's Changed

• [CHANGE] Ingester: Remove -querier.query-store-for-labels-enabled flag. Querying long-term store for labels is always enabled. #5984
• [CHANGE] Server: Instrument cortex_request_duration_seconds metric with native histogram. If native-histograms feature is enabled in monitoring Prometheus then the metric name needs to be updated in your dashboards. #6056
• [CHANGE] Distributor/Ingester: Change cortex_distributor_ingester_appends_total, cortex_distributor_ingester_append_failures_total, cortex_distributor_ingester_queries_total, and cortex_distributor_ingester_query_failures_total metrics to use the ingester ID instead of its IP as the label value. #6078
• [CHANGE] OTLP: Set AddMetricSuffixes to true to always enable metric name normalization. #6136
• [CHANGE] Querier: Deprecate and enable by default querier.ingester-metadata-streaming flag. #6147
• [CHANGE] QueryFrontend/QueryScheduler: Deprecate -querier.max-outstanding-requests-per-tenant and -query-scheduler.max-outstanding-requests-per-tenant flags. Use frontend.max-outstanding-requests-per-tenant instead. #6146
• [CHANGE] Ingesters: Enable 'snappy-block' compression on ingester clients by default. #6148
• [CHANGE] Ruler: Scheduling ruler.evaluation-delay-duration to be deprecated. Ruler will use the highest value between ruler.evaluation-delay-duration and ruler.query-offset #6149
• [CHANGE] Querier: Remove -querier.at-modifier-enabled flag. #6157
• [CHANGE] Tracing: Remove deprecated oltp_endpoint config entirely. #6158
• [CHANGE] Store Gateway: Enable store gateway zone stable shuffle sharding by default. #6161
• [FEATURE] Ingester/Distributor: Experimental: Enable native histogram ingestion via -blocks-storage.tsdb.enable-native-histograms flag. #5986 #6010 #6020
• [FEATURE] Querier: Enable querying native histogram chunks. #5944 #6031
• [FEATURE] Query Frontend: Support native histogram in query frontend response. #5996 #6043
• [FEATURE] Ruler: Support sending native histogram samples to Ingester. #6029
• [FEATURE] Ruler: Add support for filtering out alerts in ListRules API. #6011
• [FEATURE] Query Frontend: Added a query rejection mechanism to block resource-intensive queries. #6005
• [FEATURE] OTLP: Support ingesting OTLP exponential metrics as native histograms. #6071 #6135
• [FEATURE] Ingester: Add ingester.instance-limits.max-inflight-query-requests to allow limiting ingester concurrent queries. #6081
• [FEATURE] Distributor: Add validation.max-native-histogram-buckets to limit max number of bucket count. Distributor will try to automatically reduce histogram resolution until it is within the bucket limit or resolution cannot be reduced anymore. #6104
• [FEATURE] Store Gateway: Introduce token bucket limiter to enhance store gateway throttling. #6016
• [FEATURE] Ruler: Add support for query_offset field on RuleGroup and new ruler_query_offset per-tenant limit. #6085
• [ENHANCEMENT] Ruler: Add support to persist tokens in rulers. #5987
• [ENHANCEMENT] Query Frontend/Querier: Added store gateway postings touched count and touched size in Querier stats and log in Query Frontend. #5892
• [ENHANCEMENT] Query Frontend/Querier: Returns warnings on prometheus query responses. #5916
• [ENHANCEMENT] Ingester: Allowing to configure -blocks-storage.tsdb.head-compaction-interval flag up to 30 min and add a jitter on the first head compaction. #5919 #5928
• [ENHANCEMENT] Distributor: Added max_inflight_push_requests config to ingester client to protect distributor from OOMKilled. #5917
• [ENHANCEMENT] Distributor/Querier: Clean stale per-ingester metrics after ingester restarts. #5930
• [ENHANCEMENT] Distributor/Ring: Allow disabling detailed ring metrics by ring member. #5931
• [ENHANCEMENT] KV: Etcd Added etcd.ping-without-stream-allowed parameter to disable/enable PermitWithoutStream #5933
• [ENHANCEMENT] Ingester: Add a new limits_per_label_set limit. This limit functions similarly to max_series_per_metric, but allowing users to define the maximum number of series per LabelSet. #5950 #5993
• [ENHANCEMENT] Store Gateway: Log gRPC requests together with headers configured in http_request_headers_to_log. #5958
• [ENHANCEMENT] Ingester: Add a new experimental -ingester.labels-string-interning-enabled flag to enable string interning for metrics labels. #6057
• [ENHANCEMENT] Ingester: Add link to renew 10% of the ingesters tokens in the admin page. #6063
• [ENHANCEMENT] Ruler: Add support for filtering by state and health field on Rules API. #6040
• [ENHANCEMENT] Ruler: Add support for filtering by match field on Rules API. #6083
• [ENHANCEMENT] Distributor: Reduce memory usage when error volume is high. #6095
• [ENHANCEMENT] Compactor: Centralize metrics used by compactor and add user label to compactor metrics. #6096
• [ENHANCEMENT] Compactor: Add unique execution ID for each compaction cycle in log for easy debugging. #6097
• [ENHANCEMENT] Compactor: Differentiate retry and halt error and retry failed compaction only on retriable error. #6111
• [ENHANCEMENT] Ruler: Add support for filtering by state and health field on Rules API. #6040
• [ENHANCEMENT] Compactor: Split cleaner cycle for active and deleted tenants. #6112
• [ENHANCEMENT] Compactor: Introduce cleaner visit marker. #6113
• [ENHANCEMENT] Query Frontend: Add cortex_query_samples_total metric. #6142
• [ENHANCEMENT] Ingester: Implement metadata API limit. #6128
• [BUGFIX] Configsdb: Fix endline issue in db password. #5920
• [BUGFIX] Ingester: Fix user and type labels for the cortex_ingester_tsdb_head_samples_appended_total TSDB metric. #5952
• [BUGFIX] Querier: Enforce max query length check for /api/v1/series API even though ignoreMaxQueryLength is set to true. #6018
• [BUGFIX] Ingester: Fix issue with the minimize token generator where it was not taking in consideration the current ownership of an instance when generating extra tokens. #6062
• [BUGFIX] Scheduler: Fix user queue in scheduler that was not thread-safe. #6077 #6160
• [BUGFIX] Ingester: Include out-of-order head compaction when compacting TSDB head. #6108
• [BUGFIX] Ingester: Fix cortex_ingester_tsdb_mmap_chunks_total metric. #6134
• [BUGFIX] Query Frontend: Fix query rejection bug for metadata queries. #6143

New Contributors

• @Yaxhveer made their first contribution in #5920
• @KrisBuytaert made their first contribution in #5933
• @wilguo made their first contribution in #5935
• @rapphil made their first contribution in #5987
• @harshitasao made their first contribution in #6061
• @deradiri made their first contribution in #6099
• @shekeriev made their first contribution in #6125
• @klingerf made their first contribution in #6131
• @SungJin1212 made their first contribution in #6142

Full Changelog: v1.17.1...v1.18.0-rc.0