SonarQube 2025.2 installed as App Service in Azure.
Disk size is 250gb. I continue to get errors such as below 2025.04.29 13:48:58 WARN es[][o.e.c.r.a.DiskThresholdMonitor] flood stage disk watermark [95%] exceeded on [8pZIn1CYTNmHPD8mE8Axcg][sonarqube][C:\home\site\wwwroot\sonarqube-2025.2.0.105476\data\es8] free: 2.8gb[4.7%], all indices on this node will be marked read-only
When checking the disk storage space:
C:\home\site\wwwroot\sonarqube-2025.2.0.105476\data\es8>dir
Volume in drive C is Windows
Directory of C:\home\site\wwwroot\sonarqube-2025.2.0.105476\data\es8
04/29/2025 02:04 PM <DIR> .
04/29/2025 01:48 PM <DIR> ..
04/29/2025 02:04 PM <DIR> _state
04/29/2025 01:48 PM 0 node.lock
04/29/2025 01:48 PM 115 nodes
2 File(s) 115 bytes
3 Dir(s) 264,665,948,160 bytes free
I have tried deleting log files, deleting temp files, deleting the ES8 folder to reindex and nothing has worked.
ES Log:
<INFO es[][o.e.n.Node] initialized
2025.04.29 14:04:51 INFO es[][o.e.n.Node] starting ...
2025.04.29 14:04:51 INFO es[][o.e.t.TransportService] publish_address {127.0.0.1:50981}, bound_addresses {127.0.0.1:50981}
2025.04.29 14:04:52 INFO es[][o.e.b.BootstrapChecks] explicitly enforcing bootstrap checks
2025.04.29 14:04:52 INFO es[][o.e.c.c.ClusterBootstrapService] this node is locked into cluster UUID [K2SZmIVPQLmSgY3k05gLHA] and will not attempt further cluster bootstrapping
2025.04.29 14:04:52 INFO es[][o.e.c.s.MasterService] elected-as-master ([1] nodes joined in term 6)[_FINISH_ELECTION_, {sonarqube}{8pZIn1CYTNmHPD8mE8Axcg}{wAxUOzVsTHuGlyjRcgnDBw}{sonarqube}{127.0.0.1}{127.0.0.1:50981}{cdfhilmrstw}{8.16.3}{7000099-8518000} completing election], term: 6, version: 34, delta: master node changed {previous [], current [{sonarqube}{8pZIn1CYTNmHPD8mE8Axcg}{wAxUOzVsTHuGlyjRcgnDBw}{sonarqube}{127.0.0.1}{127.0.0.1:50981}{cdfhilmrstw}{8.16.3}{7000099-8518000}]}
2025.04.29 14:04:52 INFO es[][o.e.c.s.ClusterApplierService] master node changed {previous [], current [{sonarqube}{8pZIn1CYTNmHPD8mE8Axcg}{wAxUOzVsTHuGlyjRcgnDBw}{sonarqube}{127.0.0.1}{127.0.0.1:50981}{cdfhilmrstw}{8.16.3}{7000099-8518000}]}, term: 6, version: 34, reason: Publication{term=6, version=34}
2025.04.29 14:04:52 INFO es[][o.e.c.c.NodeJoinExecutor] node-join: [{sonarqube}{8pZIn1CYTNmHPD8mE8Axcg}{wAxUOzVsTHuGlyjRcgnDBw}{sonarqube}{127.0.0.1}{127.0.0.1:50981}{cdfhilmrstw}{8.16.3}{7000099-8518000}] with reason [completing election]
2025.04.29 14:04:53 INFO es[][o.e.h.AbstractHttpServerTransport] publish_address {127.0.0.1:9001}, bound_addresses {127.0.0.1:9001}
2025.04.29 14:04:53 INFO es[][o.e.n.Node] started {sonarqube}{8pZIn1CYTNmHPD8mE8Axcg}{wAxUOzVsTHuGlyjRcgnDBw}{sonarqube}{127.0.0.1}{127.0.0.1:50981}{cdfhilmrstw}{8.16.3}{7000099-8518000}{xpack.installed=true, rack_id=sonarqube}
2025.04.29 14:04:53 INFO es[][o.e.l.ClusterStateLicenseService] license [b5791f2e-b690-425b-8560-a23275c9f35d] mode [basic] - valid
2025.04.29 14:04:53 INFO es[][o.e.c.f.AbstractFileWatchingService] starting file watcher ...
2025.04.29 14:04:53 INFO es[][o.e.g.GatewayService] recovered [1] indices into cluster_state
2025.04.29 14:04:53 INFO es[][o.e.c.f.AbstractFileWatchingService] file settings service up and running [tid=36]
2025.04.29 14:04:53 INFO es[][o.e.r.s.FileSettingsService] setting file [C:\home\site\wwwroot\sonarqube-2025.2.0.105476\temp\conf\es\operator\settings.json] not found, initializing [file_settings] as empty
2025.04.29 14:04:54 INFO es[][o.e.h.n.s.HealthNodeTaskExecutor] Node [{sonarqube}{8pZIn1CYTNmHPD8mE8Axcg}] is selected as the current health node.
2025.04.29 14:04:54 INFO es[][o.e.c.r.a.AllocationService] current.health="RED" message="Cluster health status changed from [YELLOW] to [RED] (reason: [reconcile-desired-balance])." previous.health="YELLOW" reason="reconcile-desired-balance"
2025.04.29 14:06:47 INFO es[][o.e.n.NativeAccess] Using [jna] native provider and native methods for [Windows]
2025.04.29 14:06:53 INFO es[][o.e.n.Node] version[8.16.3], pid[13920], build[tar/2eb78bceb86e182dc8f45ab76a704b1bfd352c9d/2025-01-10T10:08:28.587559883Z], OS[Windows Server 2022/10.0/amd64], JVM[Microsoft/OpenJDK 64-Bit Server VM/17.0.14/17.0.14+7-LTS]
2025.04.29 14:06:53 INFO es[][o.e.n.Node] JVM home [C:\Program Files\Java\microsoft-jdk-17.0.14.7], using bundled JDK [false]
2025.04.29 14:06:53 INFO es[][o.e.n.Node] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -Djava.security.manager=allow, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, -XX:ReplayDataFile=logs/replay_pid%p.log, -Des.distribution.type=tar, -XX:+UseG1GC, -Djava.io.tmpdir=C:\home\site\wwwroot\sonarqube-2025.2.0.105476\temp, -XX:ErrorFile=C:\home\site\wwwroot\sonarqube-2025.2.0.105476\logs\es_hs_err_pid%p.log, -Xlog:disable, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djna.tmpdir=C:\home\site\wwwroot\sonarqube-2025.2.0.105476\temp, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=COMPAT, -Des.enforce.bootstrap.checks=true, -Xmx2G, -Xms2G, -XX:+HeapDumpOnOutOfMemoryError, -XX:MaxDirectMemorySize=1073741824, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, --module-path=C:\home\site\wwwroot\sonarqube-2025.2.0.105476\elasticsearch\lib, --add-modules=jdk.net, --add-modules=ALL-MODULE-PATH, -Djdk.module.main=org.elasticsearch.server]
2025.04.29 14:06:53 INFO es[][o.e.n.Node] Default Locale [en_US]
2025.04.29 14:06:56 INFO es[][o.e.p.PluginsService] loaded module [rest-root]
2025.04.29 14:06:56 INFO es[][o.e.p.PluginsService] loaded module [reindex]
2025.04.29 14:06:56 INFO es[][o.e.p.PluginsService] loaded module [x-pack-core]
2025.04.29 14:06:56 INFO es[][o.e.p.PluginsService] loaded module [x-pack-redact]
2025.04.29 14:06:56 INFO es[][o.e.p.PluginsService] loaded module [x-pack-security]
2025.04.29 14:06:56 INFO es[][o.e.p.PluginsService] loaded module [x-pack-geoip-enterprise-downloader]
2025.04.29 14:06:56 INFO es[][o.e.p.PluginsService] loaded module [x-pack-esql-core]
2025.04.29 14:06:56 INFO es[][o.e.p.PluginsService] loaded module [x-pack-apm-data]
2025.04.29 14:06:56 INFO es[][o.e.p.PluginsService] loaded module [lang-painless]
2025.04.29 14:06:56 INFO es[][o.e.p.PluginsService] loaded module [old-lucene-versions]
2025.04.29 14:06:56 INFO es[][o.e.p.PluginsService] loaded module [parent-join]
2025.04.29 14:06:56 INFO es[][o.e.p.PluginsService] loaded module [counted-keyword]
2025.04.29 14:06:56 INFO es[][o.e.p.PluginsService] loaded module [logsdb]
2025.04.29 14:06:56 INFO es[][o.e.p.PluginsService] loaded module [analysis-common]
2025.04.29 14:06:56 INFO es[][o.e.p.PluginsService] loaded module [x-pack-inference]
2025.04.29 14:06:56 INFO es[][o.e.p.PluginsService] loaded module [dot-prefix-validation]
2025.04.29 14:06:56 INFO es[][o.e.p.PluginsService] loaded module [health-shards-availability]
2025.04.29 14:06:56 INFO es[][o.e.p.PluginsService] loaded module [transport-netty4]
2025.04.29 14:06:56 INFO es[][o.e.p.PluginsService] loaded module [x-pack-otel-data]
2025.04.29 14:06:56 INFO es[][o.e.p.PluginsService] loaded module [aggregations]
2025.04.29 14:06:56 INFO es[][o.e.p.PluginsService] loaded module [apm]
2025.04.29 14:06:58 INFO es[][o.e.e.NodeEnvironment] using [1] data paths, mounts [[Windows (C:)]], net usable_space [3gb], net total_space [59.4gb], types [NTFS]
2025.04.29 14:06:58 INFO es[][o.e.e.NodeEnvironment] heap size [2gb], compressed ordinary object pointers [true]
2025.04.29 14:06:59 INFO es[][o.e.n.Node] node name [sonarqube], node ID [8pZIn1CYTNmHPD8mE8Axcg], cluster name [sonarqube], roles [data_cold, ingest, data_frozen, ml, data_hot, transform, data_content, data_warm, master, remote_cluster_client, data]
2025.04.29 14:07:03 INFO es[][o.e.i.r.RecoverySettings] using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b]
2025.04.29 14:07:03 INFO es[][o.e.f.FeatureService] Registered local node features [desired_node.version_deprecated, features_supported, file_settings, flattened.ignore_above_support, geoip.downloader.database.configuration, get_database_configuration_action.multi_node, health.extended_repository_indicator, knn_retriever_supported, license-trial-independent-version, mapper.boolean_dimension, mapper.flattened.ignore_above_with_arrays_support, mapper.ignore_above_index_level_setting, mapper.index_sorting_on_nested, mapper.keyword_dimension_ignore_above, mapper.keyword_normalizer_synthetic_source, mapper.pass_through_priority, mapper.query_index_mode, mapper.range.null_values_off_by_one_fix, mapper.segment_level_fields_stats, mapper.source.synthetic_source_copy_to_fix, mapper.source.synthetic_source_copy_to_inside_objects_fix, mapper.source.synthetic_source_fallback, mapper.source.synthetic_source_stored_fields_advance_fix, mapper.source.synthetic_source_with_copy_to_and_doc_values_false, mapper.subobjects_auto, mapper.subobjects_auto_fixes, mapper.synthetic_source_keep, mapper.track_ignored_source, mapper.vectors.bbq, mapper.vectors.bit_vectors, mapper.vectors.int4_quantization, put_database_configuration_action.ipinfo, random_reranker_retriever_supported, repositories.supports_usage_stats, rest.capabilities_action, rest.local_only_capabilities, retrievers_supported, routing.boolean_routing_path, routing.multi_value_routing_path, script.hamming, script.term_stats, search.vectors.k_param_supported, security.migration_framework, security.role_mapping_cleanup, security.roles_metadata_flattened, semantic_text.search_inference_id, simulate.component.template.substitutions, simulate.index.template.substitutions, simulate.mapping.validation, simulate.mapping.validation.templates, standard_retriever_supported, stats.include_disk_thresholds, text_similarity_reranker_retriever_composition_supported, text_similarity_reranker_retriever_supported, tsdb.ts_routing_hash_doc_value_parse_byte_ref, unified_highlighter_matched_fields, usage.data_tiers.precalculate_stats]
2025.04.29 14:07:03 INFO es[][o.e.c.m.DataStreamGlobalRetentionSettings] Updated default factory retention to [null]
2025.04.29 14:07:03 INFO es[][o.e.c.m.DataStreamGlobalRetentionSettings] Updated max factory retention to [null]
2025.04.29 14:07:04 INFO es[][o.e.x.s.Security] Security is disabled
2025.04.29 14:07:04 INFO es[][o.e.x.a.APMPlugin] APM ingest plugin is enabled
2025.04.29 14:07:04 INFO es[][o.e.x.c.t.YamlTemplateRegistry] apm index template registry is enabled
2025.04.29 14:07:04 INFO es[][o.e.x.o.OTelPlugin] OTel ingest plugin is enabled
2025.04.29 14:07:04 INFO es[][o.e.x.c.t.YamlTemplateRegistry] OpenTelemetry index template registry is enabled
2025.04.29 14:07:04 INFO es[][o.e.t.a.APM] Sending apm metrics is disabled
2025.04.29 14:07:04 INFO es[][o.e.t.a.APM] Sending apm tracing is disabled
2025.04.29 14:07:05 INFO es[][o.e.t.n.NettyAllocator] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=4mb}]
2025.04.29 14:07:05 INFO es[][o.e.d.DiscoveryModule] using discovery type [single-node] and seed hosts providers [settings]
2025.04.29 14:07:06 INFO es[][o.e.n.Node] initialized
2025.04.29 14:07:06 INFO es[][o.e.n.Node] starting ...
2025.04.29 14:07:06 INFO es[][o.e.t.TransportService] publish_address {127.0.0.1:49297}, bound_addresses {127.0.0.1:49297}
2025.04.29 14:07:07 INFO es[][o.e.b.BootstrapChecks] explicitly enforcing bootstrap checks
2025.04.29 14:07:07 INFO es[][o.e.c.c.ClusterBootstrapService] this node is locked into cluster UUID [K2SZmIVPQLmSgY3k05gLHA] and will not attempt further cluster bootstrapping
2025.04.29 14:07:07 INFO es[][o.e.c.s.MasterService] elected-as-master ([1] nodes joined in term 7)[_FINISH_ELECTION_, {sonarqube}{8pZIn1CYTNmHPD8mE8Axcg}{BqlBEVydTpmBw3mrwxEAdA}{sonarqube}{127.0.0.1}{127.0.0.1:49297}{cdfhilmrstw}{8.16.3}{7000099-8518000} completing election], term: 7, version: 40, delta: master node changed {previous [], current [{sonarqube}{8pZIn1CYTNmHPD8mE8Axcg}{BqlBEVydTpmBw3mrwxEAdA}{sonarqube}{127.0.0.1}{127.0.0.1:49297}{cdfhilmrstw}{8.16.3}{7000099-8518000}]}
2025.04.29 14:07:08 INFO es[][o.e.c.s.ClusterApplierService] master node changed {previous [], current [{sonarqube}{8pZIn1CYTNmHPD8mE8Axcg}{BqlBEVydTpmBw3mrwxEAdA}{sonarqube}{127.0.0.1}{127.0.0.1:49297}{cdfhilmrstw}{8.16.3}{7000099-8518000}]}, term: 7, version: 40, reason: Publication{term=7, version=40}
2025.04.29 14:07:08 INFO es[][o.e.c.c.NodeJoinExecutor] node-join: [{sonarqube}{8pZIn1CYTNmHPD8mE8Axcg}{BqlBEVydTpmBw3mrwxEAdA}{sonarqube}{127.0.0.1}{127.0.0.1:49297}{cdfhilmrstw}{8.16.3}{7000099-8518000}] with reason [completing election]
2025.04.29 14:07:08 INFO es[][o.e.h.AbstractHttpServerTransport] publish_address {127.0.0.1:9001}, bound_addresses {127.0.0.1:9001}
2025.04.29 14:07:08 INFO es[][o.e.n.Node] started {sonarqube}{8pZIn1CYTNmHPD8mE8Axcg}{BqlBEVydTpmBw3mrwxEAdA}{sonarqube}{127.0.0.1}{127.0.0.1:49297}{cdfhilmrstw}{8.16.3}{7000099-8518000}{rack_id=sonarqube, xpack.installed=true}
2025.04.29 14:07:08 INFO es[][o.e.l.ClusterStateLicenseService] license [b5791f2e-b690-425b-8560-a23275c9f35d] mode [basic] - valid
2025.04.29 14:07:08 INFO es[][o.e.c.f.AbstractFileWatchingService] starting file watcher ...
2025.04.29 14:07:08 INFO es[][o.e.g.GatewayService] recovered [1] indices into cluster_state
2025.04.29 14:07:08 INFO es[][o.e.c.f.AbstractFileWatchingService] file settings service up and running [tid=36]
2025.04.29 14:07:08 INFO es[][o.e.r.s.FileSettingsService] setting file [C:\home\site\wwwroot\sonarqube-2025.2.0.105476\temp\conf\es\operator\settings.json] not found, initializing [file_settings] as empty
2025.04.29 14:07:08 INFO es[][o.e.h.n.s.HealthNodeTaskExecutor] Node [{sonarqube}{8pZIn1CYTNmHPD8mE8Axcg}] is selected as the current health node.
2025.04.29 14:07:08 INFO es[][o.e.c.r.a.AllocationService] current.health="RED" message="Cluster health status changed from [YELLOW] to [RED] (reason: [reconcile-desired-balance])." previous.health="YELLOW" reason="reconcile-desired-balance"
2025.04.29 14:07:09 WARN es[][o.e.c.r.a.DiskThresholdMonitor] high disk watermark [90%] exceeded on [8pZIn1CYTNmHPD8mE8Axcg][sonarqube][C:\home\site\wwwroot\sonarqube-2025.2.0.105476\data\es8] free: 3gb[5.1%], shards will be relocated away from this node; currently relocating away shards totalling [0] bytes; the node is expected to continue to exceed the high disk watermark when these relocations are complete
2025.04.29 14:08:38 WARN es[][o.e.c.r.a.DiskThresholdMonitor] high disk watermark [90%] exceeded on [8pZIn1CYTNmHPD8mE8Axcg][sonarqube][C:\home\site\wwwroot\sonarqube-2025.2.0.105476\data\es8] free: 3gb[5.1%], shards will be relocated away from this node; currently relocating away shards totalling [0] bytes; the node is expected to continue to exceed the high disk watermark when these relocations are complete
2025.04.29 14:10:08 WARN es[][o.e.c.r.a.DiskThresholdMonitor] high disk watermark [90%] exceeded on [8pZIn1CYTNmHPD8mE8Axcg][sonarqube][C:\home\site\wwwroot\sonarqube-2025.2.0.105476\data\es8] free: 3gb[5.1%], shards will be relocated away from this node; currently relocating away shards totalling [0] bytes; the node is expected to continue to exceed the high disk watermark when these relocations are complete
2025.04.29 14:11:08 WARN es[][o.e.c.r.a.DiskThresholdMonitor] high disk watermark [90%] exceeded on [8pZIn1CYTNmHPD8mE8Axcg][sonarqube][C:\home\site\wwwroot\sonarqube-2025.2.0.105476\data\es8] free: 3gb[5.1%], shards will be relocated away from this node; currently relocating away shards totalling [0] bytes; the node is expected to continue to exceed the high disk watermark when these relocations are complete
/>