Opsiconfd läuft nicht mehr nach Update

[anha]

Ich hatte heute ein Update gemacht auf die neuste Opsi Version. Seit dem Update läuft der Opsiconfd Service immer auf einen Fehler den ich nicht rausbekomme:
Hier der Status des Service

Code: Alles auswählen

 opsiconfd.service - opsi server daemon
     Loaded: loaded (/lib/systemd/system/opsiconfd.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Mon 2022-09-26 15:41:33 CEST; 2s ago
    Process: 2122 ExecStart=/usr/bin/opsiconfd start --log-level-stderr=0 (code=exited, status=1/FAILURE)
   Main PID: 2122 (code=exited, status=1/FAILURE)
        CPU: 1.035s

Sep 26 15:41:33 sgs061005 opsiconfd[2122]:   File "opsiconfd/utils.py", line 234, in redis_client
Sep 26 15:41:33 sgs061005 opsiconfd[2122]:   File "opsiconfd/utils.py", line 222, in get_redis_connection
Sep 26 15:41:33 sgs061005 opsiconfd[2122]:   File "redis/commands/core.py", line 1132, in ping
Sep 26 15:41:33 sgs061005 opsiconfd[2122]:   File "redis/client.py", line 1235, in execute_command
Sep 26 15:41:33 sgs061005 opsiconfd[2122]:   File "redis/connection.py", line 1387, in get_connection
Sep 26 15:41:33 sgs061005 opsiconfd[2122]:   File "redis/connection.py", line 617, in connect
Sep 26 15:41:33 sgs061005 opsiconfd[2122]: redis.exceptions.ConnectionError: Error 111 connecting to localhost:6379. Connection refused.
Sep 26 15:41:33 sgs061005 systemd[1]: opsiconfd.service: Main process exited, code=exited, status=1/FAILURE
Sep 26 15:41:33 sgs061005 systemd[1]: opsiconfd.service: Failed with result 'exit-code'.
Sep 26 15:41:33 sgs061005 systemd[1]: opsiconfd.service: Consumed 1.035s CPU time.

Was noch aufgefallen ist:
Grafana läuft auch mit dem User Opsiconfd, das einloggen per WEBoberfläche funktioniert ohne Probleme.
Redis Server läuft
Sobald ich irgendwelche config routinen starte, wie zb. opsiconfd kommt dieser Fehler:

Code: Alles auswählen

[4] [2022-09-26 15:54:43.446] [               ] Failed to connect to grafana api 'http://localhost:3000': HTTPConnectionPool(host='localhost', port=3000): Max retries exceeded with url: /api/users/lookup?loginOrEmail=opsiconfd (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe4d5fc9780>: Failed to establish a new connection: [Errno 111] Connection refused'))   (grafana.py:154)
[4] [2022-09-26 15:54:45.492] [               ] Failed to setup redis downsampling: Error 111 connecting to localhost:6379. Connection refused.   (setup.py:259)

Ich weiß aktuell nicht weiter. ich bitte um Hilfe

[fkalweit]

Hallo,

ich vermute in der redis.conf fehlt die Zeile loadmodule /usr/lib/redis/modules/redistimeseries.so.
Wahrscheinlich wurde die Konfiguration beim dem Update überschrieben.
Dann startet der redis kann aber die Datenbank nicht lesen, da dort Timeseries-Daten drin stehen.


Viele Grüße
Fabian

[anha]

Leider ist das nicht der vermutete Ursprung, in der redis.conf ist der Eintrag vorhanden.

[fkalweit]

Hallo,

kann man mit

Code: Alles auswählen

redis-cli

auf den redis server zugreifen?
Wenn ja, wie sieht die Ausgabe von

Code: Alles auswählen

info

aus?

[anha]

Ausgabe von redis-cli:

Code: Alles auswählen

# Server
redis_version:6.0.16
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:6d95e1af3a2c082a
redis_mode:standalone
os:Linux 5.10.0-18-amd64 x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:10.2.1
process_id:563
run_id:ab3f13fe19a3f2d51e93c63efc94906249a0c440
tcp_port:6379
uptime_in_seconds:73213
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:3325627
executable:/usr/bin/redis-server
config_file:/etc/redis/redis.conf
io_threads_active:0

# Clients
connected_clients:1
client_recent_max_input_buffer:8
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0

# Memory
used_memory:32750176
used_memory_human:31.23M
used_memory_rss:48635904
used_memory_rss_human:46.38M
used_memory_peak:32816112
used_memory_peak_human:31.30M
used_memory_peak_perc:99.80%
used_memory_overhead:912416
used_memory_startup:860976
used_memory_dataset:31837760
used_memory_dataset_perc:99.84%
allocator_allocated:33069128
allocator_active:33419264
allocator_resident:37289984
total_system_memory:8343572480
total_system_memory_human:7.77G
used_memory_lua:41984
used_memory_lua_human:41.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.01
allocator_frag_bytes:350136
allocator_rss_ratio:1.12
allocator_rss_bytes:3870720
rss_overhead_ratio:1.30
rss_overhead_bytes:11345920
mem_fragmentation_ratio:1.49
mem_fragmentation_bytes:15928248
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:20496
mem_aof_buffer:0
mem_allocator:jemalloc-5.2.1
active_defrag_running:0
lazyfree_pending_objects:0

# Persistence
loading:0
rdb_changes_since_last_save:0
rdb_bgsave_in_progress:0
rdb_last_save_time:1664196798
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:0
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0

# Stats
total_connections_received:1
total_commands_processed:2
instantaneous_ops_per_sec:0
total_net_input_bytes:31
total_net_output_bytes:20085
instantaneous_input_kbps:0.00
instantaneous_output_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:539
evicted_keys:0
keyspace_hits:568
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:0
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_reads_processed:2
total_writes_processed:1
io_threaded_reads_processed:0
io_threaded_writes_processed:0

# Replication
role:master
connected_slaves:0
master_replid:cda2c2282c83cca1e77daa7ebae4ad4609cab4a7
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

# CPU
used_cpu_sys:26.225583
used_cpu_user:27.729178
used_cpu_sys_children:0.000000
used_cpu_user_children:0.000000

# Modules
module:name=timeseries,ver=10616,api=1,filters=0,usedby=[],using=[],options=[]

# Cluster
cluster_enabled:0

# Keyspace
db0:keys=568,expires=0,avg_ttl=0

[anha]

Ich bin jetzt weitergekommen, jetzt läuft der Confd wieder, allerdings kann ich mich noch nicht einloggen

Code: Alles auswählen

Sep 28 09:16:33  opsiconfd[788]: PAM unable to dlopen(pam_unix.so): /lib/security/pam_unix.so: cannot open shared object file: No such file or directory
Sep 28 09:16:33  opsiconfd[788]: PAM adding faulty module: pam_unix.so
Sep 28 09:16:33  opsiconfd[788]: pam_ldap: ldap_simple_bind Can't contact LDAP server
Sep 28 09:16:33  opsiconfd[788]: pam_ldap: reconnecting to LDAP server...
Sep 28 09:16:33  opsiconfd[788]: pam_ldap: ldap_simple_bind Can't contact LDAP server
Sep 28 09:30:18  opsiconfd[788]: PAM unable to dlopen(pam_unix.so): /lib/security/pam_unix.so: cannot open shared object file: No such file or directory
Sep 28 09:30:18  opsiconfd[788]: PAM adding faulty module: pam_unix.so
Sep 28 09:30:18  opsiconfd[788]: pam_ldap: ldap_simple_bind Can't contact LDAP server
Sep 28 09:30:18  opsiconfd[788]: pam_ldap: reconnecting to LDAP server...
Sep 28 09:30:18  opsiconfd[788]: pam_ldap: ldap_simple_bind Can't contact LDAP server

[anha]

So wie es aussieht wurde alles zurückgesetzt, ich versuch jetzt das Backup einzuspielen

[anha]

Nachdem jetzt das Backup wieder eingespielt wurde, kommt er wieder mit dem Fehler wie im ersten Beitrag.

Noch zur weiteren Info:
Debian 11 Bullseye

Es fehlen keinerlei Pakete, das System ist auf dem neuesten Stand.

[fkalweit]

Hallo,

welche aus gibt denn der opsiconfd aus wenn man ihn auf der console mit

Code: Alles auswählen

opsiconfd -l=6

startet.

Und was steht in der /var/log/redis/redis.log?

Viele Grüße
Fabian

[anha]

Opsiconfd -l=6:

Code: Alles auswählen

Traceback (most recent call last):
  File "redis/connection.py", line 611, in connect
  File "redis/retry.py", line 46, in call_with_retry
  File "redis/connection.py", line 612, in <lambda>
  File "redis/connection.py", line 677, in _connect
  File "redis/connection.py", line 665, in _connect
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "run-opsiconfd", line 12, in <module>
  File "opsiconfd/main.py", line 100, in main
  File "contextlib.py", line 135, in __enter__
  File "opsiconfd/utils.py", line 234, in redis_client
  File "opsiconfd/utils.py", line 222, in get_redis_connection
  File "redis/commands/core.py", line 1132, in ping
  File "redis/client.py", line 1235, in execute_command
  File "redis/connection.py", line 1387, in get_connection
  File "redis/connection.py", line 617, in connect
redis.exceptions.ConnectionError: Error 111 connecting to localhost:6379. Connec                                                                                                     tion refused.

wie erwähnt läuft der redis Server sauber:

Code: Alles auswählen

566:M 28 Sep 2022 11:36:43.562 # Server initialized
566:M 28 Sep 2022 11:36:43.562 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /et>
566:M 28 Sep 2022 11:36:43.562 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix t>
566:M 28 Sep 2022 11:36:43.569 * <timeseries> RedisTimeSeries version 10616, git_sha=16c81bbb1e93f6a9bda2395296e2ffb0cc0f898a
566:M 28 Sep 2022 11:36:43.569 * <timeseries> Redis version found by RedisTimeSeries : 6.0.16 - oss
566:M 28 Sep 2022 11:36:43.569 * <timeseries> loaded default CHUNK_SIZE_BYTES policy: 4096
566:M 28 Sep 2022 11:36:43.569 * <timeseries> loaded server DUPLICATE_POLICY: block
566:M 28 Sep 2022 11:36:43.569 * <timeseries> Setting default series ENCODING to: compressed
566:M 28 Sep 2022 11:36:43.569 * <timeseries> Detected redis oss
566:M 28 Sep 2022 11:36:43.570 * Module 'timeseries' loaded from /usr/lib/redis/modules/redistimeseries.so
566:M 28 Sep 2022 11:36:43.571 * Loading RDB produced by version 6.0.16
566:M 28 Sep 2022 11:36:43.571 * RDB age 10 seconds
566:M 28 Sep 2022 11:36:43.571 * RDB memory usage when created 11.75 Mb
566:M 28 Sep 2022 11:36:43.608 * DB loaded from disk: 0.038 seconds
566:M 28 Sep 2022 11:36:43.608 * Ready to accept connections