Incident History
Cisco CUBE
66959761
highCompletedCreatedDuration5m 39.644813sServiceNowINC2603161420WebexSentProducts
Cisco Unified Border Element (CUBE)Cisco IOS-XECisco Unified Communications Manager (CUCM)Cisco Unified Communications Manager
8c84cbb7
criticalCompletedCreatedDuration4m 58.98117500000001sServiceNowINC2603161339WebexSentProducts
Cisco Unified Communications Manager (CUCM)Cisco Real-Time Monitoring Tool (RTMT)Cisco Unified Communications Manager
5bd348af
criticalIn ProgressCreatedDuration—ServiceNow—WebexNot sentTest GKE direct
771a1d62
criticalNewCreatedDuration—ServiceNow—WebexNot sentTest Enum Failure
5b9e1d56
criticalNewCreatedDuration—ServiceNow—WebexNot sentTest Enum Failure
7e9fd986
criticalNewCreatedDuration—ServiceNow—WebexNot sentTest
9ffe0ee4
criticalNewCreatedDuration—ServiceNow—WebexNot sentTest
fbff5791
criticalNewCreatedDuration—ServiceNow—WebexNot sentTest
fb33d59a
criticalNewCreatedDuration—ServiceNow—WebexNot sentService: Redis Cache (prod-redis-01) Issue: Cache evictions causing database overload and slow queries Environment: Production (us-west-2) Error: "used_memory exceeds maxmemory, eviction policy: allkeys-lru" Impact: Database query latency increased 10x, c
463130bc
mediumCompletedCreatedDuration—ServiceNowINC0010475WebexSentProducts
RedisValkeyjemallocService: Redis Cache (prod-redis-01) Issue: Cache evictions causing database overload and slow queries Environment: Production (us-west-2) Error: "used_memory exceeds maxmemory, eviction policy: allkeys-lru" Impact: Database query latency increased 10x, c
c5ec4a81
mediumCompletedCreatedDuration—ServiceNowINC0010474WebexSentProducts
RedisAWS ElastiCacheService: Kong API Gateway (prod-apigw-01) Issue: API Gateway returning 503 Service Unavailable for 15% of requests Environment: Production (multi-region) Error: "upstream connect error or disconnect/reset before headers. reset reason: connection failure"
4277ad30
mediumIn ProgressCreatedDuration—ServiceNow—WebexNot sentService: Redis Cache (prod-redis-01) Issue: Cache evictions causing database overload and slow queries Environment: Production (us-west-2) Error: "used_memory exceeds maxmemory, eviction policy: allkeys-lru" Impact: Database query latency increased 10x, c
1e825f33
mediumIn ProgressCreatedDuration—ServiceNow—WebexNot sentService: Elasticsearch Cluster (prod-es-cluster) Issue: Cluster status yellow, 45 unassigned shards detected Environment: Production (multi-AZ) Error: "unassigned_shards: 45, reason: INDEX_CREATED, allocation_decider: disk_threshold" Impact: Search querie
b498277c
mediumCompletedCreatedDuration—ServiceNowINC0010473WebexSentProducts
ElasticsearchKibanaTest incident
00000000
mediumIn ProgressCreatedDuration—ServiceNow—WebexNot sentService: RabbitMQ Message Broker (prod-rmq-01) Issue: Order processing queue backlog of 50,000+ messages Environment: Production (us-east-1) Error: Consumer processing rate (10/sec) << producer rate (200/sec) Impact: Order processing delayed by 2+ hours,
273f112a
mediumIn ProgressCreatedDuration—ServiceNow—WebexNot sentService: Kong API Gateway (prod-apigw-01) Issue: API Gateway returning 503 Service Unavailable for 15% of requests Environment: Production (multi-region) Error: "upstream connect error or disconnect/reset before headers. reset reason: connection failure"
20af376c
mediumIn ProgressCreatedDuration—ServiceNow—WebexNot sentService: RabbitMQ Message Broker (prod-rmq-01) Issue: Order processing queue backlog of 50,000+ messages Environment: Production (us-east-1) Error: Consumer processing rate (10/sec) << producer rate (200/sec) Impact: Order processing delayed by 2+ hours,
1c3c2a66
mediumIn ProgressCreatedDuration—ServiceNow—WebexNot sentService: Redis Cache (prod-redis-01) Issue: Cache evictions causing database overload and slow queries Environment: Production (us-west-2) Error: "used_memory exceeds maxmemory, eviction policy: allkeys-lru" Impact: Database query latency increased 10x, c
b8b9ba04
mediumIn ProgressCreatedDuration—ServiceNow—WebexNot sentService: Elasticsearch Cluster (prod-es-cluster) Issue: Cluster status yellow, 45 unassigned shards detected Environment: Production (multi-AZ) Error: "unassigned_shards: 45, reason: INDEX_CREATED, allocation_decider: disk_threshold" Impact: Search querie
b394c356
mediumIn ProgressCreatedDuration—ServiceNow—WebexNot sent