Troubleshooting

Faheem Code Enterprise Replicated VM installations run in a Replicated Embedded Cluster which is a Kubernetes cluster based on k0s. Once you have access to the VM, you can use standard Kubernetes commands to inspect FCE. For Helm deployments, use your existing Kubernetes access to run the same commands.

Most FCE workloads run in the faheemcode namespace. The Replicated Admin Console runs in kotsadm, and ingress runs in traefik.

Start with a support bundle

A support bundle is the fastest way to give Faheem Code Support a snapshot of the installation. You do not need to investigate the problem yourself before opening a support ticket.

Use the Admin Console

For a Replicated VM installation:

1. Open https://admin.<your-base-domain>:30000.
2. Select Troubleshoot.
3. Select Analyze and wait for it to finish.
4. Select Download bundle.

If Send bundle to vendor is available, you can upload the bundle for us to inspect directly. Sending a support bundle does not automatically create a support ticket, so be sure to still open a support ticket and mention the support bundle upload.

Use the command line

On a Replicated VM, use the command line when the Admin Console is unavailable. For a Helm installation, run the Kubernetes command from a workstation with kubectl access.

Connect to the VM and run:

sudo /var/lib/embedded-cluster/bin/faheemcode support-bundle

If the installation did not complete, run the original installer from the directory where you extracted it:

sudo ./faheemcode support-bundle

For FCE installed with Helm in an existing Kubernetes cluster, run this command from a workstation with kubectl access:

kubectl support-bundle --load-cluster-specs --namespace faheemcode

See the Kubernetes installation guide if the support-bundle CLI is not installed.

The bundle includes cluster health, Kubernetes resource state, application logs, and FCE service checks.

Open a support ticket

Open the Faheem Code Support Portal provided during Enterprise onboarding. Please attach the generated archive. If you used Send bundle to vendor, mention the upload in the ticket. Include:

• When the problem occurred, including the time zone.
• The affected user or conversation ID, when applicable.
• The expected and actual behavior.
• Any recent upgrade or configuration change.
• Steps that reproduce the problem.

If you cannot access the Support Portal, please contact your Faheem Code representative for more assistance.

Inspect the deployment

This workflow is for practitioners who are already familiar with kubectl.

Warning

Keep your investigation read-only. Do not change Kubernetes resources unless directed by Faheem Code Support. Ad hoc kubectl changes can be overwritten during a deployment or upgrade and may leave the installation in an inconsistent state.

Get a Kubernetes session

Connect to a controller VM. On a single-node installation, this is the FCE VM. Then run:

sudo /var/lib/embedded-cluster/bin/faheemcode shell

This opens a shell with kubectl configured for the embedded cluster. Run exit when finished.

Use your existing Kubernetes access and confirm the current context:

kubectl config current-context
kubectl get pods -n faheemcode

Check overall status

Record the time, then inspect the cluster and recent events:

date -u
kubectl get nodes -o wide
kubectl get pods -n faheemcode -o wide
kubectl get deployments,statefulsets -n faheemcode
kubectl get events -n faheemcode --sort-by=.metadata.creationTimestamp

Start with the STATUS, READY, and RESTARTS columns:

• Pending usually points to scheduling, storage, or capacity problems.
• Init: means an init container has not completed. Check that container's logs.
• CrashLoopBackOff means a container repeatedly exits. Check previous logs.
• A pod that is not ready or keeps restarting usually has a failed dependency, health check, or resource limit.

If the Kubernetes Metrics API is available, check current resource usage:

kubectl top pods -n faheemcode

Inspect a pod and its logs

kubectl describe pod -n faheemcode <pod-name>

kubectl logs -n faheemcode <pod-name> \
  --all-containers=true --since=30m --timestamps

kubectl logs -n faheemcode <pod-name> \
  --all-containers=true --previous --timestamps

kubectl logs -n faheemcode <pod-name> -c <container-name> \
  --since=10m --timestamps --follow

Use --previous after a container restarts. Use -c to select a specific container, including an init container such as migrate-db.

On a Replicated VM, these logs are also written to files on the VM. See Log Collection to send them to your own observability platform.

Choose the right component

Pod names may include a release prefix and generated suffix. Match the recognizable component name to the table below.

Component	Investigate when
faheemcode	Web application, API, conversations, and general application errors.
faheem-code-integrations	Integration events and background integration work.
runtime-api	Sandbox creation, startup, pause, and cleanup.
runtime-...	A particular conversation's sandbox.
litellm	Model-provider requests and authentication.
keycloak	Login, SSO, and authentication.
kotsadm namespace	Replicated Admin Console problems.

Temporarily enable debug logging

On a Replicated VM, Log Level defaults to INFO. Use DEBUG only during a short investigation:

1. In the Admin Console, select Config.
2. Under Troubleshooting, set Log Level to DEBUG.
3. Save and deploy, then reproduce the problem.
4. Collect the logs or a support bundle.
5. Return Log Level to INFO, then save and deploy again.

Related guides

Quick start

Install an Faheem Code Enterprise VM deployment.

Admin Console configuration

Configure a Replicated VM installation.

Kubernetes installation

Install FCE into an existing Kubernetes cluster.

Resource limits

Diagnose and tune CPU, memory, replicas, and storage.

Log collection

Send VM installation logs to your own observability platform.