I’ve setup Lemmy on a VPS with LetsEncrypt. When navigating to my domain, Lemmy just says “There was an error on the server. Try refreshing your browser. If that doesn’t work, come back at a later time. If the problem persists, you can seek help in the Lemmy support community or Lemmy Matrix room.”
I’ve checked both the Lemmy API and WUI instances, but no errors are reported:
lemmy@ubuntu-s-1vcpu-2gb-sfo3-01:~$ docker compose logs
postgres |
postgres | PostgreSQL Database directory appears to contain a database; Skipping initialization
postgres |
postgres | 2025-02-17 18:21:24.644 UTC [1] LOG: starting PostgreSQL 17.3 (Debian 17.3-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
postgres | 2025-02-17 18:21:24.646 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
postgres | 2025-02-17 18:21:24.647 UTC [1] LOG: listening on IPv6 address "::", port 5432
postgres | 2025-02-17 18:21:24.652 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres | 2025-02-17 18:21:24.685 UTC [27] LOG: database system was shut down at 2025-02-17 18:21:15 UTC
postgres | 2025-02-17 18:21:24.729 UTC [1] LOG: database system is ready to accept connections
lemmy-ui-1 | Lemmy-ui v0.19.9 started listening on http://0.0.0.0:1234/
lemmy-ui-1 | 31 translation imports verified.
lemmy-ui-1 | date-fns "zh" failed: unexpected format
lemmy-ui-1 | 1 out of 94 date-fns imports failed.
lemmy-ui-1 | 4 highlight.js imports verified. (Only testing 4 samples.)
nginx-certs-1 | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
nginx-certs-1 | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
nginx-certs-1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
nginx-certs-1 | 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
nginx-certs-1 | 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
nginx-certs-1 | /docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
nginx-certs-1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
nginx-certs-1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
nginx-certs-1 | /docker-entrypoint.sh: Configuration complete; ready for start up
nginx-certs-1 | 106.75.133.150 - - [17/Feb/2025:18:21:43 +0000] "GET / HTTP/1.1" 301 169 "http://my.server.ip.addr/" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36"
nginx-certs-1 | 106.75.133.150 - - [17/Feb/2025:18:21:45 +0000] "GET /favicon.ico HTTP/1.1" 301 169 "http://my.server.ip.addr/" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36"
proxy-1 | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
proxy-1 | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
proxy-1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
proxy-1 | 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
proxy-1 | 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
proxy-1 | /docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
proxy-1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
proxy-1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
proxy-1 | /docker-entrypoint.sh: Configuration complete; ready for start up
proxy-1 | 75.154.243.157 - - [17/Feb/2025:18:21:47 +0000] "GET / HTTP/2.0" 500 5284 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36"
proxy-1 | 75.154.243.157 - - [17/Feb/2025:18:21:47 +0000] "GET /manifest.webmanifest HTTP/2.0" 500 21 "https://mydomain.xyz/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36"
proxy-1 | 75.154.243.157 - - [17/Feb/2025:18:21:48 +0000] "GET / HTTP/2.0" 500 5284 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36"
proxy-1 | 75.154.243.157 - - [17/Feb/2025:18:21:50 +0000] "GET /service-worker.js HTTP/2.0" 304 0 "https://mydomain.xyz/service-worker.js" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36"
certbot-1 | Saving debug log to /var/log/letsencrypt/letsencrypt.log
certbot-1 | Certbot doesn't know how to automatically configure the web server on this system. However, it can still get a certificate for you. Please run "certbot certonly" to do so. You'll need to manually configure your web server to use the resulting certificate.
pictrs-1 | 2025-02-17T18:21:23.900781Z INFO pict_rs: Starting pict-rs on 0.0.0.0:8080
pictrs-1 | 2025-02-17T18:21:23.901363Z INFO actix_server::builder: starting 1 workers
pictrs-1 | 2025-02-17T18:21:23.901410Z INFO actix_server::server: Tokio runtime found; starting in existing Tokio runtime
lemmy-1 | Lemmy v0.19.9
lemmy-1 | 2025-02-17T18:21:25.386757Z INFO lemmy_db_schema::schema_setup: Running Database migrations (This may take a long time)...
lemmy-1 | 2025-02-17T18:21:25.408878Z INFO lemmy_db_schema::schema_setup: Database migrations complete.
lemmy-1 | 2025-02-17T18:21:25.493688Z INFO lemmy_server::code_migrations: Running user_updates_2020_04_02
lemmy-1 | 2025-02-17T18:21:25.515048Z INFO lemmy_server::code_migrations: 0 person rows updated.
lemmy-1 | 2025-02-17T18:21:25.516901Z INFO lemmy_server::code_migrations: Running community_updates_2020_04_02
lemmy-1 | 2025-02-17T18:21:25.540687Z INFO lemmy_server::code_migrations: 0 community rows updated.
lemmy-1 | 2025-02-17T18:21:25.541173Z INFO lemmy_server::code_migrations: Running post_updates_2020_04_03
lemmy-1 | 2025-02-17T18:21:25.551391Z INFO lemmy_server::code_migrations: 0 post rows updated.
lemmy-1 | 2025-02-17T18:21:25.553517Z INFO lemmy_server::code_migrations: Running comment_updates_2020_04_03
lemmy-1 | 2025-02-17T18:21:25.571059Z INFO lemmy_server::code_migrations: 0 comment rows updated.
lemmy-1 | 2025-02-17T18:21:25.572187Z INFO lemmy_server::code_migrations: Running private_message_updates_2020_05_05
lemmy-1 | 2025-02-17T18:21:25.576403Z INFO lemmy_server::code_migrations: 0 private message rows updated.
lemmy-1 | 2025-02-17T18:21:25.578895Z INFO lemmy_server::code_migrations: Running post_thumbnail_url_updates_2020_07_27
lemmy-1 | 2025-02-17T18:21:25.681807Z INFO lemmy_server::code_migrations: 0 Post thumbnail_url rows updated.
lemmy-1 | 2025-02-17T18:21:25.684992Z INFO lemmy_server::code_migrations: Running apub_columns_2021_02_02
lemmy-1 | 2025-02-17T18:21:25.693262Z INFO lemmy_server::code_migrations: Running instance_actor_2021_09_29
lemmy-1 | 2025-02-17T18:21:25.734984Z INFO lemmy_server::code_migrations: Running regenerate_public_keys_2022_07_05
lemmy-1 | 2025-02-17T18:21:25.743673Z INFO lemmy_server::code_migrations: Running initialize_local_site_2022_10_10
lemmy-1 | Federation enabled, host is mydomain.xyz
lemmy-1 | Starting HTTP server at 0.0.0.0:8536
lemmy-1 | 2025-02-17T18:21:26.062209Z INFO lemmy_server::scheduled_tasks: Updating active site and community aggregates ...
lemmy-1 | 2025-02-17T18:21:26.233668Z INFO lemmy_server::scheduled_tasks: Done.
lemmy-1 | 2025-02-17T18:21:26.233702Z INFO lemmy_server::scheduled_tasks: Updating hot ranks for all history...
lemmy-1 | 2025-02-17T18:21:26.263454Z INFO lemmy_server::scheduled_tasks: Finished process_hot_ranks_in_batches execution for post_aggregates (processed 0 rows)
lemmy-1 | 2025-02-17T18:21:26.281667Z INFO lemmy_server::scheduled_tasks: Finished process_hot_ranks_in_batches execution for comment (processed 0 rows)
lemmy-1 | 2025-02-17T18:21:26.289482Z INFO lemmy_server::scheduled_tasks: Finished process_hot_ranks_in_batches execution for community (processed 0 rows)
lemmy-1 | 2025-02-17T18:21:26.289521Z INFO lemmy_server::scheduled_tasks: Finished hot ranks update!
lemmy-1 | 2025-02-17T18:21:26.289527Z INFO lemmy_server::scheduled_tasks: Updating banned column if it expires ...
lemmy-1 | 2025-02-17T18:21:26.310614Z INFO lemmy_server::scheduled_tasks: Clearing old activities...
lemmy-1 | 2025-02-17T18:21:26.329883Z INFO lemmy_server::scheduled_tasks: Done.
lemmy-1 | 2025-02-17T18:21:26.329909Z INFO lemmy_server::scheduled_tasks: Overwriting deleted posts...
lemmy-1 | 2025-02-17T18:21:26.345925Z INFO lemmy_server::scheduled_tasks: Done.
lemmy-1 | 2025-02-17T18:21:26.345960Z INFO lemmy_server::scheduled_tasks: Overwriting deleted comments...
lemmy-1 | 2025-02-17T18:21:26.388678Z INFO lemmy_server::scheduled_tasks: Done.
lemmy-1 | 2025-02-17T18:21:26.408661Z INFO lemmy_server::scheduled_tasks: Done.
Due to no errors showing up in logs, I’m unsure where to begin troubleshooting.
Here’s my Nginx config:
events {
}
http {
limit_req_zone $binary_remote_addr zone=mydomain.xyz_ratelimit:10m rate=1r/s;
upstream lemmy {
server "lemmy:8536";
}
upstream lemmy-ui {
server "lemmy-ui:1234";
}
server {
listen 80;
listen [::]:80;
server_name mydomain.xyz;
location /.well-known/acme-challenge/ {
root /var/www/certbot;
}
location / {
return 301 https://$host$request_uri;
}
}
server {
http2 on;
listen 443 ssl;
listen [::]:443 ssl;
server_name mydomain.xyz;
ssl_certificate /etc/letsencrypt/live/mydomain.xyz/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/mydomain.xyz/privkey.pem;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_prefer_server_ciphers on;
ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256';
ssl_session_timeout 10m;
ssl_session_cache shared:SSL:10m;
ssl_session_tickets on;
ssl_stapling on;
ssl_stapling_verify on;
server_tokens off;
gzip on;
gzip_types text/css application/javascript image/svg+xml;
gzip_vary on;
add_header Strict-Transport-Security "max-age=63072000";
add_header Referrer-Policy "same-origin";
add_header X-Content-Type-Options "nosniff";
add_header X-Frame-Options "DENY";
add_header X-XSS-Protection "1; mode=block";
client_max_body_size 20M;
location / {
set $proxpass "http://lemmy-ui/";
if ($http_accept = "application/activity+json") {
set $proxpass "http://lemmy/";
}
if ($http_accept = "application/ld+json; profile=\"https://www.w3.org/ns/activitystreams/"") {
set $proxpass "http://lemmy/";
}
if ($request_method = POST) {
set $proxpass "http://lemmy/";
}
proxy_pass $proxpass;
rewrite ^(.+)/+$ $1 permanent;
# Send actual client IP upstream
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
location ~ ^/(api|feeds|nodeinfo|.well-known) {
proxy_pass "http://lemmy/";
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
limit_req zone=mydomain.xyz_ratelimit burst=30 nodelay;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
location ~ ^/(pictrs) {
# allow browser cache, images never update, we can apply long term cache
expires 120d;
add_header Pragma "public";
add_header Cache-Control "public";
proxy_pass "http://lemmy/";
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
limit_req zone=mydomain.xyz_ratelimit burst=30 nodelay;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
location ~ /pictshare/(.*)$ {
return 301 /pictrs/image/$1;
}
}
map $remote_addr $remote_addr_anon {
~(?P<ip>\d+\.\d+\.\d+)\. $ip.0;
~(?P<ip>[^:]+:[^:]+): $ip::;
127.0.0.1 $remote_addr;
::1 $remote_addr;
default 0.0.0.0;
}
access_log /var/log/nginx/access.log combined;
}
Docker-compose:
networks:
lemmyexternalproxy:
lemmyinternal:
driver: bridge
internal: true
services:
proxy:
image: nginx:1.27.4-alpine
networks:
- lemmyinternal
- lemmyexternalproxy
ports:
- 443:443
volumes:
- ./volumes/nginx/nginx.conf:/etc/nginx/nginx.conf:ro
- ./volumes/certbot:/var/www/certbot:ro
- ./volumes/letsencrypt:/etc/letsencrypt:ro
restart: unless-stopped
depends_on:
- pictrs
- lemmy-ui
nginx-certs:
image: nginx:1.27.4-alpine
ports:
- 80:80
volumes:
- ./volumes/nginx_certs/nginx.conf:/etc/nginx/nginx.conf:ro
- ./volumes/certbot:/var/www/certbot:ro
- ./volumes/letsencrypt:/etc/letsencrypt:ro
restart: always
lemmy:
image: dessalines/lemmy:0.19.9
hostname: lemmy
networks:
- lemmyinternal
- lemmyexternalproxy
restart: unless-stopped
environment:
- RUST_LOG=verbose,lemmy_server=info,lemmy_api=info,lemmy_api_common=info,lemmy_api_crud=info,lemmy_apub=info,lemmy_db_schema=info,lemmy_db_views=info,lemmy_db_views_actor=info,lemmy_db_views_moderator=info,lemmy_routes=info,lemmy_utils=info,lemmy_websocket=info
volumes:
- ./volumes/lemmy-api/lemmy.hjson:/config/config.hjson
depends_on:
- pictrs
- postgres
lemmy-ui:
image: dessalines/lemmy-ui:0.19.9
networks:
- lemmyinternal
environment:
- LEMMY_INTERNAL_HOST=lemmy:8536
- LEMMY_EXTERNAL_HOST=mydomain.xyz
- LEMMY_HTTPS=true
depends_on:
- lemmy
restart: unless-stopped
postgres:
image: postgres:17.3
networks:
- lemmyinternal
container_name: postgres
restart: unless-stopped
environment:
POSTGRES_USER: lemmy
POSTGRES_PASSWORD: XXXXXXXXXXX
POSTGRES_DB: lemmy
volumes:
- ./volumes/postgres:/var/lib/postgresql/data
pictrs:
image: asonix/pictrs:0.5.16
hostname: pictrs
networks:
- lemmyinternal
environment:
- PICTRS__API_KEY=j9Dj3FxLxWL8mxMF
user: 0:0
volumes:
- ./volumes/pictrs:/mnt
restart: unless-stopped
certbot:
image: certbot/certbot:v3.2.0
volumes:
- ./volumes/certbot:/var/www/certbot/:rw
- ./volumes/letsencrypt:/etc/letsencrypt/:rw
And finally, Lemmy’s config:
{
setup: {
admin_username: "lemmy"
admin_password: "XXXXXXXXXXXXXXXX"
site_name: "MYDOMAIN"
}
hostname: "mydomain.xyz"
bind: "0.0.0.0"
port: 8536
tls_enabled: true
pictrs_url: "http://pictrs:8080/"
database: {
database: "lemmy"
user: "lemmy"
password: "XXXXXXXXXXXXXXXXXXXX"
host: "postgres"
port: 5432
pool_size: 5
}
}
You must log in or register to comment.
Ha, you fool. Now that I know your password is
XXXXXXXXXXXXXXXXXXXX, I’ll use it to take over your site.Seriously though, good luck.
I already took down
mydomain.xyzso good luck even using that password.
Figured it out! It was Lemmy UI. I was using an older config version. It appears that newer version of Lemmy UI use different environment variables:
I was using:
LEMMY_EXTERNAL_HOST LEMMY_INTERNAL_HOSTInstead of:
LEMMY_UI_LEMMY_INTERNAL_HOST LEMMY_UI_LEMMY_EXTERNAL_HOSTWhat are the errors in the browser console logs? Curious what the HTTP response was
Any page I go to returns a 500. I suspect it’s some nginx config not sending something to the backend or something. But still troubleshooting
When you click on any of those 500s, like the login, what does the response message say? Also look into setting the
RUST_LOGenvironment variable on the lemmy server container, like to WARN or INFO I (believe) are the settings, you will have more info there. I keep mine at the WARN level, and I’m guessing those 500s will log something then.
Just some HTML.
It seems Lemmy-UI is returning 500. This was run from a pod within the same Docker network:
# curl -I http://lemmy-ui:1234/ HTTP/1.1 500 Internal Server Error X-Powered-By: Express Content-Security-Policy: default-src 'self'; manifest-src *; connect-src *; img-src * data: blob:; script-src 'self' 'nonce-42930e15580b5234226abe8789f2c66f'; style-src 'self' 'unsafe-inline'; form-action 'self'; base-uri 'self'; frame-src *; media-src * data: Cache-Control: public, max-age=60 Content-Type: text/html; charset=utf-8 Content-Length: 5194 ETag: W/"144a-xS0fkaSDORsFqG8LgYer3xVM4T8" Date: Mon, 17 Feb 2025 19:48:22 GMT Connection: keep-alive Keep-Alive: timeout=5But I see no errors from the pod itself:
$ docker compose logs lemmy-ui lemmy-ui-1 | Lemmy-ui v0.19.9 started listening on http://0.0.0.0:1234/ lemmy-ui-1 | 31 translation imports verified. lemmy-ui-1 | date-fns "zh" failed: unexpected format lemmy-ui-1 | 1 out of 94 date-fns imports failed. lemmy-ui-1 | 4 highlight.js imports verified. (Only testing 4 samples.)NGINX logs:
75.154.243.157 - - [17/Feb/2025:19:45:43 +0000] "GET / HTTP/2.0" 500 5284 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36" 75.154.243.157 - - [17/Feb/2025:19:45:43 +0000] "GET /css/themes/darkly.css HTTP/2.0" 200 45877 "https://mydomain.xyz/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36" 75.154.243.157 - - [17/Feb/2025:19:45:43 +0000] "GET /css/themes/darkly.css.map HTTP/2.0" 400 24 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36" 75.154.243.157 - - [17/Feb/2025:19:45:43 +0000] "GET /manifest.webmanifest HTTP/2.0" 500 21 "https://mydomain.xyz/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36" 75.154.243.157 - - [17/Feb/2025:19:45:45 +0000] "GET /service-worker.js HTTP/2.0" 304 0 "https://mydomain.xyz/service-worker.js" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36" 75.154.243.157 - - [17/Feb/2025:19:45:48 +0000] "GET /login HTTP/2.0" 500 5325 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36" 75.154.243.157 - - [17/Feb/2025:19:45:48 +0000] "GET /css/themes/darkly.css HTTP/2.0" 200 45877 "https://mydomain.xyz/login" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36" 75.154.243.157 - - [17/Feb/2025:19:45:48 +0000] "GET /css/themes/darkly.css.map HTTP/2.0" 400 24 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36" 75.154.243.157 - - [17/Feb/2025:19:45:48 +0000] "GET /manifest.webmanifest HTTP/2.0" 500 21 "https://mydomain.xyz/login" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36" 75.154.243.157 - - [17/Feb/2025:19:45:50 +0000] "GET /service-worker.js HTTP/2.0" 304 0 "https://mydomain.xyz/service-worker.js" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36"Hm, that is weird, I’d say next step is narrow down if it’s the proxy, or if it’s lemmy-ui, since we can’t directly right now. Forward 1234 out of the stack temporarily, see what happens when you
curlthe lemmy-ui directly without the proxy, or try to load it directly? Just to see if you can replicate the 500.Figured it out! It was Lemmy UI. I was using an older config version. It appears that newer version of Lemmy UI use different environment variables:
I was using:
LEMMY_EXTERNAL_HOST LEMMY_INTERNAL_HOSTInstead of:
LEMMY_UI_LEMMY_INTERNAL_HOST LEMMY_UI_LEMMY_EXTERNAL_HOSTVery good find! Glad you figured it out!
Is the API meant to return 404 on the main route? Here’s the text when hitting the API, and UI directly, and also the status codes returned. This is not going through NGINX:
bastion:~# curl http://lemmy-ui:1234/ <!DOCTYPE html> <html > <head> <script nonce="undefined"> window.isoData = {"path":"\u002F","site_res":undefined,"routeData":{},"errorPageData":{},"showAdultConsentModal":false}; if (!document.documentElement.hasAttribute("data-bs-theme")) { const light = window.matchMedia("(prefers-color-scheme: light)").matches; document.documentElement.setAttribute("data-bs-theme", light ? "light" : "dark"); } </script> <!-- A remote debugging utility for mobile --> <script src="//cdn.jsdelivr.net/npm/eruda"></script><script>eruda.init();</script> <!-- Custom injected script --> <title data-inferno-helmet="true"> </title> <style> #app[data-adult-consent] { filter: blur(10px); -webkit-filter: blur(10px); -moz-filter: blur(10px); -o-filter: blur(10px); -ms-filter: blur(10px); pointer-events: none; } </style> <!-- Required meta tags --> <meta name="Description" content="Lemmy"> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no"> <link id="favicon" rel="shortcut icon" type="image/x-icon" href=/static/c15a0eb1/assets/icons/favicon.svg /> <!-- Web app manifest --> <link rel="manifest" href="/manifest.webmanifest" /> <link rel="apple-touch-icon" href=/static/c15a0eb1/assets/icons/apple-touch-icon.png /> <link rel="apple-touch-startup-image" href=/static/c15a0eb1/assets/icons/apple-touch-icon.png /> <!-- Styles --> <link rel="stylesheet" type="text/css" href="/static/c15a0eb1/styles/styles.css" /> <!-- Current theme and more --> <link rel="stylesheet" type="text/css" href="/css/themes/darkly.css" /> </head> <body > <noscript> <div class="alert alert-danger rounded-0" role="alert"> <b>Javascript is disabled. Actions will not work.</b> </div> </noscript> <div id='root'><div class="lemmy-site" id="app"><button class="btn skip-link bg-light position-absolute start-0 z-3" type="button">Jump to content</button><div class="shadow-sm"><nav class="navbar navbar-expand-md navbar-light p-0 px-3 container-lg" id="navbar"><a class="d-flex align-items-center navbar-brand me-md-3 active" aria-current="true" style="" id="navTitle" href="/"></a><button class="navbar-toggler border-0 p-1" type="button" aria-label="menu" data-tippy-content="Expand here" data-bs-toggle="collapse" data-bs-target="#navbarDropdown" aria-controls="navbarDropdown" aria-expanded="false"><svg class="icon"><use xlink:href="/static/c15a0eb1/assets/symbols.svg#icon-menu"></use><div class="visually-hidden"><title>menu</title></div></svg></button><div class="collapse navbar-collapse my-2" id="navbarDropdown"><ul class="me-auto navbar-nav" id="navbarLinks"><li class="nav-item"><a class="nav-link" title="Communities" href="/communities">Communities</a></li><li class="nav-item"><a class="nav-link" title="Create Post" href="/create_post">Create Post</a></li><li class="nav-item"><a class="nav-link d-inline-flex align-items-center d-md-inline-block" title="Support Lemmy" href="https://join-lemmy.org/donate"><svg class="icon small"><use xlink:href="/static/c15a0eb1/assets/symbols.svg#icon-heart"></use><div class="visually-hidden"><title>heart</title></div></svg><span class="d-inline ms-1 d-md-none ms-md-0">Support Lemmy</span></a></li></ul><ul class="navbar-nav" id="navbarIcons"><li class="nav-item" id="navSearch"><a class="nav-link d-inline-flex align-items-center d-md-inline-block" title="Search" href="/search"><svg class="icon"><use xlink:href="/static/c15a0eb1/assets/symbols.svg#icon-search"></use><div class="visually-hidden"><title>search</title></div></svg><span class="d-inline ms-1 d-md-none ms-md-0">Search</span></a></li><li class="nav-item"><a class="nav-link" title="Login" href="/login">Login</a></li><li class="nav-item"><a class="nav-link" title="Sign Up" href="/signup">Sign Up</a></li></ul></div></nav></div><div class="mt-4 p-0 fl-1"><div class="error-page container-lg text-center"><h1>Error!</h1><p class="p-4">There was an error on the server. Try refreshing your browser. If that doesn't work, come back at a later time. If the problem persists, you can seek help in the <a href="https://lemmy.ml/c/lemmy_support">Lemmy support community</a> or <a href="https://lemmy.ml/c/lemmy_support">Lemmy Matrix room</a>.</p></div></div><footer class="app-footer container-lg navbar navbar-expand-md navbar-light navbar-bg p-3"><div class="navbar-collapse"><ul class="navbar-nav ms-auto"><li class="nav-item"><span class="nav-link">UI: 0.19.9</span></li><li class="nav-item"><span class="nav-link">BE: </span></li><li class="nav-item"><a class="nav-link" href="/modlog">Modlog</a></li><li class="nav-item"><a class="nav-link" href="https://join-lemmy.org/docs/en/index.html">Docs</a></li><li class="nav-item"><a class="nav-link" href="https://github.com/LemmyNet">Code</a></li><li class="nav-item"><a class="nav-link" href="https://join-lemmy.org/">join-lemmy.org</a></li></ul></div></footer></div></div> <script defer src='/static/c15a0eb1/js/client.js'></script> </body> </html> bastion:~# curl http://lemmy:8536/ { "@context": [ "https://join-lemmy.org/context.json", "https://www.w3.org/ns/activitystreams" ], "type": "Application", "id": "https://mydomain.xyz/", "name": "MYDOMAIN", "preferredUsername": "mydomain.xyz", "inbox": "https://mydomain.xyz/inbox", "outbox": "https://mydomain.xyz/site_outbox", "publicKey": { "id": "https://mydomain.xyz/#main-key", "owner": "https://mydomain.xyz/", "publicKeyPem": "-----BEGIN PUBLIC KEY-----<SNIP>\n-----END PUBLIC KEY-----\n" }, "language": [], "published": "2025-02-17T15:42:53.548004Z" bastion:~# curl -I http://lemmy:8536/ HTTP/1.1 404 Not Found content-length: 0 cache-control: public, max-age=60 vary: Origin, Access-Control-Request-Method, Access-Control-Request-Headers date: Mon, 17 Feb 2025 20:06:03 GMT bastion:~# curl -I http://lemmy-ui:1234/ HTTP/1.1 500 Internal Server Error X-Powered-By: Express Content-Type: text/html; charset=utf-8 Content-Length: 5253 ETag: W/"1485-j9uo4fH1oBpLTZEATf6UDwWGY5M" Date: Mon, 17 Feb 2025 20:06:22 GMT Connection: keep-alive Keep-Alive: timeout=5
Also my proxy is a bit different, note the http_accept is a more generic anything application goes to lemmy, you’re only doing activity+json
set $proxpass "http://lemmy-ui.lemmy.svc.cluster.local:1234/"; if ($http_accept ~ "^application/.*$") { set $proxpass "http://lemmy.lemmy.svc.cluster.local:8536/"; } if ($request_method = POST) { set $proxpass "http://lemmy.lemmy.svc.cluster.local:8536/"; }(Ignore the http://, that’s custom for me)
deleted by creator
