Server error after updating to 0.18.0 · Issue #1530 · LemmyNet/lemmy-ui
github.com
Requirements This is a bug report, and if not, please post to https://lemmy.ml/c/lemmy_support instead. Please check to see if this issue already exists. It's a single bug. Do not report multiple b...

Requirements

  • [X] This is a bug report, and if not, please post to https://lemmy.ml/c/lemmy_support instead.
  • [X] Please check to see if this issue already exists.
  • [X] It’s a single bug. Do not report multiple bugs in one issue.
  • [X] It’s a frontend issue, not a backend issue; Otherwise please create an issue on the backend repo instead.

Summary

Docker setup. After updating to the 0.18.0 images for both lemmy-ui and lemmy backend, the lemmy-ui logs show a pictrs error, and the main site returns a “Server error”.

FetchError: request to https://SITE_URL_REDACTED/pictrs/image/a29da3fc-b6ce-4e59-82b0-1a9c94f8faed.webp failed, reason: connect ECONNREFUSED 127.0.1.1:443
    at ClientRequest.<anonymous> (/app/node_modules/node-fetch/lib/index.js:1505:11)
    at ClientRequest.emit (node:events:511:28)
    at TLSSocket.socketErrorListener (node:_http_client:495:9)
    at TLSSocket.emit (node:events:511:28)
    at emitErrorNT (node:internal/streams/destroy:151:8)
    at emitErrorCloseNT (node:internal/streams/destroy:116:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
  type: 'system',
  errno: 'ECONNREFUSED',
  code: 'ECONNREFUSED'
}

Steps to Reproduce

  1. Update docker-compose.yml to 0.18.0
  2. docker-compose down
  3. docker-compose up
  4. access site

Technical Details

Docker setup. After updating to the 0.18.0 images for both lemmy-ui and lemmy backend, the lemmy-ui logs show a pictrs error, and the main site returns a “Server error”.

FetchError: request to https://SITE_URL_REDACTED/pictrs/image/a29da3fc-b6ce-4e59-82b0-1a9c94f8faed.webp failed, reason: connect ECONNREFUSED 127.0.1.1:443
    at ClientRequest.<anonymous> (/app/node_modules/node-fetch/lib/index.js:1505:11)
    at ClientRequest.emit (node:events:511:28)
    at TLSSocket.socketErrorListener (node:_http_client:495:9)
    at TLSSocket.emit (node:events:511:28)
    at emitErrorNT (node:internal/streams/destroy:151:8)
    at emitErrorCloseNT (node:internal/streams/destroy:116:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
  type: 'system',
  errno: 'ECONNREFUSED',
  code: 'ECONNREFUSED'
}

Lemmy Instance Version

0.18.0

Lemmy Instance URL

No response

  • issue_tracking_bot@lemm.eeOPB
    1·
    1 year ago

    Nice! Good thing I backed up my DB. I’ll try changing the network settings and restoring my DB.

    That did not fix my instance. I had to manually set the site’s image icon to null in the database to get my site working again. There are still many issues in both lemmy-ui and lemmy be logs.

    Is your “external” NGINX proxy running on the same host server as Lemmy?

    Originally posted by madeofstown in #1530

  • issue_tracking_bot@lemm.eeOPB
    1·
    1 year ago

    Thanks! I just left that there for posterity, I added an edit to the top of my comment about how I “fixed” this.

    It was caused due to a network error, so to fix it I just gave the container internet access, even though it otherwise wouldn’t need it. That fixed things immediately.

    Still, network errors shouldn’t be fatal. Especially not for an icon.

    Originally posted by ubergeek77 in #1530

  • issue_tracking_bot@lemm.eeOPB
    1·
    1 year ago

    I think this is happening because there is not any error handling when the icon is fetched:

    https://github.com/LemmyNet/lemmy-ui/blob/9e7fec772d873e7bc46a51403d9569f70619c9b8/src/server/utils/create-ssr-html.tsx#L28

    https://github.com/LemmyNet/lemmy-ui/blob/main/src/server/utils/fetch-icon-png.ts

    So if that call fails for any reason, it breaks the SSR render.

    I think this should be changed to return a broken image icon, or a blank image, instead of aborting the entire SSR process. Otherwise, the pictrs service going down for any reason causes the entire UI to crash like this.

    That goes for any other fetches for external data too. If it’s not mission critical (and images aren’t mission critical), the SSR would ideally continue the render.

    Originally posted by ubergeek77 in #1530

  • issue_tracking_bot@lemm.eeOPB
    1·
    1 year ago

    UPDATE:

    Issue “fixed” by putting the lemmy-ui container on a network with access to the internet. It looks like lemmy-ui didn’t previously need an internet connection on 0.17.x, but now it needs one. Probably because it used to use the internal docker network to get to pictrs, but now it’s trying to resolve using the real public hostname?

    In OP’s case, it’s going to a local IP instead, but it’s using the wrong port. In both our cases, the URL it uses for pictrs is not calculated correctly for whatever reason.

    The UI could fail to contact pictrs for any variety of reasons, but that shouldn’t be fatal. It should just break images.

    Previous comment text:

    I get a very similar error but it’s not quite the same:

     FetchError: request to https://<my-instance>/pictrs/image/<uuid>.png failed, reason: getaddrinfo EAI_AGAIN <my-instance>
     at ClientRequest.<anonymous> (/app/node_modules/node-fetch/lib/index.js:1505:11)
     at ClientRequest.emit (node:events:511:28)
     at TLSSocket.socketErrorListener (node:_http_client:495:9)
     at TLSSocket.emit (node:events:511:28)
     at emitErrorNT (node:internal/streams/destroy:151:8)
     at emitErrorCloseNT (node:internal/streams/destroy:116:3)
     at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
   type: 'system',
   errno: 'EAI_AGAIN',
   code: 'EAI_AGAIN'
 }

    Just like OP, I can go to the URL that error is logging and it pulls the image just fine. No browser cache shenanigans, the request is being fully completed.

    My server is now completely inoperable and I can’t downgrade to 0.17.4 due to the database upgrades. Anything I can do to not lose my data?

    Originally posted by ubergeek77 in #1530

  • issue_tracking_bot@lemm.eeOPB
    1·
    1 year ago

    My server is now completely inoperable and I can’t downgrade to 0.17.4 due to the database upgrades. Anything I can do to not lose my data?

    @ubergeek77

    short term fix is to remove your site’s image icon.

    db query: update site set icon = null where id = 1;

    then at least your site will work again, sort of. It loads at least. I am not sure federation is working 100% though and the pictrs errors still persist in the logs.

    Originally posted by slash-zero in #1530

  • issue_tracking_bot@lemm.eeOPB
    1·
    1 year ago

    I can confirm that this issue is still present in Lemmy-ui 0.18.1-rc.2. Every single time the docker images restart I have to run the commands to delete my site logo or the site will be unreachable. I think this may be partially because my external NGINX proxy is on a different different host. When Lemmy-ui resolves the IP for my domain it is still using the local IP and port 443.

    If believe the fix would be to use the port specified by the pictures server in the internal NGINX configuration. I’m not aware of how to do this. I will test if this can be accomplished by configuring the pictures server to listen on 443 in the internal configuration.

    Originally posted by madeofstown in #1530

  • issue_tracking_bot@lemm.eeOPB
    1·
    1 year ago

    I found a more permanent fix. The error seems to be caused by having the host name of your server set to the same thing as your lemmy instance domain-name. The easy and more permanent fix is to change the host name of your server!

    Had the same problem when I upgraded to 0.18.0, this fixed it.

    Originally posted by betodealmeida in #1530

  • issue_tracking_bot@lemm.eeOPB
    1·
    1 year ago

    Nice! Good thing I backed up my DB. I’ll try changing the network settings and restoring my DB.

    That did not fix my instance. I had to manually set the site’s image icon to null in the database to get my site working again. There are still many issues in both lemmy-ui and lemmy be logs.

    Originally posted by slash-zero in #1530

  • issue_tracking_bot@lemm.eeOPB
    1·
    1 year ago

    I found a more permanent fix. The error seems to be caused by having the host name of your server set to the same thing as your lemmy instance domain-name. The easy and more permanent fix is to change the host name of your server!

    Originally posted by madeofstown in #1530

  • issue_tracking_bot@lemm.eeOPB
    1·
    1 year ago

    @randommouse:

    OK, I updated to:

    • lemmy-ui: 0.18.1-rc.7
    • lemmy be: 0.18.1-rc.4

    Then, I changed my host’s server name to not match the domain name, and rebooted. Local images were still not working after a docker-compose down and docker-compose up -d --force-rebuild. I still couldn’t upload images either, and the site banner was not working (broken image).

    The final step was to revert the lemmy.hjson back to http://pictrs:8080 (from the public domain name) and restart again, and now images are working, and having an icon for my instance no longer breaks the site.

    So, to clarify, what I did was:

    1. Change my host name in /etc/hostname so it no longer matches my site’s domain name
    2. Update lemmy.hjson to set pictrs back to the original http://pictrs:8080
    3. Reboot, because why not?
    4. “Restart” all lemmy containers via down, then up
    5. 👍

    Originally posted by slash-zero in #1530