Hi everyone!

Last year lemmy.ca ran a census for our instance, and this year we wanted to solicit feedback ahead of time. Since other instances expressed interest in running similar censuses, having some agreed upon questions would make it easier to do comparisons. (previous: lemmy.ca results, lemmy.nz results).

Timeline wise, for our instance, we are thinking of gathering feedback for the rest of the month and then running the census early in the new year. We are open to adjusting when we run it, to align with other instances.

General areas of feedback:

Self Identification / Categorization

With some questions, it is difficult to have a predetermined list of options. Last year we included a free form entry option, and then created word clouds with the full list. However, that is not as insightful, and it takes a lot of time to process. For this year, do you prefer:

  • A set of options + a free form option (same as last year)
  • A set of options only

Please share if you know of a resource that has a good/inclusive set of options for the question types. Even if a free form option is included, a better set of options as a base will lead to better data.

For context, these were the questions where this happened:

  • What is your ethnicity?
  • How do you identify? (sexual orientation)
  • What is your occupation / field of Work?

Data collection & Results

Last year we used Google forms for collection, and a spreadsheet for calculations and analysis. Visualization was done with google sheets, an open source word cloud generator (github link), and DataWrapper (link) for an interactive display on our website.

This year the plan is similar, although we’re interested to try chart.js for the visualizations. We had some issues with DataWrapper displays, and having the code be open source would make it easier for others to replicate. One consideration is data privacy (do we commit anonymized data to the repo, host it elsewhere and import it, etc.).

Sometime in the future, it might be nice to have a template and/or pipeline that everyone can use, for consistency and to minimize work. In the meantime, do you have any thoughts or recommendations?

Fun Question

Last year we asked about people’s favourite dessert. It would be cool to do something similar this year. Do you have any suggestions for a fun question? Some ideas:

  • Favourite comfort food
  • Favourite fruit
  • Favourite color (with a hex code input)

Proposed Questions

Where applicable, I’ve included changes from last year. Depending on the type of instance you run (ex. regional or not), you may want to skip, add, or modify questions.

Section 1: Where is everyone from?

Where are you from?

  • Canada? (yes / no)
    • If in Canada, which province or territory?
    • If not, which continent?

What is the size of your community? (rural / urban / suburban / other)

  • suburban and other are new options

In your opinion, what is the quality of your internet connection? (poor, reasonable, excellent)

What is your internet speed?

  • new question inspired by lemmy.nz
  • we may include a link to a speed test site for convenience, and have a free form entry

Section 2: Who are you?

What age range do you fall into? (0-9, 10-19, etc.)

Ethnicity? (options uncertain, see above)

Gender Identity? (Man, Woman, Non-binary person, Genderfluid)

  • Optionally, we could include a free form entry ‘other’ option, and list the input in a separate list.

Are you someone with trans experience (yes / no)

How do you identify? (options uncertain, see above)

Disability Status (yes / no)

  • New separate question inspired by lemmy.nz

Disability Status (free form entry)

  • Last year we manually processed the text inputs and displayed them in a list. We’re open to hearing suggestions on better ways to run this question!

Educational Background

  • Last year we had multi-select, which didn’t make sense. This year we can split it into a few questions:
    • Are you currently in school? (yes / no)
      • What are you studying? (single selection, see list below)
    • Highest level of education completed (single selection, see list below)
    • All past education (multi-select, see list below)
    • List of options:
      • No formal education
      • Some secondary school (high school)
      • Secondary school (high school) diploma
      • GED
      • Trade / technical / vocational training
      • Some Post-secondary (college, university, CÉGEP)
      • College diploma
      • Associate degree
      • Bachelor’s degree
      • Master’s degree
      • Professional degree (MD, JD, MBA, etc.)
      • Doctorate (PhD, etc.)
      • Other

Employment Status (full-time, part-time, self-employed, student, retired, unemployed, other)

  • Last year we also got responses for ‘homemaker’, ‘disabled’, ‘active duty military’. We could add these as options, or include a free form ‘other’ entry for them.

Field of Work (options uncertain, see above)

  • This question was a free form entry last year, and it was very difficult to process. Having a solid list of options would be ideal in this case.
  • We could include an ‘other’ option with a free form entry, with a note that any response will not be included in the main results and will be displayed as-is in a separate list.

Section 3: Instance Usage

How many {instance name} communities do you participate in? (0, 1-9, 10-19, 20+)

If you moderate any communities, how many? (number input)

  • Last year this was a multi-select, but a number input would be more straightforward

What community do you want to see on {instance name}? (free form entry)

What existing community do you want to see more people using? (free form entry)

  • This is a new question.

On average, how much time do you spend on Lemmy each day? (<1 hour, 1-2 hours, 2-3, 3-5, 5-8, 8-13, 13-21, 21+)

How often do you visit {instance name}? (multiple times a day, daily, weekly, monthly, rarely)

  • This is a new question.

How do you access Lemmy most often? (desktop, mobile (including tablet), both about equally)

  • (including tablet) is a new addition
  • both about equally is a new option

If you use desktop, what operating system do you use? (Windows, MacOS, Linux, I don’t use desktop, other)

  • New question

On desktop, which interfaces do you use? (default, Mlymym, Photon, Voyager, Alexandrite, I don’t use desktop)

  • I don't use desktop is a new option, and will be separate from the ‘skip this question’ option

Is there another interface you would like us to add? (free form entry)

  • New question

If you use mobile, what operating system do you use? (Android, iOS, I don’t use mobile, other)

  • I don't use mobile and other are new options

On mobile, how do you access Lemmy?

  • We will include an updated list of apps, and a free form entry for any new ones
  • We will include an option for I don't use mobile, web browser, and other, in addition to the skip this question option

If you primarily access lemmy.ca through different Lemmy/Kbin instance, please list it below (optional) (free form entry)

What other decentralized / federated platforms do you use?

  • Some potential options: none, Mbin, Kbin, PieFed, Mastodon, Bluesky, Pixelfed, Peertube, Loop, Bookwyrm, WriteFreely, Pleroma, Friendica, Misskey, Writefreely, + free form entry

Fun Question (yet to be determined)

Feedback (free form entry)

General Changes:

  • All questions will have a ‘skip this question’ option
  • Fewer random comments from me when reporting the results. I looked back at them afterwards and thought they might be a bit much.
  • hendrik@palaver.p3x.deEnglish
    4·
    2 days ago

    I’d like to know what people use Lemmy for… Reading the news, scrolling through meme pictures or talking about hobbies…

    If they subscribe to communities tailored to their interests or just watch the “All” feed.

    And maybe moderation. If users have previous experience with moderation and whether they think their experience with that was negative or positive.

    And I’m always interested to know if people still use computers. And maybe the operating system, to judge the amount of Linux nerds here.

    • Dave@lemmy.nzEnglish
      3·
      2 days ago

      I like this idea. Getting some sort of insights into Lemmy.

  • Dave@lemmy.nzEnglish
    3·
    2 days ago

    I saw another comment suggesting various things that it would be fun to know about the community. These surveys aren’t just a view into an instance, but the lemmy.ca and lemmy.nz surveys have been cited many times as they are some of the only info we have about Lemmy as a whole.

    I was thinking we should have base questions, ones that we want to watch for changes over time. And then others that could be a snapshot insight that give us a glimpse into Lemmy with an understanding that people don’t want to spend an hour filling in the survey so we can’t ask everything.

    I think the question list you have is a good set of base questions, and I don’t think there are any you’ve suggested that shouldn’t be asked each year. So I propose we pick a couple of extra questions. I think it would be nice for everyone to use the same questions and see differences across instances, but also using different ones per instance gives us wider insights. So I could be persuaded either way. The kind of things that @[email protected] suggested, like whether people mostly use Subscribed, Local, or All feeds (mindful these might have different names in different apps/frontends). Or OS they use, etc. Just making sure it’s only one or two questions so it’s not putting people off doing the survey by making it too long.

    Now to your question:

    For self-identification, free text means people are more likely to write what they actually want instead of trying to push themselves into the box of listed options, even if there is an Other option. However, it’s also a lot of work to group things, and things need to be grouped to make any decent result visualisation. Plus people should be allowed to group themselves instead of me doing it. So I suggest a predefined list with an Other free-text option.

    I think this is the case for the other similar things you list as well. For ethnicity, for our survey we used the actual list from the NZ Stats department. It has been carefully refined over years, there’s no reason to think we could do any better. But of course our list would not be very helpful for other countries, so for region-based instances, maybe something similar can be found from that region.

    The disability question was quite tricky to work out how to put into a chart. People can have multiple, but then you don’t really want them showing 10 times. I think I’d like to have a pre-defined list next time, with an Other free text option as well. Let people put themselves into their own categories instead of me trying to push them into groups.

    For the fun question, what I like about Favourite Dessert is how groups of people are likely to tend towards certain answers, but different groups of people can tend towards quite different things (especially when it’s regional). Favourite Comfort Food could be a good one, but it might be nice not to do food again. I’ll let others suggest some things. I will note that I think this Fun question is different from the suggestion above to have snapshot questions to get insights into Lemmy. Having “What OS do you use” is a census question, “What is your favourite dessert” is a fun question. I think we should have both.

    • OtterOPAEnglish
      2·
      1 day ago

      This is very detailed, thank you for your thoughts!

      That makes sense to me, to organize the questions into

      • base questions (used by everyone, repeated annually)
      • a few questions (used by everyone, on a particular census)
      • instance specific questions (up to each instance)

      I’m going to be away for a few days, but I’ll see about listing out these new questions and other changes. Maybe we can put them somewhere to make it easier to collaborate and track changes?

      For self-identification, free text means people are more likely to write what they actually want instead of trying to push themselves into the box of listed options, even if there is an Other option. However, it’s also a lot of work to group things, and things need to be grouped to make any decent result visualisation. Plus people should be allowed to group themselves instead of me doing it. So I suggest a predefined list with an Other free-text option.

      That makes sense, I’m leaning towards doing that. I’ve also gotten some suggestions on where to get the lists from so that should work :)

      • Dave@lemmy.nzEnglish
        1·
        8 hours ago

        I’m going to be away for a few days, but I’ll see about listing out these new questions and other changes. Maybe we can put them somewhere to make it easier to collaborate and track changes?

        That sounds like a good idea. Any thoughts on where?

        • OtterOPAEnglish
          2·
          5 hours ago

          Whatever is easiest for people :)

          Some options that come to mind:

          • a markdown file on git / GitHub
          • a shared Google Docs file
          • back and forth comments on Lemmy?
          • Dave@lemmy.nzEnglish
            2·
            5 hours ago

            I’m wondering if the git option is best? A public github repo is a bit more permanent, means it stays available into the future even if it stops getting maintained.

            The main issue with that is the technical hurdle for contributing, but I don’t see why people can’t comment on this post or future ones with comments if they don’t want to do a pull request.

            I prefer that over the Google docs option, and Lemmy comments are good for discussion but I think we need an “official” place to point people to. We can still discuss on Lemmy before making updates.

            • OtterOPAEnglish
              2·
              4 hours ago

              That makes sense to me :) The people maintaining it can add in the Lemmy comments as needed

              We have a GitHub organization for our instance, I’ll see if I can make a public repo and copy in the comments of this post

              • Dave@lemmy.nzEnglish
                1·
                4 hours ago

                Sounds good!

                Perhaps take on board the comments on this post, draft up a proposed template on the repo, and then perhaps do a new post asking for feedback? Is it worth creating a community for those interested to follow if they want to discuss, ask questions, propose ideas, etc, or is that better done on the repo?

    • db0@lemmy.dbzer0.comEnglish
      8·
      3 days ago

      I don’t think I got the time to set this up atm but if another admin wants to try, I’d be OK with it

      • OtterOPAEnglish
        5·
        3 days ago

        If it helps (and if your instance is ok with using Google forms too), we should be able to share the template for others to copy

          • OtterOPAEnglish
            2·
            1 day ago

            Yea we picked Google forms for convenience mostly. We want to switch to something better at some point. In my quick look around, there are a few self-hosted options that could work.

            Maybe in the future we could collectively make a few templates with the selected questions for that year. That way instances can use whichever method they have the resources to run, but still get the same format of data afterwards

            If your team does come across something better, I’d be interested in exploring further

            Does it log IP addresses of respondents?

            While the survey creator can’t see any of those details, I imagine Google may be tracking things on their end.

            • Dave@lemmy.nzEnglish
              2·
              23 hours ago

              For our census I self-hosted Lime Survey (using this not-official docker image). Took a little bit of working out but I think the survey worked out quite good. There are lots of options to pick through, including various levels of privacy settings, and I’ve also run some other community surveys using it in the past.

              • Unruffled [he/him]@lemmy.dbzer0.comEnglish
                3·
                19 hours ago

                I’ve used limesurvey a few times before and I think it’s definitely a viable option as it has an option to disable ip tracking along with an anonymous mode.

      • Dave@lemmy.nzEnglish
        6·
        3 days ago

        Cheers for tagging me, I’ll have to go through my stuff from when we did it and see if I have any feedback.

        Fewer random comments from me when reporting the results. I looked back at them afterwards and thought they might be a bit much.

        The random comments were very helpful and influenced the questions we used on lemmy.nz. I also think it’s good to understand how the data has been analysed to highlight potential biases. i.e. don’t feel bad about your rambling or I’ll feel bad about mine 😆

        • OtterOPAEnglish
          5·
          3 days ago

          Ok that’s good to hear 😄