Idea: Quick context

[LeaVerou]

This has now been superseded by #183

---

Background & Problem Statement

In State of CSS 2022 we added freeform comments to Feature questions, for respondents to provide additional context:

This came from a need to get more structured followup. It is not very useful for browsers to know that authors haven't used a feature without knowing why, or that authors have used a feature without knowing how it went. There was a lot of back and forth about what more structured followup could look like. Many brainstormed ideas also raised concerns about survey fatigue, engineering effort, compatibility of data with past years etc. In the end, we went with an MVP which only allows entirely freeform comments.

Predictably, it was not used very much: even in the question with the most comments, only a tiny fraction of its respondents left a comment (126 comments by 14114 respondents = 0.89%). Also, being entirely freeform made the data hard to analyze.

To ease discoverability, the comments field is open in the first question to demo the feature, then collapsed in subsequent questions to avoid cluttering the UI. However, it is also "out of sight, out of mind", which results in the first question getting several times more comments than any other (126 vs 15-50 for all other questions).

Goals & Requirements

• Increase % of responses that provide additional insight into authors experiences with a particular web platform feature (pain points, ease of understanding, notable browser incompatibilities etc)
• Common types of additional context should be fast and frictionless (ideally a single click)
• Should still allow for freeform input, to provide more details (e.g. "Browser incompatibilities" is useful and should be very fast to enter, but "Browser incompatibilities (Safari doesn't support the foo keyword)" is even more useful)
• Must be easy to implement, as engineering resources are limited
• Bonus if it facilitates better quantitative analysis compared to entirely freeform responses
• Bonus if it encourages more freeform comments as well

Proposed solution

Changes to Question UI

Response-sensitive buttons to quickly insert common/most useful types of feedback into the textarea, as Markdown list items. The predefined answers differ depending on the answer selected. Crude mockup:

To mitigate distraction, they should fade in slowly (unlike what is shown in the mockup). The rest of the UI should not move when they appear (this will require a creative solution for mobile).

It is important that they have visually distinct colors so that after a few questions respondents can recognize them visually and do not need to read the labels.

The comment inserted is just text, and can be edited to add additional details:

Options

Originally I was envisioning this as labels about specific issues:

1. 🤷🏼 Never heard of it / Not sure what it is

• Interested
• Not interested

2. ✅ Know what it is, but haven't used it

• Insufficient browser support
• Browser inconsistencies
• Haven’t needed it
• Hard to use

3. 👍 I've used it

• Loved it
• Hard to use
• Not powerful enough

Individual questions could override this with their own predefined answers, e.g. see Form validation for an example use case.
It is important that these one-off answers have different colors than the common ones, otherwise they may be selected by accident.

After discussing with @stubbornella about what would be most useful to browsers, it looks like it would be more high level responses primarily to express sentiment:

1. 🤷🏼 Never heard of it / Not sure what it is

• Interested
• Not interested

2. ✅ Know what it is, but haven't used it

• Want to try it
• Don’t want to try it

3. 👍 I've used it

• Would use again
• Would not use again

If there's time, we could even implement a two tier interaction, where selecting these high level sentiment labels uncovers more. E.g. selecting "would not use again" could reveal "Browser inconsistencies", "Hard to use" etc (again, these are examples).

Results UI

With a quicker way to add context, there will be a lot more comments for each question, most of them using the predefined responses. Having more data is a great problem to have, but it also introduces some challenges. With no change to the analytics pipeline or results UI, edited or entirely custom comments will be drowned out by the noise.

This is not an issue for quantitative analysis (count how many responses either are predefined or start with predefined responses), but it is for qualitative analysis, where we want to read what respondents actually wrote. This affects both our own data analysis, as well as the user-facing results, since there is an end-user facing feature to read all freeform answers from each question:

This would need to be adjusted to make unedited predefined responses easier to filter out.
In order of priority:

• MVP: Comments only consisting of predefined responses should not be displayed.
• Count at the top for each predefined feature even if edited, assuming the edit is at the end of the line, e.g. Loved it (118)
• For comments that include a mix of predefined responses, edited predefined responses, and entirely custom comments, the whole comment should be displayed. If there's time, we could fade out the non-edited predefined items.

Scope

Originally, Quick Context was designed as a way to capture more context in Part 1 of the survey (Feature questions). Since my initial proposal, use cases have arisen that make it useful for potentially any multiple choice question. However, if that complicates implementation, it could stay in Part 1 for this survey.

Discussion

Several benefits to this approach:

• Single-click feedback for common pain points — can't get faster than that
• Low engineering effort, as it’s an entirely client-side feature that does not require any backend or data model changes.
• The fact that the predefined responses can be edited allows adding additional context, which would be lost with a more structured feature.
• It enables some quantitative analysis, as most respondents would not edit the actual predefined answer (though they may add context at the end, as shown above)
• Grounds the format, and provides guidance on brevity, keeping the answers more consistent and preventing fear of blank canvas
• Increases the visibility of the comments feature as a whole, which may increase the frequency of non-canned responses too.

And a few downsides:

• If a button is accidentally pressed, undoing requires manually deleting text
• Some concerns that any predefined responses may introduce bias though hopefully Feature questions are too narrow for that to apply here.

Alternative design 1: Tags

An alternative design (proposed by @SachaG and @una) is to treat these predefined responses as "tags" that are stored separately. In that design, clicking any of them would simply highlight the tag as selected but would not insert anything in the comments area (though it could open the comment area, to implicitly invite comments).

The advantages of this approach are:

1. It makes quantitative analysis easier, since we can just count selected tags instead of matching text patterns
2. Eliminates the problem of having to filter out comments entirely generated from predefined responses (but also reduces the number of comments)
3. It is a more common UI paradigm: interfaces where structured controls (e.g. buttons) insert freeform text are relatively uncommon. The two examples I can think of are a) chat interfaces to suggest responses b) review interfaces to suggest common issues c) text editors of various types

Note that 2/3 of these advantages are about making our life easier, not the respondent's.

However, this design has several drawbacks:

1. The main issue is that the UX flow provides no obvious path between selecting one of these "tags" and adding more details about that item itself. Let’s consider a typical flow, using the example in the mockup above.
   1. With the UX proposed here, they click on "Browser inconsistencies", it inserts - Browser inconsistencies, and they can simply edit this to add "(Firefox doesn't support this on the time input)" at the end.
   2. With the tag model, they click "Browser inconsistencies", which becomes highlighted and opens the (still blank) comment field. They can just type "Firefox doesn't support this on the time input", but that's not connected to anything and requires looking at the selected tag to interpret. Worse, if multiple tags are selected, users need to manually connect their feedback to the right tag (e.g. "Browser Inconsistencies: Firefox doesn't support this on the time input"), and many will not do so.
2. Requires more engineering effort pre-launch, as it affects the data model and needs to be stored separately. In practice, this makes it less likely to be implemented at all.
3. It is well known that editing is easier than writing from scratch; having something to edit encourages more use of comments in general, even comments outside the scope of these predefined responses
4. We lose the benefit that the inserted responses act as a guide for style and brevity of the freeform comments.
5. We lose the benefit that the predefined responses steer more people towards using the comments feature.

I think the primary reason this alternative design gets proposed is that people are uncomfortable with the idea of "losing" structured data into a freeform field. However, the only reasons for respondents to edit the inserted responses is a) to add additional context or b) tweak the wording to express them better. In both of these, editability is a win!
We can verify my hypotheses and see how respondents actually use this if we implement a prototype before user testing commences.

Alternative design 2: Hybrid of tags & text macros

This is a combination of the original idea (responses act as text macros) and the tags design above, which stores separate structured data.

Selecting responses would highlight them as selected and store them separately, but also insert them in the text field to seed comments. Before storing the data, unedited responses will be removed (since we can recreate them from the structured data, we'd just lose the order) so they would not pollute the stored comments, but they can still encourage commenting, which was a crucial aspect of the original proposal.

This combines most the benefits of the original proposal, as well as most of the benefits of the Tags proposal above:

• Easy path to adding further context without having to separately reference things
• UX to steer users towards leaving more comments
• Easy quantitative analysis
• No pollution of freeform comments

It does still have some of the drawbacks

• Since the responses are stored separately, it does change the data model, and thus involves more engineering effort. But it appears to be effort @SachaG is willing to make anyway (see below).
• The structured data and the freeform comments could get out of sync if the user deletes the inserted text but does not unselect the label. Did they intend to remove the response, or did they simply not want to leave additional context, so they were "cleaning things up"? This is something that user testing can answer.

[SachaG]

This is a great start, but I thought of another downside of this specific implementation. If we have the "predefined follow-up" buttons simply insert text, then that will presumably increase the number of responses that contain one of the predefined follow-up statements. Which in turn will drown out the fully original responses that people took the time to type out (nobody wants to scroll through hundreds of identical "Insufficient browser support" comments).

Now we could work out a system to filter all of this on the back-end, but then that defeats the purpose of having a low-effort solution in the first place…

Instead the slight tweak that I would offer is that predefined follow-ups are stored separately altogether, even if we keep a relatively similar UI.

[LeaVerou]

That is not an implementation detail, keeping the predefined responses editable is core to the proposal. It allows for additional context of that item itself, and drives users towards the existing commenting feature, rather than away from it. What you are proposing is an entirely separate way to add context that simply co-exists spatially with comments.

While long carefully crafted responses are useful and maybe fun to read, they do not tell us much about what are the pain points of the majority of developers. For that, you need aggregate data, the more the merrier. Getting too many responses is a great problem to have!

Now we could work out a system to filter all of this on the back-end,

Indeed, this is trivial to filter with code.

but then that defeats the purpose of having a low-effort solution in the first place…

I don't see how. a) It is still low effort for users, which is a top priority b) Having to filter out comments entirely consisting of predefined responses is also pretty easy to implement and could even be done client side (assuming client-side JS has access to the translations): Just split each comment in lines, iterate over them, filter out the ones that entirely match predefined answers, if the resulting array is empty, hide the comment.

[SachaG]

I had time to implement a very rough version of this "follow-up question" feature in the simplest way possible, as a starting point. Here's how this implementation works:

• Clicking on any of the original predefined options triggers follow-up questions to appear underneath the selected option.
• Each option has its own set of follow-up question tags.
• Follow-up questions can be defined on a template level (and apply to e.g. all feature questions) or overridden on a question-per-question level.
• Comments work just as before, completely independently from follow-up questions.
• Follow-up questions are stored in their own subfield in the database (e.g. { features__font_display__followup: ['insufficient_browser_support', 'hard_to_use']})

[stubbornella]

I want to speak to what Web UI, my product area at Chrome, needs out of this type of survey question.

First, we need to understand awareness. Have they heard of it or not? Have they tried it?

Next, we need to understand sentiment at each level. Sentiment is a guarding metric. It means we pay attention if something goes wrong. Then, if developers expresses negative sentiment toward and API we will follow up in other ways like additional surveys, partner and developer interviews, checking bug counts, etc.

We don't want to ask why they feel that way in this question because the answers can only be simplistic (buttons, dropdowns) or ignored (text areas). We'd rather move to a high bandwidth modality, like interviews, to dig in further when we notice something is wrong.

@LeaVerou shared the rough hierarchy I need:

• haven’t heard of it
  - looks useful
  - not useful
• heard of it, but haven’t tried it
  - want to try it
  - don’t want to try it (we’ll follow up other ways)
• I’ve used it
  - want to use it again
  - do not want to use it again (we’ll follow up other ways)

Let me share 3 examples of how I have used this data in the past:

1. One API had lots of developers who hadn't tried it, but wanted to. The next year, they had moved to tried it, don't want to use it again. This kind of guarding metric is incredibly helpful for making sure we catch it if we shipped the wrong thing, it is broken, or missing support, or in that case we'd shipped a second version of the same thing and everyone was attached to the first one and didn't have an upgrade path. The answers here can be nuanced and aren't easily captured in simple pills.
2. A case where we use awareness data is to understand if our devrel efforts are working for a particular API. Ex. Color APIs aren't getting much traction yet in awareness. We can dig in to find out why via a specific survey about color. It would add far too many questions to this survey to go that deep in any one area.
3. Another example is an API that, even after years of availability, wasn't getting adoption. We were able to track awareness (high) against sentiment (very low) and realized the API wasn't expanding past its initial niche. Again, not something we could have caught with a few pills next to a response.

Given the constraints of how we asked this question in the past, we can't flatten it to 5 answers. So, it would be overwhelming to show all of these options to developers at once, please find a way to reveal the sub choices after they have made their initial selection.

@LeaVerou can you post a gif of your most recent design so I can be sure I'm following the discussion?

[SachaG]

Here's a gif of my latest prototype, which differs from Lea's in a couple areas. But overall I think the intent is the same:

(imagine that the little comment icon at the bottom is not there; this would replace it altogether)

@stubbornella my only issue with what you're proposing is that it's extremely similar in wording to the questions we have for libraries:

For the HTML survey that might be fine, since I don't know if we'll even have libraries; but for the CSS survey for example I would not want to include two question templates with such similar answers but different functionalities, I feel like it might get confusing.

As an aside I'm also not convinced by the "looks useful/not useful" follow-up since I'm not sure how people would judge that for a feature they haven't heard about? Does measuring what people think of a name they've never seen before really give us a lot of signal?

[stubbornella]

@SachaG can you please take a look?

[SachaG]

I've deployed the current implementation here (you can take the survey as a guest):

https://survey-staging.devographics.com/en-US/survey/state-of-html/2023

The first page of the survey has my implementation, the second one has the concept from Lea's post (still WIP).

[stubbornella]

Thanks for deploying this, it was helpful for understanding!

[LeaVerou]

Hi Sacha,

To recap, these were the requirements we discussed:

1. The feature should allow people to select quick positive/negative sentiment followups with a single click, plus add additional context via freeform comments.
2. The UI should not jump around when an answer is selected
3. The interaction should not be hidden behind a click
4. The UI should make it clear that the follow-ups are optional and not give people a feeling of incompletion if they don't fill them out

Here's a gif of my latest prototype, which differs from Lea's in a couple areas. But overall I think the intent is the same:

Unfortunately, while at first glance this may seem functionally equivalent, there are several issues here that will result in a significantly worse response rate. I would be surprised if it's more than 5% on average. We don’t need 100% or even 80%, but 5% is way too low (if we get 10k respondents, that’s only 500 people). Also, the higher effort the feature is to use, the more it introduces selection bias, as only the most invested of respondents will fill it in.

The biggest problem is that the whole interaction is hidden behind a "Tell us more…" button, violating one of the core requirements we discussed. Even if you teach respondents how to use the feature by automatically expanding it on the first question, that does not make a huge difference as they move on and forget about it. "Out of sight, out of mind". Also, if they need to click to see the possible followups for each answer, they are less likely to think one might be relevant to them. Whereas it's harder to resist clicking on a followup that's relevant, if it’s right there.

Second, the fact that so much of the UI moves when you do click that button makes it feel heavyweight (this is about perception, not performance), making users subconsciously hesitate a bit before using it. The initial proposal did include some UI moving after a click, but that was after the followup had been selected.

Some smaller nits:

• The phrase "Optionally, you can tell us more about why you picked this answer:" adds nothing, and increases the size of this form, making things jump around more. As a general point, you don’t improve UIs by plastering "helpful" text all over them. Users don’t use UIs by reading them top to bottom, they scan them, and are unlikely to even read the help text, so it just adds clutter. What the text is saying should be made obvious through the interaction and the UI design. Only when this is not possible, help text is a good idea. In this case, the form showed up in response to the user clicking a "Tell us more" button — so of course it's a form for them to say more! The fact that the labels are clickable should be obvious through their visual appearance (affordances & signifiers) for sighted users, and through appropriate markup for non-sighted users. As long as these are followed, this sentence is the UI equivalent of i++; // increment i.
• The placeholder in the textarea says "Other reason", however I see this as primarily useful to explain the sentiment selected, give us the whys. I have trouble imagining what kind of "other reason" people would enter there that doesn't relate to their sentiment in any way. I’d change to something that encourages them to provide this kind of additional context, ideally sensitive to the label selected, e.g.: "looks useful because…" or "don't want to use again because"
• If we only have positive/negative sentiment followups in this iteration, we should probably make them mutually exclusive (radios, not checkboxes)

One positive of this design is the proximity of the followup to the selected answer, which logically groups them (though the fact that the followups live on a separate popup dampens this). However, hiding the UI behind a click and making it so heavyweight to achieve this proximity is not an appropriate tradeoff.

For desktop, we could probably fit the two followups next to the answer itself, with a comment icon for additional context next to them (on the right of the answer), but not sure what this would look like on mobile. Maybe 👍🏼 👎🏼 icons, that would expand to labels when selected? More brainstorming needed.

@stubbornella

@LeaVerou can you post a gif of your most recent design so I can be sure I'm following the discussion?

Will do later today, update the OP, and ping you!

[SachaG]

Thanks for your feedback! At this point I think it boils down to differing priorities… My priority is not adding to the mental load of the majority of survey takers, which is why for me this follow-up UI being hidden behind a click is actually a pro, not a cons. And I'm ok with accepting lower response rates for follow-ups questions as a result, especially for a first try. I'd rather start with the least obtrusive version, and iterate from there over successive survey if the response rate really is too low.

The phrase "Optionally, you can tell us more about why you picked this answer:" adds nothing, and increases the size of this form,

It's all subjective obviously, but I disagree on that. I think UI text is very important to guide people along and avoir any frustration on their part ("what's this thing now? am I required to fill that out as well?!")

If we only have positive/negative sentiment followups in this iteration, we should probably make them mutually exclusive (radios, not checkboxes)

I think if we did only have positive/negative followups it would be a waste of this feature, and we might as well just use radio buttons and save people the trouble of dealing with an extra layer of UI. So I'm going from the assumption that we'll have ~2-4 checkbox options.

[SachaG]

Whereas it's harder to resist clicking on a followup that's relevant, if it’s right there.

Just quoting this because I think it nicely illustrates how we're looking at things differently. Making it "harder to resist clicking" is actually the opposite of what I want.

Even though all questions are optional I know many respondents will feel obliged to answer all of them anyway. The issue is that for someone who adopts that completionist mindset, adding always-visible follow-up options effectively doubles the length of the survey, which I feel risks creating a bad experience for them overall – even if they do fill out the survey.

So that's why I want to hide the UI – the message I'm going for is "hey, you can fill this out as well if you want, but you really don't have to!"

[SachaG]

Another way to look at this is in terms of priorities. From a "collect the best possible data" standpoint I 100% agree with everything you're saying.

But as important as it is to collect data that will be usable by browser vendors, that's not my top priority. Instead my top priority is for survey takers to have a (relatively) pleasant experience, so they come back year over year to take this survey – as well as other surveys we offer.

That's why I'm very wary of anything that will increase the perceived length or complexity of the survey, and why I like hiding the follow-ups behind a click. This way, at first glance the survey does not seem any more complex or long that past years.

We had a similar data-vs-experience debate about using income brackets vs asking for a specific number: the specific number is better from a data collection point of view; but it results in a worse experience for survey takers as it creates privacy worries.

P.S. I know both data and experience are extremely important to you, and that you always take both into account as demonstrated by this entire thread. I'm just saying that when they are in conflict, I will usually side on the side of experience.

[stubbornella]

I realize there is no perfect answer, and truthfully we are balancing constraints that are in opposition to each other, but you are both making really good points. I'd like to propose something that I think is at the intersection of what each of you wants and meets Chrome requirements. I think you would find more common ground in a simple solution that can better balance opposing constraints.

For this set of questions to be useful to my team, we need aspects of what both or you want.

• Awareness alone isn't enough, we need sentiment too.
• @LeaVerou makes a good point that the data won't be useful if too few developers answer the sentiment question.
• @SachaG also makes a good point that we can't overwhelm developers with options and UI in general or they won't complete the survey
• I want a two-click response to each feature so we are respectful of developer's time and attention while answering the survey.

I think we can have all of these things.

These are the things we need to remove to achieve two-click answers while keeping the UI simple:

1. Remove the emojis
2. Remove the "Tell us more" button
3. Remove the checkboxes to explain why the feature hasn't been used, ex. "I haven't needed it"
4. Remove the "other reasons" text box

Now that things are simpler, we can build a two-click experience.

Awareness [click 1]

First, the developer chooses awareness level. [first click]

Sentiment [click 2]

Choosing awareness triggers the sentiment menu to slide open automatically. The user then chooses their sentiment toward the feature. [second click]

You both make good points about the secondary menu, but ultimately you are both right! We don't want to waste developer's time on additional clicks or attention on optional fields.

Choices in menu

This wording may not be exactly right, but these are the set of options we need to include:

• Never heard of it
• Heard of it
  • Would like to learn
  • Not interested
• I’ve used it
  • Would use it again
  • Would not use it again

I agree with @SachaG that it isn't necessary to have options under "never heard of it". @LeaVerou you made the point that developers could learn about it from the survey examples. In that case, they can choose sentiment under "heard of it".

Yes @SachaG, this is intentionally very similar to the framework questions because we have found that framing of awareness and sentiment very useful. Several times we have changed product direction because of the data you gathered from asking this question. This is great:

My preference is how you asked it, five simple choices, no sub menus. Unfortunately, as I understand it, we can't ask it in the same way for features, because we would invalidate comparisons against the previous year's data. Year on year comparisons are important for judging my team's success. That said, we can still ask in a way that works well for features and is simple and easy to use. Two clicks! :)

Open ended textbox options

Finally, I'd like to ask your opinion about two options for the open ended text box. Please keep in mind both developer attention and time as you answer.

1. Should we add an open ended text box at the end of each section (rather than each question) saying something like, "anything else you'd like to share about layout"? That will give the small percentage of folks who write in open ended text boxes somewhere to share their thoughts.
2. Should we add a "tell us more" button that reveals itself after the sentiment selection has been made. It would reveal a textbox per feature. Ex. "tell us more about flexbox gap". This adds a third click, which I'm very hesitant about, even if it is optional.

Follow ups

When the survey goes out and we learn about features that score poorly for sentiment, I will do a follow up study/interviews so we can learn the most common reasons why features have poor sentiment. We can use that data next year to decide if the sentiment feature should be restructured.

[stubbornella]

Hehe, that's basically my whole job in a nutshell. Try not to make CSS and HTML features developers are forced to use but hate. 🤣

A made up, but realistic, use case could be something like: we ship CSS Nesting. Developers expect it to also scope styles, but it doesn't. So lots of them land on positive awareness but negative sentiment. We would then interview devs and run followup surveys to find out why they aren't happy and realize nesting needs a follow-on feature to also control scope. This is the kind of problem we hope to catch early with new APIs, while they can still be fixed.

When I heard "would use again" on your previous surveys I always imagined you meant on a greenfield project. Like, if I were starting a brand new project, would I choose that tool again? "Liked it" somehow seems fluffy by comparison. Ex. I liked writing Rails, would i use it on a new project? Never.

That said, I totally see your point about how developers might be forced to use has(). In some cases, even in CSS they would have alternatives like flexbox vs grid. I suppose they could just choose not to use has() and manage that in JS. I guess in the case of nesting they could choose Sass nesting instead.

@LeaVerou wdyt? I can see both sides of this.

[SachaG]

Well, I've also had some feedback over the years along the lines of:

I hate Foo but I use it as part of my day job, so I had to pick "would use again"

[stubbornella]

Aww, I feel for that person. I have definitely made bad decisions about my stack that I was still living with years later.

[LeaVerou]

@stubbornella

2. What do do with the open ended text box (maybe it stays as it is now)

I've thought a bit more about this. I think I'm leaning towards leaving it on each question; if it's per section, it's likely that by the time they get to the bottom of the section they will have forgotten what they wanted to comment on.

However, I don't see any use cases for people to leave comments on entirely unanswered questions (@SachaG, does the data confirm this?), so it may be better to only show the button when an answer is selected, next to the answer. If the intent is "we provide a textbox for those inclined" rather than "we need to capture as many comments as possible", we should make it very subtle — e.g. by using the current icon rather than a whole "Tell us more" button. Possibly with a + inside the speech bubble on hover, and a nice descriptive tooltip.

1. Are users more comfy with the 5 option menu or 3 options with simple sentiment sub-menus

What are the criteria for deciding between the 3-option menu and the 5-option menu? I'll throw a few ideas out there... but please add more!

• implementation time
• survey filling success (completion rate and completion time)
• ???

Since we’re also discussing this below, I’ll reply there.

@SachaG

For State of HTML, I would argue that it's up to us to keep the number of features down if we do adopt the 5-option format.

But wouldn’t it be a shame to miss out on opportunities to ask about more features because of the answer format we picked? Especially since the topic here is so broad, so there are so many potential features we could ask about. I know you care a lot about the experience of the developers filling out the survey; something like this would reduce the entropy of the knowledge score they get, effectively reducing the value they get out of it.
I do think that with the right design we could have our cake and eat it too; but it’s not as simple as adopting a 5 answer format.

@SachaG

Oh another caveat – the issue with "would use again/would not use again" is that people who hate something but are forced to use it might still check "would use again".

&

@stubbornella

When I heard "would use again" on your previous surveys I always imagined you meant on a greenfield project. Like, if I were starting a brand new project, would I choose that tool again? "Liked it" somehow seems fluffy by comparison. Ex. I liked writing Rails, would i use it on a new project? Never.

I see both points here. I wonder if "want to use again" might bring us the best of both worlds: if I'm forced to use it, I won't select "want to use again" because I don’t actually want to, but it's less fluffy than "liked it". This may also alleviate some of the concerns @SachaG raised later:

The whole reason why I didn't go with the same 5-option format used by libraries for features is that in my mind, options like "used it, but would not use again" simply did not apply to feature like e.g. CSS Grid. I assumed that when it comes to feature (unlike libraries), it's more about "can I use it?" and "do I need it?" than "do I like it?". So if it turns out that sentiment is important for features as well, then maybe this is a good chance to test out my original assumptions by asking about it.

If I found CSS Grid hard to use, I may not want to use it again, even if I end up doing so because there’s no alternative. Of course I could be off and maybe respondents will entirely miss this nuance — what do you both think?

[SachaG]

I think at the end of the day it really depends what you want to measure. If you want to know how enthusiastic respondents are about a feature then "want to use again" is fine. But you can easily imagine a case where someone tries using has() or any other cool new CSS feature and has a horrible experience because of browser inconsistencies, yet still checks "want to use again" because they do want to use it again – once the issues are fixed.

So if what you want to capture is respondents' actual experience in the present, then imo it's better to phrase it as "has a positive experience"/"had a negative experience", or similar.

[LeaVerou]

Alright, I spent quite some time tonight iterating on the UI concepts I discussed here.
Here’s the interactive prototype (works in browsers that support color-mix()) that was used to record the videos below.

Note: In the following there are both sentiment followups, and subsequent followups about more specific issues. This is to show how each UI could work with more specific issues as well, but whether we include these or not is orthogonal to the UI we go with.

Concept 1: Follow-ups in selected answer with comment popup

Right now, I think this is the concept I’m leaning more towards — note that it's very similar to @SachaG’s, with a few changes:

• the sentiment labels are visible without a click and color-coded
• sentiment is separated from any other issues, which are shown only after sentiment is selected
• The comment is an overlay, minimizing UI jumpiness

in-answer.mp4

Placing the followups in close proximity to the selected answer establishes a nice flow and makes the logical relationship clear. OTOH, there’s a higher risk that respondents may think that they are "expected" to respond to these compared to designs where the followups and the comment button are at the bottom of the question. Hopefully user testing can answer whether this concern is founded.

If we go this route, it will need work to figure out how to avoid UI jumping on mobile (~26% of respondents are on mobile).

Concept 2: Modified Tags

One thing I realized during these explorations is that my original concept of responses acting as text macros does not work well with the sentiment followups. I think for the sentiment followups we definitely want structured data, and they also need to be mutually exclusive.

Here is a concept of the "responses as tags" idea, with a few tweaks:

• the sentiment followups are styled differently and are mutually exclusive
• tags about more specific issues are only shown after sentiment has been selected
• Selecting any followup opens the comment area to invite more context

tags.mp4

Concept 3: Hybrid Tags + Text macros

This is similar to Concept 2 above, but the tags that correspond to issues also insert text to act as a starting point (unedited text would be stripped out before saving):

hybrid.mp4

(This could also work for Concept 1)

[LeaVerou]

That being said I think we're getting too fancy here. We're asking people to parse the main question, then the follow-ups, then understand how to relate to each other, while navigating two visually different UI languages (radio buttons + regular buttons), all at the same time. It's becoming a big cognitive load imo.

I’m not sure what you mean by "fancy", but if you’re concerned about whether people can understand how to use the followups if they are styled like buttons, that is exactly the kind of thing that the user testing phase is for.

I also tend to stay away from any hover-based interactions because of mobile, personally.

It’s common to have input methods that are suited to each device. Clicks are more expensive on desktop, so it makes sense to minimize them, taps are cheaper on mobile, so mobile users can still fall back to the 2 step interaction.

[SachaG]

I admit I was not counting on the user testing phase, as I assume it will not be possible to A) implement the UIs we want to test; B) hold the user testing and C) implement the feedback gained from user testing; all before the August 25 deadline (given that August 11-16 is a holiday here in Japan; and I am also supposed to work on the State of CSS 2023 survey results in parallel to this).

So this is why I'd rather not take any big swings interaction-wise at this time, unless we are OK with relaxing that deadline to buy us more time to iterate.

[stubbornella]

@SachaG did you have any complaints from developers that they wanted to express neutral sentiment on the 5-option format for libraries? I would strongly prefer not to present them with 7-options.

[SachaG]

Yes, it's a complaint that does pop up from time to time ("I'm kinda interested in Foo but I don't want to learn it at this time, so what do I pick??"). But I think it's fine forcing people to pick one of two options, we can't cover all possibles scenarios anyway.

[LeaVerou]

unless we are OK with relaxing that deadline to buy us more time to iterate.

I checked with Kadir, we can relax it by a few days, both because we didn't know 11-16 is a holiday in Japan when we made the roadmap, but also because we're waiting for legal before we can commence the user testing.

I would strongly prefer not to present them with 7-options.

Indeed, presenting them with 7 options is extremely overwhelming. @stubbornella did you see the mockup in #163 (reply in thread) ? It’s still collecting the same data, but presented in a much more concise way.

[michaelquiapos]

Nicely done prototype @LeaVerou. Yet one other thing that's missing as a function, just like the current MC questions in the survey, is a way to Clear selection(s).
Screen recording 2023-08-11 7.45.38 AM.webm