Gap analysis and pain points (Which elements are missing or unusable?)

[LeaVerou]

(Aug 10 update)

The more I think about this, the more it seems we need multiple separate questions:

Which existing HTML elements are you unable to use?
These are elements that you need, but even though they exist natively, you have to use libraries of custom components to satisfy your (functional, styling etc) needs. Enter e.g. <foo> for entire elements, or <foo bar="yolo"> for attributes of elements.

UI could be a list of tag names and freeform pain points. Wireframe:

Do you use any component technologies?

• Web Components
• VueJS
• React
• Lit
• Angular
• Svelte
• Solid.js
• Other

Which components do you find yourself using across multiple projects?
You can just type tag names, we’ll figure it out!

UI could be similar to the pain points above, except the second column would say "Optional details"

If you could add 3 elements to HTML, which of the following would you add?

• Multiple choice, with ability to add custom options.
• Preselected options would be selected based on elements found frequently in frameworks and/or considered for standardization (but too early to ask about in Part 1 of the survey)
• If there’s time, each option could also include a small picture, to make the purpose clear
• This needs to come last, to avoid priming effects affecting the other questions

ETA: Current draft here: https://coda.io/@leaverou/state-of-html/missing-features-and-pain-points-5

---

One of the big goals of these surveys is to identify gaps in current web platform technologies and pain points when using existing features.

In terms of HTML, this could mean:

1. What elements are needed very frequently, yet not available natively?
2. What elements are available as native HTML elements, but do not actually serve the authors use cases so they still have to reach for custom ones?

Option 1: What components have you used across 3+ projects?

This would span components regardless of ecosystem (Web Components, React, Vue, Angular etc). The "3+ projects" is to eliminate components that are used for purpose-specific templating use cases, and leave those that are general purpose elements.

The upside of phrasing it in terms of components is that for most developers, that makes it more concrete: they could even look at their projects to jog their memory. However, there are a few downsides:

1. how do people who don't use components have a say?
2. a suitable component may not exist to satisfy their needs, but they should still be able to express these needs

Option 2: What HTML elements are missing or are unusable?

With a note like: "If you use components, a good starting point would be to think about components you’ve used across 3 or more projects. This could also include elements that are available, but you still use custom ones over the built-in versions."

[Gumnos]

My biggest missing-element point of pain has been a lack of a native tab-set/tab-group markup that adheres to system conventions and accessibility. It's a common element—even as I type this, GitHub offers me a "Write" & "Preview" tab for this input box. But there's no standard-defined markup to say "this is a group of tabs, this is a tab, these are the labels for each tab." While many component libraries offer bolt-on options, accessibility (whether keyboard access or presentation in the accessibility-tree) and responsiveness to screen-size/orientation are regularly overlooked.

[LeaVerou]

Hi there, this sounds like a great answer for the question we're talking about, but I'm not sure what to take away from it in terms of designing the question itself. Do you happen to have any thoughts on that?

[SachaG]

Discussing "Which components do you find yourself using across multiple projects?" and "If you could add 3 elements to HTML, which would you add?"

"Which components do you find yourself using across multiple projects?" is both unnecessarily specific and too vague. First of all, why are we disqualifying people who only work on a single project? I'm sure there's many developers who have worked on the same app for 10+ years, I don't see the rationale for excluding them from this question.

Also, if I use a tab component from Radix on Project A and a tab component from Material UI on Project B does that count? What if I reimplement tabs from scratch myself for every new project in a slightly different, ad-hoc way? Am I "using the component across multiple projects" or not?

What we really want to do is identify the components (or UI elements if you prefer) for which people need to go beyond what plain HTML offers, so we should ask about that more directly.

There's two distinct scenarios I think, existing components (<select>) that people reimplement because they're too limited, and components that simply don't exist (tabs).

If we want to make sure we capture data about both, we could split the question in two:

• A. "Which of these existing HTML elements do you most often need to augment or reimplement using JavaScript? (pick 5)"
  • Populate the list with the top components from Bootstrap, Radix that already exist in HTML:
    • button
    • checkbox
    • textfield
    • textarea
    • select
    • radio buttons
    • slider
    • table
    • img
• B. "Which of these other UI elements do you most often use in your projects? (pick 5)"
  • Populate the list with the top components from Bootstrap, Radix, etc. excluding list A:
    • accordion
    • alert
    • calendar/date picker
    • carousel
    • combobox/autocomplete
    • command palette
    • context (right click) menu
    • data table
    • dialog
    • dropdown menu
    • progress bar
    • skeleton/loading placeholder
    • switch
    • tabs
    • toast message
    • tooltip
    • offcanvas
    • spinner/loading indicator
    • button group/segmented button
    • scroll area
    • icon
    • star rating

Note: I'm sure there's some items that are in list B that do exist, and should be in A instead

This way both are factual questions; and people always have the "other…" field to give us more freeform data.

The former doesn't prime, the latter primes first

If priming is a no-no here, we can make one or both freeform, although as usual that will mean a decreased response rate and more analysis work afterwards.

The current options are inspired from discussions with browser teams.

I'll work on a new list. I think for example Menu is way too vague (hamburger menu? dropdown menu? foobar menu?) while Product Image Carousel is way too specific (I can't imagine browsers implementing something that high-level…? Am I wrong?).

cc @stubbornella I think this is probably one of the questions you care (most) about?

[atopal]

The way the first question now appears together with the explainer "(due to e.g. browser incompatibilities, limited styling, different functionality...)", I'm worried it might be a double barreled question. How would we know afterwards why they can't use it, whether it's an interop issue or a styling issues? Maybe from context and knowing about the features themselves?

[LeaVerou]

What about adding guidance to mention what the issue is?

[LeaVerou]

(though it would be far better to be able to do this with UI…)

[atopal]

Yeah, maybe that's an option?

[LeaVerou]

Which one?

[SachaG]

@atopal did you see this? #204 (reply in thread)

I just want to be clear that our regex-based parsing will not be able to analyze something as fluid as a reason written in a freeform text field. So even if respondents write in their reason, @ShaineRosewel or somebody else would that to take care of that analysis.

[SachaG]

This is the current state of this question. There are a couple factors that would make me strongly prefer returning to a predefined checkbox format.

Issues

• Any type of freeform question is going to produce less accurate data and require more work to normalize freeform responses to canonical IDs. As such, we should not be quick to resort to them when there's an alternative.
• The downside of having checkboxes is the priming effect that favors the answers in the checkbox against custom "other…" answers. But this description also primes the respondent, which virtually guarantees that the data will be biased towards the items mentioned in the description.
• The description is also relatively long and hard to parse.

Reason for using Freeform Responses

The stated reason for why freeform responses are preferable to predefined options is:

we already know that [the items mentioned in the predefined options] are missing, we want to see if there are any blind spots.

With the idea that by not having any predefined options, we encourage more diversity in the answers and get higher respondent counts for those "blind spots".

I would argue though that:

1. The vast majority of respondents will fill in items that would've been included in the predefined list anyway.
2. I'm not sure how useful any data about "blind spots" or outliers will be anyway, since by definition those respondent counts will be rather small*, and comparable with the numbers we'd get with the predefined options + "other…" format.

*I would predict that any item not in the predefined list will not get more than 30 mentions.

Solution 1

My preferred solution would be returning to a predefined checkbox + other… option format.

Solution 2

Alternatively, if we keep the freeform list format I think we should shorten/simplify the description and remove the list of items to avoid any priming effect.

Feedback

@stubbornella @ShaineRosewel would love your feedback on this. I would especially like to know whether the goal of this question is to prioritize which elements browser vendors should work on next; or identify elements that are outside browser vendors' current awareness.

[LeaVerou]

Well, my assumption is that ~90% of the results we get will be covered by the list above, no matter if we pick checkboxes or a freeform textfield. But I could be wrong of course.

I think this assumption the root of the disagreement. I think it absolutely would make a difference in the relative proportions. Different UIs encourage different types of interactions. I'm not saying that with the freeform list people would not pick from the suggestions, but I think they're definitely more likely to enter at least one option not in the list.

If you're up for it, it would be a cool experiment to do a deployment A/B test: some users get one template, other users another, then you can see how it affects the proportion of custom to predefined answers. The benefits are twofold a) we'll get both kinds of data, just with half the responses b) you can compare them and get insights you can use in future surveys

All that being said, we could also split out the "other…" option into a wholly separate question (which is what we used to do before).

That could work (though we should reduce the number of selections to "up to 3"), though we have no way to see which ones are more important to the participant.

Finally if the key issue is that the "other…" field in checkboxes is not visible enough, we could easily make it more visible, for example make it always visible by default (unlike the current UI where you need to click the "Other" checkbox for the textfield to appear). Would that work?

A universally more prominent "Other" option is not necessarily appropriate for every question. For some questions, custom answers truly are outliers, and that's fine.

Also, that won't help in cases where we're hoping to get more than one custom answer.

[SachaG]

If you're up for it, it would be a cool experiment to do a deployment A/B test: some users get one template, other users another, then you can see how it affects the proportion of custom to predefined answers. The benefits are twofold a) we'll get both kinds of data, just with half the responses b) you can compare them and get insights you can use in future surveys

It's something to think about, I don't think I'd have time to implement that for this year though.

[SachaG]

Oh another reason why I would be in favor of predefined list is that many respondents probably don't know what already exists in HTML and what doesn't. So with freeform lists, we'll have items in list 1 that should be in list 2, and the other way around, leading to more confusing survey results for readers ("Datepicker is the number three element people would like to be in HTML? I thought it already existed, but I guess not?")

[ShaineRosewel]

Well, my assumption is that ~90% of the results we get will be covered by the list above, no matter if we pick checkboxes or a freeform textfield. But I could be wrong of course

I think this works if we have say, at least one prior research reference to back that up - so we can justify why we listed those specifically.

[ShaineRosewel]

Another option is for us to explore the results. Maybe if we find that most of the answers fall into the list used when priming, we can present that as it is. Then also try to create a separate presentation to see if anything outside the list stood out.