Add new visualization for custom X-Y axes #4185

jeffreythewang · 2018-01-09T23:08:07Z

Summary

Add a new visualization, XY Line Chart - a line chart with custom X-Y axes
Add functionality for hiding lines between data points, to create a scatterplot-like chart

Description

I added a new visualization for plotting data points on a line chart which allows the user to specify any column or metric of the query as the X-Y axis. It is based entirely on the query fields of TableViz. This way, you can easily switch between the Table View and the XY Line Chart, since they have similar form data fields. I made no changes to the query objects, and primarily made changes to the resultant pandas DataFrame. Some features include:

Ability to specify multiple columns or metrics for the Y axis
Ability to slice by groups of columns or metrics (for multiple series)

Examples

Group-By Query (with multiple aggregations plotted on the Y-Axis)

Generic Column Selection (with slicing)

Scatter Plot

Related Issues

#3769
#3184

fabianmenges

LGTM

fabianmenges · 2018-01-16T14:34:09Z

@mistercrunch @williaster any thoughts?

mistercrunch · 2018-01-17T22:12:04Z

superset/viz.py

+    viz_type = 'line_xy'
+    verbose_name = _('XY Line Chart')
+    sort_series = False
+    is_timeseries = True


Shouldn't this be False ?

This is primarily used as a small hack to implement limiting on the groupBy groupings, using the limit as the timeseries_limit in the query objects. The limit is applied on the subquery in the case of sqla, and the first phase in the case of the druid query.

The reason why limiting should be applied is so that, in some cases, the visualization doesn't come out as a large number of lines, which would crash the browser.

I realize it probably makes more sense to limit the result set after running the query, limiting on the sliceBy field, but I'm open to suggestions here. We can also forego implementing limiting for now, and encourage users to apply proper filters on their queries. Otherwise I'll leave it the way it is now and include a comment in the code.

Just to note - I decided to go with the latter solution and forego implementing limiting.

mistercrunch

Let's have a conversation as to whether supporting "NOT GROUPED BY" for this specific visualization is necessary. I'd rather keep things simple here and only allow "group by" on the X axis

mistercrunch · 2018-01-18T17:07:08Z

superset/assets/javascripts/explore/stores/controls.jsx

+    description: t('Columns or metrics to display'),
+    mapStateToProps: (state) => {
+      let choices = [];
+      if (state.controls && state.datasource) {


This looks like the exact same function as in columns_and_metrics_x, let's make this DRY

I like the idea of being able to plot non grouped-by measures. In some cases I feel that this is a missing option (e.g., box plot).

The non-grouped xy-plot would be even more useful if there were an option to just plot markers without any lines. In other words, a plain scatter plot.

Sorry, I must have mussed that the scatter plot is implemented here, already.
So, I would highly appreciate if the non-grouped code could just stay...

Trying to understand your NOT GROUPED BY use case, is it for optimization purposes in cases where the data is already aggregated? Note that you can still make columns that are used in metrics "groupbable" and group by them. Also note that the cost of grouping something that's already grouped should be fairly cheap.

I think having the two interfaces in one makes things confusing. For the case of the Table viz it was very necessary to allow NOT GROUPED BY, and I'm trying to understand what use case requires that in this context.

We, for example, need it to show raw values of our sensor products. So each data point represents one serial number.
You might say that grouping by SN should just do the job. However, grouping by, e.g., the job number, is then more useful.

In other cases there exists real xy measurement data for a product, like e.g. a transfer characteristic, and this again is not useful to be aggregated but has to be investigated for each SN separately.

Limiting the amount of data may of course be challenging then, but this is the user's responsibility in my eyes.

@mistercrunch The 'NOT GROUPED BY' use case is the same one why it is available on the table visualization. Fundamentally, this general X-Y chart lets you select any column of a table-visualization as the X axis or the Y axis. In short, if you have two numeric columns in a table you should be able to display the data as a line chart regardless of what the 'query' looked like.
The reason for the 'NOT GROUPED BY' section to be included in the table viz is to be able to visualize (list) non-aggregated data.

👍 not grouped by

What about as an alternative to the GROUPED BY / NOT GROUPED BY, if we allowed for a mix of metrics and dimensions in the dropdowns? If no metrics are selected, then we don't aggregate. Perhaps that's the way that the table view should work as well.

We're planning a fair amount of work to have a more column-centric approach in the explore view and supporting backwards compatibility on this new chart type may make that harder.

mistercrunch · 2018-01-18T17:51:23Z

superset/assets/javascripts/explore/stores/visTypes.js

+        ],
+      },
+      {
+        label: t('NOT GROUPED BY'),


Handling NOT GROUPED BY here adds a lot of complexity to this visualization, why is it necessary?

Just to note - I included NOT GROUPED BY to address the feature @rumbin mentioned above.

raghavmishra1202 · 2018-01-25T15:55:44Z

Is there anyway to make months (Jan-Dec) as x axis.
I tried to add a new column where numbers 1-12 correlated with months. It worked in terms of csv file but when I tried to visualize it, the order on the graph became either alphabetical or if i included the numbers + string as so 1, Jan 10 Oct 11 Nov 12 Dec 2 Feb 3 March etc.
Any help would be appreciated!

jeffreythewang · 2018-01-25T16:38:46Z

@raghavmishra1202 Correct me if I'm wrong, but it sounds like what you are trying to visualize is best done with a time series chart with month granularity.

If instead, you are trying to plot specific month-integer values (1-12, and not in a timestamp-like field) on the X-Axis, and want them displayed from a mapping of numbers to month-string, that seems like a display-specific change that can be implemented in in the X Axis Format section.

If we want to include similar functionality in the XY Line Chart (granularities for things other than dates), we would have to support bucketing values into ranges, which is not included in this change set.

raghavmishra1202 · 2018-01-26T15:31:06Z

@jeffreythewang thanks for your reply. Unfortunately that doesn't work in my case. the data i have is organized by basically months in numeric form in one column and data in the other. If i do the time series chart i get some errors. If i do a normal plot it shows the months as 1-12 on the x axis with corresponding data. I basically group the string version of month with numeric but now the organization of the axis gets messed up. it's weird cause the csv shows the correct order
month string_month data
1 jan 5
2 feb 3
3 mar 9

any suggestions? like i mentioned above either it arranges x axis as
1, Jan 10, Oct 11 Nov etc. --> if i can get this fixed that would be perf
or with just strings it does alphabetical

raghavmishra1202 · 2018-01-26T15:47:33Z

Also is there a way to download all the dashboard data into a single csv/zip folder?

jeffreythewang · 2018-01-30T23:30:52Z

@raghavmishra1202 The way that the XY Line Chart works under the hood is that it orders the result set of the query based on whichever column/metric is selected to be plotted on the X-axis. This way, the line(s) that is/are plotted will have continuity in the series. In other words, the dots are connected in the order that you expect to be connected. For the best visualization, quantitative columns/metrics should be plotted on the X or Y axis, and qualitative fields should go in the sliceBy part of the visualization.

For your specific use case, in my opinion, there are a few ways to go about it:

Encode your month field into a DATETIME-like column, mark it as temporal in the column edit view, and then plot it as a timeseries visualization with month granularity.
Plot the month integer on the X-axis, and label the X-axis as "Month (numerical)" or something descriptive.
Extend the X Axis Format to format numerical month-integers as string month-values.

Another solution is to allow users to specify their own ordering for the result set, but I'm not sure how useful or intuitive that would be. I'm open to suggestions here.

Also re: downloading dashboard data into a single csv/zip - As far as I know, it does not seem like it, and you'll have to export individual charts as CSV. It can be filed as a feature request though.

jeffreythewang

Note, there is an issue with rendering the XY Line Chart when a temporal column is not present, so please do not merge right now. I will update the PR when I push the fix.

UPDATE: This issue is now fixed.

williaster · 2018-02-04T01:45:48Z

curious why this needs to be a new visualization type vs just exposing more customization options (eg not grouped by, no line) in the existing time series line chart, thus making it more general and not a time series chart.

if this is a superset of functionality, why not just deprecate/migrate the line series instead of cramming more stuff in? this is more consistent with other charting frameworks where you just choose x and y and there's no assumption that x is a specifc type (eg time)

fabianmenges · 2018-02-05T16:43:57Z

@williaster there are a few reasons.

Since we want to be able to display data that is not necessarily a time series or aggregated, the controls of the Table View are much more appropriate.
To make the new "X-Y Line Chart" viz type a true 'superset' of the "Time Series - Line Chart" would require a lot of work and major refactoring. How does the 'Advance Analytics' from a "Time Series - Line Chart" work for non time series data, rolling, resampling, time shift, etc... How do Annotations work for non Time Series data?
The controls of the "X-Y Line Chart" are actually more complex than the Table or "Time Series - Line Chart" controls. You have do define your query ("GROUP BY" or "NOT GROUPED BY") and independently of that use the "Axis Options" section to pick which data should be displayed on the x-axis and which data should be displayed on the y-axis.

williaster · 2018-02-06T00:48:53Z

Since we want to be able to display data that is not necessarily a time series or aggregated, the controls of the Table View are much more appropriate.

That makes sense in general, but doesn't seem like it makes the case for a new chart type to me. It seems like matching xy-chart + table visualization controls could be a good update because it is more general.

To make the new "X-Y Line Chart" viz type a true 'superset' of the "Time Series - Line Chart" would require a lot of work and major refactoring. How does the 'Advance Analytics' from a "Time Series - Line Chart" work for non time series data, rolling, resampling, time shift, etc... How do Annotations work for non Time Series data?

We know which columns are time columns right? It seems like we could just leverage that information to toggle on / off advanced analytics + annotations when the x-axis is time? there is precedent with showing/hiding controls in your annotations work and in the future annotations could be updated to support non-time xy-charts.

while this might be considered more work, it seems like less work than a future refactor of consolidating visualizations even further if we want to move to a model where we have a generic xy chart? (which again, is extremely prevalent in other plotting frameworks) perhaps I'm wrong but I really just think there's a lot of overlap here with the existing line chart and instead of bloating the # of visualizations it seems like this would just be a good generalization to make.

mistercrunch · 2018-02-06T05:26:53Z

@williaster I was thinking that it should be different chart types as there are many controls that should become inactive when out of the time series mode. I'd say that one of the guidelines for creating new chart type as opposed to overloading an existing one is around the % of shared/relevant controls.

We currently don't have support for conditional form controls, meaning that selecting certain option in one control would affect the existence of other controls or control-sections. Until we have this I think it should be 2 different chart types.

I'm more troubled by the GROUPED BY vs NOT GROUPED BY options. When I added that to the Table viz early on in the project, I never thought it should be replicated to other chart types. The control panel section is bloated/confusing enough as it is without having 2 query-related sections. I think the short term solution here would be to have dropdowns with a mix of columns and metrics where we want to enable that. I think the longer term solution is to have a column-centric approach that allows (but doesn't always require) applying aggregate functions to the column. An example of that is that the clicks can be dragged to a Metrics section, and becomes clicks [sum] by default where an aggregation is required. You can then change it to other aggregations like clicks [avg] using a contextual popover. If the context doesn't require the column to be aggregated than it would allow turning off aggregations.

williaster · 2018-02-06T05:45:01Z

@mistercrunch I agree with all that, esp about avoiding not grouped by and worsening the data selection process 🙌most users I've spoken with (technical and non-technical) don't understand that not grouped by means atomic rows, even @john-bodley and I.

long-term having selectors be aware of aggregation sounds gr8, and we can consolidate vis's then.

sebastianbk · 2018-02-16T10:45:37Z

Hey guys. What's the progress on this? Any chance it will make the 0.23 release? What love to work with this visualization! 😃

mistercrunch · 2018-02-19T00:27:27Z

Won't make 0.23.0 for sure as the RC has been cut. Only bug fixes will make 0.23.0 from this point.

Rename vars Small fixes for XY charts on table sources XY plots work on druid datasources, not refactored Don't touch the query models, redo the UI spread syntax apply linter

BestOwed · 2018-10-11T08:14:09Z

Hey guys. What's the progress on this? What love to work with this visualization for me too! 😃

8849511588 · 2018-10-27T14:37:40Z

No need

Zhihuzhe-ye · 2018-12-27T03:49:35Z

@jeffreythewang hello , This branch 'Add new visualization for custom X-Y axes ' , Has it merged with the main branch ？ The x-axis need not necessarily be a time/date, it could just be any number. Would be great to make that possible! Can I use it?

weiqiLee · 2019-01-11T15:52:51Z

Hello everybody, what's going on? this visualization is what I want too 😃

htruongaff · 2019-01-20T15:40:35Z

Any update on this? this seems to be one of the main pain points for many folks.

jeffreythewang · 2019-01-22T14:53:47Z

@mistercrunch if we're committed to potentially merging this in, I can rebase

williaster · 2019-01-22T17:12:01Z

as we refactor all charts to use a consistent API and move them to @superset-ui as chart plugins (which users can then pick/choose from at their will), it'd be helpful for us to not add additional charts which will then need migration.

@jeffreythewang if we help with documentation for how to create such a plugin / do this migration now, would you be open to making this a chart plugin that lives outside the superset repository?

cc @kristw @xtinec @conglei

jeffreythewang · 2019-01-22T19:13:34Z

@williaster Yeah sounds good to me

stale · 2019-04-10T08:59:01Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

sumedhsakdeo · 2019-05-16T17:51:24Z

Curious if this made it into superset as a chart plugin @jeffreythewang
Can you please point me to the PR?

fabianmenges reviewed Jan 16, 2018

View reviewed changes

fabianmenges approved these changes Jan 16, 2018

View reviewed changes

mistercrunch reviewed Jan 17, 2018

View reviewed changes

mistercrunch requested changes Jan 18, 2018

View reviewed changes

jeffreythewang force-pushed the jeffreyw/custom-xy branch from 54df860 to 1f54180 Compare January 24, 2018 18:09

jeffreythewang commented Feb 2, 2018

View reviewed changes

jeffreythewang force-pushed the jeffreyw/custom-xy branch from fc2aa08 to 1f2e61c Compare February 5, 2018 18:06

jeffreythewang and others added 8 commits March 20, 2018 19:28

Add new visualization for custom x-y axis

2b9fdd5

Rename vars Small fixes for XY charts on table sources XY plots work on druid datasources, not refactored Don't touch the query models, redo the UI spread syntax apply linter

misc linter fixes

6cbeb70

Add viz thumbnail for line_xy

8f3711e

Change show lines to hide lines

fb67f82

Include range filter chart option

1311954

Validate that state control fields are defined

2eeb502

Remove limiting from xy chart, expose chart to SQL Lab

8057798

Update to new tabbed-styling

77587ea

jeffreythewang force-pushed the jeffreyw/custom-xy branch from 1f2e61c to 77587ea Compare March 20, 2018 23:37

fabianmenges mentioned this pull request Oct 30, 2018

Feature Request :- Line chart where X axis is not date but an integer/something else #3769

Closed

kristw added the enhancement:request Enhancement request submitted by anyone from the community label Jan 23, 2019

stale bot added the inactive Inactive for >= 30 days label Apr 10, 2019

stale bot closed this Apr 17, 2019

aspedrosa mentioned this pull request May 1, 2020

Create line chart without datetime axis EHDEN/NetworkDashboards#9

Closed

carstenkolve mentioned this pull request Jul 27, 2020

[SIP-50] Proposal for using ECharts as our main charting library #10418

Closed

Add new visualization for custom X-Y axes #4185

Add new visualization for custom X-Y axes #4185

Conversation

jeffreythewang commented Jan 9, 2018 • edited Loading

Summary

Description

Examples

Group-By Query (with multiple aggregations plotted on the Y-Axis)

Generic Column Selection (with slicing)

Scatter Plot

Related Issues

fabianmenges left a comment

Choose a reason for hiding this comment

fabianmenges commented Jan 16, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mistercrunch left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

raghavmishra1202 commented Jan 25, 2018

jeffreythewang commented Jan 25, 2018

raghavmishra1202 commented Jan 26, 2018

raghavmishra1202 commented Jan 26, 2018

jeffreythewang commented Jan 30, 2018

jeffreythewang left a comment • edited Loading

Choose a reason for hiding this comment

williaster commented Feb 4, 2018 • edited Loading

fabianmenges commented Feb 5, 2018

williaster commented Feb 6, 2018

mistercrunch commented Feb 6, 2018

williaster commented Feb 6, 2018

sebastianbk commented Feb 16, 2018

mistercrunch commented Feb 19, 2018

BestOwed commented Oct 11, 2018

8849511588 commented Oct 27, 2018

Zhihuzhe-ye commented Dec 27, 2018

weiqiLee commented Jan 11, 2019

htruongaff commented Jan 20, 2019

jeffreythewang commented Jan 22, 2019

williaster commented Jan 22, 2019

jeffreythewang commented Jan 22, 2019

stale bot commented Apr 10, 2019

sumedhsakdeo commented May 16, 2019

jeffreythewang commented Jan 9, 2018 •

edited

Loading

jeffreythewang left a comment •

edited

Loading

williaster commented Feb 4, 2018 •

edited

Loading