httpie changing the json fields order in the output

[altfatterz]

Wondering how can I force httpie to not change the json fields order?

curl -i http://localhost:8080/v1/notes/569766aed4c661fba8d85a12

{
  "id": "569766aed4c661fba8d85a12",
  "content": "hi"
}

with httpie

http get :8080/v1/notes/569766aed4c661fba8d85a12 

{
  "content": "hi",
  "id": "569766aed4c661fba8d85a12"
}

I prefer the id field to be always the first. Any thoughts?

[elayeek]

Note that in the json formatter, sort_keys=True
One can assume that this would be the reason

[altfatterz]

ah ok, thanks.

With the following I could disable sorting the keys (unfortunately together with the indentation, but that is not that big of a problem)

http --pretty=colors get :8080/v1/notes/569766aed4c661fba8d85a12

[elayeek]

You're welcome, although I feel that this does raise the question of wether the sorting is something that should be done in general or should it be as server intended

[altfatterz]

would it be possible to introduce another value for --pretty in oder to allow colors and formatting but without sorting the keys in the response?

[c0b]

could you make unordered as default? and an option like --sort-keys for people who want the sort keys behavior; see https://bugs.python.org/issue21650 the json.tool has an option in python3 already

➸ python3 -m json.tool -h
usage: python -m json.tool [-h] [--sort-keys] [infile] [outfile]

A simple command line interface for json module to validate and pretty-print
JSON objects.

positional arguments:
  infile       a JSON file to be validated or pretty-printed
  outfile      write the output of infile to outfile

optional arguments:
  -h, --help   show this help message and exit
  --sort-keys  sort the output of dictionaries alphabetically by key

[carlfish]

I just lost way more time than I'm comfortable admitting trying to track down a problem in my server-side JSON library because I couldn't work out why it was sending the data out in the wrong order. It didn't even occur to me that the client could be reordering stuff.

Is it even worthwhile having the option to reorder JSON? 90% of the time it's going to obfuscate rather than improve server output.

I'd submit a pull request, but it would just be changing a "True" to a "False" in one file. :) (see below)

[altfatterz]

fully agree with @carlfish

[altfatterz]

@jkbrzt any thoughts on this?

[carlfish]

Tested this locally, and it looks like if you tell the formatter not to alphabetize, you get object keys in an arbitrary order instead - I guess it's backed by an unordered dictionary? In that case, alphabetization is probably better than random.

[c0b]

I guess it's backed by an unordered dictionary?

yes, the python json.loads by default loaded into a dict, which is unpredictable order (somewhat internally used hash code), that is not better than alphabetical order;

but there is workaround, please use object_pairs_hook=OrderedDict; and drop sort_keys when calling json.dumps

• http://stackoverflow.com/a/6921760/6659271
• https://docs.python.org/2/library/json.html

>>> import json
>>> from collections import OrderedDict
>>> data = json.loads('{"foo":1, "bar": 2}', object_pairs_hook=OrderedDict)
>>> print json.dumps(data, indent=4)
{
    "foo": 1,
    "bar": 2
}
>>> 

[carlfish]

Two shaved yaks later: Pull request -> #520

[malikoth]

For what it's worth, I happen to be a user that likes seeing the keys in my JSON output sorted. My server doesn't define an order that it outputs the keys in, and if I'm looking to see if the key I expect was included in the JSON body or not, it's much easier when they're sorted. So I'd say not to just arbitrarily remove the sorting functionality, but rather make it configurable or toggleable via flag.

[c0b]

ping author @jakubroztocil is this out of maintenance?

[Baekalfen]

Is there any traction on this issue? Like @carlfish, I have just spend an embarrassingly long amount of time trying to correct a bug in my server, only to find, that httpie was the issue.

It seems very unintuitive that it would reorder/sort data from the server, without the user explicitly enabling it.

[lapo-luchini]

There is a solution in PR #520, which unfortunately has not yet been merged.

[nmtitov]

This would be really useful. Sorting is not that good when you don't want it.

[c0b]

the whole Python2 deprecated; some maintainer can take a look of the pending PRs? from contributor graphs it looks like @jakubroztocil @msabramo still active?

[opensas]

Is anybody working on this? Can we expect this to be solved?

[nmtitov]

@opensas don't know about that but possible solution is to use jq tool:

http https://jsonplaceholder.typicode.com/todos/1 | jq -C

[jkbrzt]

@opensas this feature will be included in the upcoming v2.2.0.

---

possible solution is to use jq tool:

http https://jsonplaceholder.typicode.com/todos/1 | jq -C

@nmtitov here it’s not actually jq that restores the order, it’s the act of redirecting the output which turns off colors and formatting (effectively setting --pretty=none).

[opensas]

thanks a lot for the tip, already noticed it in this thread.
Is there some way I can keep the rest of the request info (status & headers) after piping thru jq?

I mean, httpie returns this:

HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 109
Content-Type: application/json; charset=utf-8
Date: Sat, 25 Apr 2020 11:14:32 GMT
ETag: W/"6d-wWZh31xOzPgYyzU23ihgZaW8KkI"
Strict-Transport-Security: max-age=15552000; includeSubDomains
X-Content-Type-Options: nosniff
X-DNS-Prefetch-Control: off
X-Download-Options: noopen
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block

[
    {
        "id": 1,
        "text": "Read the docs"
    },
    {
        "id": 2,
        "text": "Create my first application"
    },
    {
        "id": 3,
        "text": "Write tests"
    }
]

but http | jq -C strips the first part, returning only:

[
  {
    "id": 1,
    "text": "Read the docs"
  },
  {
    "id": 2,
    "text": "Create my first application"
  },
  {
    "id": 3,
    "text": "Write tests"
  }
]

[nmtitov]

@opensas I don’t think it’s possible unless you write your custom bash wrapper

[jkbrzt]

@opensas @nmtitov piping the output has the side-effect of changing the default --print=hb (print headers and body) to --print=b (print just the body because that’s what you typically want when redirecting the output). You can explicitly ask for the headers to be included with --print=hb.

https://httpie.org/docs#output-options

[nmtitov]

@jakubroztocil this won't work because jq expects JSON body as an input

$ http --print=hb https://jsonplaceholder.typicode.com/todos/1 | jq -C
parse error: Invalid numeric literal at line 1, column 9

[jkbrzt]

I see. You could use http --download httpbin.org/get | jq

[opensas]

I found the following workqround, instead of using httpie I found curlie, which seems to

Like curl but unlike httpie, headers are written on stderr instead of stdout.

So I can use it like this:

$ curlie GET localhost:3000/tasks | jq -C
HTTP/1.1 200 OK
X-Powered-By: Express
Content-Type: application/json; charset=utf-8
Content-Length: 304
ETag: W/"130-ED1W4hQo1i7na7wy5Ewc7iKdoJc"
Date: Wed, 27 May 2020 06:28:26 GMT
Connection: keep-alive

[
  {
    "id": 2,
    "title": "new task2",
    "description": "description2",
    "status": "OPEN"
  },
  {
    "id": 3,
    "title": "new task3",
    "description": "description3",
    "status": "OPEN"
  }
]

Where as with httpie I would the headers

BTW, I created this convenient script:

$ cat ~/bin/c
curlie "$@" | jq -C

[jkbrzt]

Just released v2.2.0 that addresses this issue. Learn about about the new --unsorted, --sorted and --format-options here:

https://httpie.org/docs#colors-and-formatting