-
Notifications
You must be signed in to change notification settings - Fork 17.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
proposal: spec: add read-only slices and maps as function arguments #20443
Comments
This has been discussed before, and such big changes to the language won't be made in 1.x. I believe there were proposals to also make |
@rsc's previous evaluation of read-only slices: https://docs.google.com/document/d/1-NzIYu0qnnsshMBpMPmuO21qd8unlimHgKjRD9qwp2A/edit |
Russ Cox's evaluation applies to the second part of my possible implementation options. While I welcome full-blown read-only types, I do agree that it is a complex issue that must be thoroughly analyzed. However, what I originally required is nothing revolutionary. See the first implementation option. It can be done in Go 1.x now, but it requires boilerplate for each type. See illustration. It would be nice to have a keyword to do automatic shallow-copying for slice and map for all types without the boilerplates. The objective is to protect the original variables against accidental changes. After slicing and re-slicing and several functions later, it was crazy to track all those mutated slices to find who did what that caused the bug. |
I think this is a great idea, but I don't like the idea of shallow copying. There's nothing stopping you from creating your own types over a slice that do this, and that way at least it's very transparent to you and consumers that they're actually getting copies of data. I'd love to see go support read only types that are enforced at compile time. Slices in particular - simply have a compile error if someone tries to invoke code that would involve assigning to a slice that's const or read only. I understand it'd be a bit more involved than that, but it'd be very helpful. |
Go's slices are way too powerful and flexible that people rarely have to deal with arrays directly any more. In my opinion, a slice should be a read-only view into an array. So if people want to change the content of an array, they would need to deal with the array and not its slice. If we are going into that direction, then there is no need for new keywords to indicate immutability. |
Appending is a very common operation, how do you propose that append would with your read only slice proposal?
… On 17 Jun 2017, at 21:58, henryas ***@***.***> wrote:
Go's slices are way too powerful and flexible that people rarely have to deal with arrays directly any more. In my opinion, a slice should be a read-only view into an array. So if people want to change the content of an array, they would need to deal with the array and not its slice.
If we are going into that direction, then there is no need for new keywords to indicate immutability.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.
|
I do not agree that slices should become exclusively read only. Being able to modify slices is an attractive feature, and changing that would be a very radical breaking change. Appending to a read only slice should throw a compiler error, like any other operation that involves assignment |
How about this? You can still work with slices as usual. You can slice, re-slice, append, make it point to different values, etc. However, regardless what you do to the slice, it won't change the backing arrays and slices. You can think of a slice like a pointer. You can make it point to different values at any time, but in this case you can't change the value you are pointing at. |
What is the goal? Better documentation, or race free code? |
I agree that immutability would be great, but I think that's a little
different from read-only.
For instance, you could make copy-on-write slices that are immutable - any
reference to them points to a structure that doesn't change. That could
certainly be powerful and useful, but a bit different from what I
understand this proposal to be, which is a bit narrower in scope: simply to
have compile time checking that a slice can't be written to if passed as
readonly or created as readonly.
…On Sat, Jun 17, 2017 at 1:25 PM, leiser1960 ***@***.***> wrote:
What is the goal? Better documentation, or race free code?
Your proposal goes for the first, I assume.
You give the the guarantee that your function will have no way to alter
the underlaying array.
But you do not get the guarantee that your function is race free in
respect of changes to this array, because other goroutines may modify the
underlaying array concurrently.
Adding immutability to golang should go for both goals, i.e. full value
semantics as we have it for strings. This changes assignment as well as
comparison semantics. But would help significantly reasoning about non
sequential program behaviour.
Adding a simple form of immutibility to the type system of go is big
challenge, both for language design and implementation. But please go for
it!
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#20443 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AIOKxX3FGb5iAbgyXz0uLPvBJWLDlPZkks5sFAwagaJpZM4NhrYK>
.
|
Const and immutability are different. Obvious to me.
The idea to allow an implementation to even modify the
const parameter having but still maintaining the "const"
guarantee to the caller is very interesting.
I simply asked for more.
Why? The spirit of go is about simplicity and consistency,
not about ease of implementation and piece meal.
So const should work for slices maps all types (perhaps except channels, const channels seem useless to me).
But I support the proposal for more control over mutability,
because this is the number one problem when you have concurrency in your program.
And the more support you get from the language, compiler, race checker... the better it gets for the programmer.
COW slice implementations are well know, go strings are immutable,
I know about COW versions of balanced trees, not shure about hash maps. But your proposal should also be extensible to maps as well.
Am 17. Juni 2017 19:32:43 MESZ schrieb Dan Field <[email protected]>:
…I agree that immutability would be great, but I think that's a little
different from read-only.
For instance, you could make copy-on-write slices that are immutable -
any
reference to them points to a structure that doesn't change. That
could
certainly be powerful and useful, but a bit different from what I
understand this proposal to be, which is a bit narrower in scope:
simply to
have compile time checking that a slice can't be written to if passed
as
readonly or created as readonly.
On Sat, Jun 17, 2017 at 1:25 PM, leiser1960 ***@***.***>
wrote:
> What is the goal? Better documentation, or race free code?
> Your proposal goes for the first, I assume.
> You give the the guarantee that your function will have no way to
alter
> the underlaying array.
> But you do not get the guarantee that your function is race free in
> respect of changes to this array, because other goroutines may modify
the
> underlaying array concurrently.
> Adding immutability to golang should go for both goals, i.e. full
value
> semantics as we have it for strings. This changes assignment as well
as
> comparison semantics. But would help significantly reasoning about
non
> sequential program behaviour.
> Adding a simple form of immutibility to the type system of go is big
> challenge, both for language design and implementation. But please go
for
> it!
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <#20443 (comment)>,
or mute
> the thread
>
<https://github.com/notifications/unsubscribe-auth/AIOKxX3FGb5iAbgyXz0uLPvBJWLDlPZkks5sFAwagaJpZM4NhrYK>
> .
>
--
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
#20443 (comment)
--
Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet.
|
Would this be valid under your proposal, and what would be the final value
of x?
x := int{1,2,3}
y := x[:2]
z := append(y, 5)
…On Sat, 17 Jun 2017, 23:46 henryas ***@***.***> wrote:
How about this? You can still work with slices as usual. You can slice,
re-slice, append, make it point to different values, etc. However,
regardless what you do to the slice, it won't change the backing arrays and
slices. You can think of a slice like a pointer. You can make it point to
different values at any time, but in this case you can't change the value
you are pointing at.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#20443 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAAcA77_08XDdk2VLKuG2paKdcYfer45ks5sE9iqgaJpZM4NhrYK>
.
|
@dave: the final value of x should remain unchanged, which is {1,2,3} Anyhow, please do not take this proposal strictly as is. The idea is there, but implementation-wise I am not so certain, which is why I welcome everybody's inputs on this matter. Some form of control over a variable's mutability will certainly be beneficial to Go. |
On Sun, Jun 18, 2017 at 11:32 AM, henryas ***@***.***> wrote:
@dave <https://github.com/dave>: the final value of x should remain
unchanged, which is {1,2,3}
But how can that be, if you permit reslicing and appending?
… Anyhow, please do not take this proposal strictly as is. The idea is
there, but implementation-wise I am not so certain, which is why I welcome
everybody's inputs on this matter. Some form of control over a variable's
mutability will certainly be beneficial to Go.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#20443 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAAcA44C_XRmFuCIS371aGCzlSb1v6SKks5sFH5IgaJpZM4NhrYK>
.
|
@dave: y is a new slice that points to X's first two values. X remains as it is. Y is just a "pointer" to x. Z is a new slice that points to Y's values and 5, but it doesn't change x and y. Even if you were to do z[0] = 7, z at index 0 just points to a new value of 7, but it doesn't change x and y. But I welcome any other idea. It was just something that came to me as I was driving home from work yesterday. |
So your proposal is every reslicing operation makes a copy of the slice's
contents?
…On Sun, 18 Jun 2017, 11:55 henryas ***@***.***> wrote:
@dave <https://github.com/dave>: y is a new slice that points to X's
first two values. X remains as it is. Y is just a "pointer" to x. Z is a
new slice that points to Y's values and 5, but it doesn't change x and y.
Even if you were to do z[0] = 7, z at index 0 just points to a new value of
7, but it doesn't change x and y.
But I welcome any other idea. It was just something that came to me as I
was driving home from work yesterday.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#20443 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAAcA783fFqZILJnMDreGY5o_2I40j6Gks5sFIOagaJpZM4NhrYK>
.
|
Nope. Not a copy. If someone are to change x, the change will cascade down to y and z as long as they are still pointing to any value of x. Think of a slice like pointers to some values. When the values change, the slice will change too. However, when the slice is changed, it just points to the new values. The old values remain unaffected. |
In my example, currently x ,y and z all point to the same backing array. In
your proposal I don't see how to retain the semantics of y and z being
unable to modify the contexts of the x without reslicing and or appending
making a copy of the slice's backing array. However if that was the case
then changes to x would not be reflected in y or z.
I don't see how your proposal will be workable without removing ay least
one of append or reslicing.
…On Sun, 18 Jun 2017, 12:22 henryas ***@***.***> wrote:
Nope. Not a copy. If someone are to change x, the change will cascade down
to y and z as long as they are still pointing to any value of x.
Think of a slice like pointers to some values. When the values change, the
slice will change too. However, when the slice is changed, it just points
to the new values. The old values remain unaffected.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#20443 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAAcA5nc2oG5p44eR-6mCNZPOFZZZZJ5ks5sFInOgaJpZM4NhrYK>
.
|
Think about the table of contents (TOC) of a book. The TOC is the slice, the content of the book are the actual values. If you need to find something, you look up the TOC, find the page it refers to, and find the page and read the information. When the content of a page changes, naturally people who looks up via TOC will see the change. Hence, changes in x will cascade down to y and z. However, when you change the entries in the TOC, eg. changing the page number a TOC entry points to, people use the TOC will see the new page, but the content of the book remains unchanged. Hence, changes in z does not cascade up to x and y. |
Sure, but how to you propose to implement this at a language level.
…On Sun, Jun 18, 2017 at 12:42 PM, henryas ***@***.***> wrote:
Think about the table of contents (TOC) of a book. The TOC is the slice,
the content of the book are the actual values. If you need to find
something, you look up the TOC, find the page it refers to, and find the
page and read the information.
When the content of a page changes, naturally people who looks up via TOC
will see the change. Hence, changes in x will cascade down to y and z.
However, when you change the entries in the TOC, eg. changing the page
number a TOC entry points to, people use the TOC will see the new page, but
the content of the book remains unchanged. Hence, changes in z does not
cascade up to x and y.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#20443 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAAcA-5OwNNPG3GAZ-Wkr_NYbEsHtsEEks5sFI6KgaJpZM4NhrYK>
.
|
It seems to me that we have at least a few proposals here:
For what it's worth, I see this as parallel to some of the work going on in .NET Core 2.0 around |
The proposal here is not about immutable types, it is about your point 1 or 2 You may prohibit the last line completely, or allow it. In the second case z must point to a new array, and be of type []int. append applied to a const slice behaves as if len==cap. This would be a consistent extension, a const slice simply has no cap at all. My concerns are about the question: The question: |
I think simple The big problem with that is hidden copy operations and change of semantics. You're no longer sure exactly how your assignment/append will work without looking up the type of the slice. We kind of have the same problem with methods with value/pointer receivers but it's less obvious here. |
I'm strongly opposed to this hidden copying. There is already enough of
that in the language as it is, eg.
var ints [2^24]int
for i := range ints {
// there's a copy here
}
…On Mon, Jun 19, 2017 at 9:09 AM, Antonenko Artem ***@***.***> wrote:
I think simple const keyword like in C could be naturally (and probably
within Go 1.x) extended to a kind of immutable solution. Instead of
throwing compile errors, on any change to the const slice compiler could
emit a shallow copy operation. Changing an element in a const slice - make
a copy and make the change to that copy. Appending - make a copy and append
to that. Of course, these copy operations must be thread-safe. Otherwise
it's useless.
The big problem with that is hidden copy operations and change of
semantics. You're no longer sure exactly how your assignment/append will
work without looking up the type of the slice. We kind of have the same
problem with methods with value/pointer receivers but it's less obvious
here.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#20443 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAAcAyYgjLhCdEfWLc5cz7wmQdkhmAw0ks5sFa4fgaJpZM4NhrYK>
.
|
Read-only slice types seem like a more or less reasonable proposal -- which is why Russ's evaluation of such a proposal (linked to by @dgryski earlier) was very in-depth and involved a real implementation of the feature. By contrast, I think that immutable types which do full copies of themselves transparently are a fairly unserious feature in a language where performance is even a vague concern. Immutable types in other languages allow you to "modify" them (i.e. get a new object that's a modification of the old object while leaving the old object untouched) without doing full copies all over the place. For example, in Clojure, the persistent types such as vectors and maps are backed by tree structures which allow multiple objects to share common structure. When you append something to a vector or associate a new key/value into a map, the cost of that operation and the amount of extra space used is ~constant, not linear in the size of the vector/map. That said, I doubt that any form of immutable/persistent data structures like these fit well in the Go language (though I look forward to Russ's evaluation of immutability in the future). |
Almost; the methods return a modified copy, but the returned value is itself immutable. So any attempt to "modify" the returned value (via a "setter") results in another copy. Hence ( p4 := p2.SetAge(42)
fmt.Printf("p1: %v\n", p1)
fmt.Printf("p2: %v\n", p2)
fmt.Printf("p4: %v\n", p4) gives: p1: Person{ Name: "", Age: 0}
p2: Person{ Name: "Paul", Age: 0}
p4: Person{ Name: "Paul", Age: 42} ( |
If you turn Person into objects, what you are trying to do is already achievable without needing code generation. See https://play.golang.org/p/TUaq76pB98 I do think that by wrapping them into objects, we can take advantage of Go's interface to solve this problem. The same goes with array, slices, and maps. However, the problem with the built-in array, slices, and maps is that they have no behavior (no methods) and thus one cannot directly apply interface to them. You need to create new objects by wrapping those built-in collections. On one hand, you can say it is a good practice to wrap a collection because you are giving them a domain identity. On the other hand, it involves additional boilerplate code. Without generics, you end up creating many different collection objects for each different type. I believe we should look into using Go's interface instead of looking for new magic keywords. |
@henryas thanks, I think I get what @ianlancetaylor was referring to now by the "single method that returns a copy". So we're using the same definition for immutability, it's just different interfaces/approaches. Of course there is nothing that requires code generation; we introduced it because it was easy and it saved writing tonnes of boilerplate as you say. Taking your example, code generation would allow you to "explode": type personImpl struct {
name string
age int
} into the full implementation you have in your linked example. Writing code to write less code as someone once said 👍 |
Something which would be useful in addition to const maps and slices are const keys for maps. In other words, you declare all the keys the map can have and it'll be enforced by the compiler. While it overlaps with structs, it has the benefits of a map. E.g. Calling the field stored in a string without using reflection. And for full const maps, I don't think any of the keys or contents should be editable.
Perhaps, something like this. This would help eliminate bugs (e.g. calling the wrong key), and to give the compiler more chances to speed things up. For things which can be detected at compile time, it would error and halt compilation, otherwise it would panic. |
Just give my two cents,
|
I may have a change of opinion about this proposal since I first wrote it. I would now be very cautious about introducing mutability/immutability (or something similar) into Go. Being able to tell the compiler exactly what you want so that it can work smarter and help you catch errors is always ideal. Mutability/immutability embodies one of those ideals. However, implementation-wise, it introduces a whole lot of other problems. In order to be able to specify a type as a mutable/immutable, you also need to be able to mark their methods as such. In Go, you may do this by adding an extra keyword to the receivers. You may also need to be able to mark the method signature in an interface as mutable/immutable. Once you have mutability qualifiers all over the place, you have to give extra thought when designing your code because if you are not careful, you may hinder your code future usability due to the unnecessary constraints you place in your codes. Even after some careful consideration, you may still find yourself in an awkward situation due to the mutability constraints. In Rust, they have a number of special pointers to bail you out when everything else fails. One such smart pointer is the Cell/RefCell, which is a way to allow mutability in an immutable object. When I first learned about it, I thought I would never going to need it. It turns out that there are quite a number of occasions when I had to use it. One such example is when I needed to create a mock object. The methods being mocked specify that the receiver must be immutable (because the actual object mustn't change anything), and yet the mock object needs to update itself in order to keep track of the way the mock object is used. Hence, the need for the smart pointers. I am now of the opinion that Go's current approach, although it is far from perfect, is simpler and less mentally taxing. I would wait until something better comes along. |
Being cautious is always a good idea. |
I think immutability in Go is still a great thing to pursue. I still think we have a few different good ideas in here that would be of varying utility and difficulty to implement. The use case that drew me to this has to do with unsafe/syscall usage. I would like to be able to memory map a file in Go and pass around slices with confidence that receivers cannot modify them (without going unsafe themselves). This is more or less having an interface that utilizes something like C I can also see a great case for having runtime immutability, which would almost certainly involve some of the copying mechanisms that have been discussed above. That certainly seems like a bigger issue and something that would require some new features. It's different than the use case I mention above though. |
For that you don't need language-level immutability and it can be done today by passing only the syscall.PROT_READ flag and omitting the syscall.PROT_WRITE flag in the call to syscall.Mmap. The backing array of the byte slice viewing the file will be R/O even for crafted |
@cznic, that's fine for a runtime error, I'd like it to be able to fail at compile time. It's possible to do that in other languages, and it's a great feature when sharing code across packages/development teams/etc. For instance, I really like what .NET core is trying to do with Span and ReadOnlySpan: https://github.com/dotnet/corefxlab/blob/master/docs/specs/span.md |
Here is an extension of Brad's proposal for readonly slices, by way of replies to Russ Cox's evaluation of Brad's proposal. It is part of a set of proposals to improve Golang's security model.
There is another motivation for Quote from parent proposal:
Here's an example that doesn't need global immutability as in strings, yet benefits from increased security and compile-time type-checking w/ readonly slices. type Listener func(msg readonly []byte)
func broadcast(msg []byte, listeners []Listener) {
for listener := range listeners {
listener(readonly(msg))
}
} Without the
In this proposal,
Perhaps by introducing type Peeker interface {
Peek(int) any []byte
}
type roPeeker struct {}
func (_ roPeeker) Peek(n int) readonly []byte {...}
type rwPeeker struct {}
func (_ rwPeeker) Peek(n int) []byte {...}
xro := roPeeker{}
_, ok := xro.(interface{ Peek(int) []byte }) // not ok
_, ok = xro.(interface{ Peek(int) readonly []byte }) // ok
_, ok = xro.(interface{ Peek(int) any []byte }) // ok
xrw := rwPeeker{}
_, ok = xrw.(interface{ Peek(int) []byte }) // ok
_, ok = xrw.(interface{ Peek(int) readonly []byte }) // ok
_, ok = xrw.(interface{ Peek(int) any []byte }) // ok This should also work in unsurprising ways when there are multiple slices in the method signature. Here's a complete list of match rules for arguments:
And return values:
When a writeable slice is need, it must be provided.
I prefer the duplication approach which in the example introduces 2 new lines: type ReadInterface interface {
Less(i, j int) bool
Len() int
}
type SortInterface interface {
Swap(i, j int)
}
func Sort(data SwapInterface) bool {
… code using Less, Len, and Swap …
}
func IsSorted(data ReadInterface) bool {
… code using only Less and Len …
}
type ReadIntSlice readonly []int
func (x ReadIntSlice) Less(i, j int) bool { return x[i] < x[j] }
func (x ReadIntSlice) Len() int { return len(x) }
type SortIntSlice []int
func (x SortIntSlice) Less(i, j int) bool { return ReadIntSlice(x).Less(i, j) }
func (x SortIntSlice) Len() int { return ReadIntSlice(x).Len()) }
func (x SortIntSlice) Swap(i, j int) { x[i], x[j] = x[j], x[i] }
func Ints(a []int) { // invoked as sort.Ints
Sort(SortIntSlice(a))
}
func IntsAreSorted(a []int) bool {
return IsSorted(ReadIntSlice(a))
} We can add sugar and do: type SortIntSlice +ReadIntSlice
func (x SortIntSlice) Swap(i, j int) { x[i], x[j] = x[j], x[i] } Going back to the original problem statement,
With this proposal, if you receive a |
Syntax-wise, it seems like a 1-character prefix to the
|
Background
I recently stumbled upon a bug and after hours of searching, I managed to find the culprit, which is a function that accidentally modifies the slice argument passed to it. The problem is that as I took several subsets of the original slice and gave them new identities, I forgot that they were still pointing to the original slice. I don't think this is an uncommon issue, and hence the proposal below.
Proposal
I would like to suggest the idea of defining a read-only slice and map in the function arguments. For example, we can define a function as follows:
//function definition
func MyFunction(const mySlice []byte, const myMap map[string][]byte) []byte
//usage
mySlice := []byte {0x00, 0x01} //normal slice definition. There is no weird const whatsoever.
myMap := make(map[string][]byte) //normal map definition.
//you are still using slice and map as you normally would. Slice and map are still modifiable,
//but MyFunction cannot alter the original variables.
MyFunction(mySlice, myMap)
//you can still make changes here
myMap["Me"] = 0x01
or in the interface definition:
type MyInterface { MyFunction(const mySlice []byte) []byte }
Why slices and maps?
With structs, interfaces and built-in types, it is easy to tell whether you want them to be modifiable or not. With slices and maps, it is not so easy to tell. Maps are probably less prone to accidental changes, but slices are tricky, especially when they get passed around many functions.
Implementation Options
Read-only slices and maps are shallow-copied upon being passed to the function. Hence, any accidental change is local to the function. I understand that this will not prevent interior mutability, but exterior mutability is good enough. I think this implementation option is probably less intrusive and there are fewer breaking changes to existing codes (if any) - possibly none.
Alternatively, we may have a more elaborate system where the compiler will not compile if a function tries to modify a read-only slice/map. However, this leads to a very complex solution. What if a new sub-slice is taken from an existing read-only slice, should it be immutable too? Now, what if that sub-slice is passed into another function, should the compiler check whether the other function preserves the integrity of the sub-slice?
I personally tend to lean with the first option.
Syntax
In order not to add any new keyword, I am thinking of reusing const, and it is also more intuitive to those familiar with C-family programming languages. However, I am open to suggestions.
Let me know what you think.
Thanks
The text was updated successfully, but these errors were encountered: