updated README with better and more examples; support negative indices

issues.md → doc/issues.md

@@ -1,18 +1,29 @@
 # List of Todos and other Issues
 
-## TODOs
+- replace set_value(x) with value(x)
+- add split(":") and regex interpretations
+- /*[type().ends_with()] ?? -> or not, just replace it with #type^split("_")/[-1]=="xx"
+- /*[name|parameters|return_type] ??
+- ^fs[rw] ??
+- should blobs/bytes be part of value? they are only useful by reinterpretation
+- set value on the command line: '/username = "newuser"'
+- https://raw.githubusercontent.com/rust-lang/rust/master/src/tools/rustfmt/src/lib.rs^http^rust does not work
+- support zip, markdown
+- support 'copy source destination'
+- support ^json^tree^xml
+- support diff  ./file.json^json^tree  ./file.xml^xml^tree
+- '**[filter]' must be work as '**/*[filter]' (filter to be applied only on leaves)
+- support type selector: `hial './src/tests/rust.rs^rust/*[:function_item]'`
+- support rust/ts write: `hial './src/tests/rust.rs^rust/*[:function_item].label = "modified_fn_name"'`
+- new structure: /api, /api/impl, /interpretations/api, /interpretations/*, /search
+- add http interpretation params: method=HEAD, accept=""
+- functions
 
-! focus on releasing a first minimal version, then improve
+- release first minimal version:
     - interpretations: path+fs, json+yaml+toml+xml, rust+js, url?+http
     - explicit and implicit write support (policy, include readonly)
     - fix tests, todo!() and TODO: in code
 
-- support type selector: `hial './src/tests/rust.rs^rust/*[:function_item]'`
-- support rust/ts write: `hial './src/tests/rust.rs^rust/*[:function_item].label = "modified_fn_name"'`
-- set value on the command line
-- new structure: /api, /api/impl, /interpretations/api, /interpretations/*, /search
-
--
 
 - operations:
     - assign to variables;
@@ -30,50 +41,10 @@
 - ?explore python implementation and usage
 - ?search should return all matches embedded in a delegation cell, which has all results
     as subs and delegates write operations to all the subs
-- ?rename XCell, Cell, CellTrait to Nex/NexIn/NexInTrait
+- ?rename XCell to Xell
 - later: python, git, database, ical, zip, markdown
 
 
-### Feature implementation status
-
-| *Feature*  | *Readable* | *Writeble* |
-|------------|------------|------------|
-| url        |    yes     |    yes     |
-| path       |    yes     |    yes     |
-| fs         |    yes     |    yes     |
-| http       |    yes     |    yes     |
-| json       |    yes     |    yes     |
-| yaml       |    yes     |    yes     |
-| toml       |    yes     |    yes     |
-| xml        |    yes     |    yes     |
-| rust       |    yes     |            |
-|            |            |            |
-| git        |            |            |
-| database   |            |            |
-| ical       |            |            |
-| zip        |            |            |
-|            |            |            |
-| plain text |            |            |
-| markdown   |            |            |
-|            |            |            |
-| python     |            |            |
-| javascript |            |            |
-| go         |            |            |
-|------------|------------|------------|
-
-
-
-
-| *Feature*       | *Support* |
-|-----------------|-----------|
-| path lang       |  partial  |
-|                 |           |
-| C interop       |           |
-| Python interop  |           |
-
-
-### Todos, Issues, Problems
-
 - todo: c interop and a small c test
 - cell must implement partialeq, eq (same pointed location)
 - todo CLI:
@@ -88,9 +59,7 @@
 - todo: interpretations parameters
 - todo: custom tree datastructure?
 - todo: cell symlinks
-- todo: cell path
 - todo: path bindings
-- todo: diffs
 
 - unclear: we should have some internal language:
     - Usecase: json:  `/question[/answer_entities/*.is_empty()].count()`
@@ -100,13 +69,3 @@
     './**[.name=='config.yaml'][as composefile]^yaml/services/*/image[^string^http@status/code!=200]
     tree 'result' -> [composefile] -> image
 ```
-
-### Examples
-
-1. Extract the general structure of a rust file. Get the struct/enum/type definitions (just the name and the type) and the function definitions (just the name and the signature). Get all implementations of traits and the functions inside them, as a tree.
-
-```
-hial 'item = ./src/tests/rust.rs^rust/**[:struct_item|:enum_item|:type_item|:function_item]; item/'
-
-'
-```

doc/status.md

@@ -0,0 +1,36 @@
+# Feature implementation status
+
+### Interpretations
+
+| *Feature*  | *Readable* | *Writeble* |
+|------------|------------|------------|
+| url        |    yes     |    yes     |
+| path       |    yes     |    yes     |
+| fs         |    yes     |    yes     |
+| http       |    yes     |    yes     |
+| json       |    yes     |    yes     |
+| yaml       |    yes     |    yes     |
+| toml       |    yes     |    yes     |
+| xml        |    yes     |    yes     |
+| rust       |    yes     |            |
+|            |            |            |
+| git        |            |            |
+| database   |            |            |
+| ical       |            |            |
+| zip        |            |            |
+|            |            |            |
+| plain text |            |            |
+| markdown   |            |            |
+|            |            |            |
+| python     |            |            |
+| javascript |            |            |
+| go         |            |            |
+|------------|------------|------------|
+
+
+### Language bindings
+
+| *Feature*       | *Support* |
+|-----------------|-----------|
+| C interop       |           |
+| Python interop  |           |

doc/use-cases.md

@@ -0,0 +1,58 @@
+# List of potential use cases
+
+This list of potential use cases should drive the development of the library.
+
+See also the "### What can it do?" section in the README.md file.
+
+### Functions
+
+Special: path(cell path), label, value, serial
+General: sort, unique, filter, map, reduce, grouping
+Text: concat, regex, len, casing, substrings, split, join, replace, trim, padding, ends_with, starts_with
+Aggregation: count, sum, avg, min, max
+Math: round, abs
+Date: parse, format, add, subtract, diff, duration
+
+```
+/question[count(/answer_entities/*)==0]
+```
+
+### Python requirements
+
+Update a python module version in a requirements.txt file:
+
+```bash
+# change the version of the requests module to 1.2.3
+hial './requirements.txt^python.reqs/*[/[0]=="requests"] = "1.2.3"'
+
+# increment the minor version of the requests module
+hial 'x = ./requirements.txt^python.requirements/*[/[0]=="requests"]; $x/[2]^version/:minor += 1'
+```
+
+
+### Search with results structured into a tree
+
+Unclear: what is the accepted language?
+```
+x = './**/*[.name=='config.yaml'] (as composefile)]^yaml/services/*/image[^string^http@status/code!=200]
+tree 'result' / [composefile] / image
+```
+
+### Transform one format to another
+
+Transform a json file to an xml file and vice versa.
+
+### Structured diff between two files in different formats
+
+```
+hial 'diff x y'
+```
+
+
+### - Extract the general structure of a rust file
+
+Get the struct/enum/type definitions (just the name and the type) and the function definitions (just the name and the signature). Get all implementations of traits and the functions inside them, as a tree.
+
+```
+hial 'item = ./src/tests/rust.rs^rust/**[:struct_item|:enum_item|:type_item|:function_item]; item/'
+```

src/base/extra.rs

@@ -947,6 +947,46 @@ impl Iterator for CellIterator {
     }
 }
 
+impl DoubleEndedIterator for CellIterator {
+    fn next_back(&mut self) -> Option<Self::Item> {
+        match &mut self.cell_iterator {
+            CellIteratorKind::DynCellIterator { dyn_cell, domain } => {
+                dispatch_dyn_cell_iterator!(dyn_cell, |x| {
+                    x.next_back().map(|cell_res| Cell {
+                        dyn_cell: match cell_res {
+                            Ok(cell) => DynCell::from(cell),
+                            Err(err) => DynCell::from(err),
+                        },
+                        domain: Rc::clone(domain),
+                    })
+                })
+            }
+            CellIteratorKind::Elevation(cell_res) => match cell_res {
+                Ok(cell) => {
+                    let cell = cell.clone();
+                    *cell_res = nores();
+                    Some(cell)
+                }
+                Err(err) => {
+                    if err.kind == HErrKind::None {
+                        None
+                    } else {
+                        Some(Cell {
+                            dyn_cell: DynCell::from(err.clone()),
+                            domain: Rc::new(Domain {
+                                write_policy: cell::Cell::new(WritePolicy::ReadOnly),
+                                origin: None,
+                                dyn_root: OnceCell::new(),
+                                dirty: cell::Cell::new(false),
+                            }),
+                        })
+                    }
+                }
+            },
+        }
+    }
+}
+
 impl CellIterator {
     pub fn err(self) -> Res<CellIterator> {
         match self.cell_iterator {

src/base/intra.rs

@@ -61,7 +61,7 @@ pub trait CellWriterTrait: Debug {
 
 pub trait GroupTrait: Clone + Debug {
     type Cell: CellTrait;
-    type CellIterator: Iterator<Item = Res<Self::Cell>>;
+    type CellIterator: DoubleEndedIterator<Item = Res<Self::Cell>>;
 
     fn label_type(&self) -> LabelType;
     fn len(&self) -> Res<usize>;

src/interpretations/xml.rs

@@ -31,6 +31,7 @@ pub(crate) struct Cell {
 pub(crate) struct CellIterator {
     group: Group,
     next_pos: usize,
+    next_back_pos: usize,
     key: OwnValue,
 }
 
@@ -603,6 +604,33 @@ impl Iterator for CellIterator {
         }
     }
 }
+impl DoubleEndedIterator for CellIterator {
+    fn next_back(&mut self) -> Option<Self::Item> {
+        fn inner(this: &mut CellIterator) -> Res<Cell> {
+            loop {
+                if this.next_back_pos == 0 {
+                    return nores();
+                }
+                this.next_back_pos -= 1;
+                let cell = this.group.at(this.next_back_pos)?;
+                let reader = cell.read()?;
+                if Some(this.key.as_value()) == reader.label().ok() {
+                    return Ok(cell);
+                }
+            }
+        }
+        match inner(self) {
+            Ok(cell) => Some(Ok(cell)),
+            Err(e) => {
+                if e.kind == HErrKind::None {
+                    None
+                } else {
+                    Some(Err(e))
+                }
+            }
+        }
+    }
+}
 
 impl GroupTrait for Group {
     type Cell = Cell;
@@ -642,6 +670,7 @@ impl GroupTrait for Group {
         Ok(CellIterator {
             group: self.clone(),
             next_pos: 0,
+            next_back_pos: self.len()?,
             key: key.to_owned_value(),
         })
     }

src/pathlang/parse.rs

@@ -165,10 +165,10 @@ fn path_item_selector(input: &str) -> NomRes<&str, Selector> {
         .map(|(next_input, res)| (next_input, Selector::from(res)))
 }
 
-fn path_item_index(input: &str) -> NomRes<&str, usize> {
+fn path_item_index(input: &str) -> NomRes<&str, isize> {
     context(
         "path_item_index",
-        delimited(tag("["), number_usize, tag("]")),
+        delimited(tag("["), number_isize, tag("]")),
     )(input)
 }
 
@@ -212,8 +212,21 @@ fn path_item_start(input: &str) -> NomRes<&str, char> {
 }
 
 fn number_usize(input: &str) -> NomRes<&str, usize> {
-    context("number", recognize(many1(one_of("0123456789"))))(input).map(|(next_input, res)| {
-        let n = usize::from_str(res).unwrap_or_else(|_| panic!("parse error, logic error"));
+    context("positive number", recognize(many1(one_of("0123456789"))))(input).map(
+        |(next_input, res)| {
+            let n = usize::from_str(res).unwrap_or_else(|_| panic!("parse error, logic error"));
+            (next_input, n)
+        },
+    )
+}
+
+fn number_isize(input: &str) -> NomRes<&str, isize> {
+    context(
+        "number",
+        recognize(tuple((opt(one_of("+-")), many1(one_of("0123456789"))))),
+    )(input)
+    .map(|(next_input, res)| {
+        let n = isize::from_str(res).unwrap_or_else(|_| panic!("parse error, logic error"));
         (next_input, n)
     })
 }

src/pathlang/path.rs

@@ -19,8 +19,8 @@ pub enum PathStart<'a> {
 pub struct PathItem<'a> {
     pub(crate) relation: Relation,
     pub(crate) selector: Option<Selector<'a>>, // field name (string) or '*' or '**'
-    pub(crate) index: Option<usize>,           // or index
-    pub(crate) filters: Vec<Filter<'a>>,       // [@size>0] or [.name.endswith('.rs')]
+    pub(crate) index: Option<isize>,
+    pub(crate) filters: Vec<Filter<'a>>, // [@size>0] or [.name.endswith('.rs')]
 }
 
 #[derive(Clone, Debug, PartialEq)]

src/pathlang/search.rs

@@ -206,7 +206,16 @@ impl<'s> Searcher<'s> {
             }
             (None | Some(Selector::Star) | Some(Selector::DoubleStar), Some(index)) => {
                 ifdebug!(println!("get child by index"));
-                let cell = guard_ok!(group.at(index).err(), err => {
+                let at_index = if index < 0 {
+                    let len = group.len().unwrap_or_else(|e| {
+                        warning!("Error while searching: cannot get group length: {:?}", e);
+                        0
+                    });
+                    (len as isize + index) as usize
+                } else {
+                    index as usize
+                };
+                let cell = guard_ok!(group.at(at_index).err(), err => {
                     if err.kind != HErrKind::None {
                         warning!("Error while searching: cannot get cell: {:?}", err);
                     }
@@ -230,7 +239,12 @@ impl<'s> Searcher<'s> {
                     return ;
                 });
                 if let Some(index) = opt_index {
-                    if let Some(cell) = iter.nth(index) {
+                    let opt_cell = if index < 0 {
+                        iter.rev().nth((-index - 1) as usize)
+                    } else {
+                        iter.nth(index as usize)
+                    };
+                    if let Some(cell) = opt_cell {
                         Self::process_cell(stack, path, cell, path_index, true, next_max_path_index)
                     }
                 } else {