Improved the displays of the lists and column type header

[N-thony]

Fixes (partially) #7493
Fixes #7947
Fixes partially #8029
@rdstern I have added the column type (S) for the list, (CX) for complex and also improved the display of lists in the grid. Take a look.

[rdstern]

@N-thony that looks great, but introduces an "interesting" new problem, that I don't quite understand.

You will notice that when I now do sapply(x2,sum) and sapply(x2,mean) in the calculator I now get NA everywhere.

Earlier, as shown below I get this:

.

[N-thony]

@rdstern I understand, I didn't check this example of sapply, let me look at it. Thanks

[N-thony]

@rdstern this seems to work fine. please have a look.
@ChrisMarsh82 please could you review this?

[ChrisMarsh82]

@N-thony Looks good. I like the way you have done this. Very neat.

[rdstern]

@N-thony here is the error I get when editing.
I produced the list in the same way as usual (calculate > Integer > Divisors)
Then duplicated the variable. That worked fine.
Then edited row 6 from 1, 2, 3 - I changed the 3 to 4.

It then complains that there is a string variable:

As I mentioned somewhere, I am no longer worried if the display in reogrid is different to the R-viewer, So you may wish to display it in reogrid as c(1, 2, 3) - if that makes the editing and subsequent translation for R easier?

[N-thony]

@N-thony here is the error I get when editing. I produced the list in the same way as usual (calculate > Integer > Divisors) Then duplicated the variable. That worked fine. Then edited row 6 from 1, 2, 3 - I changed the 3 to 4.

It then complains that there is a string variable:

As I mentioned somewhere, I am no longer worried if the display in reogrid is different to the R-viewer, So you may wish to display it in reogrid as c(1, 2, 3) - if that makes the editing and subsequent translation for R easier?

@rdstern this seems to work fine now? Please have a look. Thanks

[rdstern]

@N-thony that is great, so I am approving.

1. I found I could replace a list by NA, but it complains if I make an element in the list into NA. If quick, perhaps you could check this? But otherwise let's stop for now.
2. We now need other dialogues to be able to produce lists.
3. I would also like lists be strings and possibly also matrices in the future.

But let's get this merged, and also the keyboard first.
It is really nice to be able to discuss lists, with this special case, and also to be able to use sapply in this way.

[N-thony]

@N-thony that is great, so I am approving.

1. I found I could replace a list by NA, but it complains if I make an element in the list into NA. If quick, perhaps you could check this? But otherwise let's stop for now.
2. We now need other dialogues to be able to produce lists.
3. I would also like lists be strings and possibly also matrices in the future.

But let's get this merged, and also the keyboard first. It is really nice to be able to discuss lists, with this special case, and also to be able to use sapply in this way.

@rdstern I have fixed 1. and I agree for 2. and 3. which will help on improving this.

[rdstern]

Great

[lloyddewit]

@N-thony Thank you, looks great. Just a couple of open questions

[rdstern]

@N-thony great. I approve again. I just note that the R viewer displays the first element as an empty list, while you display it as 0? Should it be 0?
If it is empty I would also be happy if it was made into NA.
Should it be 0? I tried deleting it, and it complained. Could it make NA when deleted?

[N-thony]

@rdstern I tried to delete and it works fine.

When I display the data with the grid in RStudio, the first value is numeric(0) so for the nicer display as suggested in the issue I was displaying 0 which I think it is wrong because I found that numeric(0) returns a numeric vector of length 0, so when you add anything to it you get the same result (it's basically a numeric NULL). I think it will be good to display as NA?

[rdstern]

Great - I agree. And I can use the right-click and delete - and that works now. No - not good I approved! Good that we make it NA!

[N-thony]

@rdstern I have made the changes as suggested replacing numeric(0) by NA.
@lloyddewit I resolved your comment above.
Please have look. Thanks

[lloyddewit]

@lloyddewit I resolved your comment above.

@N-thony I think there's still one unresolved comment above

[rdstern]

@lloyddewit I am approving as it seems sort of ok now & we need to move on.
I still notice that the R viewer gives a blank, where R-Instat gives NA for a list of length zero. If the code is ok, then let's accept.

[rdstern]

@N-thony I think this is an out-of date comment - so ignore for now!

@N-thony that's an improvement, but there is still a problem. If I edit any entry in the list then that row doesn't work anymore. Even if I just go into a row to edit, but don't change anything then it doesn't work.

I therefore asked Danny and David, if we are slightly opening a can of worms and should give up on all this. They feel the opposite - so we continue.

The LT column is a list column and each element of the list is a numeric vector. I think when sent to R these elements have a c( ) round them, i.e. they look more like the way we used to display them in the grid.
The R viewer distinguishes between that and the way the contents of the list are displayed to the user.

So, I am now more relaxed about having the display in reogrid different from the R-viewer than before - if that would also become simpler for you to code and clearer for the user.

But if we do display as you have now - that is still not quite the same as the R viewer - so the top element in reogrid is now 0, while it is blank in the R-viewer. Anyway we should be able to change a value in reogrid, and it still works. (Once we can do that I am particularly keen to see what happens if one of the elements is a missing value.)

I am happy that you are asking @ChrisMarsh82 to review the final code. If you are not sure on the change to make now, and Chris is also unsure, then perhaps he could check with Danny or David.

In our statistical ideas, this is going to help us explain and discuss multiple response questions.

We will stop for now with these numeric vectors, but later will then include character vectors as the elements. But that will be in another pull request.

And as a general idea, we are not taking a small step towards tibbles

[rdstern]

@N-thony here is an example, where your display of a list variable is very different from the R-viewer. And I am happy with the R-viewer, so I think your code may still not have the right methodology?

If you get a new data frame of say length 30, and include x1 to have the values 1 to 30. Then get Pascal triangles using

lapply(x1, function(x) {lapply(x, function(i) {choose(i, 0:i)})})

in the calculator. In reogrid, this just gives c for each row. In the r-viewer it gives the appropriate answer, shown in issue #8036. And you can see this is very different.

It is possible this needs a discussion with Chris, but happy if you want to try yourself first.

[N-thony]

@ChrisMarsh82 I have now made the changes I thought regarding Roger's comments above. Could you have a look?
@rdstern we can have a quick call about what you have reported above?

[rdstern]

The display seemingly looks fine now - except it is giving the wrong answers! I compare your display with the R-viewer here and you see that you are now splitting by the first digit of each number.

I guess this is a tricky task and keep thinking that @ChrisMarsh82 or @volloholic may need to add explanation. It is important, but I am concerned at the time it is taking you.

I am delighted though that you must now have your new machine working. That was quick!

[N-thony]

@ChrisMarsh82 I used the Regex expressions with this pattern .Regex.Matches(strData, "\d+|\bN\S*A\b"). I think this is much more easier or better

[rdstern]

Great - that (almost?) seems to work. I tried with a column of length 100 and with the pascal function - that will become a key.

lapply( x1, function(x) {lapply(x, function(i) {choose(i, 0:i)})})

I have approved - unless you want to try a last tweak to your code? So, first, here is the good news:

It looks fine, doesn't it? Very nice. And it doesn't go into scientific notation either - which the R-viewer does.

But later in the sequence there is a puzzle for me. I am in the middle where the numbers are very large. I assume there is an oddity here with the E notation? Maybe if you recognised e and + in the regex it would become clear? By the way I am having more trouble in the R-viewer now, than in your reogrid!

[ChrisMarsh82]

@N-thony

[ChrisMarsh82]

I would like to change how this works. Currently, you loop through all the values within the string. This could be inefficient if there are lots of values in the string. Also, GetVector does not really describe what the procedure is doing.
This is a proposed procedure that could be used instead:
Private Function GetInnerBracketedString(strData As String) As String
Dim strOutput As String = Strings.Left(strData, InStr(strData, ")") - 1) 'Get the first right bracket
strOutput = Mid(strOutput, InStrRev(strOutput, "(") + 1) 'Get the last left braket
Return strOutput
End Function

[N-thony]

@ChrisMarsh82 I just notice a small issue that the grid get bigger as the number do

[ChrisMarsh82]

@ChrisMarsh82 I just notice a small issue that the grid get bigger as the number do
The issue if it doesn't do this it could be more misleading. It looks like only a certain amount of data can fit on a row in the ReoGrid and it always shows the last row. therefore if it doesn't grow you would see something like this.

Row 44 now looks like it only contains '44, 1' when actually that's just the last 2 values.

If this is a problem for users we could add a text box at the top of the screen to show the value in the current cell?

[rdstern]

Great. It goes to e-notation at a sensible time and looks perfect.
I hope @lloyddewit or @ChrisMarsh82 can merge?

[rdstern]

@N-thony and @ChrisMarsh82 sorry I was too quick in approving and now seem to have a bug in the merged version. I only tested yesterday with the new pascal code and that still works fine.

So I had a new data frame of length 100 with x1 1 to 100 and then the code:

lapply( x1, function(x) {lapply(x, function(i) {choose(i, 0:i)})})

This (in the merged version) still correctly, (and impressively) gives as follows:

But then the command shown above, i.e. DescTools::Divisors(x1) - which was working before - gives an error and stops R-Instat.

[ChrisMarsh82]

I will look into this

[ChrisMarsh82]

@rdstern - I have a fix but I want to check. The code is designed to get anything for the inner brackets (if they exist). Therefore c(1,2) will become 1,2 and numeric(0) will become 0. Is this correct or is there an example where we need to display the whole return string including the brackets?

[rdstern]

I am not quite sure what is needed here. We don't need the whole string. We have various functions that produce numeric lists. The particular one is the divisors function on the integer keyboard, e.g. DescTools::Divisors(x1) when x1 is from 1 to 100.

I think this gives a special case when x1 = 1 and there are then no divisors. Then the R-viewer gives a blank, so I assume the list of divisors is of length zero. That now has become the value zero by eliminating the brackets. This is a rarity, so I have not been too concerned. But I think it's like a string that could be of length zero, and that's not the same as missing, etc.

But this catches me when I am finding some other oddities with the display of these numeric lists.

Here are 2 that need some attention.

a) I assumed I could easily change a list variable into a character variable. I can, by the command as.character(x2) and it then puts the c and brackets - sometimes round. I don't mind that particularly, see below. But if I right-click and use the R-Instat Convert to Character, then it makes everything missing. (Ideally I would also like to take a character variable that qualifies as a numeric list and be able to convert it back again.

b) If I filter and (say) just get the first 40 rows then the filtering works fine. But if I ask for a subset as a new data frame, then I get an error. I assume that's whenever I have a list variable. If I delete the list variable it works fine

This should be easy - I assume to fix.

[lloyddewit]

@rdstern Thank you for the comment above. This PR is merged so there's a risk that this comment will be overlooked.

Does this comment apply to one of the issues linked with this PR?
Or is the comment related to the new PR @ChrisMarsh82 made (#8050 )?

Should we move the comment?