Case modification?

[gkshenaut]

In the ksh manual page, it claims to support case modification during parameter expansion, most notable with ${parm^^} and ${parm,,}, plus additional single letter changes and so on. Anyway, ksh, even the most recent version here, doesn't do any of it. From my point of view, even implementing only ${parm^^} and ${parm,,} would be a great first step. At least this matter should be on a to-do list somewhere, since it's been in the manual page quite a while.

[McDutchie]

Where does the ksh manual page claim that? I don't see it. Could you quote the relevant text?

[McDutchie]

Ah, I see... it is not in our manual page (which was forked from 93u+ 2012-08-01), but it is in the one for ksh 93v- beta and ksh2020. And yeah, it does actually seem to be unimplemented there in ksh2020.

I'll add these to my todo list. Perhaps I'll get around to implementing these features sometime, but I'm not making any promises.

[McDutchie]

Very interesting: this work has already been done by @perette back in 2018, but ksh2020 rejected it. att#417

I think the 'backward compatibility concerns' cited there as a basis for rejecting it are completely misguided. In the last AT&T release version 93u+, these expansions are a clean syntax error. In the 93v- beta, they never worked properly. I see no backward compatibility concerns at all. Even if there was any backward compatibility need with the beta, ksh2020 killed that by killing the feature altogether!

I'm testing a backported version of @perette's patch on the ksh 93u+m dev branch code and it seems to work well. I am going to commit that to the dev branch for wider testing in the next few days. Thanks for the feature request, @gkshenaut.

[McDutchie]

The new case modification expansions feature is now on the dev branch. See c1762e0. There are no regression tests yet.

It should be compatible with bash, but also more powerful because the pattern part of the expansion can match any number of characters in the value, whereas in bash, it is only allowed to match one character.

Please test it and let me know how it goes :)

Ping @hyenias @JohnoKing @posguy99 @pghvlaans

[McDutchie]

So, about the freeze and the above workaround patch...

It occurs when matching the pattern *([A-Ma-m]). About *(…), sh.1 says:

              *(pattern-list)
                     Matches zero or more occurrences of the given patterns.

The terse comment documentation for strngrpmatch says:

ksh/src/lib/libast/string/strmatch.c

Lines 64 to 75 in b23649e

	/*
	* subgroup match
	* 0 returned if no match
	* otherwise number of subgroups matched returned
	* match group begin offsets are even elements of sub
	* match group end offsets are odd elements of sub
	* the matched string is from s+sub[0] up to but not
	* including s+sub[1]
	*/
	int
	strngrpmatch(const char* b, size_t z, const char* p, ssize_t* sub, int n, int flags)

The sub array here is the match array in macro.c. So, match[0] is the start offset of the first match and match[1] is the end offset of the first match. That's all we use here. Unused: match[2] is the start offset of the second match, and so on.

So, as the manual says,*(…) matches zero or more occurrences. Ksh makes a weird distinction between "no match" and "a match with zero occurrences". Since there is a match with zero occurrences when matching *([A-Ma-m]) against the P of PIRATES!, the start offset match[0] is set to 1, but the end offset match[1] set to 0.

I see two ways of handling a match with zero occurrences. One would be to treat it as "no match", therefore break and avoid modifying the string:

diff --git a/src/cmd/ksh93/sh/macro.c b/src/cmd/ksh93/sh/macro.c
index 599507039..d4ac1e1b6 100644
--- a/src/cmd/ksh93/sh/macro.c
+++ b/src/cmd/ksh93/sh/macro.c
@@ -1985,7 +1985,7 @@ retry2:
 						*pattern ? pattern : "?",
 						(ssize_t *)match, elementsof(match) / 2,
 						flag | STR_INT);
-					if (nmatch == 0)
+					if (nmatch == 0 || match[1] == 0)
 						break;
 					if (match[0])
 						mac_copy(mp, v, match[0]);

This means that, for arr=('Arrr' 'thar' 'be' 'PIRATES!'), the value of ${arr[@],,*([A-Ma-m])} would be ('arrr' 'thar' 'be' 'PIRATES!'), i.e., the last word would not be modified.

I'm not sure this conforms with user expectations. The other logical way would be to treat a match of zero occurrences as matching everything. The existing test would then pass. The following patch does that:

diff --git a/src/cmd/ksh93/sh/macro.c b/src/cmd/ksh93/sh/macro.c
index 599507039..bc65536c0 100644
--- a/src/cmd/ksh93/sh/macro.c
+++ b/src/cmd/ksh93/sh/macro.c
@@ -1989,6 +1989,8 @@ retry2:
 						break;
 					if (match[0])
 						mac_copy(mp, v, match[0]);
+					if (match[1] == 0)  /* match with zero occurrences as in *(...) -- match everything */
+						match[1] = vsize;
 					if (mbwide())  /* locale uses multibyte characters? */
 					{
 						int	wc, nwc;

Or maybe these are both incorrect. I'm honestly not sure what to do here, because ksh's notion of a "match with zero occurrences" really doesn't make sense to me. :-/

[McDutchie]

Or maybe these are both incorrect. I'm honestly not sure what to do here, because ksh's notion of a "match with zero occurrences" really doesn't make sense to me. :-/

Being consistent with the behaviour of other expansion operators is what we should do. A little test with ## which removes initial strings using greedy matching:

$ v=PIRATES!; echo ${v##*([A-Ma-m])}  
PIRATES!
$ v=AmXPIRATES!; echo ${v##*([A-Ma-m])}
XPIRATES!

So, a "match with zero occurrences" is treated as a non-match; no character after the non-match is removed. This suggests that a match with zero occurrences should be treated as a non-match.

The behaviour with a # is odd:

$ v=PIRATES!; echo ${v#*([A-Ma-m])} 
PIRATES!
$ v=AmXPIRATES!; echo ${v#*([A-Ma-m])} 
AmXPIRATES!

The first one is as expected. The second one also removes nothing, which is strange -- I think the first A should match the *([A-Ma-m]) pattern.

[McDutchie]

In fact, the code for ${var//pattern/string} has a workaround for the same infinite looping issue – and now I understand why it is necessary!

ksh/src/cmd/ksh93/sh/macro.c

Lines 1953 to 1968 in b23649e

	if(*v && c=='/' && type)
	{
	/* avoid infinite loop */
	if(nmatch && match[1]==0)
	{
	int sz;
	nmatch = 0;
	/* copy, and advance v by, one character */
	if ((sz = mbsize(v)) < 1)
	sz = 1;
	mac_copy(mp, v, sz);
	v += sz;
	}
	tsize -= v-oldv;
	continue;
	}

So, to be consistent, we need to do something similar for ${var^^pattern} and ${var,,pattern}:

diff --git a/src/cmd/ksh93/sh/macro.c b/src/cmd/ksh93/sh/macro.c
index 599507039..f92759fde 100644
--- a/src/cmd/ksh93/sh/macro.c
+++ b/src/cmd/ksh93/sh/macro.c
@@ -1987,6 +1987,17 @@ retry2:
 						flag | STR_INT);
 					if (nmatch == 0)
 						break;
+					if (match[1] == 0 && type)  /* avoid infinite loop */
+					{
+						int	sz;
+						/* copy, and advance v by, one character */
+						if ((sz = mbsize(v)) < 1)
+							sz = 1;
+						mac_copy(mp, v, sz);
+						v += sz;
+						vsize -= sz;
+						continue;
+					}
 					if (match[0])
 						mac_copy(mp, v, match[0]);
 					if (mbwide())  /* locale uses multibyte characters? */

With that patch, these two now have identical effects (the first one being the expansion that would cause an infinite loop without the patch):

$ arch/darwin.arm64-64/bin/ksh -c 'v=PIRATES; echo ${v,,*([A-M])}'
PiRaTeS
$ arch/darwin.arm64-64/bin/ksh -c 'v=PIRATES; echo ${v,,[A-M]}'   
PiRaTeS

...and I think that is probably the most sensible outcome.

[McDutchie]

Infinite looping bug fixed and regression tests added in b6e050c.

[hyenias]

Excellent work.

[hyenias]

@McDutchie Yep. I messed up the arr creation as you stated and updated the ${arr[@]^????};ARRR THAR be PIRATES! test as I also 100% agree. I am still researching the last ${arr[@],,*([A-Ma-m])} test. I am now using ${arr[@],,[A-Oa-o]};arrr thar be PiRaTeS! test instead as it is what I believe Perette was after visually.

Similarily for what you are investigating with using the ${var##} (remove left pattern), I presently like viewing the case modification ${var[^,^^,,]} as what is accomplished using the substitute string for pattern ${param/pattern/string} or ${param//pattern/string with each match being transformed with the singles only modifing the 1st position whereas doubles greedly altering every position.

$ ksh -c 'val="Arrr thar be PIRATES!"; typeset -a arr=($val); echo ${arr[@]/[A-Oa-o]/#}'
#rrr t#ar #e P#RATES!
$ ksh -c 'val="Arrr thar be PIRATES!"; typeset -a arr=($val); echo ${arr[@],[A-Oa-o]}'
arrr thar be PIRATES!
$ ksh -c 'val="Arrr thar be PIRATES!"; typeset -a arr=($val); echo ${arr[@]//[A-Oa-o]/#}'
#rrr t##r ## P#R#T#S!
$ ksh -c 'val="Arrr thar be PIRATES!"; typeset -a arr=($val); echo ${arr[@],,[A-Oa-o]}'
arrr thar be PiRaTeS!