FreeMarker Template Language (FTL): Split Multivalue XML Field Metadata to Group Respective Values
I have XML metadata for a people directory where a person can have multiple titles, departments, and phone numbers, etc. associated with their name.
The metadata with multiple values exists in one field each (e.g. fields : title, department, phone).
FTL appears to return the data for records in the correct order, i.e.
name (returns: name) title (returns: title0, title1, title2) department (returns: department0, department1, department2) and phone (returns: phone0, phone1, phone2)
I’d like to print the data like this :
name title0 department0 phone0 title1 department1 phone1 title2 department2 phone2
Is there some FTL facility to put the fields with multiple values in an array or something else, that would allow control of where to print (and group) their respective values (as above)?
The fields with multiple values (i.e., title, department, phone) will have normal values, for example; phone0, phone1, and phone2 — would each be a different phone number.
I found this:
https://freemarker.apache.org/docs/ref_builtins.html
The builtin for sequences seems to be somewhat relevant,
but I’m not sure how or if it would do the trick?
[UPDATE 1]
I tried the macro and code from @ddekany, changing only the names of the
entry.<xxx>
fields to match our data, in this case,
<@listGroups peopleDirectory "name"; personName, personEntries> Name: ${personName} <#list personEntries as entry> - Title: ${entry.personTitle1} - Department: ${entry.personDepartment1} - Phone: ${entry.personPhone1} </#list> </@listGroups>
where each of these
entry.<xxx>
fields might hold 0-x values, either pipe or comma delimited, but in this case, contain 4 values for each
${entry.personTitle1}, ${entry.personDepartment1}, and ${entry.personPhone1}
The 1 suffix is a remnant from when our data model had a separate field (i.e., personTitle1, personTitle2, personTitle3, etc.) for each possible multi value.
There are currently only 2 records in the set.
but I received the following error :
When calling macro "listGroups", required parameter "items" (parameter #1) was specified, but had null/missing value. ---- Tip: If the parameter value expression on the caller side is known to be legally null/missing, you may want to specify a default value for it with the "!" operator, like paramValue!defaultValue. ---- ---- FTL stack trace ("~" means nesting-related): - Failed at: #macro listGroups items groupByName [in template "conf/people/default/simple.ftl" in macro "listGroups" at line 533, column 1] - Reached through: @listGroups peopleDirectory, "name"; ... [in template "conf/people/default/simple.ftl" at line 552, column 1] ~ Reached through: #nested [in template "web/templates/modernui/search_classic.ftl" in macro "Results" at line 244, column 13] ~ Reached through: @s.Results [in template "conf/people/default/simple.ftl" at line 523, column 9] ~ Reached through: #nested [in template "web/templates/modernui/search_classic.ftl" in macro "AfterSearchOnly" at line 94, column 9] ~ Reached through: @s.AfterSearchOnly [in template "conf/people/default/simple.ftl" at line 521, column 1] ~ Reached through: #nested [in template "web/templates/modernui/search_classic.ftl" in macro "AfterSearchOnly" at line 94, column 9] ~ Reached through: @s.AfterSearchOnly [in template "conf/people/default/simple.ftl" at line 90, column 3] ----
Do I need to specify a paramValue!defaultValue? Or is there an issue with the macro?
We’d tried using a data model where any field that might contain multiple values had a separate field, and we could then group the multi value entries in their respective orders.
However, our app has a search narrowing feature we want to use called facets, which seems to only allow for grouping/searching on a single meta field (for each facet). In this case, since we’d changed our model to put multiple values (such as title) into multiple fields (i.e., title1, title2, title3, etc.), and since it seems facets can only be used to group/search on one meta field, they therefore wouldn’t match correctly those with multiple titles; which is why we went back to overloading a single field with multiple values.
Is there an XML model that would allow for our app’s facets, but not require a less than ideal solution?
[UPDATE 2]
I tried replacing the peopleDirectory variable with :
<@listGroups s.result.metaData "name"; personName, personEntries> Name: ${personName} <#list personEntries as entry> - Title: ${entry.personTitle1} - Department: ${entry.personDepartment1} - Phone: ${entry.personPhone1} </#list> </@listGroups>
but received the following error :
For "?sort_by" left-hand operand: Expected a sequence, but this has evaluated to an extended_hash (wrapper: f.t.SimpleHash): ==> items [in template "conf/people/default/simple.ftl" at line 539, column 10] ---- FTL stack trace ("~" means nesting-related): - Failed at: #list items?sort_by(groupByName) as item [in template "conf/people/default/simple.ftl" in macro "listGroups" at line 539, column 3] - Reached through: @listGroups s.result.metaData, "name"... [in template "conf/people/default/simple.ftl" at line 555, column 2] ~ Reached through: #nested [in template "web/templates/modernui/search_classic.ftl" in macro "Results" at line 244, column 13] ~ Reached through: @s.Results [in template "conf/people/default/simple.ftl" at line 523, column 9] ~ Reached through: #nested [in template "web/templates/modernui/search_classic.ftl" in macro "AfterSearchOnly" at line 94, column 9] ~ Reached through: @s.AfterSearchOnly [in template "conf/people/default/simple.ftl" at line 521, column 1] ~ Reached through: #nested [in template "web/templates/modernui/search_classic.ftl" in macro "AfterSearchOnly" at line 94, column 9] ~ Reached through: @s.AfterSearchOnly [in template "conf/people/default/simple.ftl" at line 90, column 3] ----
I also tried :
<@listGroups s.result.listMetadata "name"; personName, personEntries> Name: ${personName} <#list personEntries as entry> - Title: ${entry.personTitle1} - Department: ${entry.personDepartment1} - Phone: ${entry.personPhone1} </#list> </@listGroups>
but received :
For "?sort_by" left-hand operand: Expected a sequence, but this has evaluated to an extended_hash (com.google.common.collect.Multimaps$CustomListMultimap wrapped into com.appsearch.publicui.search.web.views.freemarker.AppsearchObjectWrapper$ListMultimapAdapter): ==> items [in template "conf/people/default/simple.ftl" at line 537, column 10] ---- FTL stack trace ("~" means nesting-related): - Failed at: #list items?sort_by(groupByName) as item [in template "conf/people/default/simple.ftl" in macro "listGroups" at line 537, column 3] - Reached through: @listGroups s.result.listMetadata, "n... [in template "conf/people/default/simple.ftl" at line 553, column 1] ~ Reached through: #nested [in template "web/templates/modernui/search_classic.ftl" in macro "Results" at line 244, column 13] ~ Reached through: @s.Results [in template "conf/people/default/simple.ftl" at line 529, column 9] ~ Reached through: #nested [in template "web/templates/modernui/search_classic.ftl" in macro "AfterSearchOnly" at line 94, column 9] ~ Reached through: @s.AfterSearchOnly [in template "conf/people/default/simple.ftl" at line 527, column 1] ~ Reached through: #nested [in template "web/templates/modernui/search_classic.ftl" in macro "AfterSearchOnly" at line 94, column 9] ~ Reached through: @s.AfterSearchOnly [in template "conf/people/default/simple.ftl" at line 90, column 3] ----
The way we’re currently printing results looks something like this :
<@s.AfterSearchOnly> <#if response.resultPacket.resultsSummary.totalMatching != 0> <@s.Results> <#if s.result.metaData["personName"]??><strong> ${s.result.metaData["personName"]!} </strong> <br/> <#else> </#if> <#if s.result.metaData["personTitle1"]??> ${s.result.metaData["personTitle1"]!} <br/> <#else> </#if> <#if s.result.metaData["personDepartment1"]??> ${s.result.metaData["personDepartment1"]!} <br/> <#else> </#if> <#if s.result.metaData["personPhone1"]??> Tel: ${s.result.metaData["personPhone1"]!} <br/> <#else> </#if> </@s.Results> </#if> </@s.AfterSearchOnly>
which prints results like :
Jane Doe Chair|Professor|Co-Site Director, World Universtiy Florence|Academic Director History Department|World Universtiy - Florence|Global Programs|Academic Directors Tel: +1 000 111 2222|+1 333 444 5555|+1 666 777 8888|+1 999 101 1111 John Doe Professor with Chair|Professor with Chair|Co-Site Director, World Universtiy|Academic Director History Department|World Universtiy - Florence|Global Programs Tel: +1 121 131 1414|+1 151 161 1717|+1 181 191 2020|+1 212 222 2323
[UPDATE 3]
We control the data-model, the data, and the entry of the data, and at one point structured the data to have a field for every value, i.e., title1, title2, title3, and in this case, could print the results as desired, like so:
Jane Doe (personName) Chair (personTitle1) History Department (personDepartment1) Tel: +1 000 111 2222 (personPhone1) Professor (personTitle2) World Universtiy - Florence (personDepartment2) Tel: +1 333 444 5555 (personPhone2) Co-Site Director, World Universtiy Florence (personTitle3) Global Programs (personDepartment3) Tel: +1 666 777 8888 (personPhone3) Academic Director (personTitle4) Academic Directors (personDepartment4) Tel: +1 999 101 1111 (personPhone4) John Doe (personName) Professor with Chair (personTitle1) History Department (personDepartment1) Tel: +1 121 131 1414 (personPhone1) Professor with Chair (personTitle2) World Universtiy - Florence (personDepartment2) +1 151 161 1717 (personPhone2) Co-Site Director, World Universtiy (personTitle3) Global Programs (personDepartment3) +1 181 191 2020 (personPhone3) Academic Director (personTitle4) <!-- data missing --> (personDepartment4) - don't know if this is a problem (could it jumble the results) +1 212 222 2323 (personPhone4)
but search facets didn’t work, as they seem only to search on a single field. If we wanted to find all with title1 = "Academic Director" we wouldn’t see those title2, title3, etc. records with "Academic Director" values. It might be worthwhile to ask the vendor to enhance facets so they’re able to search on multiple fields.
I think you might be suggesting changing the functionality of how s.Results works? I believe that is provided by the application. We’d need to request an enhancement from the vendor. Or, are you suggesting doing something different with how the data is structured? If so, is it something different to having a field for every value (as above)?
One other possibility … the vendor offers hook scripts:
https://docs.funnelback.com/develop/programming-options/hook-scripts.html
The most commonly used hook scripts are the pre and post process hooks.
Pre-process: (hook_pre_process.groovy) This runs after initial question object population, but before any of the input processing occurs. Manipulation of the query and addition or modification of most question attributes can be made at this point.
Example uses: modify the user’s query terms; convert a postcode to a geo-coordinate and add geospatial constraints
Extra searches: (hook_extra_searches.groovy) This runs after the extra search question is populated but before any extra search runs allowing modification of the extra search’s question.
Example uses: add additional constraints (such as scoping) to the extra search.
Pre-datafetch: (hook_pre_datafetch.groovy) This runs after all of the input processing is complete, but just before the query is submitted. This hook can be used to manipulate any additional data model elements that are populated by the input processing. This is most commonly used for modifying faceted navigation.
Example uses: Update metadata, gscope or facet constraints.
Post-datafetch: (hook_post_datafetch.groovy) This runs immediately after the response object is populated based on the raw XML return, but before other response elements are built. This is most commonly used to modify underlying data before the faceted navigation is built.
Example uses: Rename or sort faceted navigation categories, modify live URLs
Post-process: (hook_post_process.groovy) This is used to modify the final data model prior to rendering of the search results.
Example uses: clean titles; load additional custom data into the data model for display purposes.
An addition hook script is available for working with cached documents.
- pre-cache: (hook_pre_cache.groovy) This is used to modify the cached document prior to display.
Assuming that tittle> 1 and title(n)=phone(n)=department(n).
In the data model all metadata values are assigned to one filed separated with pipe |
. You can split first metadata ie.: personTitle1 using FTL built-in ?split('|')
and list all assigned titles as single values.
Now that you have the first field broken down into individual values, you can divide the remaining fields using the same method, and additionally, you can display the corresponding value calling index of our personTitle1 so fields will match. Take a look at the following code snippet to better understand what I mean:
<#list s.result.metaData["personTitle"]?split("|") as person> ${s.result.metaData["personTitle"]!?split('|')[person_index]!} ${s.result.metaData["personDepartment"]!?split('|')[person_index]!} Tel: ${s.result.metaData["personPhone"]!?split('|')[person_index]!} </#list>
Hope this makes sense.
The intent was that templates won’t do such grouping, and it’s the responsibility of whatever creates the data-model. So as of 2.3.30 at least, there’s no built-in to do this (but I think it will have to be added, as this comes up constantly).
For now, if we must solve this purely in the template, you can do this (although it’s quite a perverse idea to solve things like this in a template):
<#-- Splits a list to groups, and calls the nested content for each group. Do NOT use this if the size of a group is above a few dozens, as it will become slow. --> <#macro listGroups items groupByName> <#local group = []> <#list items?sort_by(groupByName) as item> <#local groupByValue = item[groupByName]!> <#if item?is_first || groupByValue != lastGroupByValue> <#if group?size != 0> <#nested lastGroupByValue group> </#if> <#local group = []> <#local lastGroupByValue = groupByValue> </#if> <#local group += [item]> <#if item?is_last> <#nested groupByValue group> </#if> </#list> </#macro>
So that’s just a macro, and to actually list something grouped do something like this:
<@listGroups peopleDirectory "name"; personName, personEntries> Name: ${personName} <#list personEntries as entry> - Title: ${entry.title} - Department: ${entry.department} - Phone: ${entry.phone} </#list> </@listGroups>