Exploratory Search with OrgBox - Interface/Aplicação para suporte a Atividades MetaCognitivas - Leitura de Artigos

Bogeum Choi, Jaime Arguello, Robert Capra, and Austin R. Ward. 2021. OrgBox: A Knowledge Representation Tool to Support Complex Search Tasks. In Proceedings of the 2021 Conference on Human Information Interaction and Retrieval (CHIIR '21). Association for Computing Machinery, New York, NY, USA, 219–228. https://doi.org/10.1145/3406522.3446029

[Atividades Metacognitivas]

These features were designed to support users with developing sub-goals, saving and (re-)organizing information, representing relationships, and monitoring their progress. ... a knowledge representation tool because it enables a searcher to externalize and modify their mental representation of a complex domain as they gather information. While there are numerous theories on how knowledge is represented in the human mind, fundamental components include facts, concepts, and relations [ 4, 6, 20 ]... In terms of metacognitive activities, prior work suggests that external knowledge representations can support goal-setting, monitoring progress, and revising approaches to the task [27].

[Em KG lidamos com alegações ao invés de fatos uma vez que estamos lidando com o Mundo Aberto Dual]

Research in cognitive psychology argues that externalizing knowledge structures can help individuals store, structure, utilize, and acquire knowledge [ 11, 17 ]. Additionally, creating visual knowledge representations can help individuals create more coherent representations of new information [ 11 ]. In education, it is argued that externalizing knowledge can help students organize, analyze, evaluate, and generate ideas [ 5, 12 , 23].

[Os mapas mentais por exemplo como ferramentas de aprendizado]

More closely related to IIR, the role of knowledge structures has also been studied in the context of sense-making, which involves "making sense" of a complex domain or task. Russell et al . [18] originally conceptualized sense-making as a cyclical process (the so called "learning loop complex"). The first stage involves searching for a good representation (i.e., the "generation loop"). The second stage involves finding and encoding information using the latest representation (i.e., the "data coverage loop"). The data coverage loop often involves identifying information that does not fit neatly into the latest representation (i.e., the "residue") resulting in a need to adjust the current representation to have better coverage (i.e., the "representation shift loop"). Russell et al . [18] called for a tighter integration between IR systems and knowledge representation tools. In a position paper, Stefik et al . [22] argued that when a sense-making task is difficult, sense-makers use external representations to store information for repeated manipulation.  

[18] Daniel M. Russell, Mark J. Stefik, Peter Pirolli, and Stuart K. Card. 1993. The Cost Structure of Sensemaking. In Proceedings of the INTERACT ’93 and CHI ’93 Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 269ś276.

3 METHODS
3.1 Study Overview & Protocol
3.2 Tasks
3.3 Search Tools
3.4 Post-task Questionnaire

[4 Research Questions]
[Grupo de participantes, Tarefas e Baseline de comparação]
[Log de Ações na Ferramenta]
[Questionário e a percepção sobre o conhecimento adquirido]
[Análise Quantitativa e Qualitativa]

4 RESULTS

Cognitive activities: Participants described four main categories of cognitive activities that were supported by the OB: (1) classifying and sorting, (2) remembering and navigating, (3) saving switching costs, and (4) creating the outline.

Complex information-seeking tasks involve a range of cognitive activities such as searching, reading, evaluating, organizing, and writing.

Metacognitive activities: Participants described three main types of metacognitive activities supported by the OB: (1) planning, (2) monitoring, and (3) revising and updating. First, many participants (n = 10) mentioned that the OB caused them to think about the outline (the end goal) at an earlier stage, which also allowed them to guide their search accordingly.

The visual representations in the OB helped them identify knowledge gaps. Specifically, the number of boxes allowed participants to monitor the breadth of information gathered, while the number of passages in each box allowed them to monitor the depth.

Leveraging information found serendipitously: Several participants (n = 3) mentioned that the OB allowed them to easily incorporate unexpected but relevant information encountered during their search.

Mismatch between boxes and information found: Participants (n = 3) reported feeling concerned when they encountered potentially useful information that did not fit into any of the boxes created in the OB.

[Aspectos do aprendizado adquiridos com a Busca]

5 DISCUSSION

Learning: Participants noted that the OB helped them to identify important topics, recognize relationships between topics, and internalize information. In terms of important topics, participants mentioned that the OB helped them keep track of recurring themes (e.g., boxes with lots of information).

Cognitive activities: Participants noted how the OB helped them with organizing information, connecting ideas, and structuring the outline. Additionally, comments suggest that the OB helped participants with switching between different cognitive activities. During the search phase of the task, the OB helped participants explore topics in parallel (versus sequentially). This is important during complex tasks since articles often contain information about different aspects of a topic. Participants also mentioned that the OB helped them to incorporate information found serendipitously and then easily re-engage with their previous search process.

Metacognitive activities: Participants described the OB as being helpful with metacognitive activities including monitoring and updating/revising. Monitoring is an important metacognitive activity that can support revising plans for subsequent searches [21].

Participants also described how the OB helped with metacognitive updating (i.e., changing/revising one’s own knowledge structures based on new information).

6 CONCLUSION

Our results also have implications for designing systems to support users with complex search tasks. First, our results suggest that support tools should include features to spatially organize information and to make annotations, allowing searchers to create knowledge structures with both visual and textual cues.

[Diretrizes para aplicações e interfaces. E o que a camada de dados deve oferecer para isto? É importante associar o contexto para facilitar a organização?]

Austin R. Ward and Robert Capra. 2021. OrgBox: Supporting Cognitive and Metacognitive Activities During Exploratory Search. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '21). Association for Computing Machinery, New York, NY, USA, 2570–2574. https://doi.org/10.1145/3404835.3462790

ABSTRACT

... search-assistance tool called the OrgBox, created to support users’ organizational, cognitive, and metacognitive activities while performing exploratory search tasks. ... also encourages users to engage in metacognitive activities during their search process (e.g., planning their next steps, monitoring their progress, evaluating the information found so far)

[O processo de busca é interativo e iterativo. Planejado em ciclos.]

1 INTRODUCTION

Modern search systems provide excellent support for fact-finding and lookup tasks, but do less to help searchers engaged in exploratory searches [13,25]. Exploratory searches are by nature complex, leading users to engage in high-level cognitive activities including learning, exploration, and synthesis [26,39]. Successful exploratory searches also require metacognitive activities including planning, monitoring, evaluation, and reflection [26,39].

Research in the fields of education and learning sciences has shown that planning, monitoring, and evaluation are specific metacognitive activities that can benefit users when they are learning about new concepts [8, 27, 42].

[Search as Learning]
[Entender, Analisar, Comparar ... mas para agir requer Confiar, para confiar é preciso perceber que existe um risco da informação não ser 100% verdadeira ou de que ela é verdadeira de acordo com o Contexto]
[Capacidade de fazer perguntas sobre as informações recuperadas: Quando, Onde, Quem, Como, Por que]

2 RELATED WORK

Much prior research has focused on helping searchers with discovery and exploration of an information space. For example, faceted search interfaces [37], methods of results visualization [21,40], and result clustering [9,22] have all been investigated as ways to help users explore an information space. Other work has focused on helping users to save information they find during an exploratory search. ... Prior work has also explored tools to help users with the process of document triage – reviewing a set of results to quickly determine which documents are of the most importance [3–7].

[Diversas abordagens para as aplicações / interfaces. Mas quais características que uma base de conhecimento e a sua linguagem de consulta devem possuir para permitir estas ações?]

3 ORGBOX

Externalizing knowledge structures – Users could externalize (and update) their current knowledge structures by creating and modifying “boxes” in which they could place information items.

[O conhecimento prévio e o que vai sendo criado apartir da busca]

Annotating information – The OrgBox also allowed users to add a note (annotation) to save with each item (Figure 3). In the main OrgBox display, notes (if present) were shown below the information item next to a yellow “note” icon (Figure 1, right). These annotations could be used to provide context about why an item was saved, how the information relates previously collected information, or to highlight how the information might be useful in the future.

Users could create hierarchies of items by dragging elements to nest them underneath other items. Items that were nested below another would be indented further than the parent item (Figure 1, right). Item hierarchies could be 3 levels deep.

Support for metacognitive activities. Metacognition is often referred to as “thinking about thinking” [17] and includes activities such as planning, monitoring, evaluation, and reflection. Metacognitive activities play important roles in learning [2].

[Conhecimento muda com a própria busca]

Users could clearly see what articles the information they collected was sourced from (e.g., was a breadth of sources used?) and how much information was collected for each sub-topic (e.g., was there enough information collected about each sub-topic?).

Evaluating and Reflection – With the collected information, users could see how information items might be related to each other (e.g., using the information item hierarchy), and how important each sub-topic might be to the overall goal (e.g., the order and position of the boxes). By showing users a visual display of the information they had found and how they had currently organized it, ...

Revising and Updating – The OrgBox enabled users to revise and update their knowledge structures through the relabeling or reordering of the boxes and through the reordering and re-positioning of the items.

4 IMPLEMENTATION

5 OBSERVED BENEFITS

... monitor the breath and depth of collected information - often influencing succeeding search activities [32]; and (3) revise/update their understanding of the collected information through restructuring and relabeling the boxes and items (which is considered to be an important part of learning during a search process [38]).

6 DISCUSSION

Cognitive psychology research has argued that externalizing knowledge structures can assist users with storing, structuring, utilizing, and acquiring knowledge [24,29]. Researchers in sense-making and knowledge structures have argued for tighter integration of IR systems and knowledge representations tools [31]. Similarly, the “search as learning” community in information retrieval investigates how learning during search occurs and how it can be supported by search interfaces and tools [11, 16, 20].

[E se a ferramenta já tivesse placeholders para informações de Contexto?]

Anita Crescenzi, Austin R. Ward, Yuan Li, and Rob Capra. 2021. Supporting Metacognition during Exploratory Search with the OrgBox. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '21). Association for Computing Machinery, New York, NY, USA, 1197–1207. https://doi.org/10.1145/3404835.3462955

1 INTRODUCTION

In addition, successful exploratory searches often require users to engage in metacognitive activities such as monitoring progress toward (sub-)goals, assessing ones’ current state of knowledge, and planning a strategy to move forward [7, 26]. Researchers in the field of education have shown that specific training and tools can encourage students to engage in metacognitive behaviors and can improve learning outcomes [8, 29, 44].

2 RELATED WORK

2.1 Support for Exploratory Search

Exploratory search is an iterative, multi-tactical process, that requires users to explore an information space to fulfill an open-ended information need [41]. Information search for exploratory tasks is complex and involves active engagement from users during the search process [27].

2.1.2 Tools to support note-taking and synthesis. To conduct learning activities (e.g., comparison, analysis, evaluation, discovery, integration/synthesis) during an information seeking process, users need to actively identify important concepts and determine relationships among the acquired information [27]. However, studies found that users often encounter problems with these tasks, including difficulties navigating between different documents, keeping track of materials, taking notes, and maintaining awareness of retrieved information and their own thoughts through the process [38].

[Recursos adicionais que ferramentas de Busca na Web não oferecem nativamente]

2.2 Metacognition

Exploratory search also includes metacognitive aspects (i.e., “thinking about thinking” [17]) as searchers engage in the search and learning processes. Metacognition refers to “the conscious self-awareness of one’s own knowledge of task, topic, and thinking, and the conscious self-management (executive control) of the related cognitive processes." [30, p.364] (describing [22]).

In the education community, researchers have found that students often do not engage in metacognitive activities of planning, evaluation, and monitoring without encouragement or support [1, 2 , 42].

Research has also indicated potential benefits of encouraging people engaged in search-based learning tasks to engage in metacognitive activities [2]. ... Participants who received metacognitive evaluation prompts performed better on post-task assessments about knowledge of sources. They also found that providing ontological categories encouraged participants to save more information and to better structure their notes.

[Motivação]

3 SYSTEM DESIGN

4 METHOD
4.1 Study Overview & Protocol
4.2 Tasks
4.3 Data collection
4.3.1 Logged system interactions
4.3.2 Questionnaires
4.4 Data analysis
4.4.1 Creating measures

5 RESULTS

6 DISCUSSION
 
We do however note that we observed a trend in our data that participants issued more queries and viewed more articles with the OrgBox than with the OrgDoc.

These findings suggest that the OrgBox encouraged people to create finer-grained knowledge structures about the topic during their search, and that they were encouraged to save more overall information into those structures.

RQ4: Perhaps our most interesting results are for RQ4. Participants reported that the OrgBox provided greater support for three specific types of cognitive and metacognitive activities: understanding and synthesizing the topic, monitoring and tracking the coverage of the notes, and evaluating the task process and progress. Support for metacognitive planning almost reached significance (𝑝=.059). These findings are important for several reasons. First, they are consistent with results of a previous study [9] which compared the OrgBox tool to a bookmarking tool baseline.

7 CONCLUSION

Our results provide evidence that tools that integrate information organization and synthesis with search can provide metacognitive support for users attempting to learn about a complex topic.

It encouraged users to save more information and to create more structures in their notes as compared against the baseline OrgDoc rich-text editor. This suggests that systems that integrate information seeking with tools to organize and save information into groups can be an effective way to encourage users to engage in more metacognitive activities during search.

[Learning to Learn]

OBSERVAÇÃO: o artigo não considera como metacognição fazer novas perguntas sobre o que for aprendendo, desenvolver competência em informação.

https://versant-pesquisadedoutorado.blogspot.com/2022/06/knowledge-context-in-search-systems.html