> For the complete documentation index, see [llms.txt](https://sunbird.gitbook.io/sunbird-spark/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://sunbird.gitbook.io/sunbird-spark/misc/reference-guides/context-indexing-flow.md).

# Context indexing flow

This page explains how content created in Spark becomes discoverable via search.

#### The flow — step by step

```
1. Content created/updated via API
         │
         ▼
2. knowlg-service writes metadata to Janus Graph
         │
         ▼
3. Neo4j kernel extension detects the write
   Appends event to the change log file:
   /opt/bitnami/janusgraph/logs/cdc-events.log
         │
         ▼
4. Logstash service reads the log file
   Publishes events to Kafka topic:
   learning.graph.events
         │
         ▼
5. Flink search-indexer job consumes the topic
   Creates or updates the document in OpenSearch
   (compositesearch index)
         │
         ▼
6. Content is now discoverable via
   /api/content/v1/search

```

#### Search indexer configuration

The search indexer's behaviour is controlled by its configuration file.\
\
**Key settings:**

* **nested.fields** — defines which metadata fields are stored as OpenSearch nested objects (affects how they can be queried)
* **restrict.objectTypes** — object types listed here are excluded from the OpenSearch index. By default, internal node types (License, Framework, Channel) are excluded — only content visible to learners is indexed.

#### Re-indexing

If the OpenSearch index becomes out of sync with Janus Graph (e.g., after an index rebuild or a Flink job failure), a full re-index can be triggered:

```
# Trigger a full Janus Graph  → OpenSearch sync
kubectl exec -it <knowlg-service-pod> -n default -- \
curl -X POST http://localhost:9000/v1/index/sync
```

A full re-index on a large deployment (millions of content items) can take several hours. The alias architecture (compositesearch → actual index) means search continues to work during re-indexing — the alias is switched to the new index atomically when the re-index is complete.

<br>


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://sunbird.gitbook.io/sunbird-spark/misc/reference-guides/context-indexing-flow.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.