> For the complete documentation index, see [llms.txt](https://sunbird.gitbook.io/sunbird-spark/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://sunbird.gitbook.io/sunbird-spark/misc/reference-guides/flink-jobs-reference.md).

# Flink jobs reference

Spark uses Apache Flink for stream processing — event-driven jobs that process Kafka events in real time. This page documents the Flink jobs that are active in a default Spark installation.

#### Namespaces

Flink jobs run in two Kubernetes namespaces:

* kp-flink-\<env> — jobs related to content knowledge management (publishing, indexing)
* flink-\<env> — jobs related to learning and data pipeline (certificates, telemetry)

### Active Jobs in Spark

{% tabs %}
{% tab title="kp-flink namespace (knowledge platform)" %}

| publish-pipeline      | Content publish API               | Processes content after it is submitted for publish. Generates ECAR packages (if enabled), processes media, and marks content as Live. Also handles QuestionSet publish — content and QuestionSet publish are merged into a single job in Spark. |
| --------------------- | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| search-indexer        | learning.graph.events Kafka topic | Syncs content metadata changes from JanusGraph to Elasticsearch.                                                                                                                                                                                 |
| audit-event-processor | Content lifecycle events          | Processes content audit events for governance tracking.                                                                                                                                                                                          |
| {% endtab %}          |                                   |                                                                                                                                                                                                                                                  |

{% tab title="flink namespace (learning and data pipeline)" %}

| collection-certificate-generator | issue.certificate.request Kafka topic | Evaluates learner completion against batch certificate rules. Generates SVG certificates and stores them in Sunbird RC. This job is a merger of the old collection-cert-pre-processor and collection-certificate-generator from Sunbird ED. |
| -------------------------------- | ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| user-cache-updater-v2            | Scheduled                             | Populates the Redis user cache used by data pipeline jobs for telemetry denormalisation (adding user metadata to raw telemetry events).                                                                                                     |
| telemetry-extractor              | telemetry.raw Kafka topic             | Extracts individual telemetry events from batch payloads submitted by client apps.                                                                                                                                                          |
| pipeline-preprocessor            | Extracted telemetry events            | Deduplicates events (checks Redis for UUID), validates events against the telemetry specification, and routes valid events forward.                                                                                                         |
| de-norm                          | Validated telemetry events            | Enriches events with denormalised metadata (user name, course name, content title) by looking up the Redis user and content caches.                                                                                                         |
| druid-validator                  | Denormalised events                   | Final validation before events are written to Druid. Only active when Obsrv add-on is enabled.                                                                                                                                              |
| {% endtab %}                     |                                       |                                                                                                                                                                                                                                             |
| {% endtabs %}                    |                                       |                                                                                                                                                                                                                                             |

#### Jobs removed from Spark

The following jobs from Sunbird ED have been removed. See the [ED → Spark Migration Guide](/sunbird-spark/use/ed-greater-than-spark-migration/flink-jobs-removed-in-spark.md) for the full list and replacements.

<br>


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://sunbird.gitbook.io/sunbird-spark/misc/reference-guides/flink-jobs-reference.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.