Skip to content

HLT2 and Sprucing

Learning Objectives

  • Build further on information established in the dataflow lesson
  • Highlight resources related to HLT2 and Sprucing

Basics of Moore

The LHCb High-Level Trigger (HLT) is made of three stages: HLT1, HLT2 and Sprucing, which are introduced in detail during the dataflow lesson. During data-taking the HLT cannot store every event and must select which ones to keep, this is done through event selection algorithms called trigger lines. If any line at each stage has a positive decision (described as a line 'triggering' or 'firing') then the event is kept, otherwise the event is discarded1. The HLT2 and Sprucing stages of the trigger contain most of the collaboration's trigger lines (several thousands), utilising tens of thousands of algorithms. This lesson is focused on signposting for Moore, the application used for HLT2 and Sprucing, as it's the most common interaction analysts will have with the trigger software. More information on HLT1, which is implemented via Allen, can be found in the Allen documentation.

The Moore documentation has detailed pages on running and developing Moore. For line development, it is best to use the trigger configuration settings suggested/provided by experts, which are chosen as they best mimic the expected data-taking period. If you are unsure what the appropriate settings are or who the appropriate experts are, it is suggested to reach out to the relevant RTA/DPA working group liaisons.

Central ideas of an HLT2 line

For each event, a trigger line typically takes two steps:

  • Attempts to construct a candidate of interest in an event, and keeps the event if one exists.

  • If fired, persists relevant information related to the event, including the found candidate(s). "Relevant information" will depend on the line and the physics use case.

Therefore creating an HLT2 line involves carefully defining these two steps.

The tools at your disposal for constructing a candidate of interest are the same tools (such as ParticleFilter and ParticleCombiner) discussed during the lesson "Introducing Particle Combiners" and the basics of writing an HLT2 line (the correct syntax etc.) are covered in the Moore documentation's section "Writing an HLT2 line".

There are strong operational constraints on the HLT2 and Sprucing stages, thus a line must select candidates of interest with high signal efficiency (if possible), good purity, and take its fair share of available resources by saving an acceptable rate and bandwidth, while not slowing down the stage's throughput.

Throughput of a selection stage

  • Throughput is the number of events that can be processed by a selection stage per second.
  • Adding/developing a well behaved line should have no/negligible negative impact to the throughput of that stage.
  • The most common reasons for a line negatively impacting throughput are discussed below in Further considerations for a line.
  • Evaluated with suggestions from "Timing and Performance".

Signal Efficiency

  • The signal efficiency is the fraction of events that the line correctly selects, given a set of simulated events containing the target physics.
  • Evaluated with HltEfficiencyChecker.

Purity

  • The purity is the fraction of signal-like events selected, given minimum-bias events (like data-taking).
  • A high purity is equivalent to having a high background-rejection.
  • Can be evaluated with data and simulation via monitoring histograms and analysis-level studies.

Rate

  • Rate is the number of positive line decisions of a line per second of data-taking.
  • A highly-efficient high-purity line selecting an exclusive decay will have a rate very close to the true signal rate for that decay.
  • An example of how to calculate the true signal rate for a given decay is described in true signal rate.
  • A rate of 1 Hz would result in approximately 1e8 candidates saved in a single data-taking year. Having poor purity can result in triggering on and then processing millions/billions of non-signal candidates!
  • Evaluated with the bandwidth testing infrastructure.

Persistency

  • The persistency of a line is the information that is kept when a candidate is successfully found within an event.
  • The default is to persist nothing aside from the reconstructed candidates that triggered the line, and has a resultant size of approximately 10 kB per saved event.
  • There is the possibility to persist much more information from an event, e.g. other reconstructed particles from the same proton-proton interaction, all reconstructed photons in the event etc.
  • However, persisting anything more than the default must be requested and carefully justified on a case-by-case basis.
  • For evaluation, see below in Bandwidth.

Bandwidth

  • Size of data written by the line per second of data-taking.
  • This is directly proportional to the line rate, and the defined persistency of the line.
  • Considering the example of highly-efficient high-rate line selecting an exclusive B decay from above, if this had the default, minimum, persistency it would save approximately 10 TB within a single data-taking year.
  • This is why persisting extra information, or requesting to add a high-rate line must be very carefully justified.
  • This has to be decided with consideration of the requirements of the intended analysis, as well as the rate of the line and the budget for further changes that are available, so this requires discussion with the relevant RTA/DPA working group liaisons.
  • Evaluated with the bandwidth testing infrastructure.

Further considerations for a line

Control flow order: impact on throughput

Consider creating a line for a decay of \(B_s^0\to (D_s^- \to K^- K^+ \pi^-) \pi^+\). Schematically the (pseudo)code would look somewhat like:

from Moore.lines import Hlt2Line
def myLine(name="Hlt2WG_MyLine):
  bs = make_bs()
  return Hlt2Line(
    name=name,
    algs=[bs],
  )

Where make_bs() is an algorithm that generates the kaons and pions then combines them first into a \(D_s^-\) and then the \(B_s^0\). So from HLT2's perspective, this line contains a single algorithm to run.

Consider instead separating this into two algorithms, like so:

from Moore.lines import Hlt2Line
def myLine(name="Hlt2WG_MyLine):
  ds = make_ds()
  bs = make_bs(ds=ds)
  return Hlt2Line(
    name=name,
    algs=[ds, bs],
  )

The benefit of this is that this line will first attempt to construct the \(D_s^-\) candidate, and only if that is successful will then continue to try to make the \(B_s^0\) candidate using the \(D_s^-\) candidate. This has no effect on physics metrics (like efficiency or purity), but does enable HLT2 to determine if the line has failed to find a candidate earlier, and thus can improve the throughput for a line.

This also has the benefit of modularising the code. This is useful as related lines will often share algorithms, and is part of following the code design guidelines.

Combinatorics: impact on throughput

Continuing to consider \(B_s^0\to (D_s^- \to K^- K^+ \pi^-) \pi^+\) as an example. The \(D_s^-\) in this decay and will have kinematic requirements (e.g. invariant mass cuts on the combination). This can be done like so:

def make_ds():
  km = make_meson(species="km")
  kp= make_meson(species="kp")
  pim = make_meson(species="pim")
  return ParticleCombiner(
    [km, kp, pim],
    name="MyDsMaker",
    DecayDescriptor="[D_s- -> K- K+ pi-]cc",
    CompositeCut=F.math.in_range(
      minimum_ds_mass, F.MASS, maximum_ds_mass
    ),
  )

Noting that there are many pions and kaons generated within a single event at the LHCb, generating a potential \(D_s^-\) candidate for every triplet before evaluating the requirements can take quite a substantial length of time and harm throughput for the selection stage.

This can be improved by reducing the multiplicity of input particles, by adding cuts motivated by the relevant physics. For instance, the pions and kaons from the \(D_s^-\) candidate will be 'detached', i.e. will not originate from the p-p collision directly but are produced later. Therefore cutting on a relevant quantity (for instance F.OWNPVIPCHI2) can help distinguish the 'prompt' and detached mesons, thus reducing the combinations done and improving throughput.

The combinatorics can also be improved by adding some cuts on the inputs before the combination. The usual cut, composite_cut, filters on the candidate after combining the inputs, including a time-consuming vertex fit of the inputs. (For more information on exactly how a combination is done in the LHCb see: the relevant documented code).

An extra cut, called combination_cut, can be added which filters on the 'naive combination' before this vertex fit is done, this saves time and improves throughput. Importantly, as the combination_cut occurs before the vertex fit, some kinematic quantities will differ before and after. Therefore it's best practice to have the composite_cut at least as tight as the combination_cut.

We can implement these two suggestions like so:

def make_ds():
  ## Make input particles via a function with an implicit cut to remove prompt particles
  km = make_detached_meson(species="km")
  kp = make_detached_meson(species="kp")
  pim = make_detached_meson(species="pim")

  # A cut on relevant quantities, to be applied before the vertex fit
  combination_cut = ...
  # A tighter cut on the same quantities, to be applied on the actual candidate, i.e. after the vertex fit
  composite_cut = ...
  return ParticleCombiner(
    [km, kp, pim],
    name="MyDsMaker",
    DecayDescriptor="[D_s- -> K- K+ pi-]cc",
    CombinationCut=combination_cut,
    CompositeCut=composite_cut,
  )

Pre-filters: impact on rate & throughput

By default, a selection line will run the contained algorithms on any event provided. However, this can be modified by use of the hlt1_filter_code (and the hlt2_filter_code which is Sprucing specific), which requires a line(s) from a previous stage have a positive decision in the event before the selection line will run.

This can have throughput improvements, as your algorithm will run less often, but it can also have physics and rate implications as it might filter out events that the line could have found a candidate in. Which hlt1/hlt2 filters are relevant for a specific line will depend on the specific physics use-case. If unsure, reach out to the relevant working group for discussion.

For example, in \(B_s^0\to (D_s^- \to K^- K^+ \pi^-) \pi^+\), one could add an HLT1 filter of "Hlt1.*Track.*MVADecision" which ensures that the HLT2 line only runs when a relevant HLT1 line (TrackMVA, TwoTrackMVA or TrackMuonMVA or etc.) line has had a positive decision in that event. As analysts typically introduce cuts on trigger related information when performing analyses to have well understood trigger efficiencies, these are encouraged to be implemented in the trigger lines explicitly to save rate, bandwidth and throughput.

  • An example of how to add this to a trigger line can be seen in this test script.

Monitoring: understanding purity

During data-taking, HLT2 automatically generates monitoring plots, and these are routinely made available to analysts to understand the performance of their lines before the data gets processed by Sprucing and becomes available to analysts. Consider again \(B_s^0\to (D_s^- \to K^- K^+ \pi^-) \pi^+\). For this line, the default monitoring will generate histograms describing the \(B_s^0\) candidate (e.g. invariant mass).

However, these defaults may not be sufficient for the specific physics intended for the line, so it might be useful to monitor other information. For example, also monitoring the intermediate states or monitoring specific relevant variables that aren't done by default. For this, there's custom monitoring available.

A common use-case for monitoring intermediate states arises for rare candidates, as the custom monitoring can be filled every time the \(D_s^-\) is successfully found, even if the combination with the \(B_s^0\) is not possible in that event. Thus monitoring the intermediate states is useful to understand the performance of the line during development and data-taking

Coding guidelines

Code should be clear, readable and maintainable, and it should be possible for any analyst to understand any line. Unnecessary duplication should be avoided, while code comments and docstrings are encouraged. This both helps future analysts, and can prevents bugs and unintended consequences by reducing maintenance burden.
There are two resources for code design of HLT2/Sprucing lines, and the relevant RTA/DPA working group liaisons will also have suggestions:

Sprucing lines

By design, writing trigger lines for Sprucing follows effectively all of the above discussion on HLT2 lines and are run in the same framework. Remember, as discussed in the dataflow lesson, HLT2 lines that go to the Turbo or TurCal stream don't have to write relevant Sprucing lines, they are instead handled via a 'passthrough' Sprucing instead.

Differences of note:

  • HLT2 lines can have a pre-filter requiring certain HLT1 line decisions. Sprucing lines can also have the same pre-filters and can pre-filter on HLT2 line decisions as well.

  • HLT2 lines have monitoring that are run during data-taking and run locally by analysts. Sprucing has the same monitoring system but it is not run during data-taking. Thus sprucing monitoring is still important but instead mostly used for tests, validations and evaluating line performance.

  • Sprucing lines have very similar syntaxes to HTL2 lines, see an almost-real example of implementing the same line at HLT2 and Sprucing below:

@register_line_builder(all_lines)
@configurable
def z_to_mu_mu_single_nomuid_line(
    name="Hlt2QEE_ZToMuMu_SingleNoMuIDFull", prescale=1, persistreco=True
):
    """Z0 boson decay to two muons line, where one requires ismuon, for efficiency studies"""

    z02mumu = make_Z_cand_SingleNoMuID()
    monitor_z0 = qee_default_monitors(
        particles=z02mumu, linename=name
    )
    return Hlt2Line(
        name=name,
        algs=upfront_reconstruction() + [z02mumu, monitor_z0],
        prescale=prescale,
        persistreco=persistreco,
        hlt1_filter_code=["Hlt1SingleHighPtMuon"],
        monitoring_variables=("pt", "eta", "n_candidates"),
    )

@register_line_builder(sprucing_lines)
@configurable
def z_to_mu_mu_single_nomuid_sprucing_line(
    name="SpruceQEE_ZToMuMu_SingleNoMuID", prescale=1, persistreco=True
):
    """Z0 boson decay to two muons line, where one requires ismuon, for MuonID efficiency studies. passthrough after Hlt2QEE_ZToMuMu_SingleNoMuIDFull"""
  z02mumu = make_Z_cand_SingleNoMuID()
  monitor_z0 = qee_default_monitors(
        particles=z02mumu, linename=name
    )
    return SpruceLine(
        name=name,
        algs=upfront_reconstruction() + [z02mumu, monitor_z0],
        prescale=prescale,
        persistreco=persistreco,
        hlt1_filter_code=["Hlt1SingleHighPtMuon"],
        hlt2_filter_code=["Hlt2QEE_ZToMuMu_SingleNoMuIDFull"],
        monitoring_variables=("pt", "eta", "n_candidates"),
    )

Summary

There are many considerations to keep in mind when developing trigger lines for HLT2 and/or Sprucing, and while this lesson aims to be thorough in explaining these conditions, it cannot be exhaustive. Any questions which arise during development that are not answered satisfactorily in this document can be answered by the relevant RTA/DPA working group liaisons who, at the least, can direct you to relevant groups of experts.
(Of course, please do feel free to suggest additions/ edits of this lesson!)

  1. Technically, passing a selection line at each stage is not the only way for an event to be saved. During data-taking, there is a small probability of keeping an event, regardless of any information about the event itself. This data, often called the 'passthrough' or 'nobias' data stream, is to allow unbiased studies for future ideas that were not considered/possible when writing the current trigger. ↩