Introduction
sched_ext is a Linux kernel feature that allows BPF programs to implement custom CPU scheduling policies, replacing or extending the default scheduler at runtime without modifying kernel source.
This book covers:
- A repository overview and high-level architecture guide
- A complete conceptual explainer of how sched_ext works
- Patch-by-patch analysis across all 42 patches in the upstream submission
Recommended Reading Path
- New to sched_ext? Start with How sched_ext Works for a complete conceptual overview of the architecture, data flow, and BPF ops interface.
- Want the big picture first? Read the High-Level Study Guide for a structured summary of the patch series goals and design decisions.
- Ready to dive into patches? Use the Patch Study section to trace the implementation patch by patch.
Commit messages and diffs are preserved verbatim from the original mailing list submissions.
sched_ext (Extensible Scheduler Class) Patch Series Documentation
This repository documents and analyzes the sched_ext BPF extensible scheduler patch series (v7, June 2024).
Overview
The project includes:
- A main study guide: sched-ext-patch-study.md
- Per-patch analysis files:
patch-study/patch-01.mdthroughpatch-study/patch-30.md - Original source material:
source/scx-v7-patch.mbox,source/scx-v7-patch.mbox.gz, andsource/original-patches/ - Utility scripts for parsing and analysis generation in
scripts/
Patch Categories
- Foundational refactoring: patches 01-07
- sched_ext core: patches 08-11
- Debugging and monitoring: patches 12-16
- CPU coordination: patches 17-19
- Task management: patches 20-23
- System integration: patches 24-28
- Documentation and testing: patches 29-30
What Each Patch File Contains
Each patch-study/patch-XX.md includes:
- Commit message and rationale
- Implementation analysis
- Diff-focused breakdown
- Link to the lore.kernel.org discussion thread
Key Concepts
- BPF-based scheduler extensibility through
sched_ext - Safety via BPF verification and watchdog mechanisms
- Automatic fallback behavior when scheduler failures occur
- Dispatch queue (
dsq) and task lifecycle control
References
- Kernel repository: https://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext.git
- sched_ext examples: https://github.com/sched-ext/scx
- Community Slack: https://bit.ly/scx_slack
Notes
- Patch series version: v7 (June 2024)
- Target kernel line: Linux 6.11+
sched_ext Patch Series: High-Level Study Guide
This guide is written to stand alone. Reading only this document should give you a solid understanding of what sched_ext is, how it works, and why it was built the way it was.
What sched_ext Is
sched_ext is a Linux scheduler class where scheduling policy lives in a BPF program
instead of the kernel. It was merged into Linux 6.11.
Traditional Linux schedulers (CFS, RT, Deadline) are static policies compiled into the kernel. Changing scheduling behavior means modifying kernel source, recompiling, and rebooting. For datacenter operators and researchers who need to tune scheduling for specific workloads (ML training jobs, game servers, low-latency services), this iteration cycle is too slow.
sched_ext solves this by separating:
- Mechanism (kernel): runqueue management, preemption, CPU affinity enforcement, safety checks
- Policy (BPF): which task runs next, on which CPU, for how long
A BPF program can be loaded and unloaded at runtime. If it misbehaves, the kernel detects it and falls back to CFS automatically — no panic, no reboot.
Where sched_ext Fits in the Scheduler Hierarchy
Linux uses a chain of struct sched_class objects. The kernel walks this chain in priority
order: higher-priority classes are checked first. sched_ext inserts ext_sched_class
between CFS and the idle class:
stop_sched_class highest priority — stop-machine tasks (SMP only)
│
dl_sched_class SCHED_DEADLINE — earliest deadline first
│
rt_sched_class SCHED_FIFO / SCHED_RR — real-time tasks
│
fair_sched_class SCHED_NORMAL / SCHED_BATCH — CFS (the common case)
│
ext_sched_class SCHED_EXT — BPF-controlled ← sched_ext
│
idle_sched_class lowest priority — per-CPU idle thread
A task uses ext_sched_class only if:
- Its scheduling policy is
SCHED_EXT(set viasched_setscheduler(2)), and - A BPF scheduler is currently loaded.
If no BPF scheduler is loaded, SCHED_EXT tasks automatically fall back to CFS. RT and
Deadline tasks are never handled by sched_ext — they always outrank it.
The BPF Interface: struct sched_ext_ops
The BPF program fills in a struct sched_ext_ops — a vtable of callbacks. The kernel calls
these callbacks at the right points in the scheduling lifecycle. Only .name is mandatory;
everything else has a sensible default.
struct sched_ext_ops {
char name[SCX_OPS_NAME_LEN]; /* required: identifies this scheduler */
/* --- CPU selection --- */
/* Pick which CPU should run task p. Return -1 to let kernel decide.
* Called when p wakes up. Can dispatch directly here (fast path). */
s32 (*select_cpu)(struct task_struct *p, s32 prev_cpu, u64 wake_flags);
/* --- Task placement --- */
/* p became runnable. BPF must call scx_bpf_dispatch() to place it in a DSQ. */
void (*enqueue)(struct task_struct *p, u64 enq_flags);
/* p is being removed (class change, migration). Remove from BPF data structures. */
void (*dequeue)(struct task_struct *p, u64 deq_flags);
/* CPU needs a task. BPF calls scx_bpf_consume() to move tasks from custom
* DSQs into this CPU's local DSQ. */
void (*dispatch)(s32 cpu, struct task_struct *prev);
/* --- Task state change notifications --- */
void (*runnable) (struct task_struct *p, u64 enq_flags); /* p became runnable */
void (*quiescent)(struct task_struct *p, u64 deq_flags); /* p blocked/exited */
void (*running) (struct task_struct *p); /* p started on CPU */
void (*stopping) (struct task_struct *p, bool runnable); /* p leaving CPU */
/* --- Task lifecycle --- */
/* Called when p first joins SCHED_EXT. Allocate per-task BPF state here.
* Return error to reject the task (it stays on CFS). */
s32 (*init_task)(struct task_struct *p, struct scx_init_task_args *args);
void (*exit_task)(struct task_struct *p, struct scx_exit_task_args *args);
void (*enable) (struct task_struct *p); /* p is now managed by this scheduler */
void (*disable) (struct task_struct *p); /* p is leaving this scheduler */
/* --- CPU lifecycle --- */
void (*cpu_online) (s32 cpu); /* CPU hotplugged in */
void (*cpu_offline)(s32 cpu); /* CPU hotplugged out */
void (*cpu_acquire)(s32 cpu, struct scx_cpu_acquire_args *a); /* CPU now exclusively ours */
void (*cpu_release)(s32 cpu, struct scx_cpu_release_args *a); /* CPU taken by CFS/RT */
/* --- Scheduler lifecycle --- */
s32 (*init)(void); /* BPF scheduler loaded */
void (*exit)(struct scx_exit_info *ei); /* BPF scheduler unloading */
/* --- Control knobs --- */
u64 flags; /* SCX_OPS_* flags */
u32 timeout_ms; /* watchdog timeout; 0 = default (30s) */
u32 exit_dump_len; /* bytes of BPF debug ring to print on exit */
};
The three most important callbacks
ops.enqueue(p, flags) — the scheduling decision.
This is called every time a task becomes runnable (wakeup, fork, preemption). The BPF program
decides where to put the task by calling scx_bpf_dispatch(p, dsq_id, slice, flags). If your
scheduler only implements this one callback, it already has a complete (FIFO) policy.
ops.dispatch(cpu, prev) — feeding CPUs.
Called when a CPU is idle and needs a task. The BPF program calls scx_bpf_consume(dsq_id)
to pull the next task from a custom DSQ into this CPU's local queue. If you only use the
global DSQ (SCX_DSQ_GLOBAL), you don't need to implement this — the kernel drains it
automatically.
ops.select_cpu(p, prev_cpu, flags) — CPU affinity.
Called when a task wakes up. Returning a valid CPU triggers a "direct dispatch" optimization:
if that CPU's local queue is empty, the task is dispatched there immediately, skipping
ops.enqueue entirely. This is a critical performance path for latency-sensitive workloads.
Dispatch Queues (DSQs): The Core Abstraction
A Dispatch Queue (DSQ) is a queue of tasks waiting to be scheduled. This is the central abstraction in sched_ext. There are three kinds:
1. Global DSQ — SCX_DSQ_GLOBAL (id = 0)
A single FIFO queue shared across all CPUs. Any idle CPU can pull from it. This is the simplest possible dispatch model:
BPF: scx_bpf_dispatch(p, SCX_DSQ_GLOBAL, SCX_SLICE_DFL, 0)
→ task sits in global FIFO
→ any idle CPU dequeues it and runs it
A scheduler that only calls scx_bpf_dispatch(p, SCX_DSQ_GLOBAL, ...) in ops.enqueue is
functionally a global FIFO scheduler with zero BPF complexity.
2. Per-CPU Local DSQs — SCX_DSQ_LOCAL (id = 1)
Each CPU has its own local queue. A CPU only runs tasks from its own local DSQ. This is how per-CPU affinity works: dispatch to a specific CPU's local DSQ and that CPU will run it.
BPF: scx_bpf_dispatch(p, SCX_DSQ_LOCAL, SCX_SLICE_DFL, 0)
→ task goes to this CPU's local DSQ
→ only this CPU will run it
To dispatch to a specific CPU's local DSQ from ops.dispatch:
scx_bpf_dispatch(p, SCX_DSQ_LOCAL_ON | target_cpu, slice, flags);
3. Custom DSQs — BPF-defined
BPF programs can create any number of DSQs:
scx_bpf_create_dsq(my_dsq_id, NUMA_node); /* in ops.init() */
scx_bpf_destroy_dsq(my_dsq_id); /* in ops.exit() */
Custom DSQs hold tasks until ops.dispatch() consumes them:
/* in ops.dispatch(): move tasks from my_dsq into this CPU's local queue */
scx_bpf_consume(my_dsq_id);
Custom DSQs can be FIFO (insertion order) or vtime-ordered (sorted by virtual time for weighted fair scheduling).
Task flow through DSQs
Task wakes up
│
▼
ops.select_cpu(p) ← BPF picks CPU; can direct-dispatch here
│
▼ (if no direct dispatch)
ops.enqueue(p) ← BPF calls scx_bpf_dispatch(p, dsq_id, slice, flags)
│
▼
[Task sits in DSQ]
│
▼ (when CPU needs work)
ops.dispatch(cpu) ← BPF calls scx_bpf_consume(dsq_id) to move task
│ from custom DSQ → CPU's local DSQ
▼
[Task in local DSQ]
│
▼
CPU picks task ← kernel: pick_next_task_scx()
│
▼
ops.running(p) ← task is executing
│
▼
Time slice expires / preempted
│
▼
ops.stopping(p) ← task leaving CPU
│
├─ still runnable? → ops.enqueue(p) again
└─ blocked? → ops.quiescent(p)
A Complete Minimal BPF Scheduler
This is a fully working sched_ext scheduler. It puts every task in the global FIFO queue.
/* minimal.bpf.c */
#include <scx/common.bpf.h>
/* Every task → global FIFO queue → any idle CPU runs it. */
void BPF_STRUCT_OPS(minimal_enqueue, struct task_struct *p, u64 enq_flags)
{
scx_bpf_dispatch(p, SCX_DSQ_GLOBAL, SCX_SLICE_DFL, enq_flags);
}
/* No ops.dispatch() needed: kernel drains SCX_DSQ_GLOBAL automatically. */
SEC(".struct_ops.link")
struct sched_ext_ops minimal_ops = {
.enqueue = (void *)minimal_enqueue,
.name = "minimal",
};
/* minimal.c — userspace loader */
#include "minimal.skel.h" /* auto-generated by bpftool */
int main(void)
{
struct minimal *skel = minimal__open_and_load();
minimal__attach(skel); /* BPF scheduler is now active */
pause(); /* run until killed */
minimal__destroy(skel);
return 0;
}
Once minimal__attach() returns, all tasks with SCHED_EXT policy are scheduled by this
BPF program. When the process exits, the BPF scheduler is unloaded and all tasks return to CFS.
Per-Task State: struct scx_entity
Every task_struct has a struct scx_entity embedded at task_struct.scx. This is
sched_ext's per-task tracking state:
struct scx_entity {
struct scx_dispatch_q *dsq; which DSQ this task is currently in
u64 dsq_vtime; virtual time (for vtime-ordered DSQs)
u64 slice; remaining time slice in nanoseconds
u32 flags; SCX_TASK_* status flags
...
}
Key flags:
SCX_TASK_QUEUED— task is sitting in a DSQSCX_TASK_RUNNABLE— task is runnable (may not be in a DSQ yet if ops.enqueue hasn't run)SCX_TASK_DISALLOW— BPF scheduler rejected this task; it runs on CFS instead
BPF programs access scx_entity via p->scx in BPF code using CO-RE (BPF Compile Once,
Run Everywhere) field access.
Safety: Three Layers of Protection
sched_ext is designed so that a buggy BPF scheduler cannot crash the kernel or starve tasks permanently. There are three complementary escape hatches:
Layer 1: BPF Verifier (load-time)
Before a BPF scheduler is allowed to run, the kernel's BPF verifier checks it for:
- Memory safety (no out-of-bounds access)
- No infinite loops
- Correct use of BPF helpers (only
scx_bpf_*helpers allowed from scheduler context)
A scheduler that fails verification never runs.
Layer 2: Watchdog Timer (runtime)
A kernel timer fires every timeout_ms / 2 (default: every 15 seconds). On each fire, it
checks: is any SCHED_EXT task runnable but hasn't run in timeout_ms?
If yes → the BPF scheduler is killed and all tasks return to CFS.
This catches the common failure mode: BPF scheduler has a bug and stops dispatching some tasks, causing them to starve indefinitely.
Layer 3: sysrq-S (manual override)
Pressing Alt+SysRq+S at the console calls scx_ops_disable() immediately. This is the
human-accessible escape hatch: if the BPF scheduler causes the system to become unresponsive
(UI frozen, shell not responding), a user at the physical console can recover without a reboot.
The disable path (triggered by any of the above)
When any escape hatch fires:
- Bypass mode activated — all new scheduling bypasses BPF, falls back to CFS-like dispatch
- DSQs drained — all tasks in sched_ext DSQs are moved to CFS runqueues
ops.exit(exit_info)called — BPF program gets a chance to log its final state- BPF program unloaded —
ext_sched_classcontinues to exist but has no BPF ops - System continues normally on CFS — zero data loss, no panic
The exit reason (SCX_EXIT_SYSRQ, SCX_EXIT_ERROR_STALL, SCX_EXIT_ERROR, etc.) is
recorded and readable from debugfs.
A More Complex Example: Priority Queue Scheduler
This shows how ops.enqueue and ops.dispatch work together for multi-queue scheduling:
#define NUM_QUEUES 5
s32 BPF_STRUCT_OPS(prio_init)
{
for (int i = 0; i < NUM_QUEUES; i++)
scx_bpf_create_dsq(i, -1); /* create 5 DSQs, NUMA-local */
return 0;
}
void BPF_STRUCT_OPS(prio_enqueue, struct task_struct *p, u64 enq_flags)
{
/* map nice value (-20..19) to queue 0 (highest) .. 4 (lowest) */
u32 q = (p->static_prio - MAX_RT_PRIO) / 8;
q = q > 4 ? 4 : q;
scx_bpf_dispatch(p, q, SCX_SLICE_DFL, enq_flags);
}
void BPF_STRUCT_OPS(prio_dispatch, s32 cpu, struct task_struct *prev)
{
/* always serve highest-priority non-empty queue */
for (int i = 0; i < NUM_QUEUES; i++)
if (scx_bpf_consume(i))
return;
}
void BPF_STRUCT_OPS(prio_exit, struct scx_exit_info *ei)
{
for (int i = 0; i < NUM_QUEUES; i++)
scx_bpf_destroy_dsq(i);
}
Step-by-step when a task wakes up:
ops.enqueue(p)is called on the CPU where p woke up- BPF checks
p->static_prio, maps to queue 0-4, callsscx_bpf_dispatch(p, q, ...) - Task sits in DSQ
q - When a CPU goes idle, kernel calls
ops.dispatch(cpu, NULL) - BPF iterates queues 0→4, calls
scx_bpf_consume(0)— if queue 0 has a task, it moves to this CPU's local DSQ andops.dispatchreturns - Kernel picks task from local DSQ and runs it
Weighted Fair Scheduling with Virtual Time
For weighted fairness (high-priority tasks get proportionally more CPU), use vtime DSQs:
void BPF_STRUCT_OPS(wfq_enqueue, struct task_struct *p, u64 enq_flags)
{
u64 vtime = p->scx.dsq_vtime;
u32 weight = p->scx.weight; /* 1..10000; default 100; higher = more CPU */
/* Advance vtime inversely proportional to weight.
* Heavy tasks (high weight) advance slowly → stay near queue front → run more.
* Light tasks (low weight) advance quickly → pushed toward back → run less. */
vtime += SCX_SLICE_DFL / weight;
scx_bpf_dispatch_vtime(p, SCX_DSQ_GLOBAL, SCX_SLICE_DFL, vtime, enq_flags);
}
The DSQ sorts tasks by vtime. The CPU always picks the task with the smallest vtime.
For new tasks, initialize p->scx.dsq_vtime to the DSQ's current minimum vtime
(scx_bpf_dsq_vtime_anchor()) so they don't jump to the front with vtime = 0.
CPU Coordination: The Central Scheduler Pattern
For policies requiring global visibility across all CPUs, sched_ext supports a "central scheduler" pattern where one CPU makes all dispatch decisions:
ops.select_cpu() → always route enqueue through CPU 0 (the central CPU)
ops.enqueue() → place task in per-CPU target queues (runs on central CPU)
ops.dispatch() → central CPU dispatches to remote CPU local queues;
worker CPUs return without dispatching (they wait for work)
scx_bpf_kick_cpu() → central CPU kicks worker CPUs to wake them up
This enables: topology-aware scheduling, NUMA-optimal placement, global fairness policies.
scx_bpf_kick_cpu(cpu, SCX_KICK_IDLE) wakes an idle CPU.
scx_bpf_kick_cpu(cpu, SCX_KICK_PREEMPT) preempts a running task on that CPU.
scx_bpf_kick_cpu(cpu, SCX_KICK_WAIT) waits until that CPU completes one schedule round.
The Patch Series in One View
The 30 patches build sched_ext in layers. Each layer depends on the previous:
LAYER 1: Foundations (Patches 01-07)
Remove hardcoded assumptions about the scheduler class set.
Add hooks the ext class will need (reweight_task, switching_to).
Factor out shared utilities (cgroup weights, PELT).
↓
LAYER 2: Core (Patches 08-12)
08: Create file skeleton and register ext_sched_class
09: Implement the full BPF scheduler (~4000 lines):
- sched_ext_ops dispatch/enqueue machinery
- DSQ implementation (global, local, custom)
- scx_bpf_* helpers
- Error exit path
10: Example schedulers (scx_simple, scx_qmap)
11: sysrq-S emergency escape
12: Watchdog timer
↓
LAYER 3: Observability (Patches 13-16, 18)
Per-task disallow flag, stack dump integration, debug ring,
scx_show_state.py, central scheduler example.
↓
LAYER 4: CPU Control (Patches 17, 19)
scx_bpf_kick_cpu(), preemption support.
Watchdog extended to dispatch() loops.
↓
LAYER 5: Task Lifecycle (Patches 20-23)
Fine-grained state callbacks (runnable/running/stopping/quiescent).
Tickless support. In-flight operation tracking. SCX_KICK_WAIT.
↓
LAYER 6: System Integration (Patches 24-28)
CPU hotplug (cpu_online/offline, cpu_acquire/release).
PM event bypass. Core scheduling (SMT security).
Vtime-ordered DSQs.
↓
LAYER 7: Documentation and Tests (Patches 29-30)
Documentation/scheduler/sched-ext.rst.
tools/testing/selftests/sched_ext/.
Key Data Structures at a Glance
| Structure | Where | Purpose |
|---|---|---|
struct sched_ext_ops | BPF program | The vtable BPF fills in; kernel calls these |
struct scx_entity | task_struct.scx | Per-task sched_ext state (DSQ, slice, flags) |
struct scx_dispatch_q | kernel/sched/ext.c | A DSQ — holds tasks waiting to be run |
ext_sched_class | kernel/sched/ext.c | The sched_class implementation that calls ops |
struct rq.scx | per-CPU runqueue | Per-CPU sched_ext state (local DSQ, stats) |
Key BPF Helpers at a Glance
| Helper | When to call | What it does |
|---|---|---|
scx_bpf_dispatch(p, dsq, slice, flags) | ops.enqueue() | Move task to a DSQ |
scx_bpf_dispatch_vtime(p, dsq, slice, vtime, flags) | ops.enqueue() | Move task to vtime DSQ |
scx_bpf_consume(dsq_id) | ops.dispatch() | Move task from DSQ → local queue |
scx_bpf_kick_cpu(cpu, flags) | Anywhere | Force CPU to reschedule |
scx_bpf_create_dsq(id, node) | ops.init() | Create a custom DSQ |
scx_bpf_destroy_dsq(id) | ops.exit() | Destroy a custom DSQ |
scx_bpf_task_running(p) | Anywhere | Is task currently executing? |
scx_bpf_exit(code, fmt, ...) | Anywhere | Voluntarily unload scheduler |
Operating Model: Safety by Design
sched_ext is designed for controlled extensibility rather than unrestricted scheduler
replacement. The key design principle: the kernel is always in control of safety; the BPF
program is only in control of policy.
- The BPF verifier enforces memory safety at load time
- The watchdog enforces forward progress at runtime
- Bypass mode ensures the system can always recover
- The disable path is atomic: all tasks atomically return to CFS
This means a BPF scheduler bug produces a warning and a CFS fallback, never a kernel panic.
What to Read Next
| Goal | Start here |
|---|---|
| Understand the full API in depth | sched-ext-explainer.md |
| Write a BPF scheduler | sched-ext-explainer.md + patch 29 (docs) |
| Understand the core implementation | patch-study/patch-30.md (PATCH 09/30) |
| Understand the file structure | patch-study/patch-09.md (PATCH 08/30 boilerplate) |
| Debug a running sched_ext scheduler | patch-study/patch-12.md through patch-study/patch-18.md |
| Understand system integration | patch-study/patch-24.md through patch-study/patch-28.md |
References
- Kernel tree: https://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext.git
- Example schedulers and tooling: https://github.com/sched-ext/scx
- Community workspace: https://bit.ly/scx_slack
Metadata
- Patch series: v7 (June 2024)
- Merged into: Linux 6.11
How sched_ext Works
A comprehensive, self-contained explanation for systems programmers who know Linux and C but are new to sched_ext. This document covers the architecture end-to-end, from the motivation through BPF scheduler authorship to kernel internals.
The Problem: Why Scheduling Policy Lives in the Kernel
The Traditional Scheduler Model
The Linux kernel ships with several scheduling classes, each implementing a distinct policy:
- CFS (
fair_sched_class,SCHED_NORMAL/SCHED_BATCH): Completely Fair Scheduler. Tracks per-task virtual runtime and always picks the task with the smallestvruntime. CPU time is distributed proportionally to task weights (derived fromnicevalues). - RT (
rt_sched_class,SCHED_FIFO/SCHED_RR): Real-time policies. Fixed-priority preemptive scheduling. A SCHED_FIFO task at priority 50 will starve every SCHED_NORMAL task indefinitely. - Deadline (
dl_sched_class,SCHED_DEADLINE): EDF-based. Tasks declare runtime budgets and deadlines; the kernel guarantees they meet them (or rejects the task at admission time).
Each of these classes is compiled into the kernel. Their code lives in:
kernel/sched/fair.c(CFS, ~7000 lines)kernel/sched/rt.c(RT, ~2500 lines)kernel/sched/deadline.c(Deadline, ~1600 lines)
The Pain Points
1. Long iteration cycles for policy experiments
If you want to test a new scheduling heuristic — say, a variant of CFS that biases toward tasks sharing cache lines, or a scheduler that tries to keep gaming threads on P-cores — you must:
- Write a kernel patch
- Build the kernel (5–30 minutes depending on your machine)
- Install it in a test environment
- Reboot the machine
- Run experiments
- If the policy was wrong, go back to step 1
A single experiment can take hours. Iterating toward a good policy for a specific workload requires dozens of experiments. This makes kernel scheduling research expensive and slow.
2. Impossibility of workload-specific tuning at runtime
Different workloads have radically different scheduling needs:
- A batch ML training job wants maximum CPU utilization, does not care about latency, and benefits from large time slices and NUMA-aware placement.
- A game render thread needs consistent sub-millisecond scheduling latency and wants to run on specific cores (e.g., the same physical core as its audio thread).
- A database server may want to prioritize query threads over background compaction threads dynamically based on current query load.
CFS handles all of these with the same algorithm, tuned via a handful of sysctl knobs. It cannot express the kind of workload-specific policy these applications need.
3. No safe way to load custom policies in production
Even if you write the perfect scheduler for your workload, you cannot deploy it to a production Linux system without running a custom kernel. This means:
- Every production machine needs a custom kernel build — no distro support, no security patches via standard channels.
- Any bug in your scheduler is a kernel bug: it can cause hangs, panics, or security vulnerabilities.
- There is no isolation between the custom scheduler and the rest of the kernel.
The net result: companies that care deeply about scheduling (Meta, Google, Microsoft, game studios) maintain large, divergent kernel trees with proprietary scheduling patches. These trees are expensive to maintain and rarely share improvements.
What We Need
The ideal solution would let a developer write scheduling policy in a safe, high-level language, load it at runtime without rebooting, have it take effect immediately for chosen workloads, and fail safely if it has bugs. That is exactly what sched_ext provides.
sched_ext in One Paragraph
sched_ext is a Linux scheduling class (ext_sched_class) where scheduling policy lives in a BPF program loaded at runtime. Tasks opt in by being assigned the SCHED_EXT scheduling policy. The BPF program implements a set of callbacks called sched_ext_ops — a vtable of function pointers that the BPF program fills in. The kernel handles mechanism: runqueue management, CPU affinity enforcement, preemption delivery, time slice accounting, task lifecycle management, and safety guarantees. The BPF program handles policy: which task runs next, on which CPU, for how long. If the BPF scheduler misbehaves — deadlocks, fails to schedule tasks, or crashes — a watchdog timer detects the problem and reverts all SCHED_EXT tasks to CFS automatically, without rebooting the machine. The BPF verifier catches memory safety bugs at load time before the scheduler runs at all.
Where sched_ext Fits in the Scheduler Hierarchy
Linux uses a sched_class chain. When the kernel needs to pick the next task to run, it walks this chain from highest to lowest priority and asks each class "do you have a runnable task for this CPU?"
stop_sched_class (highest priority: migration/stop tasks)
↓
dl_sched_class (SCHED_DEADLINE: earliest deadline first)
↓
rt_sched_class (SCHED_FIFO / SCHED_RR: real-time tasks)
↓
fair_sched_class (SCHED_NORMAL / SCHED_BATCH: CFS)
↓
ext_sched_class (SCHED_EXT: BPF-controlled) ← sched_ext
↓
idle_sched_class (lowest priority: idle tasks)
Each sched_class is defined as a struct of function pointers in the kernel:
/* kernel/sched/sched.h (illustrative, not exact) */
struct sched_class {
void (*enqueue_task) (struct rq *rq, struct task_struct *p, int flags);
void (*dequeue_task) (struct rq *rq, struct task_struct *p, int flags);
void (*yield_task) (struct rq *rq);
struct task_struct *(*pick_next_task)(struct rq *rq);
void (*put_prev_task) (struct rq *rq, struct task_struct *p);
/* ... many more ... */
};
The key rules of the hierarchy:
- RT and DL tasks are NEVER handled by sched_ext. A task on SCHED_FIFO is always managed by
rt_sched_class, regardless of whether a BPF scheduler is loaded. - A task uses sched_ext if and only if it has the
SCHED_EXTpolicy AND a BPF scheduler is currently loaded. - If no BPF scheduler is loaded, tasks with
SCHED_EXTpolicy fall back tofair_sched_class(CFS). They look like normalSCHED_NORMALtasks to the kernel. - The BPF scheduler only controls SCHED_EXT tasks. System threads, RT tasks, and regular CFS tasks are not affected.
This hierarchy means that sched_ext cannot prevent a SCHED_FIFO task from preempting your BPF-scheduled task. The BPF scheduler operates in the space below RT, just like CFS does.
The sched_class Linked List
In the kernel, the classes are linked in a doubly-linked list via next pointers. The pick_next_task() kernel function walks this list. For ext_sched_class:
/* kernel/sched/ext.c */
DEFINE_SCHED_CLASS(ext) = {
.enqueue_task = scx_enqueue,
.dequeue_task = scx_dequeue,
.pick_next_task = scx_pick_next_task,
.put_prev_task = scx_put_prev_task,
/* ... */
};
When pick_next_task() reaches ext_sched_class, it calls scx_pick_next_task(), which drains the CPU's local DSQ (more on DSQs shortly).
The BPF Interface: sched_ext_ops
The BPF program fills in a sched_ext_ops structure — a vtable of callbacks. The BPF skeleton infrastructure handles attaching this struct to the kernel.
Only .name is required; everything else has a sensible default that makes the scheduler behave like a simple global FIFO.
/* Illustrative — based on include/linux/sched/ext.h */
struct sched_ext_ops {
/* Required */
char name[SCX_OPS_NAME_LEN]; /* name of this scheduler, e.g. "my_sched" */
/* --- CPU selection: where should this task run? --- */
/* Called when a task wakes up. Return a CPU number or -1 to let the
* kernel decide. BPF can also call scx_bpf_dispatch() here directly
* (called "direct dispatch") to skip ops.enqueue(). */
s32 (*select_cpu)(struct task_struct *p, s32 prev_cpu, u64 wake_flags);
/* --- Task runqueue operations --- */
/* Task became runnable — place it into a DSQ.
* BPF MUST call scx_bpf_dispatch() (or scx_bpf_dispatch_vtime()) here,
* or the task will be lost. If BPF does nothing, the task is dispatched
* to SCX_DSQ_GLOBAL by default. */
void (*enqueue)(struct task_struct *p, u64 enq_flags);
/* Task is being removed from the scheduler (class change, migration, etc.)
* If BPF maintains external state (maps, lists), clean it up here. */
void (*dequeue)(struct task_struct *p, u64 deq_flags);
/* CPU needs a task to run — feed tasks from custom DSQs into the local queue.
* BPF calls scx_bpf_consume(dsq_id) to move tasks from a custom DSQ to this
* CPU's local queue. Called when the local queue is empty. */
void (*dispatch)(s32 cpu, struct task_struct *prev);
/* --- Task state change notifications --- */
/* Task transitioned to runnable state (was blocked, now runnable). */
void (*runnable)(struct task_struct *p, u64 enq_flags);
/* Task transitioned to a quiescent (blocking) state. */
void (*quiescent)(struct task_struct *p, u64 deq_flags);
/* Task is now actually executing on a CPU (context switch complete). */
void (*running)(struct task_struct *p);
/* Task is about to be descheduled (before context switch away). */
void (*stopping)(struct task_struct *p, bool runnable);
/* Task yielded the CPU voluntarily. */
void (*yield)(struct task_struct *from, struct task_struct *to);
/* --- Task lifecycle --- */
/* Called when task is forked or when SCHED_EXT is set on a task.
* BPF should allocate per-task state here. Return 0 on success,
* negative errno on failure (task stays on CFS). */
s32 (*init_task)(struct task_struct *p, struct scx_init_task_args *args);
/* Called when task exits or SCHED_EXT is removed. Free per-task state. */
void (*exit_task)(struct task_struct *p, struct scx_exit_task_args *args);
/* Task is being enabled for sched_ext scheduling (after init_task). */
void (*enable)(struct task_struct *p);
/* Task is being disabled (before exit_task). */
void (*disable)(struct task_struct *p);
/* Priority inheritance: task p is boosting task of due to a lock.
* BPF can requeue 'of' at higher priority. */
void (*set_weight)(struct task_struct *p, u32 weight);
/* --- CPU lifecycle --- */
/* A CPU came online (hotplug). */
void (*cpu_online)(s32 cpu);
/* A CPU went offline (hotplug). */
void (*cpu_offline)(s32 cpu);
/* BPF scheduler "acquired" this CPU: no higher-priority class has
* tasks for it, so sched_ext is now responsible for it. */
void (*cpu_acquire)(s32 cpu, struct scx_cpu_acquire_args *args);
/* A higher-priority class (RT, DL) needs this CPU back.
* BPF must stop scheduling on it. */
void (*cpu_release)(s32 cpu, struct scx_cpu_release_args *args);
/* --- Scheduler lifecycle --- */
/* Called once when the BPF scheduler is loaded. Initialize global state,
* create custom DSQs, etc. Return 0 on success. */
s32 (*init)(void);
/* Called when the scheduler is unloaded (normally or due to error).
* ei contains the reason for exit. Log final state here. */
void (*exit)(struct scx_exit_info *ei);
/* --- Control flags --- */
/* Bitmask of SCX_OPS_* flags:
* SCX_OPS_KEEP_BUILTIN_IDLE - use kernel's idle tracking alongside BPF
* SCX_OPS_ENQ_LAST - call enqueue() when there's only one task
* SCX_OPS_ENQ_EXITING - call enqueue() for exiting tasks
* SCX_OPS_SWITCH_PARTIAL - only switch tasks that opt in (not all SCHED_EXT)
* SCX_OPS_HAS_CGROUP_WEIGHT - scheduler handles cgroup weights
*/
u64 flags;
/* Watchdog timeout in milliseconds. If a SCHED_EXT task goes this long
* without being scheduled, the watchdog disables the BPF scheduler.
* Default: 30000 (30 seconds). Set to 0 to disable watchdog. */
u32 timeout_ms;
/* How many bytes of exit info to dump when the scheduler exits.
* Default is enough for typical error messages. */
u32 exit_dump_len;
};
How the BPF Program Attaches the Ops Struct
The sched_ext_ops struct is declared in the BPF program with a special ELF section annotation:
/* In the BPF C file */
SEC(".struct_ops.link")
struct sched_ext_ops my_scheduler_ops = {
.enqueue = (void *)my_enqueue,
.dispatch = (void *)my_dispatch,
.name = "my_scheduler",
};
The .struct_ops.link section tells libbpf that this struct should be "auto-attached" when skel__attach() is called. libbpf handles creating the necessary BPF maps and attaching the struct to the kernel's sched_ext_ops registration point. Once attached, ext_sched_class becomes active in the scheduler hierarchy.
Dispatch Queues (DSQs): The Core Abstraction
Dispatch Queues are the most important concept in sched_ext. Understanding DSQs is the key to understanding how the whole system works.
A Dispatch Queue (DSQ) is a queue of tasks waiting to be scheduled. It is an ordered container — tasks go in, tasks come out. The kernel provides three kinds of DSQs:
1. The Global DSQ (SCX_DSQ_GLOBAL = 0)
The global DSQ is a single FIFO queue shared across all CPUs. It is created automatically by the kernel when sched_ext is initialized — BPF does not need to do anything to use it.
Properties:
- Shared: Any idle CPU can pull tasks from it.
- FIFO ordering: Tasks are served in the order they arrive.
- Automatic draining: The kernel automatically drains the global DSQ into CPU local queues. BPF does not need to implement
ops.dispatch()to use it.
The global DSQ is the simplest possible option. A scheduler that puts everything in SCX_DSQ_GLOBAL implements pure FIFO scheduling across all CPUs — the simplest valid sched_ext scheduler.
2. Per-CPU Local DSQs (SCX_DSQ_LOCAL = 1)
Each CPU has its own local DSQ. This is the queue that the CPU actually runs tasks from — ext_sched_class.pick_next_task() dequeues from the local DSQ.
Properties:
- CPU-private: Only the owning CPU runs tasks from its local DSQ.
- Highest priority within sched_ext: A task in the local DSQ will run before anything is pulled from other DSQs.
- Direct dispatch: BPF can dispatch directly to a specific CPU's local DSQ using
SCX_DSQ_LOCAL_ON(cpu).
When BPF dispatches to SCX_DSQ_LOCAL from within ops.enqueue(), the task goes to the local DSQ of the CPU that will run it (the CPU returned by ops.select_cpu()). From ops.dispatch(), it goes to the local DSQ of the CPU calling dispatch.
3. Custom DSQs (BPF-defined)
BPF programs can create their own DSQs with arbitrary IDs. These are the building blocks for sophisticated scheduling policies.
Properties:
- Created by BPF:
scx_bpf_create_dsq(dsq_id, node)— thenodeparameter controls NUMA node allocation for the DSQ's memory. - Two orderings: FIFO (default) or virtual-time ordered (for weighted fair scheduling).
- Explicit consumption: BPF must explicitly call
scx_bpf_consume(dsq_id)fromops.dispatch()to move tasks from a custom DSQ to the local DSQ. - Destroyed by BPF:
scx_bpf_destroy_dsq(dsq_id)inops.exit().
Custom DSQ IDs are user-defined. The only constraint is that they must not collide with the built-in IDs (SCX_DSQ_GLOBAL = 0, SCX_DSQ_LOCAL = 1). Typically BPF programs use IDs starting from a high number or use a per-CPU scheme.
The Task Flow Through DSQs
This is the complete lifecycle of a task in sched_ext:
Task wakes up (e.g., from I/O completion, mutex unlock, timer)
│
▼
ops.select_cpu(p, prev_cpu, wake_flags)
│ BPF picks which CPU this task should run on.
│ Returns a CPU number (0..nr_cpus-1), or -1 to let the kernel decide.
│ BPF can also call scx_bpf_dispatch() HERE directly (direct dispatch),
│ which skips ops.enqueue() entirely.
│
▼
ops.enqueue(p, enq_flags)
│ BPF places the task into a DSQ by calling:
│ scx_bpf_dispatch(p, DSQ_ID, slice_ns, enq_flags) -- FIFO DSQ
│ scx_bpf_dispatch_vtime(p, DSQ_ID, slice_ns, vtime, enq_flags) -- vtime DSQ
│ If BPF doesn't dispatch, the task defaults to SCX_DSQ_GLOBAL.
│
▼
[Task sits in a DSQ: custom DSQ, global DSQ, or local DSQ]
│
│ (If task went to global DSQ, kernel auto-drains it to local DSQs)
│ (If task went to a custom DSQ, it waits until ops.dispatch() consumes it)
│
▼
ops.dispatch(cpu, prev_task)
│ Called when the CPU's local DSQ is empty and the CPU needs a task.
│ prev_task is the task that just ran (or NULL).
│ BPF calls:
│ scx_bpf_consume(dsq_id) -- moves one task from custom DSQ → local DSQ
│ BPF can call scx_bpf_consume() multiple times for multiple tasks.
│ BPF can also call scx_bpf_dispatch() to put more tasks into DSQs.
│
▼
[Task is now in the CPU's local DSQ]
│
▼
ext_sched_class.pick_next_task()
│ Kernel dequeues the next task from the local DSQ and selects it to run.
│
▼
ops.running(p)
│ Task is now executing on the CPU.
│ Notification only — BPF can update accounting here.
│
▼
[Task runs for its time slice, or is preempted by RT/DL task, or blocks]
│
├── Time slice expired ─────────────────────────────────────┐
│ │
├── Preempted by higher-priority class ─────────────────────┤
│ │
└── Task blocked (I/O, lock, sleep) ─────────────────────── │
│
▼
ops.stopping(p, runnable)
│ Task is about to be
│ descheduled. BPF can
│ update final accounting.
│ 'runnable' = true if task
│ will be re-enqueued (time
│ slice expired), false if
│ task is blocking.
│
▼
If runnable: ops.enqueue() again
If blocking: ops.quiescent()
[task waits for wakeup event]
When event arrives: ops.runnable()
Then: ops.enqueue()
Key BPF Helper Functions
These are the primary BPF helpers used to interact with DSQs and CPUs:
/* ========== Dispatching tasks into DSQs ========== */
/* Place task p into DSQ dsq_id with FIFO ordering.
* p: the task being dispatched
* dsq_id: destination DSQ (SCX_DSQ_GLOBAL, SCX_DSQ_LOCAL, or custom ID)
* slice: time slice in nanoseconds; SCX_SLICE_DFL for default (~20ms)
* enq_flags: pass-through from ops.enqueue(), or 0
*/
void scx_bpf_dispatch(struct task_struct *p, u64 dsq_id,
u64 slice, u64 enq_flags);
/* Place task p into a vtime-ordered DSQ.
* vtime: virtual time value; lower vtime = higher priority (runs sooner)
* All other parameters same as scx_bpf_dispatch()
*/
void scx_bpf_dispatch_vtime(struct task_struct *p, u64 dsq_id,
u64 slice, u64 vtime, u64 enq_flags);
/* ========== Consuming from DSQs (use in ops.dispatch()) ========== */
/* Move the head task from custom DSQ dsq_id to the current CPU's local DSQ.
* Returns true if a task was moved, false if the DSQ was empty. */
bool scx_bpf_consume(u64 dsq_id);
/* ========== CPU management ========== */
/* Send a scheduling event to a CPU.
* flags:
* SCX_KICK_IDLE - wake the CPU if it's idle (to run newly dispatched tasks)
* SCX_KICK_PREEMPT - preempt the currently running task
* SCX_KICK_WAIT - wait for the CPU to reschedule before returning
*/
void scx_bpf_kick_cpu(s32 cpu, u64 flags);
/* ========== DSQ management ========== */
/* Create a new custom DSQ.
* dsq_id: user-chosen ID (must not be 0 or 1)
* node: NUMA node for memory allocation, or -1 for NUMA_NO_NODE
* Returns 0 on success, negative errno on failure.
*/
s32 scx_bpf_create_dsq(u64 dsq_id, s32 node);
/* Destroy a previously created custom DSQ.
* Any tasks still in the DSQ are moved to the global DSQ. */
void scx_bpf_destroy_dsq(u64 dsq_id);
/* ========== Per-task queries ========== */
/* Returns the CPU the task is currently running on, or -1 if not running. */
s32 scx_bpf_task_running_on(struct task_struct *p);
/* Returns the CPU selected for a task by select_cpu(), or -1 if not set. */
s32 scx_bpf_task_cpu(struct task_struct *p);
/* ========== Scheduler lifecycle ========== */
/* Voluntarily exit the BPF scheduler with a reason.
* exit_code: user-defined exit code logged to exit_info
* fmt, args: printf-style message for the exit log
* This triggers the disable path cleanly (safer than a crash). */
void scx_bpf_exit(s64 exit_code, const char *fmt, ...);
/* Emit an error and trigger the disable path. */
void scx_bpf_error(const char *fmt, ...);
The select_cpu / enqueue / dispatch Triangle
The relationship between these three callbacks is the most common source of confusion for new sched_ext developers. Here is the precise interaction:
ops.select_cpu(p, prev_cpu, wake_flags)
- Called when a task becomes runnable (wakeup path)
- BPF's chance to pick a CPU before the task is enqueued
- If BPF calls
scx_bpf_dispatch()here, the task is dispatched directly —ops.enqueue()is NOT called - If BPF returns a CPU without dispatching, the task proceeds to
ops.enqueue()withp->wake_cpuset to the returned CPU - Wake flags include bits like
SCX_WAKE_FORK(task was forked),SCX_WAKE_SYNC(synchronous wakeup hint)
ops.enqueue(p, enq_flags)
- Called for every task that needs to be placed into a DSQ
- BPF MUST call
scx_bpf_dispatch()orscx_bpf_dispatch_vtime()here - If BPF returns without dispatching, the task is automatically dispatched to
SCX_DSQ_GLOBAL - Enqueue flags include
SCX_ENQ_WAKEUP(task woke from sleep),SCX_ENQ_LAST(only task on CPU),SCX_ENQ_PREEMPT(BPF preempted the current task for this one)
ops.dispatch(cpu, prev)
- Called when a CPU's local DSQ is empty and the CPU needs something to run
previs the task that just ran (may be NULL)- BPF calls
scx_bpf_consume(dsq_id)to pull tasks from custom DSQs - Can be called multiple times if the first consume was empty
- If dispatch returns without producing a task, the CPU goes idle
The Simplest Possible BPF Scheduler
The following is a complete, working minimal scheduler. It implements pure FIFO scheduling using only the global DSQ. This is essentially scx_simple from the sched_ext tools repository.
/* minimal_sched.bpf.c
* The simplest possible sched_ext scheduler.
*
* Policy: global FIFO. Every task goes into a single shared queue.
* CPUs pick tasks in arrival order. No per-task state needed.
*
* Build: clang -O2 -g -target bpf -c minimal_sched.bpf.c -o minimal_sched.bpf.o
* (In practice, use the sched_ext Makefile which handles vmlinux.h and libbpf)
*/
#include <scx/common.bpf.h>
/* char _license[] SEC("license") = "GPL"; is included via common.bpf.h */
/* ops.enqueue() is the only callback we need to implement.
*
* The default ops.dispatch() implementation knows how to drain SCX_DSQ_GLOBAL
* into per-CPU local queues, so we don't need to write dispatch().
*/
void BPF_STRUCT_OPS(minimal_enqueue, struct task_struct *p, u64 enq_flags)
{
/*
* SCX_DSQ_GLOBAL (= 0): the built-in global FIFO queue shared by all CPUs.
* SCX_SLICE_DFL: use the default time slice (tunable via /sys/kernel/sched_ext/).
* enq_flags: pass through the flags we received (contains wakeup hints etc.)
*/
scx_bpf_dispatch(p, SCX_DSQ_GLOBAL, SCX_SLICE_DFL, enq_flags);
}
/*
* The ops struct. Only .enqueue and .name are set.
* All other callbacks use their built-in defaults:
* - select_cpu: kernel picks CPU based on affinity and cache topology
* - dispatch: automatically drains SCX_DSQ_GLOBAL
* - init_task: no-op (no per-task state)
* - exit_task: no-op
* - init: no-op
* - exit: no-op
*/
SEC(".struct_ops.link")
struct sched_ext_ops minimal_ops = {
.enqueue = (void *)minimal_enqueue,
.name = "minimal",
};
What each part does:
BPF_STRUCT_OPS(minimal_enqueue, ...): A macro fromcommon.bpf.hthat annotates the function as a struct_ops callback and places it in the correct BPF ELF section.scx_bpf_dispatch(p, SCX_DSQ_GLOBAL, SCX_SLICE_DFL, enq_flags): Places the task in the global FIFO queue. When a CPU has nothing to run, the kernel calls the defaultops.dispatch()implementation, which callsscx_bpf_consume(SCX_DSQ_GLOBAL)automatically.SEC(".struct_ops.link"): Tells libbpf that this struct should be registered and auto-attached to the kernel's sched_ext_ops structure.
The userspace loader:
/* minimal_sched.c - userspace loader
*
* The .skel.h file is generated by bpftool (or the build system) from the
* compiled BPF object file. It provides type-safe C wrappers for all BPF
* operations on this specific program.
*/
#include <stdio.h>
#include <unistd.h>
#include <signal.h>
#include "minimal_sched.skel.h"
int main(void)
{
struct minimal_sched *skel;
int err;
/*
* open_and_load():
* 1. Opens the BPF object embedded in the skeleton
* 2. Runs the BPF verifier (validates memory safety, bounds, etc.)
* 3. Loads all BPF programs into the kernel
* 4. Creates any BPF maps
* Returns NULL on failure.
*/
skel = minimal_sched__open_and_load();
if (!skel) {
fprintf(stderr, "Failed to open and load BPF skeleton\n");
return 1;
}
/*
* attach():
* Registers the sched_ext_ops struct with the kernel.
* After this call, all tasks with SCHED_EXT policy are handled
* by our minimal_enqueue() callback.
*/
err = minimal_sched__attach(skel);
if (err) {
fprintf(stderr, "Failed to attach scheduler: %d\n", err);
minimal_sched__destroy(skel);
return 1;
}
printf("Minimal sched_ext scheduler loaded. Press Ctrl+C to unload.\n");
/*
* Keep the process alive. The scheduler stays active as long as
* the skel object is attached. When the process exits (or you call
* minimal_sched__destroy()), the kernel unloads the BPF scheduler
* and all SCHED_EXT tasks revert to CFS.
*/
pause();
minimal_sched__destroy(skel);
return 0;
}
To test it:
# Build (requires kernel headers and libbpf)
make minimal_sched
# Load the scheduler
sudo ./minimal_sched &
# Move a process to SCHED_EXT
sudo chrt --ext 0 -p <pid>
# Or start a new process with SCHED_EXT
sudo chrt --ext 0 <command>
# Check that the scheduler is active
cat /sys/kernel/sched_ext/root/ops
# Outputs: minimal
# Unload (kills the background process)
sudo kill %1
Per-Task State: scx_entity
Every task_struct has an scx_entity embedded in it at task_struct.scx. This is the sched_ext-specific state per task — it lives in the task itself, not in a separate allocation.
/* Illustrative — based on include/linux/sched/ext.h */
struct scx_entity {
/* DSQ membership */
struct scx_dispatch_q *dsq; /* which DSQ this task is currently in */
struct list_head dsq_node; /* linkage within the DSQ's task list */
u64 dsq_seq; /* sequence number for this DSQ slot */
/* Scheduling parameters */
u64 dsq_vtime; /* virtual time for vtime-ordered DSQs */
u64 slice; /* remaining time slice in nanoseconds */
/* Task state flags (SCX_TASK_*) */
u32 flags;
/* CPU assignment */
s32 sticky_cpu; /* CPU pinned by select_cpu(), or -1 */
s32 holding_cpu; /* CPU currently "holding" the task */
/* Weight for WFQ (derived from nice value) */
u32 weight; /* 1..10000, default 100 */
/* Ops state machine */
u32 ops_state; /* SCX_OPSS_* enum value */
/* ... more internal fields ... */
};
Key fields explained:
dsq: Pointer to the DSQ this task is currently sitting in. NULL if the task is running or transitioning.dsq_vtime: The task's virtual time within its DSQ. For vtime DSQs, this determines ordering — tasks with lowerdsq_vtimerun first. BPF reads and writes this field to implement WFQ.slice: How much time the task has left in its current time slice, in nanoseconds. The kernel decrements this each tick. When it reaches zero, the task is preempted.weight: Task weight derived from the nice value. Higher weight = more CPU time in WFQ. Maps roughly: nice -20 → weight 10000, nice 0 → weight 100, nice 19 → weight 1.sticky_cpu: Ifops.select_cpu()returned a valid CPU, this field stores it soops.enqueue()knows where the task should go.
Key flags (SCX_TASK_*):
SCX_TASK_QUEUED: Task is currently in a DSQ (between enqueue and dispatch).SCX_TASK_RUNNABLE: Task is runnable but may not be in a DSQ yet (transitioning).SCX_TASK_DISALLOW: BPF scheduler rejected this task (e.g.,init_task()returned an error). The task runs on CFS instead.SCX_TASK_INIT_DONE:ops.init_task()completed successfully.SCX_TASK_ENABLED:ops.enable()has been called; task is fully under sched_ext control.
Accessing scx_entity from BPF:
BPF programs access task->scx directly through CO-RE (Compile Once, Run Everywhere) field access. The BPF verifier checks these accesses at load time:
void BPF_STRUCT_OPS(my_enqueue, struct task_struct *p, u64 enq_flags)
{
u64 vtime = p->scx.dsq_vtime;
u32 weight = p->scx.weight;
/* ... */
}
Safety: What Happens When the BPF Scheduler Fails
sched_ext is designed with defense in depth. Multiple independent mechanisms ensure the system never hangs due to a buggy BPF scheduler.
Layer 1: The BPF Verifier (Load-Time Safety)
Before any BPF scheduler runs a single instruction, the kernel's BPF verifier performs static analysis:
- Memory safety: All pointer accesses are bounds-checked. Buffer overflows are impossible.
- Termination: All loops must have provably bounded iterations. Infinite loops are rejected at load time.
- Type safety: BPF uses BTF (BPF Type Format) to verify that struct field accesses are correct for the running kernel version.
- Helper call validation: Only approved BPF helpers can be called from sched_ext callbacks (different callbacks have different allowed helper sets).
A BPF scheduler that passes the verifier cannot corrupt kernel memory. This is the first and strongest safety layer.
Layer 2: sysrq-S (Manual Override)
Any system administrator can press Alt+SysRq+S on the keyboard (or echo s > /proc/sysrq-trigger) to:
- Immediately disable the BPF scheduler
- Move all SCHED_EXT tasks back to CFS
- Unload the BPF program
This is the emergency escape hatch for cases where the BPF scheduler is causing the system to become unresponsive. The system recovers without a reboot.
Layer 3: The Watchdog (Automatic Detection)
A kernel timer fires every timeout_ms / 2 milliseconds (default: every 15 seconds). It scans all SCHED_EXT tasks and checks if any runnable task has not been scheduled for longer than timeout_ms (default: 30 seconds).
If the watchdog finds a starved task, it calls scx_ops_error() with a message like:
sched_ext: "my_scheduler" failed: watchdog detected task stuck for 30001ms
This triggers the same disable path as sysrq-S.
The watchdog catches the most common BPF scheduler bug: a ops.dispatch() implementation that fails to consume tasks from a custom DSQ, leaving tasks stuck in limbo indefinitely.
BPF schedulers can configure or disable the watchdog:
SEC(".struct_ops.link")
struct sched_ext_ops my_ops = {
.timeout_ms = 60000, /* Give my scheduler 60 seconds before watchdog fires */
/* Set to 0 to disable watchdog entirely (dangerous!) */
.name = "my_scheduler",
};
Layer 4: BPF Scheduler Self-Reporting
The BPF scheduler itself can trigger the disable path gracefully:
/* Voluntary exit — scheduler knows it cannot continue */
void BPF_STRUCT_OPS(my_dispatch, s32 cpu, struct task_struct *prev)
{
if (some_fatal_condition) {
scx_bpf_exit(-EINVAL, "dispatch: fatal error in queue %d", cpu);
return;
}
/* ... */
}
scx_bpf_exit() and scx_bpf_error() both trigger the disable path. The difference: exit is for intentional/graceful exit, error is for unexpected errors. Both cause ops.exit(exit_info) to be called with the message you provided.
The Disable Path (Step by Step)
When any of the above triggers, the kernel executes the disable path:
-
SCX_BYPASS mode activated: A global flag is set that bypasses the BPF scheduler. New scheduling decisions are handled by CFS immediately, without calling any BPF callbacks.
-
Runqueues drained: The kernel iterates over all CPUs and moves tasks from SCX DSQs to CFS runqueues. Tasks that were waiting in custom DSQs get placed on the CFS runqueue of their affinity-preferred CPU.
-
ops.exit(exit_info)called: The BPF scheduler's exit callback is invoked.exit_infocontains:exit_info->reason: Why we're exiting (watchdog, sysrq, error, etc.)exit_info->msg: Error message (if any)exit_info->dump: Ring buffer of recent BPF prints/logs The BPF scheduler should log this to a BPF map or ring buffer for the userspace process to read.
-
BPF program unloaded: The struct_ops link is detached and the BPF programs are freed.
-
System continues on CFS: All tasks that were SCHED_EXT continue running, now managed by CFS. The kernel notes that no BPF scheduler is active; SCHED_EXT tasks behave like SCHED_NORMAL tasks until a new BPF scheduler is loaded.
The key guarantee: The kernel never panics due to a BPF scheduler bug. The combination of the BPF verifier (preventing memory corruption), the bypass mechanism (preventing scheduling deadlocks), and the watchdog (detecting starvation) means the worst-case outcome of a buggy BPF scheduler is "all tasks fall back to CFS" — not a kernel panic.
A More Complex Example: Priority Queues
This example shows how ops.enqueue() and ops.dispatch() interact for a scheduler with multiple priority levels. This is illustrative pseudocode showing the patterns; real implementations add more error handling.
/* priority_sched.bpf.c
* Priority queue scheduler: 5 priority levels based on nice value.
* Queue 0 = highest priority (nice -20..-12)
* Queue 4 = lowest priority (nice 12..19)
*
* A task in queue 0 will always run before any task in queue 1,
* regardless of arrival order. This is strict priority scheduling.
*/
#include <scx/common.bpf.h>
#define NUM_QUEUES 5
/* DSQ IDs: we use 100, 101, 102, 103, 104 to avoid colliding with
* SCX_DSQ_GLOBAL (0) and SCX_DSQ_LOCAL (1). */
#define QUEUE_BASE 100
/* Map nice value range (-20..19) to queue index (0..4).
* Nice range is 40 units wide; divide into 5 buckets of 8. */
static inline u32 nice_to_queue(struct task_struct *p)
{
/* static_prio: 100 (RT) to 139 (nice 19). Nice 0 = static_prio 120.
* MAX_RT_PRIO = 100 */
int nice_offset = p->static_prio - 120; /* -20..19 */
int queue = (nice_offset + 20) / 8; /* 0..4 */
/* Clamp to valid range */
if (queue < 0) queue = 0;
if (queue >= NUM_QUEUES) queue = NUM_QUEUES - 1;
return (u32)queue;
}
/* Create all 5 DSQs when the scheduler loads. */
s32 BPF_STRUCT_OPS(prio_init)
{
int i;
for (i = 0; i < NUM_QUEUES; i++) {
s32 ret = scx_bpf_create_dsq(QUEUE_BASE + i, -1);
if (ret < 0) {
/* If DSQ creation fails, we cannot operate.
* Return error to abort loading. */
scx_bpf_error("Failed to create DSQ %d: %d", i, ret);
return ret;
}
}
return 0; /* Success */
}
/* Place task into the appropriate priority queue. */
void BPF_STRUCT_OPS(prio_enqueue, struct task_struct *p, u64 enq_flags)
{
u32 queue = nice_to_queue(p);
scx_bpf_dispatch(p, QUEUE_BASE + queue, SCX_SLICE_DFL, enq_flags);
}
/* CPU needs a task: scan queues from highest to lowest priority. */
void BPF_STRUCT_OPS(prio_dispatch, s32 cpu, struct task_struct *prev)
{
int i;
for (i = 0; i < NUM_QUEUES; i++) {
/*
* scx_bpf_consume() moves one task from the custom DSQ to this
* CPU's local DSQ and returns true. We return immediately after
* finding a task — the CPU will pick it up from its local DSQ.
*
* If the DSQ is empty, consume() returns false and we try the
* next priority level.
*/
if (scx_bpf_consume(QUEUE_BASE + i))
return;
}
/* All queues empty: CPU will go idle. That's fine. */
}
/* Destroy DSQs when the scheduler unloads. */
void BPF_STRUCT_OPS(prio_exit, struct scx_exit_info *ei)
{
int i;
for (i = 0; i < NUM_QUEUES; i++)
scx_bpf_destroy_dsq(QUEUE_BASE + i);
/* Log exit reason */
bpf_printk("prio_sched exiting: %s", ei->msg);
}
SEC(".struct_ops.link")
struct sched_ext_ops prio_ops = {
.init = (void *)prio_init,
.enqueue = (void *)prio_enqueue,
.dispatch = (void *)prio_dispatch,
.exit = (void *)prio_exit,
.name = "priority_sched",
};
Step-by-step walkthrough when a task wakes up:
- A task with nice value
-8wakes from sleep (e.g., I/O completion). ops.select_cpu()is called — we use the default, which picks the least-loaded CPU.ops.enqueue(p, SCX_ENQ_WAKEUP)is called.nice_to_queue(p)maps nice-8to queue index1(the second-highest priority).scx_bpf_dispatch(p, QUEUE_BASE + 1, SCX_SLICE_DFL, enq_flags)places the task in DSQ 101.- The CPU's local DSQ may already be busy running something else. The task waits in DSQ 101.
- When the current task's slice expires,
ext_sched_class.pick_next_task()fires and finds the local DSQ empty. ops.dispatch(cpu, prev)is called.- We check DSQ 100 (highest priority) — it's empty.
- We check DSQ 101 — it has our task.
scx_bpf_consume(101)moves it to the local DSQ and returns true. - The CPU picks the task from its local DSQ and runs it.
- When its slice expires,
ops.stopping()fires, thenops.enqueue()is called again to re-enqueue it.
Virtual Time Scheduling: Weighted Fair Queuing
FIFO and strict-priority scheduling are easy but unfair — a flood of low-latency tasks can starve high-nice-value tasks. Weighted Fair Queuing (WFQ) provides fairness: each task gets CPU time proportional to its weight.
sched_ext supports WFQ through vtime-ordered DSQs. A vtime DSQ orders tasks by their vtime field — the task with the smallest vtime runs next.
The key insight: time advances slower for high-weight tasks. If task A has weight 200 (double the default) and task B has weight 100, and both do the same amount of work, task A's vtime advances at half the rate of task B's. So task A always appears to have "used less virtual time" and gets scheduled first, effectively getting 2x the CPU time.
/* wfq_sched.bpf.c
* Weighted Fair Queuing using a single vtime-ordered global DSQ.
*
* All tasks share one vtime DSQ. Tasks with higher weight (lower nice)
* advance their vtime more slowly, so they stay near the front of the queue
* and get proportionally more CPU time.
*/
#include <scx/common.bpf.h>
/* Global virtual time: the minimum vtime among all runnable tasks.
* We use this to "lag-limit" new tasks — they start at min_vtime so
* they don't get a burst of backlogged CPU time upon waking. */
static u64 global_min_vtime = 0;
void BPF_STRUCT_OPS(wfq_enqueue, struct task_struct *p, u64 enq_flags)
{
u64 vtime = p->scx.dsq_vtime;
u32 weight = p->scx.weight; /* 1..10000, default 100 for nice 0 */
/*
* Lag limiting: if the task's vtime has fallen far behind the global
* minimum (e.g., it was sleeping for a long time), snap it forward.
* Without this, a waking task would see a huge credit and monopolize
* the CPU for an extended period.
*
* We allow tasks to be behind by at most one default slice's worth
* of virtual time.
*/
if (vtime_before(vtime, global_min_vtime))
vtime = global_min_vtime;
/*
* Advance the task's virtual time by (slice / weight).
*
* Heavier tasks (higher weight) advance their vtime MORE slowly.
* This is the core WFQ invariant:
* - weight 200 task: vtime += 10ms / 200 = 0.05ms per real millisecond
* - weight 100 task: vtime += 10ms / 100 = 0.10ms per real millisecond
*
* The weight-100 task's vtime grows twice as fast, so the weight-200
* task always looks "further behind" and gets scheduled preferentially.
*/
vtime += SCX_SLICE_DFL / weight;
/* Update the task's stored vtime for the next scheduling decision */
p->scx.dsq_vtime = vtime;
/*
* Dispatch to the global vtime-ordered DSQ.
* The DSQ will insert this task in vtime order (smallest first).
*/
scx_bpf_dispatch_vtime(p, SCX_DSQ_GLOBAL, SCX_SLICE_DFL, vtime, enq_flags);
}
void BPF_STRUCT_OPS(wfq_running, struct task_struct *p)
{
/*
* When a task starts running, update the global minimum vtime.
* The task at the head of the vtime queue (the one that just started
* running) has the minimum vtime among all runnable tasks.
*
* NOTE: this is a simplified approximation. Production schedulers
* (like scx_rustland) use more sophisticated min-vtime tracking.
*/
if (vtime_before(global_min_vtime, p->scx.dsq_vtime))
global_min_vtime = p->scx.dsq_vtime;
}
SEC(".struct_ops.link")
struct sched_ext_ops wfq_ops = {
.enqueue = (void *)wfq_enqueue,
.running = (void *)wfq_running,
.name = "wfq_sched",
};
/* Helper: returns true if a < b, handling 64-bit wraparound */
static inline bool vtime_before(u64 a, u64 b)
{
return (s64)(a - b) < 0;
}
Why vtime DSQs instead of per-task sorting in BPF?
You might think: "I can maintain a sorted tree in a BPF map." The problem is that BPF map operations are O(log n) and have higher constant factors than kernel data structures. Using the kernel's built-in vtime DSQ (which uses a red-black tree internally) gives you correct, efficient, SMP-safe WFQ ordering without any complex BPF logic.
Tickless Operation
The Traditional Timer Tick
In a non-tickless kernel, a hardware timer fires every 1/HZ seconds (typically 4ms at HZ=250). Each tick:
- Decrements the current task's time slice (
p->scx.slice) - If the slice reaches 0, sets a reschedule flag
- At the next safe point, the kernel context-switches to the next task
This works fine for most workloads, but the interrupt overhead is measurable for compute-intensive tasks running on dedicated cores.
sched_ext and nohz_full
Linux has a "nohz_full" mode where CPUs that are running a single task suppress the periodic timer entirely — the tick is dynamically re-enabled only when needed. sched_ext integrates with this:
- Large slices: If BPF dispatches tasks with large slice values (e.g.,
SCX_SLICE_INFfor infinite), the kernel knows the task won't expire its slice soon and can suppress ticks. - BPF slice management: BPF sets
p->scx.slicewhen callingscx_bpf_dispatch(). The kernel uses this to determine when to re-enable the tick. - Slice expiry callback: When a task's slice does expire mid-tick-suppression, the hardware timer that does fire handles it correctly.
/* Give latency-sensitive tasks a normal slice, compute tasks a large one */
void BPF_STRUCT_OPS(tickless_enqueue, struct task_struct *p, u64 enq_flags)
{
u64 slice;
if (p->flags & PF_KTHREAD) {
/* Kernel threads: use default slice */
slice = SCX_SLICE_DFL;
} else if (task_is_latency_sensitive(p)) {
/* Interactive tasks: small slice for quick response */
slice = 1 * NSEC_PER_MSEC;
} else {
/* Batch/compute tasks: large slice, suppress ticks */
slice = 100 * NSEC_PER_MSEC;
}
scx_bpf_dispatch(p, SCX_DSQ_GLOBAL, slice, enq_flags);
}
The benefit: a CPU running a long-running compute task with a 100ms slice will only receive a timer interrupt every 100ms instead of every 4ms. For compute-bound workloads, this reduces scheduling overhead from ~250 interrupts/second to ~10 interrupts/second.
The ops.stopping() / ops.running() Pattern for Precise Accounting
BPF schedulers that do vtime-based scheduling need to know exactly how much time a task actually used. The running/stopping pair enables this:
void BPF_STRUCT_OPS(my_running, struct task_struct *p)
{
/* Record when the task started executing */
p->scx.dsq_vtime = bpf_ktime_get_ns(); /* repurpose for start time */
}
void BPF_STRUCT_OPS(my_stopping, struct task_struct *p, bool runnable)
{
u64 now = bpf_ktime_get_ns();
u64 used = now - p->scx.dsq_vtime; /* actual CPU time consumed */
/* Update virtual time based on actual usage, not allocated slice */
p->scx.dsq_vtime = compute_new_vtime(p, used);
}
CPU Coordination: Multi-CPU Scheduling
For true global scheduling policies, BPF needs to coordinate decisions across CPUs. sched_ext provides two patterns for this.
The scx_central Pattern (Centralized Dispatch)
In this pattern, one designated CPU acts as the "scheduler CPU" and makes all scheduling decisions for all other CPUs:
All tasks enqueue here
│
▼
Central CPU
┌──────────────┐
│ ops.dispatch │
│ (for all CPUs│
│ via │
│ dispatch_ │
│ local_on()) │
└──────┬───────┘
│
┌───────────┼───────────┐
│ │ │
▼ ▼ ▼
CPU 0's CPU 1's CPU 2's
local DSQ local DSQ local DSQ
Implementation sketch:
/* central_sched.bpf.c - simplified central scheduler pattern */
#define CENTRAL_CPU 0
/* Per-CPU task queues: tasks for each CPU */
struct {
__uint(type, BPF_MAP_TYPE_QUEUE);
__uint(max_entries, 4096);
__type(value, u32); /* task pid or task pointer */
} cpu_queues[MAX_CPUS] SEC(".maps");
void BPF_STRUCT_OPS(central_enqueue, struct task_struct *p, u64 enq_flags)
{
s32 target_cpu = p->scx.sticky_cpu;
if (target_cpu < 0)
target_cpu = bpf_get_smp_processor_id();
/* Put task in a per-CPU queue for the central scheduler to process */
scx_bpf_dispatch(p, CPU_DSQ(target_cpu), SCX_SLICE_DFL, enq_flags);
}
void BPF_STRUCT_OPS(central_dispatch, s32 cpu, struct task_struct *prev)
{
if (cpu != CENTRAL_CPU) {
/*
* Worker CPUs don't dispatch themselves.
* They wait for the central CPU to push tasks into their local DSQ.
* If the central CPU is busy, this CPU may briefly go idle —
* the central CPU will kick it with scx_bpf_kick_cpu().
*/
return;
}
/* Central CPU: dispatch tasks to all CPUs */
int target;
bpf_for(target, 0, nr_cpus) {
if (!scx_bpf_consume(CPU_DSQ(target)))
continue; /* No task for this CPU */
if (target == CENTRAL_CPU) {
/* Task for us: it's now in our local DSQ, we'll pick it up */
continue;
}
/*
* Dispatch directly to target CPU's local DSQ.
* scx_bpf_kick_cpu() wakes the CPU if it's idle.
*/
/* Note: in real code, use scx_bpf_dispatch_local_on() here */
scx_bpf_kick_cpu(target, SCX_KICK_IDLE);
}
}
The central pattern enables global policies that are difficult to implement with distributed per-CPU decisions — for example, ensuring that exactly N tasks of type X are running at any moment across the entire system.
CPU Acquire/Release: Knowing When You Own CPUs
The kernel calls ops.cpu_acquire() and ops.cpu_release() to tell the BPF scheduler which CPUs it currently "owns":
void BPF_STRUCT_OPS(my_cpu_acquire, s32 cpu, struct scx_cpu_acquire_args *args)
{
/*
* The kernel has given us this CPU: no RT/DL tasks are runnable on it.
* We can schedule sched_ext tasks here.
* Update our internal CPU availability bitmap.
*/
atomic_or(&available_cpus, (1ULL << cpu));
}
void BPF_STRUCT_OPS(my_cpu_release, s32 cpu, struct scx_cpu_release_args *args)
{
/*
* An RT or DL task needs this CPU. We must stop scheduling here.
* The kernel will call pick_next_task() for the higher class next.
* args->reason tells us why: SCX_CPU_RELEASE_PREEMPT or similar.
*/
atomic_and(&available_cpus, ~(1ULL << cpu));
}
These callbacks are essential for BPF schedulers that implement NUMA-aware placement, power management (C-state optimization), or work-stealing algorithms that need accurate CPU availability information.
Work-Stealing with scx_bpf_kick_cpu
When a BPF scheduler enqueues a task but the target CPU is busy, it needs to notify an idle CPU to steal the work:
void BPF_STRUCT_OPS(stealing_enqueue, struct task_struct *p, u64 enq_flags)
{
/* Place task in global queue */
scx_bpf_dispatch(p, SCX_DSQ_GLOBAL, SCX_SLICE_DFL, enq_flags);
/* If there are idle CPUs, wake one of them to pick up this task */
s32 idle_cpu = scx_bpf_pick_idle_cpu(p->cpus_ptr, 0);
if (idle_cpu >= 0)
scx_bpf_kick_cpu(idle_cpu, SCX_KICK_IDLE);
}
scx_bpf_pick_idle_cpu() searches the CPU idle mask (maintained by the kernel or BPF, depending on flags) for an idle CPU within the task's affinity mask.
Core Scheduling Integration
The SMT Problem
Modern CPUs use Simultaneous Multithreading (SMT, also known as Hyperthreading). A physical core contains 2 or 4 "logical CPUs" (hardware threads) that share the core's execution resources including L1/L2 caches and execution pipelines.
This sharing creates a security problem: a malicious task on one hardware thread can use microarchitectural side channels (Spectre, MDS) to read data being processed by a task on the sibling hardware thread. This is the threat model that "core scheduling" addresses.
Core Scheduling
Core scheduling (CONFIG_SCHED_CORE) requires that SMT siblings always run tasks from the same security domain (tagged with the same core_sched_cookie). If two tasks have different cookies (different trust levels), they cannot run on sibling HTs simultaneously — one HT must idle or run an idle thread while the other HT runs.
Without core scheduling integration, sched_ext could accidentally schedule:
- Task A (cookie X, user process) on CPU 0
- Task B (cookie Y, potentially hostile) on CPU 1 (sibling of CPU 0)
This would be a security vulnerability.
sched_ext's integration:
sched_ext participates in core scheduling through the ops.core_sched_before() callback and the cookie-based task pairing mechanism:
/* BPF can implement task affinity for core scheduling.
* This callback is called to decide if task 'a' should run before
* task 'b' on the same physical core, considering their cookies.
*
* Return true if 'a' should run before 'b'.
* Return false if 'b' should run before 'a' or they're equivalent.
*
* The kernel uses this to ensure compatible tasks are co-scheduled
* on SMT siblings. */
bool BPF_STRUCT_OPS(my_core_sched_before,
struct task_struct *a, struct task_struct *b)
{
/* For security: prefer tasks with the same cookie as the sibling */
if (a->core_cookie == b->core_cookie)
return false; /* equivalent priority */
/* Otherwise, use vtime ordering */
return vtime_before(a->scx.dsq_vtime, b->scx.dsq_vtime);
}
If a BPF scheduler does not implement core_sched_before(), the kernel uses default cookie comparison rules.
Writing a Production BPF Scheduler: Key Principles
1. Handle init_task() Errors Correctly
ops.init_task() is called for every task that joins sched_ext. If you allocate per-task memory here (from a BPF map, for example) and the allocation fails, you must return a negative errno:
struct {
__uint(type, BPF_MAP_TYPE_HASH);
__uint(max_entries, 65536);
__type(key, pid_t);
__type(value, struct my_task_data);
} task_data SEC(".maps");
s32 BPF_STRUCT_OPS(my_init_task, struct task_struct *p,
struct scx_init_task_args *args)
{
struct my_task_data data = { .vtime = 0, .weight = p->scx.weight };
pid_t pid = p->pid;
if (bpf_map_update_elem(&task_data, &pid, &data, BPF_NOEXIST) < 0) {
/*
* Map is full. Return -ENOMEM to signal failure.
* The kernel will set SCX_TASK_DISALLOW on this task,
* and it will be handled by CFS instead.
* DO NOT return 0 here — that would lie to the kernel.
*/
scx_bpf_error("task_data map full");
return -ENOMEM;
}
return 0;
}
Tasks that fail init_task() get the SCX_TASK_DISALLOW flag and are transparently redirected to CFS. This is safe — the system keeps running, the task just doesn't use your scheduler.
2. Always Handle ops.dequeue()
When a task leaves sched_ext (policy change, migration, exit), ops.dequeue() is called. If your BPF scheduler maintains external state — anything beyond p->scx.* fields — you must clean it up here:
void BPF_STRUCT_OPS(my_dequeue, struct task_struct *p, u64 deq_flags)
{
pid_t pid = p->pid;
/*
* Remove per-task data from our map.
* If we don't do this, the map entry leaks.
* For BPF_MAP_TYPE_HASH with max_entries, leaks eventually cause
* bpf_map_update_elem() to fail, breaking the scheduler for new tasks.
*/
bpf_map_delete_elem(&task_data, &pid);
}
Missing ops.dequeue() cleanup is the most common production bug in BPF schedulers.
3. Keep ops.dispatch() Bounded
ops.dispatch() must not loop indefinitely. The BPF verifier will catch truly infinite loops, but you can still write a dispatch that takes too long or starves the CPU:
/* BAD: could spin consuming tasks forever if the DSQ is constantly refilled */
void BPF_STRUCT_OPS(bad_dispatch, s32 cpu, struct task_struct *prev)
{
while (scx_bpf_consume(my_dsq)) {
/* This will keep running until the DSQ is empty... */
/* but if enqueue() keeps adding tasks, we might never return */
}
}
/* GOOD: consume at most N tasks per dispatch call */
void BPF_STRUCT_OPS(good_dispatch, s32 cpu, struct task_struct *prev)
{
int i;
for (i = 0; i < 8; i++) { /* BPF loop bound */
if (!scx_bpf_consume(my_dsq))
break;
}
/* Return normally. The CPU will call dispatch() again if needed. */
}
4. Use scx_bpf_exit() for Graceful Shutdown
When the scheduler detects an unrecoverable error, calling scx_bpf_exit() is better than letting the watchdog fire:
s32 BPF_STRUCT_OPS(my_init)
{
if (scx_bpf_create_dsq(MY_DSQ, -1) < 0) {
/* Don't return -ENOMEM silently — the kernel won't know why */
scx_bpf_exit(-ENOMEM, "Failed to create DSQ during init");
return -ENOMEM;
}
return 0;
}
scx_bpf_exit() is called from within ops.init() or any callback. The kernel receives the exit message and triggers the disable path immediately, logging the message for the userspace process to read via ops.exit().
5. Test with the sched_ext Selftests
The kernel tree includes selftests in tools/testing/selftests/sched_ext/. These tests cover:
- Basic load/unload of example schedulers
- Watchdog trigger tests
- sysrq-S functionality
- Migration and CPU hotplug edge cases
- ops.init_task() failure handling
Run them with:
cd tools/testing/selftests/sched_ext
sudo make && sudo ./runner.py
6. Monitor via /sys/kernel/sched_ext
When a scheduler is loaded, the kernel exposes information at:
/sys/kernel/sched_ext/
├── root/
│ ├── ops # current scheduler name
│ ├── state # enabled/disabled/initializing
│ └── stats/ # per-scheduler statistics
└── ...
Read state to check if your scheduler is still active:
cat /sys/kernel/sched_ext/root/ops # "my_scheduler" or empty
7. Build with libbpf Skeletons
The recommended build pattern:
# Makefile excerpt
%.bpf.o: %.bpf.c vmlinux.h
clang -O2 -g -target bpf \
-D__TARGET_ARCH_$(ARCH) \
-I$(LIBBPF_INCLUDE) \
-c $< -o $@
%.skel.h: %.bpf.o
bpftool gen skeleton $< > $@
%: %.c %.skel.h
$(CC) -O2 -g $< -o $@ -lbpf -lelf -lz
The skeleton header (*.skel.h) generated by bpftool gen skeleton provides:
my_sched__open()— parse the BPF object, prepare mapsmy_sched__load()— run the verifier, load into kernelmy_sched__attach()— register the struct_ops and activate the schedulermy_sched__destroy()— clean up and unload
The ops_state Machine: How Tasks Join and Leave sched_ext
Every task has an ops_state field (part of scx_entity) that tracks its relationship with the BPF scheduler. The transitions are driven by the kernel, not BPF.
[Not an SCX task]
│
│ SCHED_EXT policy set (via sched_setscheduler() / prctl())
│ OR task forked from an SCX task
│
▼
[SCX_OPSS_NONE]
│ Kernel calls ops.init_task(p, args)
│
├─── init_task() returned 0 (success) ──────────────────────────────────┐
│ │
├─── init_task() returned -errno ─────────────────────────────────────┐ │
│ SCX_TASK_DISALLOW set; task falls back to CFS │ │
│ (ops.exit_task() NOT called for init failures) │ │
│ ▼ ▼
│ [SCX_OPSS_INIT_DONE]
│ │
│ Kernel calls ops.enable(p)
│ │
│ ▼
│ [SCX_OPSS_ENABLED]
│ ◄────────────────
│ │ Normal operating
│ │ state. All
│ │ scheduling ops
│ │ active.
│ ─────────────────►
│
│ (Task policy changed away from SCHED_EXT, task exits,
│ or BPF scheduler is unloaded)
│
│ Kernel calls ops.disable(p)
│ │
│ ▼
│ [SCX_OPSS_DISABLED]
│ │
│ Kernel calls ops.exit_task(p)
│ │
│ ▼
│ [Not an SCX task]
Key transition notes:
ops.init_task()failure does NOT callops.exit_task(). The task never fully joined, so there is nothing to clean up on the exit path.ops.enable()is called afterops.init_task()succeeds. This is when the task first becomes schedulable by your BPF program.- The
INIT_DONE → ENABLEDtransition may be delayed if the task is being migrated. The kernel ensuresenable()is called on the task's destination CPU. - When the BPF scheduler is unloaded, the kernel calls
ops.disable()on every SCHED_EXT task simultaneously (in parallel), then callsops.exit_task()in sequence.
Kernel Entry Points: Where sched_ext Hooks In
Understanding where sched_ext integrates into the kernel helps with debugging and understanding overhead.
Task Lifecycle Hooks
sched_fork() → scx_fork()
Called when a new task is created via fork()/clone(). Allocates the scx_entity structure (already embedded in task_struct, but needs initialization) and calls ops.init_task() if the task will use SCHED_EXT.
/* kernel/sched/core.c */
int sched_fork(unsigned long clone_flags, struct task_struct *p)
{
/* ... other initialization ... */
scx_fork(p); /* -> calls ops.init_task() */
return 0;
}
do_exit() → scx_exit_task()
Called when a task exits. Calls ops.stopping() (if running), ops.dequeue() (if queued), ops.disable(), and ops.exit_task() in order.
sched_setscheduler() → __setscheduler() → class switch
When userspace calls sched_setscheduler(pid, SCHED_EXT, ¶m), the kernel:
- Calls
check_class_changing()to validate the transition - Removes the task from its current class's runqueue
- Sets
p->sched_class = &ext_sched_class - Calls
scx_ops_enable_task()→ops.init_task()thenops.enable() - Enqueues the task on the ext runqueue
Scheduling Hooks
schedule() → __schedule() → pick_next_task()
The main scheduling path. When the kernel decides to context-switch:
- Calls
put_prev_task()on the outgoing task →ops.stopping() - Walks the sched_class chain to find the next task
- If
ext_sched_classhas tasks, callsscx_pick_next_task() scx_pick_next_task()dequeues from the CPU's local DSQ- If local DSQ is empty, calls
ops.dispatch(cpu, prev)first - Returns the selected task →
ops.running()called
try_to_wake_up() (ttwu)
The wakeup path. Called when a sleeping task should become runnable:
- Calls
select_task_rq_ext()→ops.select_cpu() - Moves task to the selected CPU's runqueue
- Calls
enqueue_task_ext()→ops.runnable()thenops.enqueue() - May kick the target CPU:
scx_bpf_kick_cpu(target, SCX_KICK_IDLE)
scheduler_tick()
Called by the timer interrupt every 1/HZ seconds:
- Calls
task_tick_ext()for the current ext task - Decrements
p->scx.slice - If slice expired: sets
TIF_NEED_RESCHEDflag → nextschedule()call preempts the task - Checks watchdog: if any SCHED_EXT task has been runnable but unscheduled for
>timeout_ms, triggersscx_ops_error()
The Per-CPU Runqueue
Each CPU has a struct rq (runqueue) that contains all scheduling state for that CPU:
/* Simplified kernel/sched/sched.h */
struct rq {
/* ... */
struct cfs_rq cfs; /* CFS runqueue */
struct rt_rq rt; /* RT runqueue */
struct dl_rq dl; /* Deadline runqueue */
struct scx_rq scx; /* sched_ext per-CPU state */
/* ... */
};
struct scx_rq {
struct scx_dispatch_q local_dsq; /* per-CPU local DSQ */
u64 flags;
/* ... */
};
The scx.local_dsq is the per-CPU local DSQ. ext_sched_class.pick_next_task() dequeues from this. The only way to get a task into this DSQ is:
- BPF calls
scx_bpf_dispatch(p, SCX_DSQ_LOCAL, ...)orscx_bpf_dispatch(p, SCX_DSQ_LOCAL_ON(cpu), ...) - The kernel auto-drains
SCX_DSQ_GLOBALinto local DSQs - BPF calls
scx_bpf_consume(custom_dsq)fromops.dispatch(), which moves the task to the local DSQ
Putting It All Together: The scx_simple Scheduler
The scx_simple scheduler in the sched_ext/scx tools repository is the canonical minimal example. Here is an annotated version that ties together all the concepts:
Load time:
1. Userspace calls minimal_sched__open_and_load()
2. BPF verifier runs — validates all callbacks
3. Kernel calls ops.init() — BPF creates any global state
4. For each existing SCHED_EXT task:
ops.init_task(p, ...) → ops.enable(p)
Run time (task wakeup):
1. Task T wakes from sleep (I/O, timer, etc.)
2. Kernel calls ops.select_cpu(T, prev_cpu, SCX_WAKE_TTWU)
→ BPF returns a CPU (or -1 for kernel to decide)
3. Kernel calls ops.runnable(T, SCX_ENQ_WAKEUP)
→ BPF notes T is now runnable (accounting only)
4. Kernel calls ops.enqueue(T, SCX_ENQ_WAKEUP)
→ BPF calls scx_bpf_dispatch(T, SCX_DSQ_GLOBAL, SCX_SLICE_DFL, ...)
5. T is now in the global DSQ
Run time (CPU needs a task):
1. CPU N's local DSQ is empty; it's about to go idle
2. ext_sched_class.pick_next_task() is called
3. Local DSQ is empty; kernel calls ops.dispatch(N, prev_task)
4. BPF calls scx_bpf_consume(SCX_DSQ_GLOBAL)
→ Moves T from global DSQ to CPU N's local DSQ
→ Returns true
5. kernel picks T from local DSQ
6. ops.running(T) called — T is now executing
Run time (task runs out of time):
1. T's p->scx.slice reaches 0 (decremented by timer tick)
2. TIF_NEED_RESCHED set; T preempted at next safe point
3. ops.stopping(T, true) called — T still runnable
4. ops.enqueue(T, 0) called — T re-enqueued
5. [cycle repeats]
Unload time:
1. Userspace calls minimal_sched__destroy()
2. Kernel activates SCX_BYPASS mode
3. All runqueue tasks migrated to CFS
4. ops.exit(exit_info) called
5. BPF program freed
6. SCHED_EXT tasks continue running on CFS
Glossary
BPF verifier: The kernel's static analyzer that validates BPF programs at load time. Checks memory safety, bounds, types, and termination before any code runs.
bypass mode (SCX_BYPASS): A kernel state where the BPF scheduler is skipped and all tasks fall back to CFS. Activated during the disable path (watchdog, sysrq-S, or scx_bpf_exit()).
CO-RE (Compile Once, Run Everywhere): The BPF mechanism that allows BPF programs compiled against one kernel version's BTF type info to run on different kernel versions, with field offsets adjusted at load time.
core scheduling: A Linux security feature (CONFIG_SCHED_CORE) that prevents tasks with different core_sched_cookie values from running simultaneously on SMT siblings. Mitigates MDS/Spectre-class side-channel attacks.
direct dispatch: Calling scx_bpf_dispatch() from within ops.select_cpu() instead of from ops.enqueue(). When used, ops.enqueue() is skipped for that task, reducing callback overhead.
DSQ (Dispatch Queue): A queue of tasks waiting to be scheduled. The fundamental data structure in sched_ext. Three kinds: global (SCX_DSQ_GLOBAL), per-CPU local (SCX_DSQ_LOCAL), and custom (BPF-created).
ext_sched_class: The kernel sched_class struct that implements the sched_ext scheduling class. Lives in kernel/sched/ext.c. Fourth in the scheduler hierarchy, below RT and above idle.
ops_state: The state machine tracking a task's relationship with the BPF scheduler. Values: SCX_OPSS_NONE, SCX_OPSS_INIT_DONE, SCX_OPSS_ENABLED, SCX_OPSS_DISABLED.
scx_entity: Per-task sched_ext state embedded in task_struct at task_struct.scx. Contains DSQ membership, vtime, slice, weight, and state flags.
sched_ext_ops: The BPF vtable. A struct of function pointers that the BPF program fills in. The kernel calls these functions to implement scheduling policy. The BPF program registers this via the .struct_ops.link ELF section.
SCX_DSQ_GLOBAL (= 0): The built-in global FIFO DSQ shared by all CPUs. The kernel automatically drains it into per-CPU local DSQs. The simplest way to use sched_ext.
SCX_DSQ_LOCAL (= 1): The symbolic ID meaning "this CPU's local DSQ." Each CPU has a private local DSQ; ext_sched_class.pick_next_task() dequeues exclusively from it.
SCX_DSQ_LOCAL_ON(cpu): A macro that produces the DSQ ID for a specific CPU's local DSQ. Used to dispatch a task directly to a target CPU, bypassing the usual enqueue flow.
SCX_SLICE_DFL: A constant representing the default time slice duration. Configurable via sysfs. Approximately 20ms by default.
SCX_SLICE_INF: A constant representing an infinite time slice. Used when the BPF scheduler manages preemption explicitly rather than relying on periodic timer ticks.
SCX_TASK_DISALLOW: A flag on scx_entity indicating that the BPF scheduler rejected this task (e.g., ops.init_task() returned an error). The task runs on CFS instead.
skeleton (*.skel.h): A generated C header file produced by bpftool gen skeleton. Provides type-safe C wrappers for opening, loading, attaching, and destroying a specific BPF program.
slice: The duration (in nanoseconds) allocated to a task for one scheduling quantum. Set by BPF via the slice parameter of scx_bpf_dispatch(). Decremented by timer ticks. When it reaches zero, the task is preempted.
struct_ops: A BPF program type that implements a kernel "struct of callbacks" interface. sched_ext uses struct_ops to implement the sched_ext_ops vtable in BPF. The BPF verifier applies callback-specific rules to each function in the struct.
vtime (virtual time): A monotonically increasing per-task counter used for weighted fair scheduling. In a vtime-ordered DSQ, tasks with smaller vtime run first. High-weight tasks advance their vtime more slowly, giving them higher effective priority.
watchdog: A kernel timer that fires every timeout_ms / 2 milliseconds and checks if any SCHED_EXT task has been runnable but unscheduled for longer than timeout_ms. If detected, triggers the disable path and logs an error.
Further Reading
For readers who want to go deeper, the following are the canonical sources:
- Kernel source:
kernel/sched/ext.c— the sched_ext implementation (~4000 lines) - Kernel headers:
include/linux/sched/ext.h—sched_ext_ops,scx_entity, and all public types - BPF helpers:
include/uapi/linux/bpf.h— helper function declarations - Example schedulers:
tools/sched_ext/in the kernel tree (or thesched-ext/scxGitHub repo)scx_simple: minimal FIFO scheduler (global DSQ only)scx_central: centralized dispatch patternscx_flatcg: cgroup-aware schedulerscx_rustland: userspace scheduling via io_uring, written in Rustscx_layered: production-quality layered scheduler (used at Meta)
- Selftests:
tools/testing/selftests/sched_ext/ - Design document:
Documentation/scheduler/sched-ext.rstin the kernel tree
Patch Study
Patch-by-patch analysis grouped by theme.
Foundational Refactoring (Patches 01–07)
Overview
Before a single line of sched_ext itself could be written, seven preparatory patches had to reshape
the existing Linux scheduler infrastructure. These patches do not add any new scheduling policy.
They are purely structural: they remove assumptions baked into the scheduler core that would
have made it impossible to insert a new, dynamically loadable scheduler class between
fair_sched_class and idle_sched_class, and they add hooks that the ext class will call at
specific points in a task's lifecycle.
Understanding these patches deeply is essential for a maintainer because they reveal the implicit contracts that had accumulated in the scheduler over decades of evolution. Each one exposes a place where the kernel had encoded assumptions about the set of scheduler classes, and each fix generalizes that assumption into something extensible. That generalization work is unglamorous but is the reason sched_ext could be merged cleanly into mainline rather than requiring invasive surgery to the core scheduler paths.
Why These Patches Are Needed
The Linux scheduler is organized as a linked list of sched_class objects:
stop_sched_class → dl_sched_class → rt_sched_class → fair_sched_class → idle_sched_class
Each class is a statically allocated struct sched_class with function pointers for every
scheduling operation. The kernel traverses this list in priority order—a class only runs tasks
when all higher-priority classes have no runnable work.
sched_ext introduces ext_sched_class, which sits between fair_sched_class and
idle_sched_class. Inserting a new class into this list sounds simple until you examine what
the existing code assumes:
-
The set of classes is fixed at compile time. Several validation checks used linker-section addresses to compare class priority rather than a dedicated comparison function. These checks would silently misorder the new class.
-
Fork is infallible for scheduling purposes.
sched_cgroup_fork()was void. sched_ext needs to allocate per-task BPF state during fork, which can fail withENOMEM. Propagating that error required making the call site aware that fork-time scheduler initialization can fail. -
Priority changes on enqueued tasks are handled internally. No
sched_classhook existed for the moment a task's weight changes while it is already on a run queue. sched_ext needs to notify the BPF program when a task's nice value changes so the BPF program can re-sort its own data structures. -
Class transitions are managed by fair/rt directly. When a task moves from one scheduler class to another, the old class's
put_prev_task()and the new class'senqueue_task()are called, but there was no "you are about to receive this task" notification to the incoming class. sched_ext needs to initialize per-task BPF state before the task is enqueued into the ext class. -
Cgroup weight conversion and load-average helpers are fair-private. sched_ext needs to expose the same load metrics and weight calculations to BPF programs, requiring these helpers to be factored out of
fair.cinto shared code.
Key Concepts
PATCH 01 — sched_class_above()
The sched_class structures are laid out in a specific order in the kernel's .data section.
Historically, code in kernel/sched/core.c and kernel/sched/fair.c checked class priority by
comparing pointer values directly:
/* OLD, brittle */
if (p->sched_class > &fair_sched_class)
...
This works only if the linker places the class objects in the expected order and there are no gaps.
Inserting ext_sched_class between fair_sched_class and idle_sched_class would shift pointer
values and break every such comparison.
Patch 01 introduces sched_class_above(a, b), a semantic predicate that returns true if class a
has higher priority than class b. Internally it uses the existing linker ordering (or a generated
priority field), but critically, all call sites now express their intent — "is this task's class
above fair?" — rather than encoding linker layout knowledge. After this patch, inserting a new class
only requires updating sched_class_above(), not auditing every comparison in the tree.
PATCH 02 — Fallible Fork
sched_cgroup_fork() is called from copy_process() during fork(2). Before this patch it was
declared void — if it failed internally, it could only BUG() or silently drop the error.
The patch changes the signature to return int and threads that return value back up through
copy_process(). A companion function, sched_cancel_fork(), is added to undo partial
initialization when a later step of copy_process() fails after sched_cgroup_fork() has
succeeded.
For sched_ext, this matters because ops.enable() — the BPF callback that initializes per-task
state — needs to run during fork. BPF programs may allocate a BPF_MAP entry per task; that
allocation can fail. Without a fallible fork path, the only option would have been to defer the
allocation to the first time the task runs, introducing a window where the task exists without
BPF state.
PATCH 03 — reweight_task()
When a task's priority changes via setpriority(2) or nice(2), the kernel calls
set_user_nice() → reweight_task() (for CFS, this updates the task's load weight in the RB
tree). The existing sched_class interface had no hook for this event.
Patch 03 adds sched_class->reweight_task(rq, p, prio). The CFS implementation is refactored
to use this new hook rather than doing the reweighting inline. For sched_ext, the hook generates a
call to ops.reweight_task(), allowing a BPF scheduler to update its internal priority queues
when a task's nice value changes while that task is enqueued.
Without this hook, a BPF scheduler that maintains, say, a priority queue keyed on task weight
would see stale weights after a nice() call, leading to incorrect scheduling decisions.
PATCH 04 — switching_to() and check_class_changing/changed()
When a task moves from one scheduler class to another (e.g., from CFS to sched_ext via
SCHED_EXT policy, or between any classes), the scheduler calls:
check_class_changing(rq, p, prev_class)— called with the task still in the old class, before dequeuing from the old class.check_class_changed(rq, p, prev_class, oldprio)— called after the task has been moved to the new class and potentially enqueued.
The problem: check_class_changed() calls switched_to() on the new class, but by that point
the task is already enqueued. There was no hook called on the new class before the task arrived.
Patch 04 adds sched_class->switching_to(rq, p), called on the incoming class while the task is
still in the old class. For sched_ext, this is where the kernel can invoke ops.enable() on the
BPF program, giving it a chance to allocate and initialize per-task state before the task's first
enqueue_task_scx() call. Without this ordering guarantee, the enqueue callback would be called
with uninitialized BPF task state, requiring ugly NULL checks in every enqueue path.
PATCHES 05–06 — Factored Utilities
fair.c contains two categories of code that sched_ext needs to use:
-
Cgroup weight conversion:
sched_prio_to_weight[]and related helpers translate cgroup CPU weight settings (which follow a specific scale) to scheduler load weights. sched_ext exposes these weights to BPF programs so they can implement weighted fair scheduling. -
Load average tracking: The PELT (Per-Entity Load Tracking) infrastructure in CFS tracks load, utilization, and runnable averages with exponential decay. sched_ext can leverage these same signals so BPF programs can make power-aware or utilization-aware decisions without reimplementing load tracking from scratch.
These patches extract the relevant functions from fair.c into kernel/sched/sched.h or
kernel/sched/pelt.h so that ext.c can call them without introducing a dependency on
fair.c internals.
PATCH 07 — normal_policy()
A small but important helper: normal_policy(policy) returns true if the scheduling policy is
one of SCHED_NORMAL, SCHED_BATCH, or SCHED_IDLE — the three "normal" (non-realtime, non-ext)
policies that map to CFS. Before this patch, these checks were open-coded as multi-condition
expressions scattered across core.c and fair.c.
sched_ext needs to make this determination in several places: for example, when a task that was
previously using SCHED_EXT calls sched_setscheduler() to switch back to SCHED_NORMAL. A
named predicate is clearer than a repeated chain of policy == SCHED_NORMAL || policy == SCHED_BATCH || policy == SCHED_IDLE.
Connections Between Patches
These seven patches form a dependency chain that flows into the core sched_ext implementation:
PATCH 01 (sched_class_above)
└─→ Required by ext.c wherever it needs to check "is this task above/below ext class?"
PATCH 02 (fallible fork)
└─→ Required by PATCH 04: switching_to() / ops.enable() may allocate; fork must propagate errors
PATCH 03 (reweight_task)
└─→ Required by ext.c: ops.reweight_task() BPF callback
PATCH 04 (switching_to)
└─→ Required by ext.c: ops.enable() must run before enqueue_task_scx()
PATCHES 05-06 (factored utilities)
└─→ Required by ext.c: scx_bpf_task_cgroup_weight(), PELT load metrics exposed to BPF
PATCH 07 (normal_policy)
└─→ Used throughout ext.c: determining when a task is returning to CFS
Notice that none of these patches reference sched_ext at all. They are pure scheduler infrastructure improvements. This was a deliberate design choice: each patch is independently justifiable and could be accepted on its own merits. The sched_ext patchset was structured this way to make review easier — reviewers of the scheduler core did not have to understand BPF to evaluate patches 01–07.
What to Focus On
For a maintainer, the most important lessons from this group are:
-
Implicit contracts in sched_class. Before this series, the scheduler had several places where the contract was "there are exactly these N scheduler classes in this order." Patches 01 and 04 make those contracts explicit and extensible. When reviewing future scheduler patches, watch for new places where such implicit contracts re-emerge.
-
Error propagation in lifecycle hooks. Patch 02's fallible fork is a template for any future scheduler hook that runs during
copy_process(). The pattern — make the hook returnint, add a cancel/cleanup companion, thread the error throughcopy_process()— should be followed whenever a new lifecycle hook might fail. -
Ordering of class-transition callbacks. Patch 04 reveals that class transitions had a subtle ordering gap: the new class was not notified before the task arrived.
switching_to()fills this gap. When reviewing any future change that touches class transitions, verify thatswitching_to(),switched_to(), andcheck_class_changing/changed()are called in the correct order and that no class-specific state is accessed before the appropriate notification. -
Code factoring for cross-class reuse. Patches 05–06 establish the precedent that helpers which multiple scheduler classes need should live in shared headers, not buried in
fair.c. When sched_ext (or any future class) needs a capability that CFS already has, the right fix is to promote the CFS implementation to shared code rather than duplicating it. -
Semantic naming over structural checks. Patch 07's
normal_policy()and patch 01'ssched_class_above()both replace structural knowledge (linker addresses, repeated conditionals) with named predicates. This is a general principle: when the same structural check appears more than twice, it belongs in a named function that captures the semantic intent.
[PATCH 01/30] sched: Restructure sched_class order sanity checks in sched_init()
View on Lore: https://lore.kernel.org/all/20240618212056.2833381-2-tj@kernel.org
Commit Message
Currently, sched_init() checks that the sched_class'es are in the expected
order by testing each adjacency which is a bit brittle and makes it
cumbersome to add optional sched_class'es. Instead, let's verify whether
they're in the expected order using sched_class_above() which is what
matters.
Signed-off-by: Tejun Heo <tj@kernel.org>
Suggested-by: Peter Zijlstra <peterz@infradead.org>
Reviewed-by: David Vernet <dvernet@meta.com>
---
kernel/sched/core.c | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 0935f9d4bb7b..b4d4551bc7f2 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -8164,12 +8164,12 @@ void __init sched_init(void)
int i;
/* Make sure the linker didn't screw up */
- BUG_ON(&idle_sched_class != &fair_sched_class + 1 ||
- &fair_sched_class != &rt_sched_class + 1 ||
- &rt_sched_class != &dl_sched_class + 1);
#ifdef CONFIG_SMP
- BUG_ON(&dl_sched_class != &stop_sched_class + 1);
+ BUG_ON(!sched_class_above(&stop_sched_class, &dl_sched_class));
#endif
+ BUG_ON(!sched_class_above(&dl_sched_class, &rt_sched_class));
+ BUG_ON(!sched_class_above(&rt_sched_class, &fair_sched_class));
+ BUG_ON(!sched_class_above(&fair_sched_class, &idle_sched_class));
wait_bit_init();
--
2.45.2
Diff
---
kernel/sched/core.c | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 0935f9d4bb7b..b4d4551bc7f2 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -8164,12 +8164,12 @@ void __init sched_init(void)
int i;
/* Make sure the linker didn't screw up */
- BUG_ON(&idle_sched_class != &fair_sched_class + 1 ||
- &fair_sched_class != &rt_sched_class + 1 ||
- &rt_sched_class != &dl_sched_class + 1);
#ifdef CONFIG_SMP
- BUG_ON(&dl_sched_class != &stop_sched_class + 1);
+ BUG_ON(!sched_class_above(&stop_sched_class, &dl_sched_class));
#endif
+ BUG_ON(!sched_class_above(&dl_sched_class, &rt_sched_class));
+ BUG_ON(!sched_class_above(&rt_sched_class, &fair_sched_class));
+ BUG_ON(!sched_class_above(&fair_sched_class, &idle_sched_class));
wait_bit_init();
--
2.45.2
Implementation Analysis
Overview
This patch replaces fragile pointer-arithmetic sanity checks in sched_init() with semantic checks using sched_class_above(). While it appears small, it is the foundational prerequisite that makes inserting ext_sched_class between fair_sched_class and idle_sched_class possible: the old checks hardcoded strict memory adjacency between every existing class pair, so adding any new class in the middle would have caused a boot-time BUG().
Background: The Linux Scheduler Class Hierarchy
The Linux kernel uses a chain of struct sched_class objects to implement scheduler policy dispatch. Each class handles a specific scheduling policy (e.g., real-time, CFS, deadline). The chain is ordered by priority — highest-priority classes are checked first when the kernel selects the next task to run.
The ordering at the time of this patch (highest to lowest):
stop_sched_class (SMP only — stop-machine tasks)
|
dl_sched_class (SCHED_DEADLINE)
|
rt_sched_class (SCHED_FIFO, SCHED_RR)
|
fair_sched_class (SCHED_NORMAL, SCHED_BATCH — CFS)
|
idle_sched_class (SCHED_IDLE — per-CPU idle thread)
The sched_ext series inserts ext_sched_class between fair_sched_class and idle_sched_class, making the final chain:
stop → dl → rt → fair → ext → idle
The kernel walks this chain using sched_class_above(a, b), which returns true if class a has higher scheduling priority than class b. Internally, the chain is implemented by placing the sched_class structs in a specific linker section so that pointer arithmetic (adding 1 to a class pointer) walks to the next lower-priority class. sched_class_above() exploits this layout — but that is an implementation detail that sched_init() must not rely on directly when validating the chain.
The Problem Being Solved
Before this patch, sched_init() validated the class ordering by checking pointer adjacency:
BUG_ON(&idle_sched_class != &fair_sched_class + 1 ||
&fair_sched_class != &rt_sched_class + 1 ||
&rt_sched_class != &dl_sched_class + 1);
#ifdef CONFIG_SMP
BUG_ON(&dl_sched_class != &stop_sched_class + 1);
#endif
This checks that each pair of adjacent classes is separated by exactly one struct-sized step in memory. It is an implicit assertion that no other class exists between them. Inserting ext_sched_class between fair_sched_class and idle_sched_class would cause &idle_sched_class != &fair_sched_class + 1 to be true, triggering BUG() at boot. The check must be removed or changed before sched_ext can be added.
Additionally, the check is semantically wrong: what actually matters for correctness is not whether two structs are adjacent in memory, but whether one class has higher priority than the other. The old code tests a storage layout, not the scheduling semantics.
Code Walkthrough
The old block is removed entirely:
- BUG_ON(&idle_sched_class != &fair_sched_class + 1 ||
- &fair_sched_class != &rt_sched_class + 1 ||
- &rt_sched_class != &dl_sched_class + 1);
-#ifdef CONFIG_SMP
- BUG_ON(&dl_sched_class != &stop_sched_class + 1);
-#endif
It is replaced with individual pairwise checks using sched_class_above():
#ifdef CONFIG_SMP
BUG_ON(!sched_class_above(&stop_sched_class, &dl_sched_class));
#endif
BUG_ON(!sched_class_above(&dl_sched_class, &rt_sched_class));
BUG_ON(!sched_class_above(&rt_sched_class, &fair_sched_class));
BUG_ON(!sched_class_above(&fair_sched_class, &idle_sched_class));
Two structural differences are worth noting:
-
The
#ifdef CONFIG_SMPnow guards only thestop/dlcheck (stop is an SMP-only class), whereas the old code had the#ifdefwrapping a separateBUG_ONfor the same reason — this is functionally equivalent. -
The new checks assert non-adjacency-dependent priority ordering. They will remain true even after
ext_sched_classis inserted betweenfair_sched_classandidle_sched_class, becausesched_class_above(&fair_sched_class, &idle_sched_class)is still true with ext in between —fairis still aboveidlein the ordering.
No check is added for the fair/ext or ext/idle boundary here; that is left to a later patch when ext_sched_class is actually defined.
Why sched_ext Needs This
Without this patch, the very first thing sched_init() does after wait_bit_init() would be to BUG() as soon as ext_sched_class is inserted. This patch is the minimal, non-functional blocker that must land before any patch that touches the linker-section placement of scheduler classes can be merged.
The change also establishes a clean contract: the scheduler class ordering is validated by its logical meaning (sched_class_above), not by a storage artifact. New optional classes can be inserted anywhere in the chain without touching this validation code.
Connection to Other Patches
This is patch 01 in the series and has no dependencies on subsequent patches. All later patches in this series that add or rearrange scheduler classes depend on this change being in place. Without it, any patch that changes the physical layout of the sched_class linker section would produce a boot-time kernel panic.
Key Data Structures / Functions Modified
sched_init()(kernel/sched/core.c): The kernel's scheduler initialization function called once at boot. The sanity-check block near its top is the only thing changed.sched_class_above(a, b): An existing inline helper defined inkernel/sched/sched.hthat returns true if scheduler classahas higher priority than classb. It works by comparing the addresses of the two structs in the linker-ordered section — higher address means lower priority in the layout used by this kernel. This patch starts using it for validation rather than raw pointer arithmetic.
[PATCH 02/30] sched: Allow sched_cgroup_fork() to fail and introduce sched_cancel_fork()
View on Lore: https://lore.kernel.org/all/20240618212056.2833381-3-tj@kernel.org
Commit Message
A new BPF extensible sched_class will need more control over the forking
process. It wants to be able to fail from sched_cgroup_fork() after the new
task's sched_task_group is initialized so that the loaded BPF program can
prepare the task with its cgroup association is established and reject fork
if e.g. allocation fails.
Allow sched_cgroup_fork() to fail by making it return int instead of void
and adding sched_cancel_fork() to undo sched_fork() in the error path.
sched_cgroup_fork() doesn't fail yet and this patch shouldn't cause any
behavior changes.
v2: Patch description updated to detail the expected use.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: David Vernet <dvernet@meta.com>
Acked-by: Josh Don <joshdon@google.com>
Acked-by: Hao Luo <haoluo@google.com>
Acked-by: Barret Rhoden <brho@google.com>
---
include/linux/sched/task.h | 3 ++-
kernel/fork.c | 15 ++++++++++-----
kernel/sched/core.c | 8 +++++++-
3 files changed, 19 insertions(+), 7 deletions(-)
diff --git a/include/linux/sched/task.h b/include/linux/sched/task.h
index d362aacf9f89..4df2f9055587 100644
--- a/include/linux/sched/task.h
+++ b/include/linux/sched/task.h
@@ -63,7 +63,8 @@ extern asmlinkage void schedule_tail(struct task_struct *prev);
extern void init_idle(struct task_struct *idle, int cpu);
extern int sched_fork(unsigned long clone_flags, struct task_struct *p);
-extern void sched_cgroup_fork(struct task_struct *p, struct kernel_clone_args *kargs);
+extern int sched_cgroup_fork(struct task_struct *p, struct kernel_clone_args *kargs);
+extern void sched_cancel_fork(struct task_struct *p);
extern void sched_post_fork(struct task_struct *p);
extern void sched_dead(struct task_struct *p);
diff --git a/kernel/fork.c b/kernel/fork.c
index 99076dbe27d8..e601fdf787c3 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -2363,7 +2363,7 @@ __latent_entropy struct task_struct *copy_process(
retval = perf_event_init_task(p, clone_flags);
if (retval)
- goto bad_fork_cleanup_policy;
+ goto bad_fork_sched_cancel_fork;
retval = audit_alloc(p);
if (retval)
goto bad_fork_cleanup_perf;
@@ -2496,7 +2496,9 @@ __latent_entropy struct task_struct *copy_process(
* cgroup specific, it unconditionally needs to place the task on a
* runqueue.
*/
- sched_cgroup_fork(p, args);
+ retval = sched_cgroup_fork(p, args);
+ if (retval)
+ goto bad_fork_cancel_cgroup;
/*
* From this point on we must avoid any synchronous user-space
@@ -2542,13 +2544,13 @@ __latent_entropy struct task_struct *copy_process(
/* Don't start children in a dying pid namespace */
if (unlikely(!(ns_of_pid(pid)->pid_allocated & PIDNS_ADDING))) {
retval = -ENOMEM;
- goto bad_fork_cancel_cgroup;
+ goto bad_fork_core_free;
}
/* Let kill terminate clone/fork in the middle */
if (fatal_signal_pending(current)) {
retval = -EINTR;
- goto bad_fork_cancel_cgroup;
+ goto bad_fork_core_free;
}
/* No more failure paths after this point. */
@@ -2622,10 +2624,11 @@ __latent_entropy struct task_struct *copy_process(
return p;
-bad_fork_cancel_cgroup:
+bad_fork_core_free:
sched_core_free(p);
spin_unlock(¤t->sighand->siglock);
write_unlock_irq(&tasklist_lock);
+bad_fork_cancel_cgroup:
cgroup_cancel_fork(p, args);
bad_fork_put_pidfd:
if (clone_flags & CLONE_PIDFD) {
@@ -2664,6 +2667,8 @@ __latent_entropy struct task_struct *copy_process(
audit_free(p);
bad_fork_cleanup_perf:
perf_event_free_task(p);
+bad_fork_sched_cancel_fork:
+ sched_cancel_fork(p);
bad_fork_cleanup_policy:
lockdep_free_task(p);
#ifdef CONFIG_NUMA
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index b4d4551bc7f2..095604490c26 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -4609,7 +4609,7 @@ int sched_fork(unsigned long clone_flags, struct task_struct *p)
return 0;
}
-void sched_cgroup_fork(struct task_struct *p, struct kernel_clone_args *kargs)
+int sched_cgroup_fork(struct task_struct *p, struct kernel_clone_args *kargs)
{
unsigned long flags;
@@ -4636,6 +4636,12 @@ void sched_cgroup_fork(struct task_struct *p, struct kernel_clone_args *kargs)
if (p->sched_class->task_fork)
p->sched_class->task_fork(p);
raw_spin_unlock_irqrestore(&p->pi_lock, flags);
+
+ return 0;
+}
+
+void sched_cancel_fork(struct task_struct *p)
+{
}
void sched_post_fork(struct task_struct *p)
--
2.45.2
Diff
---
include/linux/sched/task.h | 3 ++-
kernel/fork.c | 15 ++++++++++-----
kernel/sched/core.c | 8 +++++++-
3 files changed, 19 insertions(+), 7 deletions(-)
diff --git a/include/linux/sched/task.h b/include/linux/sched/task.h
index d362aacf9f89..4df2f9055587 100644
--- a/include/linux/sched/task.h
+++ b/include/linux/sched/task.h
@@ -63,7 +63,8 @@ extern asmlinkage void schedule_tail(struct task_struct *prev);
extern void init_idle(struct task_struct *idle, int cpu);
extern int sched_fork(unsigned long clone_flags, struct task_struct *p);
-extern void sched_cgroup_fork(struct task_struct *p, struct kernel_clone_args *kargs);
+extern int sched_cgroup_fork(struct task_struct *p, struct kernel_clone_args *kargs);
+extern void sched_cancel_fork(struct task_struct *p);
extern void sched_post_fork(struct task_struct *p);
extern void sched_dead(struct task_struct *p);
diff --git a/kernel/fork.c b/kernel/fork.c
index 99076dbe27d8..e601fdf787c3 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -2363,7 +2363,7 @@ __latent_entropy struct task_struct *copy_process(
retval = perf_event_init_task(p, clone_flags);
if (retval)
- goto bad_fork_cleanup_policy;
+ goto bad_fork_sched_cancel_fork;
retval = audit_alloc(p);
if (retval)
goto bad_fork_cleanup_perf;
@@ -2496,7 +2496,9 @@ __latent_entropy struct task_struct *copy_process(
* cgroup specific, it unconditionally needs to place the task on a
* runqueue.
*/
- sched_cgroup_fork(p, args);
+ retval = sched_cgroup_fork(p, args);
+ if (retval)
+ goto bad_fork_cancel_cgroup;
/*
* From this point on we must avoid any synchronous user-space
@@ -2542,13 +2544,13 @@ __latent_entropy struct task_struct *copy_process(
/* Don't start children in a dying pid namespace */
if (unlikely(!(ns_of_pid(pid)->pid_allocated & PIDNS_ADDING))) {
retval = -ENOMEM;
- goto bad_fork_cancel_cgroup;
+ goto bad_fork_core_free;
}
/* Let kill terminate clone/fork in the middle */
if (fatal_signal_pending(current)) {
retval = -EINTR;
- goto bad_fork_cancel_cgroup;
+ goto bad_fork_core_free;
}
/* No more failure paths after this point. */
@@ -2622,10 +2624,11 @@ __latent_entropy struct task_struct *copy_process(
return p;
-bad_fork_cancel_cgroup:
+bad_fork_core_free:
sched_core_free(p);
spin_unlock(¤t->sighand->siglock);
write_unlock_irq(&tasklist_lock);
+bad_fork_cancel_cgroup:
cgroup_cancel_fork(p, args);
bad_fork_put_pidfd:
if (clone_flags & CLONE_PIDFD) {
@@ -2664,6 +2667,8 @@ __latent_entropy struct task_struct *copy_process(
audit_free(p);
bad_fork_cleanup_perf:
perf_event_free_task(p);
+bad_fork_sched_cancel_fork:
+ sched_cancel_fork(p);
bad_fork_cleanup_policy:
lockdep_free_task(p);
#ifdef CONFIG_NUMA
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index b4d4551bc7f2..095604490c26 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -4609,7 +4609,7 @@ int sched_fork(unsigned long clone_flags, struct task_struct *p)
return 0;
}
-void sched_cgroup_fork(struct task_struct *p, struct kernel_clone_args *kargs)
+int sched_cgroup_fork(struct task_struct *p, struct kernel_clone_args *kargs)
{
unsigned long flags;
@@ -4636,6 +4636,12 @@ void sched_cgroup_fork(struct task_struct *p, struct kernel_clone_args *kargs)
if (p->sched_class->task_fork)
p->sched_class->task_fork(p);
raw_spin_unlock_irqrestore(&p->pi_lock, flags);
+
+ return 0;
+}
+
+void sched_cancel_fork(struct task_struct *p)
+{
}
void sched_post_fork(struct task_struct *p)
--
2.45.2
Implementation Analysis
Overview
This patch makes sched_cgroup_fork() return an error code instead of void, and introduces a new sched_cancel_fork() cleanup function. Neither function does anything new yet — sched_cgroup_fork() always returns 0 and sched_cancel_fork() is empty — but the plumbing through copy_process() is wired up correctly so that a later patch can fill them in. The immediate driver is sched_ext's need to allocate per-task BPF state during fork at the point where the task's cgroup association is already known, with a clean rollback path on failure.
Background: The Linux Scheduler Class Hierarchy
The fork path in the Linux kernel (copy_process() in kernel/fork.c) calls several scheduler hooks in sequence:
sched_fork()— called early; initializes the new task's scheduling state, assigns it to the parent's scheduler class.sched_cgroup_fork()— called later, after cgroup subsystem state is attached; places the task on a runqueue for the first time.sched_post_fork()— called after the task is fully set up.
The key distinction between sched_fork() and sched_cgroup_fork() is timing: sched_cgroup_fork() is called after the task's sched_task_group is initialized, meaning the task's cgroup association is finalized. For sched_ext, this is the earliest safe point at which BPF programs can be invoked to prepare per-task state — and the only point where an allocation failure can be reported back to copy_process() as a fork failure.
The Problem Being Solved
Before this patch, sched_cgroup_fork() returned void. There was no way for the scheduler to signal that fork preparation failed. If sched_ext needed to allocate per-task BPF storage (e.g., task-local data in a BPF map) during the fork path, it had no mechanism to return -ENOMEM to userspace and abort the fork. Equally, if sched_fork() had allocated resources and then a later step failed, there was no dedicated sched_cancel_fork() hook to release those resources.
The existing error label bad_fork_cleanup_policy was used as the target for perf event failure, meaning sched_fork state was left in place during that cleanup — which would be wrong once sched_fork can acquire resources that need releasing.
Code Walkthrough
include/linux/sched/task.h — the public API changes:
-extern void sched_cgroup_fork(struct task_struct *p, struct kernel_clone_args *kargs);
+extern int sched_cgroup_fork(struct task_struct *p, struct kernel_clone_args *kargs);
+extern void sched_cancel_fork(struct task_struct *p);
sched_cgroup_fork() gains an int return type. sched_cancel_fork() is declared as the undo function for sched_fork().
kernel/sched/core.c — the implementations:
-void sched_cgroup_fork(struct task_struct *p, struct kernel_clone_args *kargs)
+int sched_cgroup_fork(struct task_struct *p, struct kernel_clone_args *kargs)
{
...
+ return 0;
}
+void sched_cancel_fork(struct task_struct *p)
+{
+}
Both functions are stubs for now. The return 0 in sched_cgroup_fork() means no behavior change.
kernel/fork.c — copy_process() error handling is restructured in three places:
First, the sched_cgroup_fork() call site gains error checking:
- sched_cgroup_fork(p, args);
+ retval = sched_cgroup_fork(p, args);
+ if (retval)
+ goto bad_fork_cancel_cgroup;
Second, the cleanup labels are reordered to correctly sequence the teardown. The old bad_fork_cancel_cgroup label was reached by both the dying-pidns check and the fatal-signal check. Those two cases must now jump to bad_fork_core_free (which runs sched_core_free and releases the tasklist lock) before falling through to bad_fork_cancel_cgroup (cgroup cleanup). The new label ordering:
-bad_fork_cancel_cgroup:
+bad_fork_core_free:
sched_core_free(p);
spin_unlock(¤t->sighand->siglock);
write_unlock_irq(&tasklist_lock);
+bad_fork_cancel_cgroup:
cgroup_cancel_fork(p, args);
Third, a new label is inserted before the perf cleanup to call sched_cancel_fork():
bad_fork_cleanup_perf:
perf_event_free_task(p);
+bad_fork_sched_cancel_fork:
+ sched_cancel_fork(p);
bad_fork_cleanup_policy:
The perf event failure path, which previously jumped to bad_fork_cleanup_policy (bypassing any sched_fork cleanup), now jumps to bad_fork_sched_cancel_fork:
retval = perf_event_init_task(p, clone_flags);
if (retval)
- goto bad_fork_cleanup_policy;
+ goto bad_fork_sched_cancel_fork;
This ensures that whenever perf init fails after sched_fork() has run, sched_cancel_fork() is called — a correct invariant whether or not sched_cancel_fork() currently does anything.
Why sched_ext Needs This
sched_ext tracks every task with per-task BPF storage. That storage must be allocated before the task is visible to the scheduler, but after the task's cgroup is known (so the BPF program can make cgroup-aware decisions during task initialization). sched_cgroup_fork() is exactly that window.
If the allocation fails (e.g., out of memory, or the BPF program explicitly rejects the fork), sched_ext must be able to return an error. The int return type and the goto bad_fork_cancel_cgroup wiring provide this. sched_cancel_fork() will later be implemented to free any resources sched_fork() allocated, completing the symmetry.
Connection to Other Patches
This patch depends on nothing earlier in the series. Later sched_ext patches will fill in the bodies of both sched_cgroup_fork() and sched_cancel_fork() with real allocation and teardown logic. Without this infrastructure, those patches would have nowhere to return errors from and no cleanup hook to call.
Key Data Structures / Functions Modified
sched_cgroup_fork()(kernel/sched/core.c, declared ininclude/linux/sched/task.h): Scheduler hook called duringcopy_process()after cgroup state is attached. Changed fromvoidtoint.sched_cancel_fork()(kernel/sched/core.c, declared ininclude/linux/sched/task.h): New function; the undo counterpart tosched_fork(). Currently empty.copy_process()(kernel/fork.c): The main process/thread creation function. Its error-label chain is restructured to correctly invokesched_cancel_fork()on any failure that occurs aftersched_fork().bad_fork_sched_cancel_fork/bad_fork_core_free/bad_fork_cancel_cgroup: Goto labels incopy_process()that form the cleanup ladder. Their order determines which cleanup functions run on each failure path.
[PATCH 03/30] sched: Add sched_class->reweight_task()
View on Lore: https://lore.kernel.org/all/20240618212056.2833381-4-tj@kernel.org
Commit Message
Currently, during a task weight change, sched core directly calls
reweight_task() defined in fair.c if @p is on CFS. Let's make it a proper
sched_class operation instead. CFS's reweight_task() is renamed to
reweight_task_fair() and now called through sched_class.
While it turns a direct call into an indirect one, set_load_weight() isn't
called from a hot path and this change shouldn't cause any noticeable
difference. This will be used to implement reweight_task for a new BPF
extensible sched_class so that it can keep its cached task weight
up-to-date.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: David Vernet <dvernet@meta.com>
Acked-by: Josh Don <joshdon@google.com>
Acked-by: Hao Luo <haoluo@google.com>
Acked-by: Barret Rhoden <brho@google.com>
---
kernel/sched/core.c | 4 ++--
kernel/sched/fair.c | 3 ++-
kernel/sched/sched.h | 4 ++--
3 files changed, 6 insertions(+), 5 deletions(-)
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 095604490c26..48f9d00d0666 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -1343,8 +1343,8 @@ void set_load_weight(struct task_struct *p, bool update_load)
* SCHED_OTHER tasks have to update their load when changing their
* weight
*/
- if (update_load && p->sched_class == &fair_sched_class) {
- reweight_task(p, prio);
+ if (update_load && p->sched_class->reweight_task) {
+ p->sched_class->reweight_task(task_rq(p), p, prio);
} else {
load->weight = scale_load(sched_prio_to_weight[prio]);
load->inv_weight = sched_prio_to_wmult[prio];
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 41b58387023d..18ecd4f908e4 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -3835,7 +3835,7 @@ static void reweight_entity(struct cfs_rq *cfs_rq, struct sched_entity *se,
}
}
-void reweight_task(struct task_struct *p, int prio)
+static void reweight_task_fair(struct rq *rq, struct task_struct *p, int prio)
{
struct sched_entity *se = &p->se;
struct cfs_rq *cfs_rq = cfs_rq_of(se);
@@ -13221,6 +13221,7 @@ DEFINE_SCHED_CLASS(fair) = {
.task_tick = task_tick_fair,
.task_fork = task_fork_fair,
+ .reweight_task = reweight_task_fair,
.prio_changed = prio_changed_fair,
.switched_from = switched_from_fair,
.switched_to = switched_to_fair,
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 62fd8bc6fd08..a2399ccf259a 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -2324,6 +2324,8 @@ struct sched_class {
*/
void (*switched_from)(struct rq *this_rq, struct task_struct *task);
void (*switched_to) (struct rq *this_rq, struct task_struct *task);
+ void (*reweight_task)(struct rq *this_rq, struct task_struct *task,
+ int newprio);
void (*prio_changed) (struct rq *this_rq, struct task_struct *task,
int oldprio);
@@ -2509,8 +2511,6 @@ extern void init_sched_dl_class(void);
extern void init_sched_rt_class(void);
extern void init_sched_fair_class(void);
-extern void reweight_task(struct task_struct *p, int prio);
-
extern void resched_curr(struct rq *rq);
extern void resched_cpu(int cpu);
--
2.45.2
Diff
---
kernel/sched/core.c | 4 ++--
kernel/sched/fair.c | 3 ++-
kernel/sched/sched.h | 4 ++--
3 files changed, 6 insertions(+), 5 deletions(-)
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 095604490c26..48f9d00d0666 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -1343,8 +1343,8 @@ void set_load_weight(struct task_struct *p, bool update_load)
* SCHED_OTHER tasks have to update their load when changing their
* weight
*/
- if (update_load && p->sched_class == &fair_sched_class) {
- reweight_task(p, prio);
+ if (update_load && p->sched_class->reweight_task) {
+ p->sched_class->reweight_task(task_rq(p), p, prio);
} else {
load->weight = scale_load(sched_prio_to_weight[prio]);
load->inv_weight = sched_prio_to_wmult[prio];
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 41b58387023d..18ecd4f908e4 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -3835,7 +3835,7 @@ static void reweight_entity(struct cfs_rq *cfs_rq, struct sched_entity *se,
}
}
-void reweight_task(struct task_struct *p, int prio)
+static void reweight_task_fair(struct rq *rq, struct task_struct *p, int prio)
{
struct sched_entity *se = &p->se;
struct cfs_rq *cfs_rq = cfs_rq_of(se);
@@ -13221,6 +13221,7 @@ DEFINE_SCHED_CLASS(fair) = {
.task_tick = task_tick_fair,
.task_fork = task_fork_fair,
+ .reweight_task = reweight_task_fair,
.prio_changed = prio_changed_fair,
.switched_from = switched_from_fair,
.switched_to = switched_to_fair,
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 62fd8bc6fd08..a2399ccf259a 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -2324,6 +2324,8 @@ struct sched_class {
*/
void (*switched_from)(struct rq *this_rq, struct task_struct *task);
void (*switched_to) (struct rq *this_rq, struct task_struct *task);
+ void (*reweight_task)(struct rq *this_rq, struct task_struct *task,
+ int newprio);
void (*prio_changed) (struct rq *this_rq, struct task_struct *task,
int oldprio);
@@ -2509,8 +2511,6 @@ extern void init_sched_dl_class(void);
extern void init_sched_rt_class(void);
extern void init_sched_fair_class(void);
-extern void reweight_task(struct task_struct *p, int prio);
-
extern void resched_curr(struct rq *rq);
extern void resched_cpu(int cpu);
--
2.45.2
Implementation Analysis
Overview
This patch promotes reweight_task() from a CFS-private function called directly by the scheduler core into a proper sched_class vtable operation. The core's set_load_weight() previously hard-coded a check for fair_sched_class before calling the function; now it tests whether the class implements the operation at all and dispatches through the vtable. This is required so that sched_ext can receive notification when a task's nice value (and thus its scheduling weight) changes while the task is running under the ext class.
Background: The Linux Scheduler Class Hierarchy
The struct sched_class in kernel/sched/sched.h is a vtable — a set of function pointers that the scheduler core calls to implement scheduling operations. Each scheduling class (stop, dl, rt, fair, idle — and in this series, ext) fills in the operations it supports. Operations that a class does not implement are left as NULL, and callers check for NULL before invoking them.
When the user changes a task's nice value (via nice() or setpriority()), the kernel calls set_load_weight() to update the task's load_weight fields. For a CFS task, the weight also needs to be reflected in the task's position in the red-black tree via reweight_entity(). Before this patch, set_load_weight() knew it had to call into CFS by checking p->sched_class == &fair_sched_class — a hardcoded class comparison that cannot work for a new class like ext.
The Problem Being Solved
set_load_weight() in kernel/sched/core.c contained this pattern:
if (update_load && p->sched_class == &fair_sched_class) {
reweight_task(p, prio);
}
This is a direct, class-specific call bypassing the vtable. It breaks in two ways for sched_ext:
- When a task runs under
ext_sched_classand the user changes its nice value,p->sched_class == &fair_sched_classis false, soreweight_task()is never called. The ext scheduler's cached copy of the task weight goes stale. - Even if sched_ext wanted to hook into this path, there was no vtable slot for it to fill in.
Code Walkthrough
kernel/sched/sched.h — new vtable slot added to struct sched_class:
+ void (*reweight_task)(struct rq *this_rq, struct task_struct *task,
+ int newprio);
It is placed after switched_to and before prio_changed, which are the other class-switch and priority-change callbacks. The signature gains a struct rq * parameter compared to the old standalone function — consistent with the convention used by the rest of the vtable.
The old global declaration is removed:
-extern void reweight_task(struct task_struct *p, int prio);
kernel/sched/core.c — the dispatch in set_load_weight() is made class-agnostic:
- if (update_load && p->sched_class == &fair_sched_class) {
- reweight_task(p, prio);
+ if (update_load && p->sched_class->reweight_task) {
+ p->sched_class->reweight_task(task_rq(p), p, prio);
} else {
load->weight = scale_load(sched_prio_to_weight[prio]);
load->inv_weight = sched_prio_to_wmult[prio];
The condition no longer names a specific class. It checks whether the task's current class implements reweight_task at all. If it does, it calls through the vtable. The else branch (which updates load->weight and load->inv_weight directly) now runs for any class that does not implement the operation — this covers rt, dl, stop, idle, and initially ext.
kernel/sched/fair.c — CFS's implementation is renamed and wired into the vtable:
-void reweight_task(struct task_struct *p, int prio)
+static void reweight_task_fair(struct rq *rq, struct task_struct *p, int prio)
The function becomes static (no longer needs external linkage since it is accessed only through the vtable) and gains the rq parameter to match the new signature. It is registered in DEFINE_SCHED_CLASS(fair):
+ .reweight_task = reweight_task_fair,
Why sched_ext Needs This
When a task's nice value changes while it is under sched_ext, the ext class needs to update its own per-task weight cache so that the BPF program can make correct scheduling decisions (e.g., proportional-share accounting). With this vtable slot, the ext class can implement reweight_task and receive the notification. Without it, the ext class would permanently see the weight the task had at the time it joined the ext class, even after the user changed the task's nice value.
The commit message explicitly notes that set_load_weight() is not a hot path, so the additional indirection through the vtable has no measurable overhead.
Connection to Other Patches
This patch is self-contained from the perspective of what it removes. It depends on patch 01 (which restructures the class hierarchy checks) having established that the series is moving toward proper vtable dispatch. The ext class patch later in the series will implement .reweight_task using the slot added here. Without this patch, the ext class would receive no notification when a task's weight changes.
Key Data Structures / Functions Modified
struct sched_class(kernel/sched/sched.h): The scheduler vtable. Gains a newreweight_taskfunction pointer.set_load_weight()(kernel/sched/core.c): Called whenever a task's scheduling priority changes (nice value, policy change). The class-specific dispatch for weight updates is made generic here.reweight_task_fair()(kernel/sched/fair.c, formerlyreweight_task()): CFS's implementation ofreweight_task. Updates a task'ssched_entityweight in the CFS red-black tree viareweight_entity().DEFINE_SCHED_CLASS(fair)(kernel/sched/fair.c): The CFS vtable definition. Gains the.reweight_task = reweight_task_fairentry.
[PATCH 04/30] sched: Add sched_class->switching_to() and expose check_class_changing/changed()
View on Lore: https://lore.kernel.org/all/20240618212056.2833381-5-tj@kernel.org
Commit Message
When a task switches to a new sched_class, the prev and new classes are
notified through ->switched_from() and ->switched_to(), respectively, after
the switching is done.
A new BPF extensible sched_class will have callbacks that allow the BPF
scheduler to keep track of relevant task states (like priority and cpumask).
Those callbacks aren't called while a task is on a different sched_class.
When a task comes back, we wanna tell the BPF progs the up-to-date state
before the task gets enqueued, so we need a hook which is called before the
switching is committed.
This patch adds ->switching_to() which is called during sched_class switch
through check_class_changing() before the task is restored. Also, this patch
exposes check_class_changing/changed() in kernel/sched/sched.h. They will be
used by the new BPF extensible sched_class to implement implicit sched_class
switching which is used e.g. when falling back to CFS when the BPF scheduler
fails or unloads.
This is a prep patch and doesn't cause any behavior changes. The new
operation and exposed functions aren't used yet.
v3: Refreshed on top of tip:sched/core.
v2: Improve patch description w/ details on planned use.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: David Vernet <dvernet@meta.com>
Acked-by: Josh Don <joshdon@google.com>
Acked-by: Hao Luo <haoluo@google.com>
Acked-by: Barret Rhoden <brho@google.com>
---
kernel/sched/core.c | 12 ++++++++++++
kernel/sched/sched.h | 3 +++
kernel/sched/syscalls.c | 1 +
3 files changed, 16 insertions(+)
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 48f9d00d0666..b088fbeaf26d 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -2035,6 +2035,17 @@ inline int task_curr(const struct task_struct *p)
return cpu_curr(task_cpu(p)) == p;
}
+/*
+ * ->switching_to() is called with the pi_lock and rq_lock held and must not
+ * mess with locking.
+ */
+void check_class_changing(struct rq *rq, struct task_struct *p,
+ const struct sched_class *prev_class)
+{
+ if (prev_class != p->sched_class && p->sched_class->switching_to)
+ p->sched_class->switching_to(rq, p);
+}
+
/*
* switched_from, switched_to and prio_changed must _NOT_ drop rq->lock,
* use the balance_callback list if you want balancing.
@@ -7021,6 +7032,7 @@ void rt_mutex_setprio(struct task_struct *p, struct task_struct *pi_task)
}
__setscheduler_prio(p, prio);
+ check_class_changing(rq, p, prev_class);
if (queued)
enqueue_task(rq, p, queue_flag);
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index a2399ccf259a..0ed4271cedf5 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -2322,6 +2322,7 @@ struct sched_class {
* cannot assume the switched_from/switched_to pair is serialized by
* rq->lock. They are however serialized by p->pi_lock.
*/
+ void (*switching_to) (struct rq *this_rq, struct task_struct *task);
void (*switched_from)(struct rq *this_rq, struct task_struct *task);
void (*switched_to) (struct rq *this_rq, struct task_struct *task);
void (*reweight_task)(struct rq *this_rq, struct task_struct *task,
@@ -3608,6 +3609,8 @@ extern void set_load_weight(struct task_struct *p, bool update_load);
extern void enqueue_task(struct rq *rq, struct task_struct *p, int flags);
extern void dequeue_task(struct rq *rq, struct task_struct *p, int flags);
+extern void check_class_changing(struct rq *rq, struct task_struct *p,
+ const struct sched_class *prev_class);
extern void check_class_changed(struct rq *rq, struct task_struct *p,
const struct sched_class *prev_class,
int oldprio);
diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
index ae1b42775ef9..cf189bc3dd18 100644
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -797,6 +797,7 @@ int __sched_setscheduler(struct task_struct *p,
__setscheduler_prio(p, newprio);
}
__setscheduler_uclamp(p, attr);
+ check_class_changing(rq, p, prev_class);
if (queued) {
/*
--
2.45.2
Diff
---
kernel/sched/core.c | 12 ++++++++++++
kernel/sched/sched.h | 3 +++
kernel/sched/syscalls.c | 1 +
3 files changed, 16 insertions(+)
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 48f9d00d0666..b088fbeaf26d 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -2035,6 +2035,17 @@ inline int task_curr(const struct task_struct *p)
return cpu_curr(task_cpu(p)) == p;
}
+/*
+ * ->switching_to() is called with the pi_lock and rq_lock held and must not
+ * mess with locking.
+ */
+void check_class_changing(struct rq *rq, struct task_struct *p,
+ const struct sched_class *prev_class)
+{
+ if (prev_class != p->sched_class && p->sched_class->switching_to)
+ p->sched_class->switching_to(rq, p);
+}
+
/*
* switched_from, switched_to and prio_changed must _NOT_ drop rq->lock,
* use the balance_callback list if you want balancing.
@@ -7021,6 +7032,7 @@ void rt_mutex_setprio(struct task_struct *p, struct task_struct *pi_task)
}
__setscheduler_prio(p, prio);
+ check_class_changing(rq, p, prev_class);
if (queued)
enqueue_task(rq, p, queue_flag);
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index a2399ccf259a..0ed4271cedf5 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -2322,6 +2322,7 @@ struct sched_class {
* cannot assume the switched_from/switched_to pair is serialized by
* rq->lock. They are however serialized by p->pi_lock.
*/
+ void (*switching_to) (struct rq *this_rq, struct task_struct *task);
void (*switched_from)(struct rq *this_rq, struct task_struct *task);
void (*switched_to) (struct rq *this_rq, struct task_struct *task);
void (*reweight_task)(struct rq *this_rq, struct task_struct *task,
@@ -3608,6 +3609,8 @@ extern void set_load_weight(struct task_struct *p, bool update_load);
extern void enqueue_task(struct rq *rq, struct task_struct *p, int flags);
extern void dequeue_task(struct rq *rq, struct task_struct *p, int flags);
+extern void check_class_changing(struct rq *rq, struct task_struct *p,
+ const struct sched_class *prev_class);
extern void check_class_changed(struct rq *rq, struct task_struct *p,
const struct sched_class *prev_class,
int oldprio);
diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
index ae1b42775ef9..cf189bc3dd18 100644
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -797,6 +797,7 @@ int __sched_setscheduler(struct task_struct *p,
__setscheduler_prio(p, newprio);
}
__setscheduler_uclamp(p, attr);
+ check_class_changing(rq, p, prev_class);
if (queued) {
/*
--
2.45.2
Implementation Analysis
Overview
This patch adds a switching_to() callback to the sched_class vtable that fires just before a task's class transition is committed, and exposes check_class_changing() alongside the already-existing check_class_changed() so both can be called from sched_ext's own implicit-switch code paths. The critical distinction from the existing switched_to() callback is timing: switching_to() fires while both the old and new class identity are known but before the task is enqueued, giving the new class a window to synchronize its state before the task becomes runnable under it.
Background: The Linux Scheduler Class Hierarchy
When a task moves from one scheduler class to another — for example, from CFS to sched_ext, or from rt back to CFS when an RT mutex is released — the kernel calls a pair of callbacks:
switched_from(rq, p): called on the old class, after the switch is doneswitched_to(rq, p): called on the new class, after the switch is done
Both callbacks are called by check_class_changed(), which is invoked after p->sched_class has already been updated. At this point the task may already be enqueued.
sched_ext maintains callbacks that track per-task state changes: priority, cpumask, etc. While a task is running under a different class, those callbacks are not invoked — sched_ext deliberately does not track tasks that do not belong to it. When a task returns to sched_ext, the BPF program needs to receive a full state sync before it ever sees the task in an enqueue callback, otherwise its view of the task's priority and cpumask will be stale.
switched_to() is too late for this: at that point, the task may already be enqueued. What is needed is a hook that fires after p->sched_class is updated but before the task is placed on a runqueue — which is what switching_to() provides.
The Problem Being Solved
There was no hook in the class-switch path that ran before the task became runnable under the new class. The two existing hooks (switched_from, switched_to) both run after the switch is committed and the task may already be on a runqueue. For sched_ext, this creates a race: the BPF program could receive an enqueue event before it has had a chance to synchronize the task's priority, cpumask, or other state.
Additionally, sched_ext will implement implicit class switching: when the BPF scheduler fails or unloads, tasks that were running under ext_sched_class must be silently migrated back to CFS. This migration is triggered from within sched_ext itself, not from __sched_setscheduler(). The existing check_class_changing() and check_class_changed() functions existed but were static or module-internal — not exposed in sched.h — so sched_ext could not call them directly.
Code Walkthrough
kernel/sched/sched.h — new vtable slot:
+ void (*switching_to) (struct rq *this_rq, struct task_struct *task);
void (*switched_from)(struct rq *this_rq, struct task_struct *task);
void (*switched_to) (struct rq *this_rq, struct task_struct *task);
switching_to is inserted immediately before switched_from in the vtable, making the trio switching_to → switched_from → switched_to read in chronological order. Called with pi_lock and rq_lock held (as noted in the comment added to core.c), so implementations must not acquire locks.
kernel/sched/core.c — check_class_changing() is added:
+void check_class_changing(struct rq *rq, struct task_struct *p,
+ const struct sched_class *prev_class)
+{
+ if (prev_class != p->sched_class && p->sched_class->switching_to)
+ p->sched_class->switching_to(rq, p);
+}
The function checks two things: (1) the class actually changed (comparing prev_class to the now-updated p->sched_class), and (2) the new class implements switching_to. Note that at call time, p->sched_class has already been updated to the new class — the new class is calling switching_to on itself. prev_class is the caller's saved snapshot of the old class pointer.
check_class_changing() is then inserted at two sites where a class change can occur:
In rt_mutex_setprio() (priority inheritance — a task inherits a higher priority from a lock holder):
__setscheduler_prio(p, prio);
+ check_class_changing(rq, p, prev_class);
if (queued)
enqueue_task(rq, p, queue_flag);
In __sched_setscheduler() (explicit policy/priority change from syscall):
__setscheduler_uclamp(p, attr);
+ check_class_changing(rq, p, prev_class);
if (queued) {
In both cases, check_class_changing() is called after the class pointer is updated but before enqueue_task() — precisely the window sched_ext needs.
kernel/sched/sched.h — both functions are now declared as extern:
+extern void check_class_changing(struct rq *rq, struct task_struct *p,
+ const struct sched_class *prev_class);
extern void check_class_changed(struct rq *rq, struct task_struct *p,
const struct sched_class *prev_class,
int oldprio);
Exposing check_class_changing() here allows sched_ext (which lives in its own file) to call it during implicit class switches without duplicating the logic.
Why sched_ext Needs This
When a task re-enters sched_ext (e.g., after being temporarily boosted to a real-time priority via priority inheritance), the BPF program needs to know the task's current state before it can correctly schedule it. switching_to() is the hook where sched_ext will call back into the BPF program to push the current priority, cpumask, and any other cached state before the first enqueue. Without this hook, the BPF program would receive an enqueue event for a task whose state it has not seen updated since the task last left the ext class — potentially scheduling it with wrong weight or wrong CPU affinity.
The exposure of check_class_changing() in sched.h enables sched_ext to trigger the switching_to → switched_from + switched_to sequence correctly during its own implicit class switches (fallback to CFS on BPF program failure).
Connection to Other Patches
This patch depends on patch 03 having added the reweight_task slot, establishing the pattern of adding vtable operations for sched_ext's needs. The sched_ext class implementation later in the series implements switching_to to sync BPF task state. check_class_changing() will also be called from sched_ext's own class-switch code paths. Without this patch, sched_ext has no pre-enqueue synchronization point.
Key Data Structures / Functions Modified
struct sched_class(kernel/sched/sched.h): Gains theswitching_tofunction pointer, placed beforeswitched_fromin declaration order.check_class_changing()(kernel/sched/core.c, now exposed inkernel/sched/sched.h): Dispatchesswitching_towhen a class change is detected. Called afterp->sched_classis updated but beforeenqueue_task().check_class_changed()(kernel/sched/sched.h): The existing post-switch dispatcher forswitched_fromandswitched_to. Now declared alongsidecheck_class_changing()in the header to make the pair's roles clear.rt_mutex_setprio()(kernel/sched/core.c): Handles priority changes due to priority inheritance (PI mutexes). Now callscheck_class_changing()after updatingp->sched_class.__sched_setscheduler()(kernel/sched/syscalls.c): Handlessched_setscheduler()andsched_setattr()syscalls. Also now callscheck_class_changing()after updatingp->sched_class.
[PATCH 05/30] sched: Factor out cgroup weight conversion functions
View on Lore: https://lore.kernel.org/all/20240618212056.2833381-6-tj@kernel.org
Commit Message
Factor out sched_weight_from/to_cgroup() which convert between scheduler
shares and cgroup weight. No functional change. The factored out functions
will be used by a new BPF extensible sched_class so that the weights can be
exposed to the BPF programs in a way which is consistent cgroup weights and
easier to interpret.
The weight conversions will be used regardless of cgroup usage. It's just
borrowing the cgroup weight range as it's more intuitive.
CGROUP_WEIGHT_MIN/DFL/MAX constants are moved outside CONFIG_CGROUPS so that
the conversion helpers can always be defined.
v2: The helpers are now defined regardless of COFNIG_CGROUPS.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: David Vernet <dvernet@meta.com>
Acked-by: Josh Don <joshdon@google.com>
Acked-by: Hao Luo <haoluo@google.com>
Acked-by: Barret Rhoden <brho@google.com>
---
include/linux/cgroup.h | 4 ++--
kernel/sched/core.c | 28 +++++++++++++---------------
kernel/sched/sched.h | 18 ++++++++++++++++++
3 files changed, 33 insertions(+), 17 deletions(-)
diff --git a/include/linux/cgroup.h b/include/linux/cgroup.h
index 2150ca60394b..3cdaec701600 100644
--- a/include/linux/cgroup.h
+++ b/include/linux/cgroup.h
@@ -29,8 +29,6 @@
struct kernel_clone_args;
-#ifdef CONFIG_CGROUPS
-
/*
* All weight knobs on the default hierarchy should use the following min,
* default and max values. The default value is the logarithmic center of
@@ -40,6 +38,8 @@ struct kernel_clone_args;
#define CGROUP_WEIGHT_DFL 100
#define CGROUP_WEIGHT_MAX 10000
+#ifdef CONFIG_CGROUPS
+
enum {
CSS_TASK_ITER_PROCS = (1U << 0), /* walk only threadgroup leaders */
CSS_TASK_ITER_THREADED = (1U << 1), /* walk all threaded css_sets in the domain */
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index b088fbeaf26d..0bfbceebc4e9 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -9552,29 +9552,27 @@ static int cpu_local_stat_show(struct seq_file *sf,
}
#ifdef CONFIG_FAIR_GROUP_SCHED
+
+static unsigned long tg_weight(struct task_group *tg)
+{
+ return scale_load_down(tg->shares);
+}
+
static u64 cpu_weight_read_u64(struct cgroup_subsys_state *css,
struct cftype *cft)
{
- struct task_group *tg = css_tg(css);
- u64 weight = scale_load_down(tg->shares);
-
- return DIV_ROUND_CLOSEST_ULL(weight * CGROUP_WEIGHT_DFL, 1024);
+ return sched_weight_to_cgroup(tg_weight(css_tg(css)));
}
static int cpu_weight_write_u64(struct cgroup_subsys_state *css,
- struct cftype *cft, u64 weight)
+ struct cftype *cft, u64 cgrp_weight)
{
- /*
- * cgroup weight knobs should use the common MIN, DFL and MAX
- * values which are 1, 100 and 10000 respectively. While it loses
- * a bit of range on both ends, it maps pretty well onto the shares
- * value used by scheduler and the round-trip conversions preserve
- * the original value over the entire range.
- */
- if (weight < CGROUP_WEIGHT_MIN || weight > CGROUP_WEIGHT_MAX)
+ unsigned long weight;
+
+ if (cgrp_weight < CGROUP_WEIGHT_MIN || cgrp_weight > CGROUP_WEIGHT_MAX)
return -ERANGE;
- weight = DIV_ROUND_CLOSEST_ULL(weight * 1024, CGROUP_WEIGHT_DFL);
+ weight = sched_weight_from_cgroup(cgrp_weight);
return sched_group_set_shares(css_tg(css), scale_load(weight));
}
@@ -9582,7 +9580,7 @@ static int cpu_weight_write_u64(struct cgroup_subsys_state *css,
static s64 cpu_weight_nice_read_s64(struct cgroup_subsys_state *css,
struct cftype *cft)
{
- unsigned long weight = scale_load_down(css_tg(css)->shares);
+ unsigned long weight = tg_weight(css_tg(css));
int last_delta = INT_MAX;
int prio, delta;
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 0ed4271cedf5..656a63c0d393 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -244,6 +244,24 @@ static inline void update_avg(u64 *avg, u64 sample)
#define shr_bound(val, shift) \
(val >> min_t(typeof(shift), shift, BITS_PER_TYPE(typeof(val)) - 1))
+/*
+ * cgroup weight knobs should use the common MIN, DFL and MAX values which are
+ * 1, 100 and 10000 respectively. While it loses a bit of range on both ends, it
+ * maps pretty well onto the shares value used by scheduler and the round-trip
+ * conversions preserve the original value over the entire range.
+ */
+static inline unsigned long sched_weight_from_cgroup(unsigned long cgrp_weight)
+{
+ return DIV_ROUND_CLOSEST_ULL(cgrp_weight * 1024, CGROUP_WEIGHT_DFL);
+}
+
+static inline unsigned long sched_weight_to_cgroup(unsigned long weight)
+{
+ return clamp_t(unsigned long,
+ DIV_ROUND_CLOSEST_ULL(weight * CGROUP_WEIGHT_DFL, 1024),
+ CGROUP_WEIGHT_MIN, CGROUP_WEIGHT_MAX);
+}
+
/*
* !! For sched_setattr_nocheck() (kernel) only !!
*
--
2.45.2
Diff
---
include/linux/cgroup.h | 4 ++--
kernel/sched/core.c | 28 +++++++++++++---------------
kernel/sched/sched.h | 18 ++++++++++++++++++
3 files changed, 33 insertions(+), 17 deletions(-)
diff --git a/include/linux/cgroup.h b/include/linux/cgroup.h
index 2150ca60394b..3cdaec701600 100644
--- a/include/linux/cgroup.h
+++ b/include/linux/cgroup.h
@@ -29,8 +29,6 @@
struct kernel_clone_args;
-#ifdef CONFIG_CGROUPS
-
/*
* All weight knobs on the default hierarchy should use the following min,
* default and max values. The default value is the logarithmic center of
@@ -40,6 +38,8 @@ struct kernel_clone_args;
#define CGROUP_WEIGHT_DFL 100
#define CGROUP_WEIGHT_MAX 10000
+#ifdef CONFIG_CGROUPS
+
enum {
CSS_TASK_ITER_PROCS = (1U << 0), /* walk only threadgroup leaders */
CSS_TASK_ITER_THREADED = (1U << 1), /* walk all threaded css_sets in the domain */
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index b088fbeaf26d..0bfbceebc4e9 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -9552,29 +9552,27 @@ static int cpu_local_stat_show(struct seq_file *sf,
}
#ifdef CONFIG_FAIR_GROUP_SCHED
+
+static unsigned long tg_weight(struct task_group *tg)
+{
+ return scale_load_down(tg->shares);
+}
+
static u64 cpu_weight_read_u64(struct cgroup_subsys_state *css,
struct cftype *cft)
{
- struct task_group *tg = css_tg(css);
- u64 weight = scale_load_down(tg->shares);
-
- return DIV_ROUND_CLOSEST_ULL(weight * CGROUP_WEIGHT_DFL, 1024);
+ return sched_weight_to_cgroup(tg_weight(css_tg(css)));
}
static int cpu_weight_write_u64(struct cgroup_subsys_state *css,
- struct cftype *cft, u64 weight)
+ struct cftype *cft, u64 cgrp_weight)
{
- /*
- * cgroup weight knobs should use the common MIN, DFL and MAX
- * values which are 1, 100 and 10000 respectively. While it loses
- * a bit of range on both ends, it maps pretty well onto the shares
- * value used by scheduler and the round-trip conversions preserve
- * the original value over the entire range.
- */
- if (weight < CGROUP_WEIGHT_MIN || weight > CGROUP_WEIGHT_MAX)
+ unsigned long weight;
+
+ if (cgrp_weight < CGROUP_WEIGHT_MIN || cgrp_weight > CGROUP_WEIGHT_MAX)
return -ERANGE;
- weight = DIV_ROUND_CLOSEST_ULL(weight * 1024, CGROUP_WEIGHT_DFL);
+ weight = sched_weight_from_cgroup(cgrp_weight);
return sched_group_set_shares(css_tg(css), scale_load(weight));
}
@@ -9582,7 +9580,7 @@ static int cpu_weight_write_u64(struct cgroup_subsys_state *css,
static s64 cpu_weight_nice_read_s64(struct cgroup_subsys_state *css,
struct cftype *cft)
{
- unsigned long weight = scale_load_down(css_tg(css)->shares);
+ unsigned long weight = tg_weight(css_tg(css));
int last_delta = INT_MAX;
int prio, delta;
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 0ed4271cedf5..656a63c0d393 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -244,6 +244,24 @@ static inline void update_avg(u64 *avg, u64 sample)
#define shr_bound(val, shift) \
(val >> min_t(typeof(shift), shift, BITS_PER_TYPE(typeof(val)) - 1))
+/*
+ * cgroup weight knobs should use the common MIN, DFL and MAX values which are
+ * 1, 100 and 10000 respectively. While it loses a bit of range on both ends, it
+ * maps pretty well onto the shares value used by scheduler and the round-trip
+ * conversions preserve the original value over the entire range.
+ */
+static inline unsigned long sched_weight_from_cgroup(unsigned long cgrp_weight)
+{
+ return DIV_ROUND_CLOSEST_ULL(cgrp_weight * 1024, CGROUP_WEIGHT_DFL);
+}
+
+static inline unsigned long sched_weight_to_cgroup(unsigned long weight)
+{
+ return clamp_t(unsigned long,
+ DIV_ROUND_CLOSEST_ULL(weight * CGROUP_WEIGHT_DFL, 1024),
+ CGROUP_WEIGHT_MIN, CGROUP_WEIGHT_MAX);
+}
+
/*
* !! For sched_setattr_nocheck() (kernel) only !!
*
--
2.45.2
Implementation Analysis
Overview
This patch factors the arithmetic for converting between the kernel scheduler's internal load weight ("shares") and the cgroup weight scale into two shared inline helpers — sched_weight_from_cgroup() and sched_weight_to_cgroup() — and moves the CGROUP_WEIGHT_MIN/DFL/MAX constants outside CONFIG_CGROUPS so the helpers are available unconditionally. The CFS cgroup code is updated to use them. sched_ext will use these same helpers to present task weights to BPF programs in a human-readable scale (1–10000) rather than the raw internal shares value.
The Problem Being Solved
Before this patch, the conversion between cgroup weight and scheduler shares was inlined at each use site in cpu_weight_read_u64() and cpu_weight_write_u64():
// read path:
return DIV_ROUND_CLOSEST_ULL(weight * CGROUP_WEIGHT_DFL, 1024);
// write path:
weight = DIV_ROUND_CLOSEST_ULL(weight * 1024, CGROUP_WEIGHT_DFL);
This is fine for one consumer, but sched_ext needs to perform the same conversions when surfacing task weights through BPF maps. Duplicating the arithmetic formula in two separate places invites divergence if the conversion formula ever changes. Additionally, the constants CGROUP_WEIGHT_MIN/DFL/MAX were gated by #ifdef CONFIG_CGROUPS, making them unavailable in configurations that build sched_ext without cgroup support — even though sched_ext wants to use the cgroup weight range as a user-visible scale regardless of whether cgroups are enabled.
Code Walkthrough
include/linux/cgroup.h — constants are moved out of the CONFIG_CGROUPS guard:
-#ifdef CONFIG_CGROUPS
-
/*
* All weight knobs on the default hierarchy should use the following min,
* default and max values. ...
*/
#define CGROUP_WEIGHT_MIN 1
#define CGROUP_WEIGHT_DFL 100
#define CGROUP_WEIGHT_MAX 10000
+#ifdef CONFIG_CGROUPS
The three #defines are now unconditionally available. The #ifdef CONFIG_CGROUPS guard is moved down to cover only the cgroup-specific enum and struct definitions that follow. This allows sched.h to use CGROUP_WEIGHT_DFL in inline helpers without requiring CONFIG_CGROUPS.
kernel/sched/sched.h — two new inline helpers are added:
static inline unsigned long sched_weight_from_cgroup(unsigned long cgrp_weight)
{
return DIV_ROUND_CLOSEST_ULL(cgrp_weight * 1024, CGROUP_WEIGHT_DFL);
}
static inline unsigned long sched_weight_to_cgroup(unsigned long weight)
{
return clamp_t(unsigned long,
DIV_ROUND_CLOSEST_ULL(weight * CGROUP_WEIGHT_DFL, 1024),
CGROUP_WEIGHT_MIN, CGROUP_WEIGHT_MAX);
}
The conversion formula is straightforward: the kernel's internal weight scale for CFS uses 1024 as the unit for a "normal" (nice-0) task, while the cgroup weight scale uses 100 as the default. The conversion is a proportional rescaling. The to_cgroup direction also clamps the result to [CGROUP_WEIGHT_MIN, CGROUP_WEIGHT_MAX] using clamp_t — a small correctness improvement over the original inline code which did not clamp.
kernel/sched/core.c — the CFS cgroup read/write paths are updated to use the new helpers:
static u64 cpu_weight_read_u64(struct cgroup_subsys_state *css,
struct cftype *cft)
{
- struct task_group *tg = css_tg(css);
- u64 weight = scale_load_down(tg->shares);
- return DIV_ROUND_CLOSEST_ULL(weight * CGROUP_WEIGHT_DFL, 1024);
+ return sched_weight_to_cgroup(tg_weight(css_tg(css)));
}
static int cpu_weight_write_u64(struct cgroup_subsys_state *css,
- struct cftype *cft, u64 weight)
+ struct cftype *cft, u64 cgrp_weight)
{
- if (weight < CGROUP_WEIGHT_MIN || weight > CGROUP_WEIGHT_MAX)
+ if (cgrp_weight < CGROUP_WEIGHT_MIN || cgrp_weight > CGROUP_WEIGHT_MAX)
return -ERANGE;
- weight = DIV_ROUND_CLOSEST_ULL(weight * 1024, CGROUP_WEIGHT_DFL);
+ weight = sched_weight_from_cgroup(cgrp_weight);
return sched_group_set_shares(css_tg(css), scale_load(weight));
}
The parameter rename from weight to cgrp_weight in the write path disambiguates the two weight representations within the same function, which was previously a source of confusion. A small tg_weight() helper is also extracted to avoid repeating scale_load_down(tg->shares):
+static unsigned long tg_weight(struct task_group *tg)
+{
+ return scale_load_down(tg->shares);
+}
Why sched_ext Needs This
sched_ext exposes task weights to BPF programs through BPF map entries or task struct fields. The internal load_weight representation (based on 1024 as the unit for a nice-0 task) is opaque and not documented in any user-facing ABI. The cgroup weight scale (1–10000, default 100) is already user-visible and documented in cgroup v2 documentation, so using it as the BPF-facing representation keeps the interface consistent with cgroup tooling.
Without these shared helpers, the ext scheduler would have to either duplicate the arithmetic or expose the raw internal weight. By factoring the helpers into sched.h and making the constants unconditional, sched_ext can call sched_weight_to_cgroup() and sched_weight_from_cgroup() in its implementation regardless of whether CONFIG_CGROUPS is set.
Connection to Other Patches
This patch does not depend on any earlier patch in the series. It is a self-contained refactoring. The sched_ext implementation later in the series calls sched_weight_to_cgroup() when populating the per-task weight field that is visible to BPF programs.
Key Data Structures / Functions Modified
CGROUP_WEIGHT_MIN / DFL / MAX(include/linux/cgroup.h): Constants defining the cgroup weight scale (1, 100, 10000). Moved outsideCONFIG_CGROUPSto be universally available.sched_weight_from_cgroup()(kernel/sched/sched.h): New inline helper. Converts a cgroup-scale weight (1–10000) to the scheduler's internal shares representation.sched_weight_to_cgroup()(kernel/sched/sched.h): New inline helper. Converts the scheduler's internal shares representation to a cgroup-scale weight. Clamps the result to the valid range.tg_weight()(kernel/sched/core.c): New local helper that returns the scaled-down shares value from atask_group.cpu_weight_read_u64()/cpu_weight_write_u64()(kernel/sched/core.c): CFS cgroupcpu.weightread/write handlers. Updated to call the new shared helpers instead of open-coding the arithmetic.
[PATCH 06/30] sched: Factor out update_other_load_avgs() from __update_blocked_others()
View on Lore: https://lore.kernel.org/all/20240618212056.2833381-7-tj@kernel.org
Commit Message
RT, DL, thermal and irq load and utilization metrics need to be decayed and
updated periodically and before consumption to keep the numbers reasonable.
This is currently done from __update_blocked_others() as a part of the fair
class load balance path. Let's factor it out to update_other_load_avgs().
Pure refactor. No functional changes.
This will be used by the new BPF extensible scheduling class to ensure that
the above metrics are properly maintained.
v2: Refreshed on top of tip:sched/core.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: David Vernet <dvernet@meta.com>
---
kernel/sched/fair.c | 16 +++-------------
kernel/sched/sched.h | 4 ++++
kernel/sched/syscalls.c | 19 +++++++++++++++++++
3 files changed, 26 insertions(+), 13 deletions(-)
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 18ecd4f908e4..715d7c1f55df 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -9352,28 +9352,18 @@ static inline void update_blocked_load_status(struct rq *rq, bool has_blocked) {
static bool __update_blocked_others(struct rq *rq, bool *done)
{
- const struct sched_class *curr_class;
- u64 now = rq_clock_pelt(rq);
- unsigned long hw_pressure;
- bool decayed;
+ bool updated;
/*
* update_load_avg() can call cpufreq_update_util(). Make sure that RT,
* DL and IRQ signals have been updated before updating CFS.
*/
- curr_class = rq->curr->sched_class;
-
- hw_pressure = arch_scale_hw_pressure(cpu_of(rq));
-
- decayed = update_rt_rq_load_avg(now, rq, curr_class == &rt_sched_class) |
- update_dl_rq_load_avg(now, rq, curr_class == &dl_sched_class) |
- update_hw_load_avg(now, rq, hw_pressure) |
- update_irq_load_avg(rq, 0);
+ updated = update_other_load_avgs(rq);
if (others_have_blocked(rq))
*done = false;
- return decayed;
+ return updated;
}
#ifdef CONFIG_FAIR_GROUP_SCHED
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 656a63c0d393..a5a4f59151db 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -3074,6 +3074,8 @@ static inline void cpufreq_update_util(struct rq *rq, unsigned int flags) { }
#ifdef CONFIG_SMP
+bool update_other_load_avgs(struct rq *rq);
+
unsigned long effective_cpu_util(int cpu, unsigned long util_cfs,
unsigned long *min,
unsigned long *max);
@@ -3117,6 +3119,8 @@ static inline unsigned long cpu_util_rt(struct rq *rq)
return READ_ONCE(rq->avg_rt.util_avg);
}
+#else /* !CONFIG_SMP */
+static inline bool update_other_load_avgs(struct rq *rq) { return false; }
#endif /* CONFIG_SMP */
#ifdef CONFIG_UCLAMP_TASK
diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
index cf189bc3dd18..050215ef8fa4 100644
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -259,6 +259,25 @@ int sched_core_idle_cpu(int cpu)
#endif
#ifdef CONFIG_SMP
+/*
+ * Load avg and utiliztion metrics need to be updated periodically and before
+ * consumption. This function updates the metrics for all subsystems except for
+ * the fair class. @rq must be locked and have its clock updated.
+ */
+bool update_other_load_avgs(struct rq *rq)
+{
+ u64 now = rq_clock_pelt(rq);
+ const struct sched_class *curr_class = rq->curr->sched_class;
+ unsigned long hw_pressure = arch_scale_hw_pressure(cpu_of(rq));
+
+ lockdep_assert_rq_held(rq);
+
+ return update_rt_rq_load_avg(now, rq, curr_class == &rt_sched_class) |
+ update_dl_rq_load_avg(now, rq, curr_class == &dl_sched_class) |
+ update_hw_load_avg(now, rq, hw_pressure) |
+ update_irq_load_avg(rq, 0);
+}
+
/*
* This function computes an effective utilization for the given CPU, to be
* used for frequency selection given the linear relation: f = u * f_max.
--
2.45.2
Diff
---
kernel/sched/fair.c | 16 +++-------------
kernel/sched/sched.h | 4 ++++
kernel/sched/syscalls.c | 19 +++++++++++++++++++
3 files changed, 26 insertions(+), 13 deletions(-)
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 18ecd4f908e4..715d7c1f55df 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -9352,28 +9352,18 @@ static inline void update_blocked_load_status(struct rq *rq, bool has_blocked) {
static bool __update_blocked_others(struct rq *rq, bool *done)
{
- const struct sched_class *curr_class;
- u64 now = rq_clock_pelt(rq);
- unsigned long hw_pressure;
- bool decayed;
+ bool updated;
/*
* update_load_avg() can call cpufreq_update_util(). Make sure that RT,
* DL and IRQ signals have been updated before updating CFS.
*/
- curr_class = rq->curr->sched_class;
-
- hw_pressure = arch_scale_hw_pressure(cpu_of(rq));
-
- decayed = update_rt_rq_load_avg(now, rq, curr_class == &rt_sched_class) |
- update_dl_rq_load_avg(now, rq, curr_class == &dl_sched_class) |
- update_hw_load_avg(now, rq, hw_pressure) |
- update_irq_load_avg(rq, 0);
+ updated = update_other_load_avgs(rq);
if (others_have_blocked(rq))
*done = false;
- return decayed;
+ return updated;
}
#ifdef CONFIG_FAIR_GROUP_SCHED
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 656a63c0d393..a5a4f59151db 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -3074,6 +3074,8 @@ static inline void cpufreq_update_util(struct rq *rq, unsigned int flags) { }
#ifdef CONFIG_SMP
+bool update_other_load_avgs(struct rq *rq);
+
unsigned long effective_cpu_util(int cpu, unsigned long util_cfs,
unsigned long *min,
unsigned long *max);
@@ -3117,6 +3119,8 @@ static inline unsigned long cpu_util_rt(struct rq *rq)
return READ_ONCE(rq->avg_rt.util_avg);
}
+#else /* !CONFIG_SMP */
+static inline bool update_other_load_avgs(struct rq *rq) { return false; }
#endif /* CONFIG_SMP */
#ifdef CONFIG_UCLAMP_TASK
diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
index cf189bc3dd18..050215ef8fa4 100644
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -259,6 +259,25 @@ int sched_core_idle_cpu(int cpu)
#endif
#ifdef CONFIG_SMP
+/*
+ * Load avg and utiliztion metrics need to be updated periodically and before
+ * consumption. This function updates the metrics for all subsystems except for
+ * the fair class. @rq must be locked and have its clock updated.
+ */
+bool update_other_load_avgs(struct rq *rq)
+{
+ u64 now = rq_clock_pelt(rq);
+ const struct sched_class *curr_class = rq->curr->sched_class;
+ unsigned long hw_pressure = arch_scale_hw_pressure(cpu_of(rq));
+
+ lockdep_assert_rq_held(rq);
+
+ return update_rt_rq_load_avg(now, rq, curr_class == &rt_sched_class) |
+ update_dl_rq_load_avg(now, rq, curr_class == &dl_sched_class) |
+ update_hw_load_avg(now, rq, hw_pressure) |
+ update_irq_load_avg(rq, 0);
+}
+
/*
* This function computes an effective utilization for the given CPU, to be
* used for frequency selection given the linear relation: f = u * f_max.
--
2.45.2
Implementation Analysis
Overview
This patch extracts the per-CPU load average update logic for RT, DL, hardware pressure, and IRQ subsystems from __update_blocked_others() in fair.c into a standalone function update_other_load_avgs() in syscalls.c, declared in sched.h. The function is pure refactoring — no behavior changes. sched_ext needs it because the ext class has its own load balancing path that does not go through CFS's __update_blocked_others(), yet it still must keep these metrics current.
The Problem Being Solved
Before this patch, the code that updates RT, DL, hardware pressure, and IRQ load averages lived exclusively inside __update_blocked_others(), a static function in kernel/sched/fair.c:
curr_class = rq->curr->sched_class;
hw_pressure = arch_scale_hw_pressure(cpu_of(rq));
decayed = update_rt_rq_load_avg(now, rq, curr_class == &rt_sched_class) |
update_dl_rq_load_avg(now, rq, curr_class == &dl_sched_class) |
update_hw_load_avg(now, rq, hw_pressure) |
update_irq_load_avg(rq, 0);
This function is called as part of CFS's blocked-load update path, which runs during CFS load balancing. The comment in __update_blocked_others() already acknowledged the coupling: RT, DL, and IRQ signals need to be updated before CFS updates its own load averages (because update_load_avg() can trigger cpufreq_update_util()).
sched_ext has its own load balancing code path that does not involve CFS's blocked-load update. If update_other_load_avgs() is only reachable through CFS, then when an ext-class CPU is idle or lightly loaded and CFS's path is not being run, the RT/DL/IRQ load metrics stagnate. Stale metrics affect CPU frequency selection and other consumers of these signals. The fix is to make the function independently callable.
Code Walkthrough
kernel/sched/syscalls.c — the new function is defined here (under #ifdef CONFIG_SMP):
bool update_other_load_avgs(struct rq *rq)
{
u64 now = rq_clock_pelt(rq);
const struct sched_class *curr_class = rq->curr->sched_class;
unsigned long hw_pressure = arch_scale_hw_pressure(cpu_of(rq));
lockdep_assert_rq_held(rq);
return update_rt_rq_load_avg(now, rq, curr_class == &rt_sched_class) |
update_dl_rq_load_avg(now, rq, curr_class == &dl_sched_class) |
update_hw_load_avg(now, rq, hw_pressure) |
update_irq_load_avg(rq, 0);
}
The logic is identical to what was in __update_blocked_others(). Notable additions compared to the original inline code:
lockdep_assert_rq_held(rq): An assertion that the caller holds the runqueue lock. This is a documentation-in-code improvement that catches misuse at compile/debug time.- The function is placed in
syscalls.crather thanfair.c, making it neutral with respect to scheduler class — it is not CFS-specific infrastructure.
kernel/sched/fair.c — __update_blocked_others() is simplified:
static bool __update_blocked_others(struct rq *rq, bool *done)
{
- const struct sched_class *curr_class;
- u64 now = rq_clock_pelt(rq);
- unsigned long hw_pressure;
- bool decayed;
+ bool updated;
/*
* update_load_avg() can call cpufreq_update_util(). Make sure that RT,
* DL and IRQ signals have been updated before updating CFS.
*/
- curr_class = rq->curr->sched_class;
- hw_pressure = arch_scale_hw_pressure(cpu_of(rq));
- decayed = update_rt_rq_load_avg(...) | update_dl_rq_load_avg(...) | ...;
+ updated = update_other_load_avgs(rq);
if (others_have_blocked(rq))
*done = false;
- return decayed;
+ return updated;
}
The body is replaced by a single call site. The variable rename from decayed to updated is cosmetic but more accurate — the return value indicates whether any metric was updated, not specifically whether any load average decayed.
kernel/sched/sched.h — declaration and no-op stub:
#ifdef CONFIG_SMP
bool update_other_load_avgs(struct rq *rq);
...
#else /* !CONFIG_SMP */
static inline bool update_other_load_avgs(struct rq *rq) { return false; }
#endif /* CONFIG_SMP */
The !CONFIG_SMP stub returns false (no updates occurred), which is correct because on a single-CPU system these per-runqueue averages are not meaningful for load balancing.
Why sched_ext Needs This
sched_ext performs its own per-CPU scheduling decisions and implements a load balancing path independent of CFS. If the CFS blocked-load update path (update_blocked_averages()) is not running for a given CPU — which can happen when the CPU is under ext-class control — then RT, DL, hardware pressure, and IRQ load averages will not be updated. These metrics are consumed by effective_cpu_util() and the cpufreq governor, so stale values can cause incorrect frequency scaling.
By factoring out update_other_load_avgs(), the sched_ext load balancing code can call it directly to ensure the metrics stay current regardless of whether the CFS path runs.
Connection to Other Patches
This patch does not depend on earlier patches in the series. It is a prerequisite for the sched_ext load balancing implementation later in the series, which will call update_other_load_avgs() from the ext class's equivalent of update_blocked_averages().
Key Data Structures / Functions Modified
update_other_load_avgs()(kernel/sched/syscalls.c, declared inkernel/sched/sched.h): New function. Updates RT, DL, hardware pressure, and IRQ load averages for a given runqueue. Requires the runqueue lock to be held.__update_blocked_others()(kernel/sched/fair.c): CFS internal function called during blocked-load accounting. Simplified to delegate toupdate_other_load_avgs().update_rt_rq_load_avg()/update_dl_rq_load_avg()/update_hw_load_avg()/update_irq_load_avg(): Existing per-subsystem load average update functions now called throughupdate_other_load_avgs(). Their signatures are unchanged.
[PATCH 07/30] sched: Add normal_policy()
View on Lore: https://lore.kernel.org/all/20240618212056.2833381-8-tj@kernel.org
Commit Message
A new BPF extensible sched_class will need to dynamically change how a task
picks its sched_class. For example, if the loaded BPF scheduler progs fail,
the tasks will be forced back on CFS even if the task's policy is set to the
new sched_class. To support such mapping, add normal_policy() which wraps
testing for %SCHED_NORMAL. This doesn't cause any behavior changes.
v2: Update the description with more details on the expected use.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: David Vernet <dvernet@meta.com>
Acked-by: Josh Don <joshdon@google.com>
Acked-by: Hao Luo <haoluo@google.com>
Acked-by: Barret Rhoden <brho@google.com>
---
kernel/sched/fair.c | 2 +-
kernel/sched/sched.h | 7 ++++++-
2 files changed, 7 insertions(+), 2 deletions(-)
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 715d7c1f55df..d59537416865 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -8391,7 +8391,7 @@ static void check_preempt_wakeup_fair(struct rq *rq, struct task_struct *p, int
* Batch and idle tasks do not preempt non-idle tasks (their preemption
* is driven by the tick):
*/
- if (unlikely(p->policy != SCHED_NORMAL) || !sched_feat(WAKEUP_PREEMPTION))
+ if (unlikely(!normal_policy(p->policy)) || !sched_feat(WAKEUP_PREEMPTION))
return;
find_matching_se(&se, &pse);
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index a5a4f59151db..25660dc9f639 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -192,9 +192,14 @@ static inline int idle_policy(int policy)
return policy == SCHED_IDLE;
}
+static inline int normal_policy(int policy)
+{
+ return policy == SCHED_NORMAL;
+}
+
static inline int fair_policy(int policy)
{
- return policy == SCHED_NORMAL || policy == SCHED_BATCH;
+ return normal_policy(policy) || policy == SCHED_BATCH;
}
static inline int rt_policy(int policy)
--
2.45.2
Diff
---
kernel/sched/fair.c | 2 +-
kernel/sched/sched.h | 7 ++++++-
2 files changed, 7 insertions(+), 2 deletions(-)
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 715d7c1f55df..d59537416865 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -8391,7 +8391,7 @@ static void check_preempt_wakeup_fair(struct rq *rq, struct task_struct *p, int
* Batch and idle tasks do not preempt non-idle tasks (their preemption
* is driven by the tick):
*/
- if (unlikely(p->policy != SCHED_NORMAL) || !sched_feat(WAKEUP_PREEMPTION))
+ if (unlikely(!normal_policy(p->policy)) || !sched_feat(WAKEUP_PREEMPTION))
return;
find_matching_se(&se, &pse);
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index a5a4f59151db..25660dc9f639 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -192,9 +192,14 @@ static inline int idle_policy(int policy)
return policy == SCHED_IDLE;
}
+static inline int normal_policy(int policy)
+{
+ return policy == SCHED_NORMAL;
+}
+
static inline int fair_policy(int policy)
{
- return policy == SCHED_NORMAL || policy == SCHED_BATCH;
+ return normal_policy(policy) || policy == SCHED_BATCH;
}
static inline int rt_policy(int policy)
--
2.45.2
Implementation Analysis
Overview
This patch adds a normal_policy() inline helper that returns true when a scheduling policy is SCHED_NORMAL, mirroring the existing idle_policy(), fair_policy(), and rt_policy() helpers. It is a tiny change that unifies the policy-test pattern and specifically enables sched_ext to check whether a task belongs to SCHED_NORMAL — the policy that sched_ext will claim when a BPF scheduler is loaded — as distinct from SCHED_BATCH and SCHED_IDLE, which CFS handles but which cannot use sched_ext.
Background: The Linux Scheduler Class Hierarchy
Linux defines several scheduling policies that userspace can assign to a task:
| Policy | Class | Meaning |
|---|---|---|
SCHED_NORMAL (0) | fair or ext | Normal timesharing tasks |
SCHED_FIFO (1) | rt | Real-time FIFO |
SCHED_RR (2) | rt | Real-time round-robin |
SCHED_BATCH (3) | fair | Batch/CPU-bound, no wakeup preemption |
SCHED_IDLE (5) | fair | Very low priority; not related to idle class |
SCHED_DEADLINE (6) | dl | EDF deadline tasks |
SCHED_EXT (7) | ext | BPF-extensible scheduler |
fair_policy() currently encompasses both SCHED_NORMAL and SCHED_BATCH because both run under CFS. The distinction matters for wakeup preemption: SCHED_BATCH tasks intentionally do not trigger preemption on wakeup (they want to run in long bursts). check_preempt_wakeup_fair() already tested for p->policy != SCHED_NORMAL specifically to skip batch tasks.
sched_ext only accepts SCHED_NORMAL tasks (or tasks explicitly assigned SCHED_EXT). When a BPF scheduler is loaded, SCHED_NORMAL tasks will be redirected to ext_sched_class instead of fair_sched_class. SCHED_BATCH and SCHED_IDLE tasks stay in CFS. The ext class code therefore needs a clean way to ask "is this task a normal-policy task?" without testing the raw integer constant everywhere.
The Problem Being Solved
Before this patch, the kernel had a family of policy-test helpers:
static inline int idle_policy(int policy) { return policy == SCHED_IDLE; }
static inline int fair_policy(int policy) { return policy == SCHED_NORMAL || policy == SCHED_BATCH; }
static inline int rt_policy(int policy) { return policy == SCHED_FIFO || policy == SCHED_RR; }
static inline int dl_policy(int policy) { return policy == SCHED_DEADLINE; }
There was no normal_policy(). Code that specifically needed SCHED_NORMAL (not SCHED_BATCH) had to test p->policy != SCHED_NORMAL directly, as check_preempt_wakeup_fair() did. For sched_ext, which must frequently check whether a task is eligible for the ext class, having to hardcode == SCHED_NORMAL every time is fragile — if the set of policies considered "normal" ever changes, every site must be updated manually.
Code Walkthrough
kernel/sched/sched.h — normal_policy() is added and fair_policy() is updated to use it:
+static inline int normal_policy(int policy)
+{
+ return policy == SCHED_NORMAL;
+}
+
static inline int fair_policy(int policy)
{
- return policy == SCHED_NORMAL || policy == SCHED_BATCH;
+ return normal_policy(policy) || policy == SCHED_BATCH;
}
normal_policy() is placed immediately before fair_policy(), and fair_policy() is updated to delegate to it. This makes the relationship explicit: fair_policy is a superset of normal_policy. The insertion point (after idle_policy, before fair_policy) keeps the helpers in policy-hierarchy order.
kernel/sched/fair.c — one existing open-coded test is updated to use the helper:
- if (unlikely(p->policy != SCHED_NORMAL) || !sched_feat(WAKEUP_PREEMPTION))
+ if (unlikely(!normal_policy(p->policy)) || !sched_feat(WAKEUP_PREEMPTION))
This is in check_preempt_wakeup_fair(), which decides whether a waking task should preempt the current task. SCHED_BATCH tasks should not trigger wakeup preemption; only SCHED_NORMAL tasks should. The semantics are identical — this is purely a style improvement.
Why sched_ext Needs This
When the ext class implements its task-to-class mapping logic, it must determine whether a given task should use ext_sched_class or stay with fair_sched_class. The rule is: tasks with SCHED_NORMAL policy are eligible for ext; tasks with SCHED_BATCH or SCHED_IDLE policy remain in CFS even when a BPF scheduler is loaded.
The ext class implementation calls normal_policy(p->policy) rather than p->policy == SCHED_NORMAL to express this intent. Having fair_policy() also call normal_policy() internally ensures that if the definition of "normal" is ever extended (e.g., if SCHED_EXT is treated as a superset of SCHED_NORMAL for some purposes), only normal_policy() needs updating.
Connection to Other Patches
This is the last of the seven preparatory patches. It does not depend on any of the earlier six patches in the series. The sched_ext class implementation later in the series uses normal_policy() in its task-class assignment logic — specifically in the function that decides whether a newly-forked or policy-changed task should go to the ext class or fall back to CFS.
Key Data Structures / Functions Modified
normal_policy()(kernel/sched/sched.h): New inline helper. Returns true ifpolicy == SCHED_NORMAL. The single authoritative definition of what "normal policy" means.fair_policy()(kernel/sched/sched.h): Updated to callnormal_policy()internally. Now expresses that CFS handles normal tasks (vianormal_policy) and batch tasks.check_preempt_wakeup_fair()(kernel/sched/fair.c): The CFS wakeup-preemption check. Updated to callnormal_policy()instead of testingp->policy != SCHED_NORMALdirectly.
sched_ext Core Implementation (Patches 08–12)
Overview
This group contains the actual sched_ext implementation — the new scheduler class, its BPF interface, its safety infrastructure, and the first example schedulers. If the foundational refactoring patches (01–07) cleared the terrain, this group builds the structure. The cover letter (patch-08.md) contextualizes the submission as the v7 patchset, the version that was ultimately merged into Linux 6.11.
The five patches in this group are not equal in size or complexity. The boilerplate patch (patch-08) establishes the file skeleton and class registration. The core implementation patch (patch-09, ~4000 lines) is where almost all of the scheduling logic lives — it is the patch a maintainer must understand most deeply. Patches 10–12 add the safety envelope: example schedulers for verification, a sysrq escape hatch, and a watchdog timer.
Why These Patches Are Needed
A BPF-based scheduler class requires:
- A kernel-side implementation of
struct sched_classthat delegates decisions to BPF programs through a well-defined callback interface (struct sched_ext_ops). - Per-task state to track where each task sits in the scheduler's queues.
- Dispatch queues — the mechanism by which BPF programs move tasks to CPUs.
- BPF helper functions — the kernel-side API that BPF programs call to perform actions.
- Graceful failure — when the BPF program misbehaves, the kernel must recover without a panic.
- Example code that demonstrates the interface is usable and correctly designed.
This group delivers all six. Understanding why each piece exists requires understanding the problem sched_ext is solving: allow arbitrary scheduling policy in BPF while keeping the kernel safe from misbehaving BPF programs.
Key Concepts
The Cover Letter (patch-08.md) — v7 and the Merge Path
The v7 patchset was submitted after six earlier rounds of review. The cover letter is important reading for a maintainer because it documents the design decisions that were debated and resolved during review — the reasons certain choices were made over alternatives. It also announces that sched_ext would be merged into Linux 6.11 via the tip/sched/core tree.
Key design decisions the cover letter defends:
- Why
SCHED_EXTis a new scheduling policy (not a flag onSCHED_NORMAL). - Why BPF programs communicate through
sched_ext_opscallbacks rather than a ring buffer. - Why the dispatch queue abstraction (DSQ) exists as an intermediary rather than having BPF
programs directly set
curron CPUs.
The Boilerplate Patch — File Structure and Class Registration
The boilerplate patch creates the scaffolding that the core patch fills in:
include/linux/sched/ext.h— userspace-visible header:SCHED_EXTpolicy number,struct sched_ext_opsdeclaration (the BPF program fills this struct), and theSCX_DSQ_*constants.kernel/sched/ext.h— kernel-internal header:scx_entity(per-task state embedded intask_struct), internal DSQ structures, and declarations for functionscore.ccalls.kernel/sched/ext.c— the implementation file, initially stub functions.
The critical registration step: ext_sched_class is defined with .next = &idle_sched_class
and fair_sched_class.next is changed to &ext_sched_class. This inserts the new class at
the correct priority level. Hooks are added to kernel/fork.c (sched_ext_free() is called
from free_task()), kernel/sched/core.c (for class transitions), and kernel/sched/idle.c
(so idle CPUs check the SCX global DSQ).
PATCH 09 — The Core Implementation
This is the heart of sched_ext. The key data structures and mechanisms are:
struct scx_dispatch_q (DSQ)
A DSQ is an ordered list of tasks waiting to run. There are three kinds:
SCX_DSQ_GLOBAL(ID 0): A single global FIFO queue. Any CPU can dispatch from it. This is the simplest possible dispatch model — a BPF scheduler that only uses the global DSQ is functionally similar to a simple FIFO scheduler.SCX_DSQ_LOCAL(IDSCX_DSQ_LOCAL_ON | cpu): Per-CPU local queues. Tasks in a CPU's local DSQ will run on that specific CPU. The kernel dispatches from the local DSQ before looking elsewhere.- User-defined DSQs: BPF programs can create arbitrary DSQs with
scx_bpf_create_dsq(id, node). These allow BPF programs to implement complex queuing policies (priority queues, round-robin groups, deadline queues) entirely in BPF.
struct scx_entity
Embedded in task_struct (alongside struct sched_entity se for CFS). Tracks:
- Which DSQ the task is currently in (
dsqpointer). - The task's position in the DSQ (
dsq_node). - Slice remaining (
slice— how many nanoseconds the task can run before it's preempted). - Virtual time for vtime-ordered DSQs (
dsq_vtime). - Various flags:
SCX_TASK_QUEUED,SCX_TASK_RUNNING,SCX_TASK_DISALLOW, etc.
The dispatch path
When the kernel needs to pick the next task for a CPU, pick_next_task_scx() is called:
- Check the CPU's local DSQ (
rq->scx.local_dsq). If non-empty, take the first task. - Call
ops.dispatch(cpu, prev)— give the BPF program a chance to fill the local DSQ. - After
ops.dispatch()returns, check the local DSQ again. - Check
SCX_DSQ_GLOBALas a fallback. - If still empty, return NULL (kernel will try
idle_sched_class).
The enqueue path
When a task becomes runnable, enqueue_task_scx() is called:
- Call
ops.select_cpu(p, prev_cpu, wake_flags)— BPF program can return a preferred CPU. - If the preferred CPU's local DSQ is empty and the CPU is idle, dispatch there directly
(this is the "direct dispatch" optimization that avoids an extra
ops.dispatch()round). - Otherwise, call
ops.enqueue(p, enq_flags). The BPF program must callscx_bpf_dispatch(p, dsq_id, slice, enq_flags)to place the task in a DSQ.
BPF helper functions (scx_bpf_*)
These are the kernel-side API that BPF programs call:
scx_bpf_dispatch(p, dsq_id, slice, enq_flags)— move taskpto DSQdsq_idwith time sliceslice.scx_bpf_dispatch_vtime(p, dsq_id, vtime, slice, enq_flags)— dispatch with virtual time ordering.scx_bpf_consume(dsq_id)— inops.dispatch(), pull the next task from a user DSQ into the local DSQ.scx_bpf_create_dsq(dsq_id, node)— create a new DSQ.scx_bpf_destroy_dsq(dsq_id)— destroy a DSQ (BPF programs clean up inops.exit()).scx_bpf_task_running(p)— is the task currently executing on a CPU?scx_bpf_cpu_rq(cpu)— get thestruct rq *for a CPU (used to enqueue to local DSQs).
Error exit (scx_ops_error())
If the BPF program misbehaves (e.g., tries to dispatch to a non-existent DSQ, returns an
invalid CPU from select_cpu, or causes a kernel BUG), the kernel calls scx_ops_error().
This:
- Sets an error state in
scx_ops_exit_kind. - Schedules
scx_ops_disable_workfn()on a work queue. - The work function disables the BPF scheduler: moves all tasks back to CFS, calls
ops.exit(), and clearsext_sched_classfrom the class list.
The error state captures a human-readable reason string and is readable from debugfs, which feeds the monitoring tooling in later patches.
PATCH 10 — Example Schedulers
Two example BPF schedulers are provided in tools/sched_ext/:
scx_simple: The simplest possible sched_ext scheduler. It implements only ops.enqueue()
and ops.dispatch(). Every task is dispatched to SCX_DSQ_GLOBAL. This is the "hello world"
of sched_ext and demonstrates the minimum viable implementation.
scx_example_qmap: A more sophisticated scheduler using 5 per-priority queues implemented
as a BPF array map. ops.enqueue() places tasks in the queue corresponding to their nice value.
ops.dispatch() drains queues in priority order. This demonstrates:
- How to create and manage BPF-defined DSQs.
- How
scx_bpf_consume()is used to pull from custom DSQs into the local DSQ. - How per-task data can be stored in BPF maps keyed by
task_struct *.
These examples are not just demos — they are the primary validation that the API is usable and that the kernel-side implementation correctly handles the BPF callbacks.
PATCH 11 — sysrq-S Emergency Fallback
The keyboard shortcut Alt+SysRq+S is registered to call scx_ops_disable(). This provides
a human-accessible escape hatch: if a BPF scheduler causes the system to become unresponsive
(tasks not scheduled, UI frozen), a user at the console can press Alt+SysRq+S to forcibly
kill the BPF scheduler and return all tasks to CFS.
Mechanically, this calls scx_ops_error() with a "sysrq" reason, which triggers the same
disable path as an internal error. The distinction is in the exit reason recorded: SCX_EXIT_SYSRQ
vs SCX_EXIT_ERROR. This is important for post-mortem analysis — was the scheduler disabled
intentionally or due to a bug?
The sysrq handler is registered in scx_init() and deregistered in scx_exit().
PATCH 12 — Watchdog Timer
The watchdog timer addresses a failure mode that sysrq cannot catch: the system is technically running but the BPF scheduler is not scheduling tasks in a timely manner. For example, a BPF program might have a bug where certain tasks are never dequeued from a user-defined DSQ, starving them indefinitely.
The watchdog implementation:
- A per-CPU
struct scx_watchdogcontains ahrtimerand tracks when the CPU's SCX tasks were last dispatched. - The timer fires every
scx_watchdog_timeout / 2(default: 15 seconds, half the 30-second timeout). - On each fire, it checks whether any runnable SCX task has not been scheduled within
scx_watchdog_timeout. - If a stall is detected,
scx_ops_error()is called with "runnable task stall detected".
The watchdog uses a generation counter per task (scx_entity.runnable_at) that is updated
each time a task is dispatched. The watchdog checks whether now - runnable_at > timeout for
any runnable task.
The watchdog is enabled when scx_ops_enable() succeeds and disabled as part of
scx_ops_disable_workfn(). It cannot fire during class transitions, avoiding a window
where a task might appear stalled simply because it is being migrated.
Connections Between Patches
The patches in this group build on each other in a specific order:
PATCH 08 (boilerplate)
└─→ Creates the files and stubs that PATCH 09 fills in
└─→ Registers ext_sched_class, making the class visible to the scheduler
PATCH 09 (core)
└─→ Implements the full dispatch/enqueue/select_cpu machinery
└─→ Provides scx_ops_error() which PATCH 11 and PATCH 12 use
└─→ Provides scx_ops_enable/disable which PATCH 12's watchdog wraps
PATCH 10 (examples)
└─→ Validates that PATCH 09's API is correct and complete
└─→ The simplest test that the dispatch path works end-to-end
PATCH 11 (sysrq)
└─→ Uses scx_ops_error() from PATCH 09
└─→ Provides human-accessible escape hatch tested before watchdog
PATCH 12 (watchdog)
└─→ Uses scx_ops_error() from PATCH 09
└─→ Closes the failure mode that PATCH 11 cannot catch (no human present)
Connections to Foundational Patches
This group directly consumes every change from patches 01–07:
sched_class_above()(PATCH 01) is called inext.cwherever class priority comparisons occur.- Fallible fork (PATCH 02) enables
scx_cgroup_can_attach()andops.enable()during fork to propagateENOMEM. reweight_task()(PATCH 03) is implemented inext.cto callops.reweight_task().switching_to()(PATCH 04) is implemented inext.cto callops.enable()before enqueue.- Cgroup/PELT helpers (PATCHES 05–06) are called from
scx_bpf_task_cgroup_weight()and the PELT integration inscx_entity. normal_policy()(PATCH 07) is used in several places inext.cto check task policy.
What to Focus On
For a maintainer, the critical areas to understand in this group:
-
The dispatch contract. The rule is: a task must end up in a DSQ (via
scx_bpf_dispatch()) before the kernel can run it. Ifops.enqueue()is called and the BPF program does not callscx_bpf_dispatch(), the task is considered to have been "consumed" without being dispatched, which triggers an error exit. Understanding this contract is necessary to evaluate any future change to the enqueue/dispatch path. -
DSQ lifecycle. User-defined DSQs are created with
scx_bpf_create_dsq()and must be destroyed inops.exit(). If a BPF program forgets to destroy a DSQ, the kernel detects leaked DSQs duringscx_ops_disable_workfn()and logs an error. Future patches touching DSQ lifecycle (e.g., per-NUMA DSQs, per-cgroup DSQs) must respect this cleanup contract. -
The error exit machinery.
scx_ops_error()can be called from any context — interrupt, softirq, process context — so it must be lock-free in its initial steps. The actual disable work is deferred to a workqueue. Understanding this deferred-disable pattern is essential for reviewing any change that adds new error conditions. -
scx_entityintask_struct. Adding fields totask_structis always scrutinized heavily in kernel review because it increases memory usage for every task on the system, not just SCX tasks. Study howscx_entityis conditionally compiled (CONFIG_SCHED_CLASS_EXT) and what techniques are used to minimize its footprint. -
The select_cpu / direct dispatch optimization. The path where
ops.select_cpu()returns a CPU whose local DSQ is empty, and the task is dispatched there without going throughops.enqueue()at all, is a critical performance optimization. Any change to wakeup paths must preserve this fast path or explicitly justify removing it. -
Example schedulers as specification. The example schedulers in PATCH 10 are not just documentation — they are the canonical test of whether a proposed API change is usable. When reviewing future sched_ext API changes, check whether the examples would need to change and whether the changes make the examples simpler or more complex.
[PATCHSET v7] sched: Implement BPF extensible scheduler class
View on Lore: https://lore.kernel.org/all/20240618212056.2833381-1-tj@kernel.org
Commit Message
Updates and Changes
-------------------
This is v7 and the final posting of the sched_ext (SCX) patchset. sched_ext
is scheduled to be merged in the coming v6.11 merge window. Development from
this point on will take place in the below korg tree seeded with this
patchset and follow the usual kernel development protocol.
https://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext.git/
Please refer to the v6 patchset thread for the discussions and conclusion:
https://lore.kernel.org/all/20240501151312.635565-1-tj@kernel.org/#r
If you're interested in getting your hands dirty, the following repository
contains example and practical schedulers along with documentation on how to
get started:
https://github.com/sched-ext/scx
The kernel and scheduler packages are available for Ubuntu, CachyOS and Arch
(through CachyOS repo). Fedora packaging is in the works.
There are also a slack and weekly office hour:
https://bit.ly/scx_slack
Office Hour: Mondays at 16:00 UTC (8:00 AM PST, 16:00 UTC, 17:00 CEST,
1:00 AM KST). Please see the #office-hours channel for the
zoom invite.
Changes from v6
(https://lkml.kernel.org/r/20240501151312.635565-1-tj@kernel.org):
- Rebased on top of tip/sched/core + bpf/for-next as of 2024-06-24.
- A bug in CPU hotplug support is fixed. This made
0009-sched-Add-reason-to-sched_class-rq_-on-off-line.patch unnecessary.
Dropped.
- Features that interact with other subsystems or have other dependencies
are separated out and will be posted as separate patchsets.
- cgroup support:
0001-cgroup-Implement-cgroup_show_cftypes.patch
0007-sched-Expose-css_tg-and-__setscheduler_prio.patch
0008-sched-Enumerate-CPU-cgroup-file-types.patch
0028-sched_ext-Add-cgroup-support.patch
0029-sched_ext-Add-a-cgroup-scheduler-which-uses-flattene.patch
- cpufreq support:
0011-cpufreq_schedutil-Refactor-sugov_cpu_is_busy.patch
0037-sched_ext-Add-cpuperf-support.patch
- DSQ iterator:
0036-sched_ext-Implement-DSQ-iterator.patch
Combined with the dropping of 0009, this reduces the total number of
patches in this series to 30.
- sched_ext_ops.dump*() operations are added so that the BPF scheduler can
dump information specific to the scheduler implementation. Dump can now
also be initiated using SysRq-D and read through a tracepoint.
- Fixes for bugs including a race condition around TASK_DEAD handling in the
enable path and possible deadlock in the disable path.
- Cleanups including dropping backward compat helpers as this patchset will
be the new baseline.
Overview
--------
This patch set proposes a new scheduler class called ‘ext_sched_class’, or
sched_ext, which allows scheduling policies to be implemented as BPF programs.
More details will be provided on the overall architecture of sched_ext
throughout the various patches in this set, as well as in the “How” section
below. We realize that this patch set is a significant proposal, so we will be
going into depth in the following “Motivation” section to explain why we think
it’s justified. That section is laid out as follows, touching on three main
axes where we believe that sched_ext provides significant value:
1. Ease of experimentation and exploration: Enabling rapid iteration of new
scheduling policies.
2. Customization: Building application-specific schedulers which implement
policies that are not applicable to general-purpose schedulers.
3. Rapid scheduler deployments: Non-disruptive swap outs of scheduling
policies in production environments.
After the motivation section, we’ll provide a more detailed (but still
high-level) overview of how sched_ext works.
Motivation
----------
1. Ease of experimentation and exploration
*Why is exploration important?*
Scheduling is a challenging problem space. Small changes in scheduling
behavior can have a significant impact on various components of a system, with
the corresponding effects varying widely across different platforms,
architectures, and workloads.
While complexities have always existed in scheduling, they have increased
dramatically over the past 10-15 years. In the mid-late 2000s, cores were
typically homogeneous and further apart from each other, with the criteria for
scheduling being roughly the same across the entire die.
Systems in the modern age are by comparison much more complex. Modern CPU
designs, where the total power budget of all CPU cores often far exceeds the
power budget of the socket, with dynamic frequency scaling, and with or
without chiplets, have significantly expanded the scheduling problem space.
Cache hierarchies have become less uniform, with Core Complex (CCX) designs
such as recent AMD processors having multiple shared L3 caches within a single
socket. Such topologies resemble NUMA sans persistent NUMA node stickiness.
Use-cases have become increasingly complex and diverse as well. Applications
such as mobile and VR have strict latency requirements to avoid missing
deadlines that impact user experience. Stacking workloads in servers is
constantly pushing the demands on the scheduler in terms of workload isolation
and resource distribution.
Experimentation and exploration are important for any non-trivial problem
space. However, given the recent hardware and software developments, we
believe that experimentation and exploration are not just important, but
_critical_ in the scheduling problem space.
Indeed, other approaches in industry are already being explored. AMD has
proposed an experimental patch set [0] which enables userspace to provide
hints to the scheduler via “Userspace Hinting”. The approach adds a prctl()
API which allows callers to set a numerical “hint” value on a struct
task_struct. This hint is then optionally read by the scheduler to adjust the
cost calculus for various scheduling decisions.
[0]: https://lore.kernel.org/lkml/20220910105326.1797-1-kprateek.nayak@amd.com/
Huawei have also expressed interest [1] in enabling some form of programmable
scheduling. While we’re unaware of any patch sets which have been sent to the
upstream list for this proposal, it similarly illustrates the need for more
flexibility in the scheduler.
[1]: https://lore.kernel.org/bpf/dedc7b72-9da4-91d0-d81d-75360c177188@huawei.com/
Additionally, Google has developed ghOSt [2] with the goal of enabling custom,
userspace driven scheduling policies. Prior presentations at LPC [3] have
discussed ghOSt and how BPF can be used to accelerate scheduling.
[2]: https://dl.acm.org/doi/pdf/10.1145/3477132.3483542
[3]: https://lpc.events/event/16/contributions/1365/
*Why can’t we just explore directly with CFS?*
Experimenting with CFS directly or implementing a new sched_class from scratch
is of course possible, but is often difficult and time consuming. Newcomers to
the scheduler often require years to understand the codebase and become
productive contributors. Even for seasoned kernel engineers, experimenting
with and upstreaming features can take a very long time. The iteration process
itself is also time consuming, as testing scheduler changes on real hardware
requires reinstalling the kernel and rebooting the host.
Core scheduling is an example of a feature that took a significant amount of
time and effort to integrate into the kernel. Part of the difficulty with core
scheduling was the inherent mismatch in abstraction between the desire to
perform core-wide scheduling, and the per-cpu design of the kernel scheduler.
This caused issues, for example ensuring proper fairness between the
independent runqueues of SMT siblings.
The high barrier to entry for working on the scheduler is an impediment to
academia as well. Master’s/PhD candidates who are interested in improving the
scheduler will spend years ramping-up, only to complete their degrees just as
they’re finally ready to make significant changes. A lower entrance barrier
would allow researchers to more quickly ramp up, test out hypotheses, and
iterate on novel ideas. Research methodology is also severely hampered by the
high barrier of entry to make modifications; for example, the Shenango [4] and
Shinjuku scheduling policies used sched affinity to replicate the desired
policy semantics, due to the difficulty of incorporating these policies into
the kernel directly.
[4]: https://www.usenix.org/system/files/nsdi19-ousterhout.pdf
The iterative process itself also imposes a significant cost to working on the
scheduler. Testing changes requires developers to recompile and reinstall the
kernel, reboot their machines, rewarm their workloads, and then finally rerun
their benchmarks. Though some of this overhead could potentially be mitigated
by enabling schedulers to be implemented as kernel modules, a machine crash or
subtle system state corruption is always only one innocuous mistake away.
These problems are exacerbated when testing production workloads in a
datacenter environment as well, where multiple hosts may be involved in an
experiment; requiring a significantly longer ramp up time. Warming up memcache
instances in the Meta production environment takes hours, for example.
*How does sched_ext help with exploration?*
sched_ext attempts to address all of the problems described above. In this
section, we’ll describe the benefits to experimentation and exploration that
are afforded by sched_ext, provide real-world examples of those benefits, and
discuss some of the trade-offs and considerations in our design choices.
One of our main goals was to lower the barrier to entry for experimenting
with the scheduler. sched_ext provides ergonomic callbacks and helpers to
ease common operations such as managing idle CPUs, scheduling tasks on
arbitrary CPUs, handling preemptions from other scheduling classes, and
more. While sched_ext does require some ramp-up, the complexity is
self-contained, and the learning curve gradual. Developers can ramp up by
first implementing simple policies such as global weighted vtime scheduling
in only tens of lines of code, and then continue to learn the APIs and
building blocks available with sched_ext as they build more featureful and
complex schedulers.
Another critical advantage provided by sched_ext is the use of BPF. BPF
provides strong safety guarantees by statically analyzing programs at load
time to ensure that they cannot corrupt or crash the system. sched_ext
guarantees system integrity no matter what BPF scheduler is loaded, and
provides mechanisms to safely disable the current BPF scheduler and migrate
tasks back to a trusted scheduler. For example, we also implement in-kernel
safety mechanisms to guarantee that a misbehaving scheduler cannot
indefinitely starve tasks. BPF also enables sched_ext to significantly improve
iteration speed for running experiments. Loading and unloading a BPF scheduler
is simply a matter of running and terminating a sched_ext binary.
BPF also provides programs with a rich set of APIs, such as maps, kfuncs,
and BPF helpers. In addition to providing useful building blocks to programs
that run entirely in kernel space (such as many of our example schedulers),
these APIs also allow programs to leverage user space in making scheduling
decisions. Specifically, the Atropos sample scheduler has a relatively
simple weighted vtime or FIFO scheduling layer in BPF, paired with a load
balancing component in userspace written in Rust. As described in more
detail below, we also built a more general user-space scheduling framework
called “rhone” by leveraging various BPF features.
On the other hand, BPF does have shortcomings, as can be plainly seen from
the complexity in some of the example schedulers. scx_pair.bpf.c illustrates
this point well. To start, it requires a good amount of code to emulate
cgroup-local-storage. In the kernel proper, this would simply be a matter of
adding another pointer to the struct cgroup, but in BPF, it requires a
complex juggling of data amongst multiple different maps, a good amount of
boilerplate code, and some unwieldy bpf_loop()‘s and atomics. The code is
also littered with explicit and often unnecessary sanity checks to appease
the verifier.
That being said, BPF is being rapidly improved. For example, Yonghong Song
recently upstreamed a patch set [5] to add a cgroup local storage map type,
allowing scx_pair.bpf.c to be simplified. There are plans to address other
issues as well, such as providing statically-verified locking, and avoiding
the need for unnecessary sanity checks. Addressing these shortcomings is a
high priority for BPF, and as progress continues to be made, we expect most
deficiencies to be addressed in the not-too-distant future.
[5]: https://lore.kernel.org/bpf/20221026042835.672317-1-yhs@fb.com/
Yet another exploration advantage of sched_ext is helping widening the scope
of experiments. For example, sched_ext makes it easy to defer CPU assignment
until a task starts executing, allowing schedulers to share scheduling queues
at any granularity (hyper-twin, CCX and so on). Additionally, higher level
frameworks can be built on top to further widen the scope. For example, the
aforementioned “rhone” [6] library allows implementing scheduling policies in
user-space by encapsulating the complexity around communicating scheduling
decisions with the kernel. This allows taking advantage of a richer
programming environment in user-space, enabling experimenting with, for
instance, more complex mathematical models.
[6]: https://github.com/Decave/rhone
sched_ext also allows developers to leverage machine learning. At Meta, we
experimented with using machine learning to predict whether a running task
would soon yield its CPU. These predictions can be used to aid the scheduler
in deciding whether to keep a runnable task on its current CPU rather than
migrating it to an idle CPU, with the hope of avoiding unnecessary cache
misses. Using a tiny neural net model with only one hidden layer of size 16,
and a decaying count of 64 syscalls as a feature, we were able to achieve a
15% throughput improvement on an Nginx benchmark, with an 87% inference
accuracy.
2. Customization
This section discusses how sched_ext can enable users to run workloads on
application-specific schedulers.
*Why deploy custom schedulers rather than improving CFS?*
Implementing application-specific schedulers and improving CFS are not
conflicting goals. Scheduling features explored with sched_ext which yield
beneficial results, and which are sufficiently generalizable, can and should
be integrated into CFS. However, CFS is fundamentally designed to be a general
purpose scheduler, and thus is not conducive to being extended with some
highly targeted application or hardware specific changes.
Targeted, bespoke scheduling has many potential use cases. For example, VM
scheduling can make certain optimizations that are infeasible in CFS due to
the constrained problem space (scheduling a static number of long-running
VCPUs versus an arbitrary number of threads). Additionally, certain
applications might want to make targeted policy decisions based on hints
directly from the application (for example, a service that knows the different
deadlines of incoming RPCs).
Google has also experimented with some promising, novel scheduling policies.
One example is “central” scheduling, wherein a single CPU makes all
scheduling decisions for the entire system. This allows most cores on the
system to be fully dedicated to running workloads, and can have significant
performance improvements for certain use cases. For example, central
scheduling with VCPUs can avoid expensive vmexits and cache flushes, by
instead delegating the responsibility of preemption checks from the tick to
a single CPU. See scx_central.bpf.c for a simple example of a central
scheduling policy built in sched_ext.
Some workloads also have non-generalizable constraints which enable
optimizations in a scheduling policy which would otherwise not be feasible.
For example,VM workloads at Google typically have a low overcommit ratio
compared to the number of physical CPUs. This allows the scheduler to support
bounded tail latencies, as well as longer blocks of uninterrupted time.
Yet another interesting use case is the scx_flatcg scheduler, which is in
0024-sched_ext-Add-cgroup-support.patch and provides a flattened
hierarchical vtree for cgroups. This scheduler does not account for
thundering herd problems among cgroups, and therefore may not be suitable
for inclusion in CFS. However, in a simple benchmark using wrk[8] on apache
serving a CGI script calculating sha1sum of a small file, it outperformed
CFS by ~3% with CPU controller disabled and by ~10% with two apache
instances competing with 2:1 weight ratio nested four level deep.
[7] https://github.com/wg/wrk
Certain industries require specific scheduling behaviors that do not apply
broadly. For example, ARINC 653 defines scheduling behavior that is widely
used by avionic software, and some out-of-tree implementations
(https://ieeexplore.ieee.org/document/7005306) have been built. While the
upstream community may decide to merge one such implementation in the future,
it would also be entirely reasonable to not do so given the narrowness of
use-case, and non-generalizable, strict requirements. Such cases can be well
served by sched_ext in all stages of the software development lifecycle --
development, testing, deployment and maintenance.
There are also classes of policy exploration, such as machine learning, or
responding in real-time to application hints, that are significantly harder
(and not necessarily appropriate) to integrate within the kernel itself.
*Won’t this increase fragmentation?*
We acknowledge that to some degree, sched_ext does run the risk of
increasing the fragmentation of scheduler implementations. As a result of
exploration, however, we believe that enabling the larger ecosystem to
innovate will ultimately accelerate the overall development and performance
of Linux.
BPF programs are required to be GPLv2, which is enforced by the verifier on
program loads. With regards to API stability, just as with other semi-internal
interfaces such as BPF kfuncs, we won’t be providing any API stability
guarantees to BPF schedulers. While we intend to make an effort to provide
compatibility when possible, we will not provide any explicit, strong
guarantees as the kernel typically does with e.g. UAPI headers. For users who
decide to keep their schedulers out-of-tree,the licensing and maintenance
overheads will be fundamentally the same as for carrying out-of-tree patches.
With regards to the schedulers included in this patch set, and any other
schedulers we implement in the future, both Meta and Google will open-source
all of the schedulers we implement which have any relevance to the broader
upstream community. We expect that some of these, such as the simple example
schedulers and scx_rusty scheduler, will be upstreamed as part of the kernel
tree. Distros will be able to package and release these schedulers with the
kernel, allowing users to utilize these schedulers out-of-the-box without
requiring any additional work or dependencies such as clang or building the
scheduler programs themselves. Other schedulers and scheduling frameworks
such as rhone may be open-sourced through separate per-project repos.
3. Rapid scheduler deployments
Rolling out kernel upgrades is a slow and iterative process. At a large scale
it can take months to roll a new kernel out to a fleet of servers. While this
latency is expected and inevitable for normal kernel upgrades, it can become
highly problematic when kernel changes are required to fix bugs. Livepatch [8]
is available to quickly roll out critical security fixes to large fleets, but
the scope of changes that can be applied with livepatching is fairly limited,
and would likely not be usable for patching scheduling policies. With
sched_ext, new scheduling policies can be rapidly rolled out to production
environments.
[8]: https://www.kernel.org/doc/html/latest/livepatch/livepatch.html
As an example, one of the variants of the L1 Terminal Fault (L1TF) [9]
vulnerability allows a VCPU running a VM to read arbitrary host kernel
memory for pages in L1 data cache. The solution was to implement core
scheduling, which ensures that tasks running as hypertwins have the same
“cookie”.
[9]: https://www.intel.com/content/www/us/en/architecture-and-technology/l1tf.html
While core scheduling works well, it took a long time to finalize and land
upstream. This long rollout period was painful, and required organizations to
make difficult choices amongst a bad set of options. Some companies such as
Google chose to implement and use their own custom L1TF-safe scheduler, others
chose to run without hyper-threading enabled, and yet others left
hyper-threading enabled and crossed their fingers.
Once core scheduling was upstream, organizations had to upgrade the kernels on
their entire fleets. As downtime is not an option for many, these upgrades had
to be gradually rolled out, which can take a very long time for large fleets.
An example of an sched_ext scheduler that illustrates core scheduling
semantics is scx_pair.bpf.c, which co-schedules pairs of tasks from the same
cgroup, and is resilient to L1TF vulnerabilities. While this example
scheduler is certainly not suitable for production in its current form, a
similar scheduler that is more performant and featureful could be written
and deployed if necessary.
Rapid scheduling deployments can similarly be useful to quickly roll-out new
scheduling features without requiring kernel upgrades. At Google, for example,
it was observed that some low-priority workloads were causing degraded
performance for higher-priority workloads due to consuming a disproportionate
share of memory bandwidth. While a temporary mitigation was to use sched
affinity to limit the footprint of this low-priority workload to a small
subset of CPUs, a preferable solution would be to implement a more featureful
task-priority mechanism which automatically throttles lower-priority tasks
which are causing memory contention for the rest of the system. Implementing
this in CFS and rolling it out to the fleet could take a very long time.
sched_ext would directly address these gaps. If another hardware bug or
resource contention issue comes in that requires scheduler support to
mitigate, sched_ext can be used to experiment with and test different
policies. Once a scheduler is available, it can quickly be rolled out to as
many hosts as necessary, and function as a stop-gap solution until a
longer-term mitigation is upstreamed.
How
---
sched_ext is a new sched_class which allows scheduling policies to be
implemented in BPF programs.
sched_ext leverages BPF’s struct_ops feature to define a structure which
exports function callbacks and flags to BPF programs that wish to implement
scheduling policies. The struct_ops structure exported by sched_ext is struct
sched_ext_ops, and is conceptually similar to struct sched_class. The role of
sched_ext is to map the complex sched_class callbacks to the more simple and
ergonomic struct sched_ext_ops callbacks.
Unlike some other BPF program types which have ABI requirements due to
exporting UAPIs, struct_ops has no ABI requirements whatsoever. This provides
us with the flexibility to change the APIs provided to schedulers as
necessary. BPF struct_ops is also already being used successfully in other
subsystems, such as in support of TCP congestion control.
The only struct_ops field that is required to be specified by a scheduler is
the ‘name’ field. Otherwise, sched_ext will provide sane default behavior,
such as automatically choosing an idle CPU on the task wakeup path if
.select_cpu() is missing.
*Dispatch queues*
To bridge the workflow imbalance between the scheduler core and sched_ext_ops
callbacks, sched_ext uses simple FIFOs called dispatch queues (dsq's). By
default, there is one global dsq (SCX_DSQ_GLOBAL), and one local per-CPU dsq
(SCX_DSQ_LOCAL). SCX_DSQ_GLOBAL is provided for convenience and need not be
used by a scheduler that doesn't require it. As described in more detail
below, SCX_DSQ_LOCAL is the per-CPU FIFO that sched_ext pulls from when
putting the next task on the CPU. The BPF scheduler can manage an arbitrary
number of dsq's using scx_bpf_create_dsq() and scx_bpf_destroy_dsq().
*Scheduling cycle*
The following briefly shows a typical workflow for how a waking task is
scheduled and executed.
1. When a task is waking up, .select_cpu() is the first operation invoked.
This serves two purposes. It both allows a scheduler to optimize task
placement by specifying a CPU where it expects the task to eventually be
scheduled, and the latter is that the selected CPU will be woken if it’s
idle.
2. Once the target CPU is selected, .enqueue() is invoked. It can make one of
the following decisions:
- Immediately dispatch the task to either the global dsq (SCX_DSQ_GLOBAL)
or the current CPU’s local dsq (SCX_DSQ_LOCAL).
- Immediately dispatch the task to a user-created dispatch queue.
- Queue the task on the BPF side, e.g. in an rbtree map for a vruntime
scheduler, with the intention of dispatching it at a later time from
.dispatch().
3. When a CPU is ready to schedule, it first looks at its local dsq. If empty,
it invokes .consume() which should make one or more scx_bpf_consume() calls
to consume tasks from dsq's. If a scx_bpf_consume() call succeeds, the CPU
has the next task to run and .consume() can return. If .consume() is not
defined, sched_ext will by-default consume from only the built-in
SCX_DSQ_GLOBAL dsq.
4. If there's still no task to run, .dispatch() is invoked which should make
one or more scx_bpf_dispatch() calls to dispatch tasks from the BPF
scheduler to one of the dsq's. If more than one task has been dispatched,
go back to the previous consumption step.
*Verifying callback behavior*
sched_ext always verifies that any value returned from a callback is valid,
and will issue an error and unload the scheduler if it is not. For example, if
.select_cpu() returns an invalid CPU, or if an attempt is made to invoke the
scx_bpf_dispatch() with invalid enqueue flags. Furthermore, if a task remains
runnable for too long without being scheduled, sched_ext will detect it and
error-out the scheduler.
Closing Thoughts
----------------
Both Meta and Google have experimented quite a lot with schedulers in the
last several years. Google has benchmarked various workloads using user
space scheduling, and have achieved performance wins by trading off
generality for application specific needs. At Meta, we are actively
experimenting with multiple production workloads and seeing significant
performance gains, and are in the process of deploying sched_ext schedulers
on production workloads at scale. We expect to leverage it extensively to
run various experiments and develop customized schedulers for a number of
critical workloads.
In closing, both Meta and Google believe that sched_ext will significantly
evolve how the broader community explores the scheduling problem space,
while also enabling targeted policies for custom applications. We’ll be able
to experiment easier and faster, explore uncharted areas, and deploy
emergency scheduler changes when necessary. The same applies to anyone who
wants to work on the scheduler, including academia and specialized
industries. sched_ext will push forward the state of the art when it comes
to scheduling and performance in Linux.
Written By
----------
David Vernet <dvernet@meta.com>
Josh Don <joshdon@google.com>
Tejun Heo <tj@kernel.org>
Barret Rhoden <brho@google.com>
Supported By
------------
Paul Turner <pjt@google.com>
Neel Natu <neelnatu@google.com>
Patrick Bellasi <derkling@google.com>
Hao Luo <haoluo@google.com>
Dimitrios Skarlatos <dskarlat@cs.cmu.edu>
Patchset
--------
This patchset is on top of tip/sched/core + bpf/for-next as of 2024-06-24:
c793a62823d1 ("sched/core: Drop spinlocks on contention iff kernel is preemptible")
f6afdaf72af7 ("Merge branch 'bpf-support-resilient-split-btf'")
and contains the following patches:
NOTE: The doc added by 0029 contains a high-level overview and might be good
place to start.
0001-sched-Restructure-sched_class-order-sanity-checks-in.patch
0002-sched-Allow-sched_cgroup_fork-to-fail-and-introduce-.patch
0003-sched-Add-sched_class-reweight_task.patch
0004-sched-Add-sched_class-switching_to-and-expose-check_.patch
0005-sched-Factor-out-cgroup-weight-conversion-functions.patch
0006-sched-Factor-out-update_other_load_avgs-from-__updat.patch
0007-sched-Add-normal_policy.patch
0008-sched_ext-Add-boilerplate-for-extensible-scheduler-c.patch
0009-sched_ext-Implement-BPF-extensible-scheduler-class.patch
0010-sched_ext-Add-scx_simple-and-scx_example_qmap-exampl.patch
0011-sched_ext-Add-sysrq-S-which-disables-the-BPF-schedul.patch
0012-sched_ext-Implement-runnable-task-stall-watchdog.patch
0013-sched_ext-Allow-BPF-schedulers-to-disallow-specific-.patch
0014-sched_ext-Print-sched_ext-info-when-dumping-stack.patch
0015-sched_ext-Print-debug-dump-after-an-error-exit.patch
0016-tools-sched_ext-Add-scx_show_state.py.patch
0017-sched_ext-Implement-scx_bpf_kick_cpu-and-task-preemp.patch
0018-sched_ext-Add-a-central-scheduler-which-makes-all-sc.patch
0019-sched_ext-Make-watchdog-handle-ops.dispatch-looping-.patch
0020-sched_ext-Add-task-state-tracking-operations.patch
0021-sched_ext-Implement-tickless-support.patch
0022-sched_ext-Track-tasks-that-are-subjects-of-the-in-fl.patch
0023-sched_ext-Implement-SCX_KICK_WAIT.patch
0024-sched_ext-Implement-sched_ext_ops.cpu_acquire-releas.patch
0025-sched_ext-Implement-sched_ext_ops.cpu_online-offline.patch
0026-sched_ext-Bypass-BPF-scheduler-while-PM-events-are-i.patch
0027-sched_ext-Implement-core-sched-support.patch
0028-sched_ext-Add-vtime-ordered-priority-queue-to-dispat.patch
0029-sched_ext-Documentation-scheduler-Document-extensibl.patch
0030-sched_ext-Add-selftests.patch
0001-0007: Scheduler prep.
0008-0010: sched_ext core implementation and a couple example BPF scheduler.
0011-0016: Utility features including safety mechanisms, switch-all and
printing sched_ext state when dumping backtraces.
0017-0023: Kicking and preempting other CPUs, task state transition tracking
and tickless support. Demonstrated with an example central
scheduler which makes all scheduling decisions on one CPU.
0024-0026: Add CPU preemption and hotplug and power-management support.
0027 : Add core-sched support.
0028 : Add DSQ rbtree support.
0029-0030: Add documentation and selftests.
This patchset is applied to the following git branch:
git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext.git for-6.11
diffstat follows.
Documentation/scheduler/index.rst | 1
Documentation/scheduler/sched-ext.rst | 314 +
MAINTAINERS | 13
Makefile | 8
drivers/tty/sysrq.c | 1
include/asm-generic/vmlinux.lds.h | 1
include/linux/cgroup.h | 4
include/linux/sched.h | 5
include/linux/sched/ext.h | 203 +
include/linux/sched/task.h | 3
include/trace/events/sched_ext.h | 32
include/uapi/linux/sched.h | 1
init/init_task.c | 12
kernel/Kconfig.preempt | 26
kernel/fork.c | 17
kernel/sched/build_policy.c | 10
kernel/sched/core.c | 187
kernel/sched/debug.c | 3
kernel/sched/ext.c | 6128 +++++++++++++++++++++++++++++++
kernel/sched/ext.h | 114
kernel/sched/fair.c | 21
kernel/sched/idle.c | 2
kernel/sched/sched.h | 75
kernel/sched/syscalls.c | 26
lib/dump_stack.c | 1
tools/Makefile | 10
tools/sched_ext/.gitignore | 2
tools/sched_ext/Makefile | 246 +
tools/sched_ext/README.md | 258 +
tools/sched_ext/include/bpf-compat/gnu/stubs.h | 11
tools/sched_ext/include/scx/common.bpf.h | 394 +
tools/sched_ext/include/scx/common.h | 75
tools/sched_ext/include/scx/compat.bpf.h | 28
tools/sched_ext/include/scx/compat.h | 186
tools/sched_ext/include/scx/user_exit_info.h | 111
tools/sched_ext/scx_central.bpf.c | 361 +
tools/sched_ext/scx_central.c | 135
tools/sched_ext/scx_qmap.bpf.c | 524 ++
tools/sched_ext/scx_qmap.c | 131
tools/sched_ext/scx_show_state.py | 39
tools/sched_ext/scx_simple.bpf.c | 156
tools/sched_ext/scx_simple.c | 107
tools/testing/selftests/sched_ext/.gitignore | 6
tools/testing/selftests/sched_ext/Makefile | 218 +
tools/testing/selftests/sched_ext/config | 9
tools/testing/selftests/sched_ext/create_dsq.bpf.c | 58
tools/testing/selftests/sched_ext/create_dsq.c | 57
tools/testing/selftests/sched_ext/ddsp_bogus_dsq_fail.bpf.c | 42
tools/testing/selftests/sched_ext/ddsp_bogus_dsq_fail.c | 57
tools/testing/selftests/sched_ext/ddsp_vtimelocal_fail.bpf.c | 39
tools/testing/selftests/sched_ext/ddsp_vtimelocal_fail.c | 56
tools/testing/selftests/sched_ext/dsp_local_on.bpf.c | 65
tools/testing/selftests/sched_ext/dsp_local_on.c | 58
tools/testing/selftests/sched_ext/enq_last_no_enq_fails.bpf.c | 21
tools/testing/selftests/sched_ext/enq_last_no_enq_fails.c | 60
tools/testing/selftests/sched_ext/enq_select_cpu_fails.bpf.c | 43
tools/testing/selftests/sched_ext/enq_select_cpu_fails.c | 61
tools/testing/selftests/sched_ext/exit.bpf.c | 84
tools/testing/selftests/sched_ext/exit.c | 55
tools/testing/selftests/sched_ext/exit_test.h | 20
tools/testing/selftests/sched_ext/hotplug.bpf.c | 61
tools/testing/selftests/sched_ext/hotplug.c | 168
tools/testing/selftests/sched_ext/hotplug_test.h | 15
tools/testing/selftests/sched_ext/init_enable_count.bpf.c | 53
tools/testing/selftests/sched_ext/init_enable_count.c | 166
tools/testing/selftests/sched_ext/maximal.bpf.c | 132
tools/testing/selftests/sched_ext/maximal.c | 51
tools/testing/selftests/sched_ext/maybe_null.bpf.c | 36
tools/testing/selftests/sched_ext/maybe_null.c | 49
tools/testing/selftests/sched_ext/maybe_null_fail_dsp.bpf.c | 25
tools/testing/selftests/sched_ext/maybe_null_fail_yld.bpf.c | 28
tools/testing/selftests/sched_ext/minimal.bpf.c | 21
tools/testing/selftests/sched_ext/minimal.c | 58
tools/testing/selftests/sched_ext/prog_run.bpf.c | 32
tools/testing/selftests/sched_ext/prog_run.c | 78
tools/testing/selftests/sched_ext/reload_loop.c | 75
tools/testing/selftests/sched_ext/runner.c | 201 +
tools/testing/selftests/sched_ext/scx_test.h | 131
tools/testing/selftests/sched_ext/select_cpu_dfl.bpf.c | 40
tools/testing/selftests/sched_ext/select_cpu_dfl.c | 72
tools/testing/selftests/sched_ext/select_cpu_dfl_nodispatch.bpf.c | 89
tools/testing/selftests/sched_ext/select_cpu_dfl_nodispatch.c | 72
tools/testing/selftests/sched_ext/select_cpu_dispatch.bpf.c | 41
tools/testing/selftests/sched_ext/select_cpu_dispatch.c | 70
tools/testing/selftests/sched_ext/select_cpu_dispatch_bad_dsq.bpf.c | 37
tools/testing/selftests/sched_ext/select_cpu_dispatch_bad_dsq.c | 56
tools/testing/selftests/sched_ext/select_cpu_dispatch_dbl_dsp.bpf.c | 38
tools/testing/selftests/sched_ext/select_cpu_dispatch_dbl_dsp.c | 56
tools/testing/selftests/sched_ext/select_cpu_vtime.bpf.c | 92
tools/testing/selftests/sched_ext/select_cpu_vtime.c | 59
tools/testing/selftests/sched_ext/test_example.c | 49
tools/testing/selftests/sched_ext/util.c | 71
tools/testing/selftests/sched_ext/util.h | 13
93 files changed, 13160 insertions(+), 66 deletions(-)
Patchset History
----------------
v5 (http://lkml.kernel.org/r/20231111024835.2164816-1-tj@kernel.org) -> v6:
- scx_pair, scx_userland, scx_next and the rust schedulers are removed from
kernel tree and now hosted in https://github.com/sched-ext/scx along with
all other schedulers.
- SCX_OPS_DISABLING state is replaced with the new bypass mechanism which
allows temporarily putting the system into simple FIFO scheduling mode. In
addition to the shut down path, this is used to isolate the BPF scheduler
across power management events.
- ops.prep_enable() is replaced with ops.init_task() and
ops.enable/disable() are now called whenever the task enters and leaves
sched_ext instead of when the task becomes schedulable on sched_ext and
stops being so. A new operation - ops.exit_task() - is called when the
task stops being schedulable on sched_ext.
- scx_bpf_dispatch() can now be called from ops.select_cpu() too. This
removes the need for communicating local dispatch decision made by
ops.select_cpu() to ops.enqueue() via per-task storage.
- SCX_TASK_ENQ_LOCAL which told the BPF scheduler that scx_select_cpu_dfl()
wants the task to be dispatched to the local DSQ was removed. Instead,
scx_bpf_select_cpu_dfl() now dispatches directly if it finds a suitable
idle CPU. If such behavior is not desired, users can use
scx_bpf_select_cpu_dfl() which returns the verdict in a bool out param.
- Dispatch decisions made in ops.dispatch() may now be cancelled with a new
scx_bpf_dispatch_cancel() kfunc.
- A new SCX_KICK_IDLE flag is available for use with scx_bpf_kick_cpu() to
only send a resched IPI if the target CPU is idle.
- exit_code added to scx_exit_info. This is used to indicate different exit
conditions on non-error exits and enables e.g. handling CPU hotplugs by
restarting the scheduler.
- Debug dump added. When the BPF scheduler gets aborted, the states of all
runqueues and runnable tasks are captured and sent to the scheduler binary
to aid debugging. See https://github.com/sched-ext/scx/issues/234 for an
example of the debug dump being used to root cause a bug in scx_lavd.
- The BPF scheduler can now iterate DSQs and consume specific tasks.
- CPU frequency scaling support added through cpuperf kfunc interface.
- The current state of sched_ext can now be monitored through files under
/sys/sched_ext instead of /sys/kernel/debug/sched/ext. This is to enable
monitoring on kernels which don't enable debugfs. A drgn script
tools/sched_ext/scx_show_state.py is added for additional visibility.
- tools/sched_ext/include/scx/compat[.bpf].h and other facilities to allow
schedulers to be loaded on older kernels are added. The current tentative
target is maintaining backward compatibility for at least one major kernel
release where reasonable.
- Code reorganized so that only the parts necessary to integrate with the
rest of the kernel are in the header files.
v4 (http://lkml.kernel.org/r/20230711011412.100319-1-tj@kernel.org) -> v5:
- Updated to rebase on top of the current bpf/for-next (2023-11-06).
'0002-0010: Scheduler prep' were simply rebased on top of new EEVDF
scheduler which demonstrate clean cut API boundary between sched-ext and
sched core.
- To accommodate 32bit configs, fields which use atomic ops and
store_release/load_acquire are switched from 64bits to longs.
- To help triaging, if sched_ext is enabled, backtrace dumps now show the
currently active scheduler along with some debug information.
- Fixes for bugs including p->scx.flags corruption due to unsynchronized
SCX_TASK_DSQ_ON_PRIQ changes, and overly permissive BTF struct and scx_bpf
kfunc access checks.
- Other misc changes including renaming "type" to "kind" in scx_exit_info to
ease usage from rust and other languages in which "type" is a reserved
keyword.
- scx_atropos is renamed to scx_rusty and received signficant updates to
improve scalability. Load metrics are now tracked in BPF and accessed only
as necessary from userspace.
- Misc example scheduler improvements including the usage of resizable BPF
.bss array, the introduction of SCX_BUG[_ON](), and timer CPU pinning in
scx_central.
- Improve Makefile and documentation for example schedulers.
v3 (https://lkml.kernel.org/r/20230317213333.2174969-1-tj@kernel.org) -> v4:
- There aren't any significant changes to the sched_ext API even though we
kept experimenting heavily with a couple BPF scheduler implementations
indicating that the core API reached a level of maturity.
- 0002-sched-Encapsulate-task-attribute-change-sequence-int.patch which
implemented custom guard scope for scheduler attribute changes dropped as
upstream is moving towards a more generic implementation.
- Build fixes with different CONFIG combinations.
- Core code cleanups and improvements including how idle CPU is selected and
disabling ttwu_queue for tasks on SCX to avoid confusing BPF schedulers
expecting ->select_cpu() call. See
0012-sched_ext-Implement-BPF-extensible-scheduler-class.patch for more
details.
- "_example" dropped from the example schedulers as the distinction between
the example-only and practically-useful isn't black-and-white. Instead,
each scheduler has detailed comments and there's also a README file.
- scx_central, scx_pair and scx_flatcg are moved into their own patches as
suggested by Josh Don.
- scx_atropos received sustantial updates including fixes for bugs that
could cause temporary stalls and improvements in load balancing and wakeup
target CPU selection. For details, See
0034-sched_ext-Add-a-rust-userspace-hybrid-example-schedu.patch.
v2 (http://lkml.kernel.org/r/20230128001639.3510083-1-tj@kernel.org) -> v3:
- ops.set_weight() added to allow BPF schedulers to track weight changes
without polling p->scx.weight.
- scx_bpf_task_cgroup() kfunc added to allow BPF scheduler to reliably
determine the current cpu cgroup under rq lock protection. This required
improving the kf_mask SCX operation verification mechanism and adding
0023-sched_ext-Track-tasks-that-are-subjects-of-the-in-fl.patch.
- Updated to use the latest BPF improvements including KF_RCU and the inline
iterator.
- scx_example_flatcg added to 0024-sched_ext-Add-cgroup-support.patch. It
uses the new BPF RB tree support to implement flattened cgroup hierarchy.
- A DSQ now also contains an rbtree so that it can be used to implement
vtime based scheduling among tasks sharing a DSQ conveniently and
efficiently. For more details, see
0029-sched_ext-Add-vtime-ordered-priority-queue-to-dispat.patch. All
eligible example schedulers are updated to default to weighted vtime
scheduilng.
- atropos scheduler's userspace code is substantially restructred and
rewritten. The binary is renamed to scx_atropos and can auto-config the
domains according to the cache topology.
- Various other example scheduler updates including scx_example_dummy being
renamed to scx_example_simple, the example schedulers defaulting to
enabling switch_all and clarifying performance expectation of each example
scheduler.
- A bunch of fixes and improvements. Please refer to each patch for details.
v1 (http://lkml.kernel.org/r/20221130082313.3241517-1-tj@kernel.org) -> v2:
- Rebased on top of bpf/for-next - a5f6b9d577eb ("Merge branch 'Enable
struct_ops programs to be sleepable'"). There were several missing
features including generic cpumask helpers and sleepable struct_ops
operation support that v1 was working around. The rebase gets rid of all
SCX specific temporary helpers.
- Some kfunc helpers are context-sensitive and can only be called from
specific operations. v1 didn't restrict kfunc accesses allowing them to be
misused which can lead to crashes and other malfunctions. v2 makes more
kfuncs safe to be called from anywhere and implements per-task mask based
runtime access control for the rest. The longer-term plan is to make the
BPF verifier enforce these restrictions. Combined with the above, sans
mistakes and bugs, it shouldn't be possible to crash the machine through
SCX and its helpers.
- Core-sched support. While v1 implemented the pick_task operation, there
were multiple missing pieces for working core-sched support. v2 adds
0027-sched_ext-Implement-core-sched-support.patch. SCX by default
implements global FIFO ordering and allows the BPF schedulers to implement
custom ordering via scx_ops.core_sched_before(). scx_example_qmap is
updated so that the five queues' relative priorities are correctly
reflected when core-sched is enabled.
- Dropped balance_scx_on_up() which was called from put_prev_task_balance().
UP support is now contained in SCX proper.
- 0002-sched-Encapsulate-task-attribute-change-sequence-int.patch adds
SCHED_CHANGE_BLOCK() which encapsulates the preparation and restoration
sequences used for task attribute changes. For SCX, this replaces
sched_deq_and_put_task() and sched_enq_and_set_task() from v1.
- 0011-sched-Add-reason-to-sched_move_task.patch dropped from v1. SCX now
distinguishes cgroup and autogroup tg's using task_group_is_autogroup().
- Other misc changes including fixes for bugs that Julia Lawall noticed and
patch descriptions updates with more details on how the introduced changes
are going to be used.
- MAINTAINERS entries added.
The followings are discussion points which were raised but didn't result in
code changes in this iteration.
- There were discussions around exposing __setscheduler_prio() and, in v2,
SCHED_CHANGE_BLOCK() in kernel/sched/sched.h. Switching scheduler
implementations is innate for SCX. At the very least, it needs to be able
to turn on and off the BPF scheduler which requires something equivalent
to SCHED_CHANGE_BLOCK(). The use of __setscheduler_prio() depends on the
behavior we want to present to userspace. The current one of using CFS as
the fallback when BPF scheduler is not available seems more friendly and
less error-prone to other options.
- Another discussion point was around for_each_active_class() and friends
which skip over CFS or SCX when it's known that the sched_class must be
empty. I left it as-is for now as it seems to be cleaner and more robust
than trying to plug each operation which may added unnecessary overheads.
Thanks.
--
tejun
Diff
No diff found.
Implementation Analysis
Overview
This is the cover letter for the 30-patch v7 series that introduced sched_ext into Linux 6.11. It is not a code patch — there is no diff. Its purpose is to establish the design rationale, describe the overall architecture, and document what changed from v6. Reading this cover letter before any code patch is the correct starting point for understanding why every subsequent decision was made the way it was.
The series was the result of a multi-year collaboration between Meta (Tejun Heo, David Vernet) and Google (Josh Don, Barret Rhoden) and went through seven public revision rounds before being accepted.
Architecture Context
sched_ext introduces ext_sched_class, a new Linux scheduler class that sits between fair_sched_class and idle_sched_class in the class hierarchy. Its defining characteristic is that the actual scheduling policy is provided at runtime by a BPF program rather than being fixed in the kernel. The BPF program implements struct sched_ext_ops, a set of function-pointer callbacks that mirror (but simplify) the struct sched_class interface.
The three axes the cover letter uses to justify the proposal are worth internalizing because they map directly to the design constraints you will see enforced throughout the code:
-
Experimentation speed: Uploading a new BPF scheduler takes milliseconds; rebuilding and rebooting a kernel takes hours. sched_ext enables the iteration loop that kernel scheduler research requires.
-
Application-specific customization: CFS is a general-purpose scheduler. Some workloads (VM scheduling, latency-sensitive services, ARINC 653 avionics) have constraints that are not generalizable. sched_ext allows those policies to be expressed without modifying the kernel.
-
Rapid production deployment: Rolling a new scheduler policy to a large fleet via a BPF program update is orders of magnitude faster than a kernel upgrade. The L1TF example in the cover letter is the canonical case: organizations needed core-scheduling semantics quickly and could not wait for an upstream kernel cycle.
Code Walkthrough
There is no diff. The cover letter is a design document. However, it describes the overall control flow precisely enough that it is worth mapping to code you will later see in the series:
Dispatch Queues (DSQs): The cover letter introduces these as "simple FIFOs that bridge the workflow imbalance between the scheduler core and sched_ext_ops callbacks." In the implementation you will find three built-in DSQs: SCX_DSQ_GLOBAL (a global FIFO any CPU can consume from), SCX_DSQ_LOCAL (a per-CPU FIFO consumed only by that CPU), and SCX_DSQ_LOCAL_ON (dispatch to a specific CPU's local DSQ). BPF schedulers can also create arbitrary custom DSQs via scx_bpf_create_dsq().
The scheduling cycle, as described in the cover letter, is:
ops.select_cpu()— choose a target CPU on wakeup and optionally wake itops.enqueue()— dispatch the task to a DSQ or hold it in BPF-side data structuresops.consume()— a CPU needing work callsscx_bpf_consume()to pull from a DSQops.dispatch()— if still no task, BPF dispatches tasks it was holding
Safety mechanisms: The cover letter explicitly calls out two mechanisms that will be implemented in later patches: the watchdog (PATCH 12/30, patch-11.md in this study) which aborts the BPF scheduler if any SCX task stays runnable longer than timeout_ms, and sysrq-S (PATCH 11/30, patch-10.md) which gives a system administrator an emergency escape hatch to revert all tasks to CFS without rebooting.
Key Concepts Introduced
struct sched_ext_ops: The BPF-visible interface. Uses the BPF_STRUCT_OPS mechanism (the same mechanism used for TCP congestion control in tcp_bbr). Only the .name field is required; all callbacks have sensible defaults. This is intentional — a BPF scheduler that implements nothing still provides a working (global FIFO) scheduler.
BPF struct_ops with no ABI stability: Unlike UAPI headers, sched_ext_ops carries no ABI guarantee. The cover letter states this explicitly. This is a deliberate trade-off: it allows the kernel to evolve the interface as scheduling needs change, at the cost of requiring BPF schedulers to track kernel versions. The tools/sched_ext/include/scx/compat.bpf.h helpers added in this patch series are the practical response to this constraint.
v7 scope reduction: The cover letter documents exactly what was dropped from v6 to keep the initial merge tractable: cgroup support (5 patches), cpufreq/cpuperf support (2 patches), and the DSQ iterator (1 patch). These were separated into follow-on patchsets. Understanding what was intentionally excluded from v7 is important for reading the code — you will encounter stubs and forward references whose implementations live in those follow-on series.
Why This Matters for Maintainers
The cover letter establishes the invariants that the entire codebase is designed to enforce:
-
System integrity is unconditional: BPF safety guarantees and the watchdog/sysrq mechanisms mean a buggy BPF scheduler cannot permanently damage the system. Maintainers must preserve these guarantees when reviewing any change to the disable/fallback path.
-
The ops interface has no ABI: Any change to
struct sched_ext_opsis permitted but must be documented in the cover letter of the patchset making the change. Downstream BPF scheduler authors (in thesched-ext/scxrepo) need to be notified. -
GPL-only: BPF programs are required to be GPLv2. The verifier enforces this at program load time. This is a hard constraint, not a convention.
-
The "only .name is required" invariant: Defaults must remain sensible. A scheduler that implements only
.namemust continue to be a functional (if not efficient) scheduler. Any patch that breaks this invariant is a regression.
Connection to Other Patches
This cover letter is the entry point for the entire series. The patch numbering it lists maps to:
- PATCH 08/30 (patch-09.md): The boilerplate that creates the file and symbol scaffolding with
#error "NOT IMPLEMENTED YET"guards. - PATCH 09/30: The full
ext_sched_classimplementation — the core of the series. - PATCH 10/30 (patch-15.md):
scx_simpleandscx_qmapexample schedulers that demonstrate the API. - PATCH 11/30 (patch-10.md): sysrq-S emergency escape hatch.
- PATCH 12/30 (patch-11.md): Watchdog for runnable task stall detection.
- PATCH 29/30: The RST documentation, which the cover letter recommends reading first for high-level orientation.
The cover letter's description of what was dropped from v6 also explains why cgroup-aware and cpufreq-aware scheduling are absent from the v7 implementation you will study here.
[PATCH 08/30] sched_ext: Add boilerplate for extensible scheduler class
View on Lore: https://lore.kernel.org/all/20240618212056.2833381-9-tj@kernel.org
Commit Message
This adds dummy implementations of sched_ext interfaces which interact with
the scheduler core and hook them in the correct places. As they're all
dummies, this doesn't cause any behavior changes. This is split out to help
reviewing.
v2: balance_scx_on_up() dropped. This will be handled in sched_ext proper.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: David Vernet <dvernet@meta.com>
Acked-by: Josh Don <joshdon@google.com>
Acked-by: Hao Luo <haoluo@google.com>
Acked-by: Barret Rhoden <brho@google.com>
---
include/linux/sched/ext.h | 12 ++++++++++++
kernel/fork.c | 2 ++
kernel/sched/core.c | 32 ++++++++++++++++++++++++--------
kernel/sched/ext.h | 24 ++++++++++++++++++++++++
kernel/sched/idle.c | 2 ++
kernel/sched/sched.h | 2 ++
6 files changed, 66 insertions(+), 8 deletions(-)
create mode 100644 include/linux/sched/ext.h
create mode 100644 kernel/sched/ext.h
diff --git a/include/linux/sched/ext.h b/include/linux/sched/ext.h
new file mode 100644
index 000000000000..a05dfcf533b0
--- /dev/null
+++ b/include/linux/sched/ext.h
@@ -0,0 +1,12 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_SCHED_EXT_H
+#define _LINUX_SCHED_EXT_H
+
+#ifdef CONFIG_SCHED_CLASS_EXT
+#error "NOT IMPLEMENTED YET"
+#else /* !CONFIG_SCHED_CLASS_EXT */
+
+static inline void sched_ext_free(struct task_struct *p) {}
+
+#endif /* CONFIG_SCHED_CLASS_EXT */
+#endif /* _LINUX_SCHED_EXT_H */
diff --git a/kernel/fork.c b/kernel/fork.c
index e601fdf787c3..741d962db0d9 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -23,6 +23,7 @@
#include <linux/sched/task.h>
#include <linux/sched/task_stack.h>
#include <linux/sched/cputime.h>
+#include <linux/sched/ext.h>
#include <linux/seq_file.h>
#include <linux/rtmutex.h>
#include <linux/init.h>
@@ -971,6 +972,7 @@ void __put_task_struct(struct task_struct *tsk)
WARN_ON(refcount_read(&tsk->usage));
WARN_ON(tsk == current);
+ sched_ext_free(tsk);
io_uring_free(tsk);
cgroup_free(tsk);
task_numa_free(tsk, true);
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 0bfbceebc4e9..d8c963fea9eb 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -4559,6 +4559,8 @@ late_initcall(sched_core_sysctl_init);
*/
int sched_fork(unsigned long clone_flags, struct task_struct *p)
{
+ int ret;
+
__sched_fork(clone_flags, p);
/*
* We mark the process as NEW here. This guarantees that
@@ -4595,12 +4597,16 @@ int sched_fork(unsigned long clone_flags, struct task_struct *p)
p->sched_reset_on_fork = 0;
}
- if (dl_prio(p->prio))
- return -EAGAIN;
- else if (rt_prio(p->prio))
+ scx_pre_fork(p);
+
+ if (dl_prio(p->prio)) {
+ ret = -EAGAIN;
+ goto out_cancel;
+ } else if (rt_prio(p->prio)) {
p->sched_class = &rt_sched_class;
- else
+ } else {
p->sched_class = &fair_sched_class;
+ }
init_entity_runnable_average(&p->se);
@@ -4618,6 +4624,10 @@ int sched_fork(unsigned long clone_flags, struct task_struct *p)
RB_CLEAR_NODE(&p->pushable_dl_tasks);
#endif
return 0;
+
+out_cancel:
+ scx_cancel_fork(p);
+ return ret;
}
int sched_cgroup_fork(struct task_struct *p, struct kernel_clone_args *kargs)
@@ -4648,16 +4658,18 @@ int sched_cgroup_fork(struct task_struct *p, struct kernel_clone_args *kargs)
p->sched_class->task_fork(p);
raw_spin_unlock_irqrestore(&p->pi_lock, flags);
- return 0;
+ return scx_fork(p);
}
void sched_cancel_fork(struct task_struct *p)
{
+ scx_cancel_fork(p);
}
void sched_post_fork(struct task_struct *p)
{
uclamp_post_fork(p);
+ scx_post_fork(p);
}
unsigned long to_ratio(u64 period, u64 runtime)
@@ -5800,7 +5812,7 @@ static void put_prev_task_balance(struct rq *rq, struct task_struct *prev,
* We can terminate the balance pass as soon as we know there is
* a runnable task of @class priority or higher.
*/
- for_class_range(class, prev->sched_class, &idle_sched_class) {
+ for_balance_class_range(class, prev->sched_class, &idle_sched_class) {
if (class->balance(rq, prev, rf))
break;
}
@@ -5818,6 +5830,9 @@ __pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
const struct sched_class *class;
struct task_struct *p;
+ if (scx_enabled())
+ goto restart;
+
/*
* Optimization: we know that if all tasks are in the fair class we can
* call that function directly, but only if the @prev task wasn't of a
@@ -5858,7 +5873,7 @@ __pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
if (prev->dl_server)
prev->dl_server = NULL;
- for_each_class(class) {
+ for_each_active_class(class) {
p = class->pick_next_task(rq);
if (p)
return p;
@@ -5891,7 +5906,7 @@ static inline struct task_struct *pick_task(struct rq *rq)
const struct sched_class *class;
struct task_struct *p;
- for_each_class(class) {
+ for_each_active_class(class) {
p = class->pick_task(rq);
if (p)
return p;
@@ -8355,6 +8370,7 @@ void __init sched_init(void)
balance_push_set(smp_processor_id(), false);
#endif
init_sched_fair_class();
+ init_sched_ext_class();
psi_init();
diff --git a/kernel/sched/ext.h b/kernel/sched/ext.h
new file mode 100644
index 000000000000..6a93c4825339
--- /dev/null
+++ b/kernel/sched/ext.h
@@ -0,0 +1,24 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+#ifdef CONFIG_SCHED_CLASS_EXT
+#error "NOT IMPLEMENTED YET"
+#else /* CONFIG_SCHED_CLASS_EXT */
+
+#define scx_enabled() false
+
+static inline void scx_pre_fork(struct task_struct *p) {}
+static inline int scx_fork(struct task_struct *p) { return 0; }
+static inline void scx_post_fork(struct task_struct *p) {}
+static inline void scx_cancel_fork(struct task_struct *p) {}
+static inline void init_sched_ext_class(void) {}
+
+#define for_each_active_class for_each_class
+#define for_balance_class_range for_class_range
+
+#endif /* CONFIG_SCHED_CLASS_EXT */
+
+#if defined(CONFIG_SCHED_CLASS_EXT) && defined(CONFIG_SMP)
+#error "NOT IMPLEMENTED YET"
+#else
+static inline void scx_update_idle(struct rq *rq, bool idle) {}
+#endif
diff --git a/kernel/sched/idle.c b/kernel/sched/idle.c
index 6e78d071beb5..c7a218123b7a 100644
--- a/kernel/sched/idle.c
+++ b/kernel/sched/idle.c
@@ -452,11 +452,13 @@ static void wakeup_preempt_idle(struct rq *rq, struct task_struct *p, int flags)
static void put_prev_task_idle(struct rq *rq, struct task_struct *prev)
{
+ scx_update_idle(rq, false);
}
static void set_next_task_idle(struct rq *rq, struct task_struct *next, bool first)
{
update_idle_core(rq);
+ scx_update_idle(rq, true);
schedstat_inc(rq->sched_goidle);
}
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 25660dc9f639..c52ad5fdd096 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -3658,4 +3658,6 @@ static inline void balance_callbacks(struct rq *rq, struct balance_callback *hea
#endif
+#include "ext.h"
+
#endif /* _KERNEL_SCHED_SCHED_H */
--
2.45.2
Diff
---
include/linux/sched/ext.h | 12 ++++++++++++
kernel/fork.c | 2 ++
kernel/sched/core.c | 32 ++++++++++++++++++++++++--------
kernel/sched/ext.h | 24 ++++++++++++++++++++++++
kernel/sched/idle.c | 2 ++
kernel/sched/sched.h | 2 ++
6 files changed, 66 insertions(+), 8 deletions(-)
create mode 100644 include/linux/sched/ext.h
create mode 100644 kernel/sched/ext.h
diff --git a/include/linux/sched/ext.h b/include/linux/sched/ext.h
new file mode 100644
index 000000000000..a05dfcf533b0
--- /dev/null
+++ b/include/linux/sched/ext.h
@@ -0,0 +1,12 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_SCHED_EXT_H
+#define _LINUX_SCHED_EXT_H
+
+#ifdef CONFIG_SCHED_CLASS_EXT
+#error "NOT IMPLEMENTED YET"
+#else /* !CONFIG_SCHED_CLASS_EXT */
+
+static inline void sched_ext_free(struct task_struct *p) {}
+
+#endif /* CONFIG_SCHED_CLASS_EXT */
+#endif /* _LINUX_SCHED_EXT_H */
diff --git a/kernel/fork.c b/kernel/fork.c
index e601fdf787c3..741d962db0d9 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -23,6 +23,7 @@
#include <linux/sched/task.h>
#include <linux/sched/task_stack.h>
#include <linux/sched/cputime.h>
+#include <linux/sched/ext.h>
#include <linux/seq_file.h>
#include <linux/rtmutex.h>
#include <linux/init.h>
@@ -971,6 +972,7 @@ void __put_task_struct(struct task_struct *tsk)
WARN_ON(refcount_read(&tsk->usage));
WARN_ON(tsk == current);
+ sched_ext_free(tsk);
io_uring_free(tsk);
cgroup_free(tsk);
task_numa_free(tsk, true);
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 0bfbceebc4e9..d8c963fea9eb 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -4559,6 +4559,8 @@ late_initcall(sched_core_sysctl_init);
*/
int sched_fork(unsigned long clone_flags, struct task_struct *p)
{
+ int ret;
+
__sched_fork(clone_flags, p);
/*
* We mark the process as NEW here. This guarantees that
@@ -4595,12 +4597,16 @@ int sched_fork(unsigned long clone_flags, struct task_struct *p)
p->sched_reset_on_fork = 0;
}
- if (dl_prio(p->prio))
- return -EAGAIN;
- else if (rt_prio(p->prio))
+ scx_pre_fork(p);
+
+ if (dl_prio(p->prio)) {
+ ret = -EAGAIN;
+ goto out_cancel;
+ } else if (rt_prio(p->prio)) {
p->sched_class = &rt_sched_class;
- else
+ } else {
p->sched_class = &fair_sched_class;
+ }
init_entity_runnable_average(&p->se);
@@ -4618,6 +4624,10 @@ int sched_fork(unsigned long clone_flags, struct task_struct *p)
RB_CLEAR_NODE(&p->pushable_dl_tasks);
#endif
return 0;
+
+out_cancel:
+ scx_cancel_fork(p);
+ return ret;
}
int sched_cgroup_fork(struct task_struct *p, struct kernel_clone_args *kargs)
@@ -4648,16 +4658,18 @@ int sched_cgroup_fork(struct task_struct *p, struct kernel_clone_args *kargs)
p->sched_class->task_fork(p);
raw_spin_unlock_irqrestore(&p->pi_lock, flags);
- return 0;
+ return scx_fork(p);
}
void sched_cancel_fork(struct task_struct *p)
{
+ scx_cancel_fork(p);
}
void sched_post_fork(struct task_struct *p)
{
uclamp_post_fork(p);
+ scx_post_fork(p);
}
unsigned long to_ratio(u64 period, u64 runtime)
@@ -5800,7 +5812,7 @@ static void put_prev_task_balance(struct rq *rq, struct task_struct *prev,
* We can terminate the balance pass as soon as we know there is
* a runnable task of @class priority or higher.
*/
- for_class_range(class, prev->sched_class, &idle_sched_class) {
+ for_balance_class_range(class, prev->sched_class, &idle_sched_class) {
if (class->balance(rq, prev, rf))
break;
}
@@ -5818,6 +5830,9 @@ __pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
const struct sched_class *class;
struct task_struct *p;
+ if (scx_enabled())
+ goto restart;
+
/*
* Optimization: we know that if all tasks are in the fair class we can
* call that function directly, but only if the @prev task wasn't of a
@@ -5858,7 +5873,7 @@ __pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
if (prev->dl_server)
prev->dl_server = NULL;
- for_each_class(class) {
+ for_each_active_class(class) {
p = class->pick_next_task(rq);
if (p)
return p;
@@ -5891,7 +5906,7 @@ static inline struct task_struct *pick_task(struct rq *rq)
const struct sched_class *class;
struct task_struct *p;
- for_each_class(class) {
+ for_each_active_class(class) {
p = class->pick_task(rq);
if (p)
return p;
@@ -8355,6 +8370,7 @@ void __init sched_init(void)
balance_push_set(smp_processor_id(), false);
#endif
init_sched_fair_class();
+ init_sched_ext_class();
psi_init();
diff --git a/kernel/sched/ext.h b/kernel/sched/ext.h
new file mode 100644
index 000000000000..6a93c4825339
--- /dev/null
+++ b/kernel/sched/ext.h
@@ -0,0 +1,24 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+#ifdef CONFIG_SCHED_CLASS_EXT
+#error "NOT IMPLEMENTED YET"
+#else /* CONFIG_SCHED_CLASS_EXT */
+
+#define scx_enabled() false
+
+static inline void scx_pre_fork(struct task_struct *p) {}
+static inline int scx_fork(struct task_struct *p) { return 0; }
+static inline void scx_post_fork(struct task_struct *p) {}
+static inline void scx_cancel_fork(struct task_struct *p) {}
+static inline void init_sched_ext_class(void) {}
+
+#define for_each_active_class for_each_class
+#define for_balance_class_range for_class_range
+
+#endif /* CONFIG_SCHED_CLASS_EXT */
+
+#if defined(CONFIG_SCHED_CLASS_EXT) && defined(CONFIG_SMP)
+#error "NOT IMPLEMENTED YET"
+#else
+static inline void scx_update_idle(struct rq *rq, bool idle) {}
+#endif
diff --git a/kernel/sched/idle.c b/kernel/sched/idle.c
index 6e78d071beb5..c7a218123b7a 100644
--- a/kernel/sched/idle.c
+++ b/kernel/sched/idle.c
@@ -452,11 +452,13 @@ static void wakeup_preempt_idle(struct rq *rq, struct task_struct *p, int flags)
static void put_prev_task_idle(struct rq *rq, struct task_struct *prev)
{
+ scx_update_idle(rq, false);
}
static void set_next_task_idle(struct rq *rq, struct task_struct *next, bool first)
{
update_idle_core(rq);
+ scx_update_idle(rq, true);
schedstat_inc(rq->sched_goidle);
}
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 25660dc9f639..c52ad5fdd096 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -3658,4 +3658,6 @@ static inline void balance_callbacks(struct rq *rq, struct balance_callback *hea
#endif
+#include "ext.h"
+
#endif /* _KERNEL_SCHED_SCHED_H */
--
2.45.2
Implementation Analysis
Overview
This patch (PATCH 08/30) establishes the compile-time scaffolding for sched_ext without implementing any real behavior. Every function added here is either an empty stub or a #error "NOT IMPLEMENTED YET" guard. The patch exists specifically to let reviewers verify that the hook sites in the rest of the kernel are correct before the large ext.c implementation lands in the next patch (PATCH 09/30).
This is a standard kernel technique for large patch series: split the "where do we hook in" question from the "what does the hook do" question so each can be reviewed independently.
Architecture Context
sched_ext needs hook points in four areas of the kernel that this patch touches:
-
Task lifecycle (
kernel/fork.c): sched_ext must track every task from creation to destruction to manage itssched_ext_entityembedded intask_struct. -
Scheduler selection (
kernel/sched/core.c): The scheduler core has fast-path optimizations that assume only CFS tasks exist. sched_ext must disable those optimizations when it is active. -
Class iteration (
kernel/sched/core.c): When sched_ext is loaded,ext_sched_classmust participate in the class-priority walk; when it is not loaded, it must be invisible with zero overhead. -
Idle CPU tracking (
kernel/sched/idle.c): sched_ext needs to know which CPUs are idle in order to implementops.select_cpu()efficiently. The idle entry/exit hooks feed a per-CPU idle mask that the BPF scheduler can query.
Code Walkthrough
include/linux/sched/ext.h — the public header
#ifdef CONFIG_SCHED_CLASS_EXT
#error "NOT IMPLEMENTED YET"
#else
static inline void sched_ext_free(struct task_struct *p) {}
#endif
The #ifdef CONFIG_SCHED_CLASS_EXT branch is intentionally broken. This forces any build with CONFIG_SCHED_CLASS_EXT=y to fail until PATCH 09/30 replaces the #error with real declarations. The !CONFIG branch provides the no-op sched_ext_free() that __put_task_struct() will call unconditionally, so the call site in kernel/fork.c compiles cleanly in all configurations.
kernel/sched/ext.h — the internal header
#define scx_enabled() false
static inline void scx_pre_fork(struct task_struct *p) {}
static inline int scx_fork(struct task_struct *p) { return 0; }
static inline void scx_post_fork(struct task_struct *p) {}
static inline void scx_cancel_fork(struct task_struct *p) {}
static inline void init_sched_ext_class(void) {}
#define for_each_active_class for_each_class
#define for_balance_class_range for_class_range
When CONFIG_SCHED_CLASS_EXT=n, scx_enabled() is a compile-time false, which lets the compiler eliminate every branch guarded by it. The for_each_active_class and for_balance_class_range macros fall back to the existing for_each_class and for_class_range macros respectively, preserving existing behavior.
When CONFIG_SCHED_CLASS_EXT=y (and CONFIG_SMP=y), there is a second #error "NOT IMPLEMENTED YET" block for scx_update_idle(). This function will ultimately notify sched_ext when a CPU transitions into or out of idle, feeding the idle-CPU tracking mechanism.
kernel/fork.c — task destruction hook
sched_ext_free(tsk);
io_uring_free(tsk);
sched_ext_free() is placed before io_uring_free() and cgroup_free() in __put_task_struct(). This ordering matters in the real implementation: sched_ext needs to clean up its per-task state (remove from DSQ, call ops.disable()) before the task's cgroup reference is dropped, because ops.disable() may read cgroup information.
kernel/sched/core.c — fork lifecycle hooks
The fork path acquires four new hook calls:
scx_pre_fork(p); /* in sched_fork(), before class assignment */
/* ... class assignment ... */
out_cancel:
scx_cancel_fork(p); /* on error in sched_fork() */
return scx_fork(p); /* replaces return 0 in sched_cgroup_fork() */
scx_cancel_fork(p); /* in sched_cancel_fork() */
scx_post_fork(p); /* in sched_post_fork() */
The fork sequence is: sched_fork() → sched_cgroup_fork() → sched_post_fork(). sched_ext needs to know about a task early (in scx_pre_fork) to initialize its sched_ext_entity, must be able to propagate failure (the return scx_fork(p) that now allows sched_cgroup_fork to fail), and must clean up if the fork is cancelled at any point.
The reason sched_fork() gains a goto out_cancel structure instead of early returns is precisely to guarantee scx_cancel_fork() is always called on the error path — a classic resource-cleanup pattern applied to scheduler state.
kernel/sched/core.c — scheduling fast-path bypass
if (scx_enabled())
goto restart;
__pick_next_task() has a well-known optimization: if the previous task was a CFS task and no higher-priority tasks exist, it skips the full class walk and calls fair_sched_class.pick_next_task() directly. When sched_ext is active this optimization is invalid because an SCX task could be waiting. The goto restart forces the full for_each_active_class() walk. The scx_enabled() check is a static key in the real implementation, so when sched_ext is not loaded the branch is a single no-prediction-needed not-taken jump.
kernel/sched/idle.c — idle tracking hooks
static void put_prev_task_idle(struct rq *rq, struct task_struct *prev)
{
scx_update_idle(rq, false); /* CPU leaving idle */
}
static void set_next_task_idle(struct rq *rq, struct task_struct *next, bool first)
{
update_idle_core(rq);
scx_update_idle(rq, true); /* CPU entering idle */
schedstat_inc(rq->sched_goidle);
}
put_prev_task_idle is called when the idle task is being replaced (CPU waking up from idle). set_next_task_idle is called when the idle task is being scheduled (CPU going idle). The true/false arguments tell sched_ext whether to set or clear this CPU's bit in the idle cpumask that BPF schedulers will use for ops.select_cpu().
kernel/sched/sched.h — header inclusion
#include "ext.h"
This single line, added at the very end of sched.h, makes all the stubs from kernel/sched/ext.h available to all files that include sched.h (which is the entire scheduler subsystem). The placement at the end is deliberate: ext.h uses types declared earlier in sched.h, so it must come last.
Key Concepts Introduced
scx_enabled() as a static key: In the stub it is #define scx_enabled() false. In the real implementation (PATCH 09/30) it becomes a static key — a runtime-patchable NOP instruction that only costs one cycle when sched_ext is inactive. This is the mechanism that makes the __pick_next_task() fast-path bypass zero-cost when no BPF scheduler is loaded.
The for_each_active_class / for_balance_class_range macros: These are the mechanism by which ext_sched_class is conditionally included in scheduler class walks. When sched_ext is disabled they expand to the standard macros and incur no overhead. When it is enabled they use a custom iterator that starts from the highest-priority active class and skips ext_sched_class if it is being disabled. This is the invariant that guarantees tasks always have a fallback to CFS.
The four-phase fork lifecycle: pre_fork → fork (cgroup path) → cancel_fork (error path) → post_fork. This decomposition reflects the fact that task creation in the kernel is not atomic — it can fail at multiple points, and any resource acquired during fork must be releasable from any failure point.
Why This Matters for Maintainers
The #error "NOT IMPLEMENTED YET" pattern: You will see this in several places. It is not laziness — it is a deliberate device to make a partial implementation fail loudly at compile time rather than silently misbehave at runtime. When reviewing new sched_ext features that follow the same boilerplate pattern, verify that the #error guards are replaced atomically with functional code.
Hook ordering in __put_task_struct: sched_ext_free() runs before cgroup_free(). Any future change to move these calls must preserve this ordering, because the real sched_ext_free() implementation may read cgroup state during ops.disable() or ops.exit_task().
scx_update_idle() in the SMP branch: The stub for scx_update_idle() is in a separate #if defined(CONFIG_SCHED_CLASS_EXT) && defined(CONFIG_SMP) block. The idle-CPU tracking mechanism is inherently SMP-only (there is nothing to track on a uniprocessor), so this is a correctness constraint, not just a performance optimization.
The goto restart in __pick_next_task(): This is a subtle correctness point. Without it, a system running an SCX scheduler could exhibit rare scheduling pathologies where CFS tasks are selected ahead of higher-priority SCX tasks due to the fast-path optimization. Any change to the scheduling fast path must account for this.
Connection to Other Patches
This patch is explicitly described in its commit message as "split out to help reviewing." Its sole purpose is to create the correct hook sites so that PATCH 09/30 (the full ext.c implementation) can be reviewed as pure logic, not as a mix of hook-site correctness and algorithm correctness.
The #error "NOT IMPLEMENTED YET" in include/linux/sched/ext.h and kernel/sched/ext.h will be replaced by PATCH 09/30, which fills in:
- The full
struct sched_ext_entityembedded in everytask_struct - The real
ext_sched_classwith allsched_classcallbacks - The DSQ implementation
- The
scx_ops_enable()andscx_ops_disable()state machine - The real
scx_enabled()static key
The scx_update_idle() stub (SMP branch) will be replaced with actual idle-cpumask management, which feeds scx_bpf_select_cpu_dfl() — the default idle CPU picker available to BPF schedulers.
[PATCH 09/30] sched_ext: Implement BPF extensible scheduler class
View on Lore: https://lore.kernel.org/all/20240618212056.2833381-10-tj@kernel.org
Commit Message
Implement a new scheduler class sched_ext (SCX), which allows scheduling
policies to be implemented as BPF programs to achieve the following:
1. Ease of experimentation and exploration: Enabling rapid iteration of new
scheduling policies.
2. Customization: Building application-specific schedulers which implement
policies that are not applicable to general-purpose schedulers.
3. Rapid scheduler deployments: Non-disruptive swap outs of scheduling
policies in production environments.
sched_ext leverages BPF’s struct_ops feature to define a structure which
exports function callbacks and flags to BPF programs that wish to implement
scheduling policies. The struct_ops structure exported by sched_ext is
struct sched_ext_ops, and is conceptually similar to struct sched_class. The
role of sched_ext is to map the complex sched_class callbacks to the more
simple and ergonomic struct sched_ext_ops callbacks.
For more detailed discussion on the motivations and overview, please refer
to the cover letter.
Later patches will also add several example schedulers and documentation.
This patch implements the minimum core framework to enable implementation of
BPF schedulers. Subsequent patches will gradually add functionalities
including safety guarantee mechanisms, nohz and cgroup support.
include/linux/sched/ext.h defines struct sched_ext_ops. With the comment on
top, each operation should be self-explanatory. The followings are worth
noting:
- Both "sched_ext" and its shorthand "scx" are used. If the identifier
already has "sched" in it, "ext" is used; otherwise, "scx".
- In sched_ext_ops, only .name is mandatory. Every operation is optional and
if omitted a simple but functional default behavior is provided.
- A new policy constant SCHED_EXT is added and a task can select sched_ext
by invoking sched_setscheduler(2) with the new policy constant. However,
if the BPF scheduler is not loaded, SCHED_EXT is the same as SCHED_NORMAL
and the task is scheduled by CFS. When the BPF scheduler is loaded, all
tasks which have the SCHED_EXT policy are switched to sched_ext.
- To bridge the workflow imbalance between the scheduler core and
sched_ext_ops callbacks, sched_ext uses simple FIFOs called dispatch
queues (dsq's). By default, there is one global dsq (SCX_DSQ_GLOBAL), and
one local per-CPU dsq (SCX_DSQ_LOCAL). SCX_DSQ_GLOBAL is provided for
convenience and need not be used by a scheduler that doesn't require it.
SCX_DSQ_LOCAL is the per-CPU FIFO that sched_ext pulls from when putting
the next task on the CPU. The BPF scheduler can manage an arbitrary number
of dsq's using scx_bpf_create_dsq() and scx_bpf_destroy_dsq().
- sched_ext guarantees system integrity no matter what the BPF scheduler
does. To enable this, each task's ownership is tracked through
p->scx.ops_state and all tasks are put on scx_tasks list. The disable path
can always recover and revert all tasks back to CFS. See p->scx.ops_state
and scx_tasks.
- A task is not tied to its rq while enqueued. This decouples CPU selection
from queueing and allows sharing a scheduling queue across an arbitrary
subset of CPUs. This adds some complexities as a task may need to be
bounced between rq's right before it starts executing. See
dispatch_to_local_dsq() and move_task_to_local_dsq().
- One complication that arises from the above weak association between task
and rq is that synchronizing with dequeue() gets complicated as dequeue()
may happen anytime while the task is enqueued and the dispatch path might
need to release the rq lock to transfer the task. Solving this requires a
bit of complexity. See the logic around p->scx.sticky_cpu and
p->scx.ops_qseq.
- Both enable and disable paths are a bit complicated. The enable path
switches all tasks without blocking to avoid issues which can arise from
partially switched states (e.g. the switching task itself being starved).
The disable path can't trust the BPF scheduler at all, so it also has to
guarantee forward progress without blocking. See scx_ops_enable() and
scx_ops_disable_workfn().
- When sched_ext is disabled, static_branches are used to shut down the
entry points from hot paths.
v7: - scx_ops_bypass() was incorrectly and unnecessarily trying to grab
scx_ops_enable_mutex which can lead to deadlocks in the disable path.
Fixed.
- Fixed TASK_DEAD handling bug in scx_ops_enable() path which could lead
to use-after-free.
- Consolidated per-cpu variable usages and other cleanups.
v6: - SCX_NR_ONLINE_OPS replaced with SCX_OPI_*_BEGIN/END so that multiple
groups can be expressed. Later CPU hotplug operations are put into
their own group.
- SCX_OPS_DISABLING state is replaced with the new bypass mechanism
which allows temporarily putting the system into simple FIFO
scheduling mode bypassing the BPF scheduler. In addition to the shut
down path, this will also be used to isolate the BPF scheduler across
PM events. Enabling and disabling the bypass mode requires iterating
all runnable tasks. rq->scx.runnable_list addition is moved from the
later watchdog patch.
- ops.prep_enable() is replaced with ops.init_task() and
ops.enable/disable() are now called whenever the task enters and
leaves sched_ext instead of when the task becomes schedulable on
sched_ext and stops being so. A new operation - ops.exit_task() - is
called when the task stops being schedulable on sched_ext.
- scx_bpf_dispatch() can now be called from ops.select_cpu() too. This
removes the need for communicating local dispatch decision made by
ops.select_cpu() to ops.enqueue() via per-task storage.
SCX_KF_SELECT_CPU is added to support the change.
- SCX_TASK_ENQ_LOCAL which told the BPF scheudler that
scx_select_cpu_dfl() wants the task to be dispatched to the local DSQ
was removed. Instead, scx_bpf_select_cpu_dfl() now dispatches directly
if it finds a suitable idle CPU. If such behavior is not desired,
users can use scx_bpf_select_cpu_dfl() which returns the verdict in a
bool out param.
- scx_select_cpu_dfl() was mishandling WAKE_SYNC and could end up
queueing many tasks on a local DSQ which makes tasks to execute in
order while other CPUs stay idle which made some hackbench numbers
really bad. Fixed.
- The current state of sched_ext can now be monitored through files
under /sys/sched_ext instead of /sys/kernel/debug/sched/ext. This is
to enable monitoring on kernels which don't enable debugfs.
- sched_ext wasn't telling BPF that ops.dispatch()'s @prev argument may
be NULL and a BPF scheduler which derefs the pointer without checking
could crash the kernel. Tell BPF. This is currently a bit ugly. A
better way to annotate this is expected in the future.
- scx_exit_info updated to carry pointers to message buffers instead of
embedding them directly. This decouples buffer sizes from API so that
they can be changed without breaking compatibility.
- exit_code added to scx_exit_info. This is used to indicate different
exit conditions on non-error exits and will be used to handle e.g. CPU
hotplugs.
- The patch "sched_ext: Allow BPF schedulers to switch all eligible
tasks into sched_ext" is folded in and the interface is changed so
that partial switching is indicated with a new ops flag
%SCX_OPS_SWITCH_PARTIAL. This makes scx_bpf_switch_all() unnecessasry
and in turn SCX_KF_INIT. ops.init() is now called with
SCX_KF_SLEEPABLE.
- Code reorganized so that only the parts necessary to integrate with
the rest of the kernel are in the header files.
- Changes to reflect the BPF and other kernel changes including the
addition of bpf_sched_ext_ops.cfi_stubs.
v5: - To accommodate 32bit configs, p->scx.ops_state is now atomic_long_t
instead of atomic64_t and scx_dsp_buf_ent.qseq which uses
load_acquire/store_release is now unsigned long instead of u64.
- Fix the bug where bpf_scx_btf_struct_access() was allowing write
access to arbitrary fields.
- Distinguish kfuncs which can be called from any sched_ext ops and from
anywhere. e.g. scx_bpf_pick_idle_cpu() can now be called only from
sched_ext ops.
- Rename "type" to "kind" in scx_exit_info to make it easier to use on
languages in which "type" is a reserved keyword.
- Since cff9b2332ab7 ("kernel/sched: Modify initial boot task idle
setup"), PF_IDLE is not set on idle tasks which haven't been online
yet which made scx_task_iter_next_filtered() include those idle tasks
in iterations leading to oopses. Update scx_task_iter_next_filtered()
to directly test p->sched_class against idle_sched_class instead of
using is_idle_task() which tests PF_IDLE.
- Other updates to match upstream changes such as adding const to
set_cpumask() param and renaming check_preempt_curr() to
wakeup_preempt().
v4: - SCHED_CHANGE_BLOCK replaced with the previous
sched_deq_and_put_task()/sched_enq_and_set_tsak() pair. This is
because upstream is adaopting a different generic cleanup mechanism.
Once that lands, the code will be adapted accordingly.
- task_on_scx() used to test whether a task should be switched into SCX,
which is confusing. Renamed to task_should_scx(). task_on_scx() now
tests whether a task is currently on SCX.
- scx_has_idle_cpus is barely used anymore and replaced with direct
check on the idle cpumask.
- SCX_PICK_IDLE_CORE added and scx_pick_idle_cpu() improved to prefer
fully idle cores.
- ops.enable() now sees up-to-date p->scx.weight value.
- ttwu_queue path is disabled for tasks on SCX to avoid confusing BPF
schedulers expecting ->select_cpu() call.
- Use cpu_smt_mask() instead of topology_sibling_cpumask() like the rest
of the scheduler.
v3: - ops.set_weight() added to allow BPF schedulers to track weight changes
without polling p->scx.weight.
- move_task_to_local_dsq() was losing SCX-specific enq_flags when
enqueueing the task on the target dsq because it goes through
activate_task() which loses the upper 32bit of the flags. Carry the
flags through rq->scx.extra_enq_flags.
- scx_bpf_dispatch(), scx_bpf_pick_idle_cpu(), scx_bpf_task_running()
and scx_bpf_task_cpu() now use the new KF_RCU instead of
KF_TRUSTED_ARGS to make it easier for BPF schedulers to call them.
- The kfunc helper access control mechanism implemented through
sched_ext_entity.kf_mask is improved. Now SCX_CALL_OP*() is always
used when invoking scx_ops operations.
v2: - balance_scx_on_up() is dropped. Instead, on UP, balance_scx() is
called from put_prev_taks_scx() and pick_next_task_scx() as necessary.
To determine whether balance_scx() should be called from
put_prev_task_scx(), SCX_TASK_DEQD_FOR_SLEEP flag is added. See the
comment in put_prev_task_scx() for details.
- sched_deq_and_put_task() / sched_enq_and_set_task() sequences replaced
with SCHED_CHANGE_BLOCK().
- Unused all_dsqs list removed. This was a left-over from previous
iterations.
- p->scx.kf_mask is added to track and enforce which kfunc helpers are
allowed. Also, init/exit sequences are updated to make some kfuncs
always safe to call regardless of the current BPF scheduler state.
Combined, this should make all the kfuncs safe.
- BPF now supports sleepable struct_ops operations. Hacky workaround
removed and operations and kfunc helpers are tagged appropriately.
- BPF now supports bitmask / cpumask helpers. scx_bpf_get_idle_cpumask()
and friends are added so that BPF schedulers can use the idle masks
with the generic helpers. This replaces the hacky kfunc helpers added
by a separate patch in V1.
- CONFIG_SCHED_CLASS_EXT can no longer be enabled if SCHED_CORE is
enabled. This restriction will be removed by a later patch which adds
core-sched support.
- Add MAINTAINERS entries and other misc changes.
Signed-off-by: Tejun Heo <tj@kernel.org>
Co-authored-by: David Vernet <dvernet@meta.com>
Acked-by: Josh Don <joshdon@google.com>
Acked-by: Hao Luo <haoluo@google.com>
Acked-by: Barret Rhoden <brho@google.com>
Cc: Andrea Righi <andrea.righi@canonical.com>
---
MAINTAINERS | 13 +
include/asm-generic/vmlinux.lds.h | 1 +
include/linux/sched.h | 5 +
include/linux/sched/ext.h | 141 +-
include/uapi/linux/sched.h | 1 +
init/init_task.c | 11 +
kernel/Kconfig.preempt | 25 +-
kernel/sched/build_policy.c | 9 +
kernel/sched/core.c | 66 +-
kernel/sched/debug.c | 3 +
kernel/sched/ext.c | 4256 +++++++++++++++++++++++++++++
kernel/sched/ext.h | 73 +-
kernel/sched/sched.h | 17 +
kernel/sched/syscalls.c | 2 +
14 files changed, 4616 insertions(+), 7 deletions(-)
create mode 100644 kernel/sched/ext.c
diff --git a/MAINTAINERS b/MAINTAINERS
index cd3277a98cfe..9e3ee22f015e 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -19936,6 +19936,19 @@ F: include/linux/wait.h
F: include/uapi/linux/sched.h
F: kernel/sched/
+SCHEDULER - SCHED_EXT
+R: Tejun Heo <tj@kernel.org>
+R: David Vernet <void@manifault.com>
+L: linux-kernel@vger.kernel.org
+S: Maintained
+W: https://github.com/sched-ext/scx
+T: git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext.git
+F: include/linux/sched/ext.h
+F: kernel/sched/ext.h
+F: kernel/sched/ext.c
+F: tools/sched_ext/
+F: tools/testing/selftests/sched_ext
+
SCSI LIBSAS SUBSYSTEM
R: John Garry <john.g.garry@oracle.com>
R: Jason Yan <yanaijie@huawei.com>
diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
index 5703526d6ebf..2e712183ba09 100644
--- a/include/asm-generic/vmlinux.lds.h
+++ b/include/asm-generic/vmlinux.lds.h
@@ -133,6 +133,7 @@
*(__dl_sched_class) \
*(__rt_sched_class) \
*(__fair_sched_class) \
+ *(__ext_sched_class) \
*(__idle_sched_class) \
__sched_class_lowest = .;
diff --git a/include/linux/sched.h b/include/linux/sched.h
index 90691d99027e..06beb8a6e0ca 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -80,6 +80,8 @@ struct task_group;
struct task_struct;
struct user_event_mm;
+#include <linux/sched/ext.h>
+
/*
* Task state bitmask. NOTE! These bits are also
* encoded in fs/proc/array.c: get_task_state().
@@ -802,6 +804,9 @@ struct task_struct {
struct sched_rt_entity rt;
struct sched_dl_entity dl;
struct sched_dl_entity *dl_server;
+#ifdef CONFIG_SCHED_CLASS_EXT
+ struct sched_ext_entity scx;
+#endif
const struct sched_class *sched_class;
#ifdef CONFIG_SCHED_CORE
diff --git a/include/linux/sched/ext.h b/include/linux/sched/ext.h
index a05dfcf533b0..c1530a7992cc 100644
--- a/include/linux/sched/ext.h
+++ b/include/linux/sched/ext.h
@@ -1,9 +1,148 @@
/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2022 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2022 Tejun Heo <tj@kernel.org>
+ * Copyright (c) 2022 David Vernet <dvernet@meta.com>
+ */
#ifndef _LINUX_SCHED_EXT_H
#define _LINUX_SCHED_EXT_H
#ifdef CONFIG_SCHED_CLASS_EXT
-#error "NOT IMPLEMENTED YET"
+
+#include <linux/llist.h>
+#include <linux/rhashtable-types.h>
+
+enum scx_public_consts {
+ SCX_OPS_NAME_LEN = 128,
+
+ SCX_SLICE_DFL = 20 * 1000000, /* 20ms */
+};
+
+/*
+ * DSQ (dispatch queue) IDs are 64bit of the format:
+ *
+ * Bits: [63] [62 .. 0]
+ * [ B] [ ID ]
+ *
+ * B: 1 for IDs for built-in DSQs, 0 for ops-created user DSQs
+ * ID: 63 bit ID
+ *
+ * Built-in IDs:
+ *
+ * Bits: [63] [62] [61..32] [31 .. 0]
+ * [ 1] [ L] [ R ] [ V ]
+ *
+ * 1: 1 for built-in DSQs.
+ * L: 1 for LOCAL_ON DSQ IDs, 0 for others
+ * V: For LOCAL_ON DSQ IDs, a CPU number. For others, a pre-defined value.
+ */
+enum scx_dsq_id_flags {
+ SCX_DSQ_FLAG_BUILTIN = 1LLU << 63,
+ SCX_DSQ_FLAG_LOCAL_ON = 1LLU << 62,
+
+ SCX_DSQ_INVALID = SCX_DSQ_FLAG_BUILTIN | 0,
+ SCX_DSQ_GLOBAL = SCX_DSQ_FLAG_BUILTIN | 1,
+ SCX_DSQ_LOCAL = SCX_DSQ_FLAG_BUILTIN | 2,
+ SCX_DSQ_LOCAL_ON = SCX_DSQ_FLAG_BUILTIN | SCX_DSQ_FLAG_LOCAL_ON,
+ SCX_DSQ_LOCAL_CPU_MASK = 0xffffffffLLU,
+};
+
+/*
+ * Dispatch queue (dsq) is a simple FIFO which is used to buffer between the
+ * scheduler core and the BPF scheduler. See the documentation for more details.
+ */
+struct scx_dispatch_q {
+ raw_spinlock_t lock;
+ struct list_head list; /* tasks in dispatch order */
+ u32 nr;
+ u64 id;
+ struct rhash_head hash_node;
+ struct llist_node free_node;
+ struct rcu_head rcu;
+};
+
+/* scx_entity.flags */
+enum scx_ent_flags {
+ SCX_TASK_QUEUED = 1 << 0, /* on ext runqueue */
+ SCX_TASK_BAL_KEEP = 1 << 1, /* balance decided to keep current */
+ SCX_TASK_RESET_RUNNABLE_AT = 1 << 2, /* runnable_at should be reset */
+ SCX_TASK_DEQD_FOR_SLEEP = 1 << 3, /* last dequeue was for SLEEP */
+
+ SCX_TASK_STATE_SHIFT = 8, /* bit 8 and 9 are used to carry scx_task_state */
+ SCX_TASK_STATE_BITS = 2,
+ SCX_TASK_STATE_MASK = ((1 << SCX_TASK_STATE_BITS) - 1) << SCX_TASK_STATE_SHIFT,
+
+ SCX_TASK_CURSOR = 1 << 31, /* iteration cursor, not a task */
+};
+
+/* scx_entity.flags & SCX_TASK_STATE_MASK */
+enum scx_task_state {
+ SCX_TASK_NONE, /* ops.init_task() not called yet */
+ SCX_TASK_INIT, /* ops.init_task() succeeded, but task can be cancelled */
+ SCX_TASK_READY, /* fully initialized, but not in sched_ext */
+ SCX_TASK_ENABLED, /* fully initialized and in sched_ext */
+
+ SCX_TASK_NR_STATES,
+};
+
+/*
+ * Mask bits for scx_entity.kf_mask. Not all kfuncs can be called from
+ * everywhere and the following bits track which kfunc sets are currently
+ * allowed for %current. This simple per-task tracking works because SCX ops
+ * nest in a limited way. BPF will likely implement a way to allow and disallow
+ * kfuncs depending on the calling context which will replace this manual
+ * mechanism. See scx_kf_allow().
+ */
+enum scx_kf_mask {
+ SCX_KF_UNLOCKED = 0, /* not sleepable, not rq locked */
+ /* all non-sleepables may be nested inside SLEEPABLE */
+ SCX_KF_SLEEPABLE = 1 << 0, /* sleepable init operations */
+ /* ops.dequeue (in REST) may be nested inside DISPATCH */
+ SCX_KF_DISPATCH = 1 << 2, /* ops.dispatch() */
+ SCX_KF_ENQUEUE = 1 << 3, /* ops.enqueue() and ops.select_cpu() */
+ SCX_KF_SELECT_CPU = 1 << 4, /* ops.select_cpu() */
+ SCX_KF_REST = 1 << 5, /* other rq-locked operations */
+
+ __SCX_KF_RQ_LOCKED = SCX_KF_DISPATCH |
+ SCX_KF_ENQUEUE | SCX_KF_SELECT_CPU | SCX_KF_REST,
+};
+
+/*
+ * The following is embedded in task_struct and contains all fields necessary
+ * for a task to be scheduled by SCX.
+ */
+struct sched_ext_entity {
+ struct scx_dispatch_q *dsq;
+ struct list_head dsq_node;
+ u32 flags; /* protected by rq lock */
+ u32 weight;
+ s32 sticky_cpu;
+ s32 holding_cpu;
+ u32 kf_mask; /* see scx_kf_mask above */
+ atomic_long_t ops_state;
+
+ struct list_head runnable_node; /* rq->scx.runnable_list */
+
+ u64 ddsp_dsq_id;
+ u64 ddsp_enq_flags;
+
+ /* BPF scheduler modifiable fields */
+
+ /*
+ * Runtime budget in nsecs. This is usually set through
+ * scx_bpf_dispatch() but can also be modified directly by the BPF
+ * scheduler. Automatically decreased by SCX as the task executes. On
+ * depletion, a scheduling event is triggered.
+ */
+ u64 slice;
+
+ /* cold fields */
+ /* must be the last field, see init_scx_entity() */
+ struct list_head tasks_node;
+};
+
+void sched_ext_free(struct task_struct *p);
+
#else /* !CONFIG_SCHED_CLASS_EXT */
static inline void sched_ext_free(struct task_struct *p) {}
diff --git a/include/uapi/linux/sched.h b/include/uapi/linux/sched.h
index 3bac0a8ceab2..359a14cc76a4 100644
--- a/include/uapi/linux/sched.h
+++ b/include/uapi/linux/sched.h
@@ -118,6 +118,7 @@ struct clone_args {
/* SCHED_ISO: reserved but not implemented yet */
#define SCHED_IDLE 5
#define SCHED_DEADLINE 6
+#define SCHED_EXT 7
/* Can be ORed in to make sure the process is reverted back to SCHED_NORMAL on fork */
#define SCHED_RESET_ON_FORK 0x40000000
diff --git a/init/init_task.c b/init/init_task.c
index eeb110c65fe2..c6804396fe12 100644
--- a/init/init_task.c
+++ b/init/init_task.c
@@ -6,6 +6,7 @@
#include <linux/sched/sysctl.h>
#include <linux/sched/rt.h>
#include <linux/sched/task.h>
+#include <linux/sched/ext.h>
#include <linux/init.h>
#include <linux/fs.h>
#include <linux/mm.h>
@@ -98,6 +99,16 @@ struct task_struct init_task __aligned(L1_CACHE_BYTES) = {
#endif
#ifdef CONFIG_CGROUP_SCHED
.sched_task_group = &root_task_group,
+#endif
+#ifdef CONFIG_SCHED_CLASS_EXT
+ .scx = {
+ .dsq_node = LIST_HEAD_INIT(init_task.scx.dsq_node),
+ .sticky_cpu = -1,
+ .holding_cpu = -1,
+ .runnable_node = LIST_HEAD_INIT(init_task.scx.runnable_node),
+ .ddsp_dsq_id = SCX_DSQ_INVALID,
+ .slice = SCX_SLICE_DFL,
+ },
#endif
.ptraced = LIST_HEAD_INIT(init_task.ptraced),
.ptrace_entry = LIST_HEAD_INIT(init_task.ptrace_entry),
diff --git a/kernel/Kconfig.preempt b/kernel/Kconfig.preempt
index c2f1fd95a821..39ecfc2b5a1c 100644
--- a/kernel/Kconfig.preempt
+++ b/kernel/Kconfig.preempt
@@ -133,4 +133,27 @@ config SCHED_CORE
which is the likely usage by Linux distributions, there should
be no measurable impact on performance.
-
+config SCHED_CLASS_EXT
+ bool "Extensible Scheduling Class"
+ depends on BPF_SYSCALL && BPF_JIT && !SCHED_CORE
+ help
+ This option enables a new scheduler class sched_ext (SCX), which
+ allows scheduling policies to be implemented as BPF programs to
+ achieve the following:
+
+ - Ease of experimentation and exploration: Enabling rapid
+ iteration of new scheduling policies.
+ - Customization: Building application-specific schedulers which
+ implement policies that are not applicable to general-purpose
+ schedulers.
+ - Rapid scheduler deployments: Non-disruptive swap outs of
+ scheduling policies in production environments.
+
+ sched_ext leverages BPF struct_ops feature to define a structure
+ which exports function callbacks and flags to BPF programs that
+ wish to implement scheduling policies. The struct_ops structure
+ exported by sched_ext is struct sched_ext_ops, and is conceptually
+ similar to struct sched_class.
+
+ For more information:
+ https://github.com/sched-ext/scx
diff --git a/kernel/sched/build_policy.c b/kernel/sched/build_policy.c
index 39c315182b35..f0c148fcd2df 100644
--- a/kernel/sched/build_policy.c
+++ b/kernel/sched/build_policy.c
@@ -21,13 +21,18 @@
#include <linux/cpuidle.h>
#include <linux/jiffies.h>
+#include <linux/kobject.h>
#include <linux/livepatch.h>
+#include <linux/pm.h>
#include <linux/psi.h>
+#include <linux/rhashtable.h>
+#include <linux/seq_buf.h>
#include <linux/seqlock_api.h>
#include <linux/slab.h>
#include <linux/suspend.h>
#include <linux/tsacct_kern.h>
#include <linux/vtime.h>
+#include <linux/percpu-rwsem.h>
#include <uapi/linux/sched/types.h>
@@ -52,4 +57,8 @@
#include "cputime.c"
#include "deadline.c"
+#ifdef CONFIG_SCHED_CLASS_EXT
+# include "ext.c"
+#endif
+
#include "syscalls.c"
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index d8c963fea9eb..6042ce3bfee0 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -3790,6 +3790,15 @@ bool cpus_share_resources(int this_cpu, int that_cpu)
static inline bool ttwu_queue_cond(struct task_struct *p, int cpu)
{
+ /*
+ * The BPF scheduler may depend on select_task_rq() being invoked during
+ * wakeups. In addition, @p may end up executing on a different CPU
+ * regardless of what happens in the wakeup path making the ttwu_queue
+ * optimization less meaningful. Skip if on SCX.
+ */
+ if (task_on_scx(p))
+ return false;
+
/*
* Do not complicate things with the async wake_list while the CPU is
* in hotplug state.
@@ -4357,6 +4366,10 @@ static void __sched_fork(unsigned long clone_flags, struct task_struct *p)
p->rt.on_rq = 0;
p->rt.on_list = 0;
+#ifdef CONFIG_SCHED_CLASS_EXT
+ init_scx_entity(&p->scx);
+#endif
+
#ifdef CONFIG_PREEMPT_NOTIFIERS
INIT_HLIST_HEAD(&p->preempt_notifiers);
#endif
@@ -4604,6 +4617,10 @@ int sched_fork(unsigned long clone_flags, struct task_struct *p)
goto out_cancel;
} else if (rt_prio(p->prio)) {
p->sched_class = &rt_sched_class;
+#ifdef CONFIG_SCHED_CLASS_EXT
+ } else if (task_should_scx(p)) {
+ p->sched_class = &ext_sched_class;
+#endif
} else {
p->sched_class = &fair_sched_class;
}
@@ -5511,8 +5528,10 @@ void sched_tick(void)
wq_worker_tick(curr);
#ifdef CONFIG_SMP
- rq->idle_balance = idle_cpu(cpu);
- sched_balance_trigger(rq);
+ if (!scx_switched_all()) {
+ rq->idle_balance = idle_cpu(cpu);
+ sched_balance_trigger(rq);
+ }
#endif
}
@@ -6902,6 +6921,10 @@ void __setscheduler_prio(struct task_struct *p, int prio)
p->sched_class = &dl_sched_class;
else if (rt_prio(prio))
p->sched_class = &rt_sched_class;
+#ifdef CONFIG_SCHED_CLASS_EXT
+ else if (task_should_scx(p))
+ p->sched_class = &ext_sched_class;
+#endif
else
p->sched_class = &fair_sched_class;
@@ -8203,6 +8226,10 @@ void __init sched_init(void)
BUG_ON(!sched_class_above(&dl_sched_class, &rt_sched_class));
BUG_ON(!sched_class_above(&rt_sched_class, &fair_sched_class));
BUG_ON(!sched_class_above(&fair_sched_class, &idle_sched_class));
+#ifdef CONFIG_SCHED_CLASS_EXT
+ BUG_ON(!sched_class_above(&fair_sched_class, &ext_sched_class));
+ BUG_ON(!sched_class_above(&ext_sched_class, &idle_sched_class));
+#endif
wait_bit_init();
@@ -10337,3 +10364,38 @@ void sched_mm_cid_fork(struct task_struct *t)
t->mm_cid_active = 1;
}
#endif
+
+#ifdef CONFIG_SCHED_CLASS_EXT
+void sched_deq_and_put_task(struct task_struct *p, int queue_flags,
+ struct sched_enq_and_set_ctx *ctx)
+{
+ struct rq *rq = task_rq(p);
+
+ lockdep_assert_rq_held(rq);
+
+ *ctx = (struct sched_enq_and_set_ctx){
+ .p = p,
+ .queue_flags = queue_flags,
+ .queued = task_on_rq_queued(p),
+ .running = task_current(rq, p),
+ };
+
+ update_rq_clock(rq);
+ if (ctx->queued)
+ dequeue_task(rq, p, queue_flags | DEQUEUE_NOCLOCK);
+ if (ctx->running)
+ put_prev_task(rq, p);
+}
+
+void sched_enq_and_set_task(struct sched_enq_and_set_ctx *ctx)
+{
+ struct rq *rq = task_rq(ctx->p);
+
+ lockdep_assert_rq_held(rq);
+
+ if (ctx->queued)
+ enqueue_task(rq, ctx->p, ctx->queue_flags | ENQUEUE_NOCLOCK);
+ if (ctx->running)
+ set_next_task(rq, ctx->p);
+}
+#endif /* CONFIG_SCHED_CLASS_EXT */
diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c
index c1eb9a1afd13..c057ef46c5f8 100644
--- a/kernel/sched/debug.c
+++ b/kernel/sched/debug.c
@@ -1090,6 +1090,9 @@ void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns,
P(dl.runtime);
P(dl.deadline);
}
+#ifdef CONFIG_SCHED_CLASS_EXT
+ __PS("ext.enabled", task_on_scx(p));
+#endif
#undef PN_SCHEDSTAT
#undef P_SCHEDSTAT
diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c
new file mode 100644
index 000000000000..49b115f5b052
--- /dev/null
+++ b/kernel/sched/ext.c
@@ -0,0 +1,4256 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2022 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2022 Tejun Heo <tj@kernel.org>
+ * Copyright (c) 2022 David Vernet <dvernet@meta.com>
+ */
+#define SCX_OP_IDX(op) (offsetof(struct sched_ext_ops, op) / sizeof(void (*)(void)))
+
+enum scx_consts {
+ SCX_DSP_DFL_MAX_BATCH = 32,
+
+ SCX_EXIT_BT_LEN = 64,
+ SCX_EXIT_MSG_LEN = 1024,
+};
+
+enum scx_exit_kind {
+ SCX_EXIT_NONE,
+ SCX_EXIT_DONE,
+
+ SCX_EXIT_UNREG = 64, /* user-space initiated unregistration */
+ SCX_EXIT_UNREG_BPF, /* BPF-initiated unregistration */
+ SCX_EXIT_UNREG_KERN, /* kernel-initiated unregistration */
+
+ SCX_EXIT_ERROR = 1024, /* runtime error, error msg contains details */
+ SCX_EXIT_ERROR_BPF, /* ERROR but triggered through scx_bpf_error() */
+};
+
+/*
+ * scx_exit_info is passed to ops.exit() to describe why the BPF scheduler is
+ * being disabled.
+ */
+struct scx_exit_info {
+ /* %SCX_EXIT_* - broad category of the exit reason */
+ enum scx_exit_kind kind;
+
+ /* exit code if gracefully exiting */
+ s64 exit_code;
+
+ /* textual representation of the above */
+ const char *reason;
+
+ /* backtrace if exiting due to an error */
+ unsigned long *bt;
+ u32 bt_len;
+
+ /* informational message */
+ char *msg;
+};
+
+/* sched_ext_ops.flags */
+enum scx_ops_flags {
+ /*
+ * Keep built-in idle tracking even if ops.update_idle() is implemented.
+ */
+ SCX_OPS_KEEP_BUILTIN_IDLE = 1LLU << 0,
+
+ /*
+ * By default, if there are no other task to run on the CPU, ext core
+ * keeps running the current task even after its slice expires. If this
+ * flag is specified, such tasks are passed to ops.enqueue() with
+ * %SCX_ENQ_LAST. See the comment above %SCX_ENQ_LAST for more info.
+ */
+ SCX_OPS_ENQ_LAST = 1LLU << 1,
+
+ /*
+ * An exiting task may schedule after PF_EXITING is set. In such cases,
+ * bpf_task_from_pid() may not be able to find the task and if the BPF
+ * scheduler depends on pid lookup for dispatching, the task will be
+ * lost leading to various issues including RCU grace period stalls.
+ *
+ * To mask this problem, by default, unhashed tasks are automatically
+ * dispatched to the local DSQ on enqueue. If the BPF scheduler doesn't
+ * depend on pid lookups and wants to handle these tasks directly, the
+ * following flag can be used.
+ */
+ SCX_OPS_ENQ_EXITING = 1LLU << 2,
+
+ /*
+ * If set, only tasks with policy set to SCHED_EXT are attached to
+ * sched_ext. If clear, SCHED_NORMAL tasks are also included.
+ */
+ SCX_OPS_SWITCH_PARTIAL = 1LLU << 3,
+
+ SCX_OPS_ALL_FLAGS = SCX_OPS_KEEP_BUILTIN_IDLE |
+ SCX_OPS_ENQ_LAST |
+ SCX_OPS_ENQ_EXITING |
+ SCX_OPS_SWITCH_PARTIAL,
+};
+
+/* argument container for ops.init_task() */
+struct scx_init_task_args {
+ /*
+ * Set if ops.init_task() is being invoked on the fork path, as opposed
+ * to the scheduler transition path.
+ */
+ bool fork;
+};
+
+/* argument container for ops.exit_task() */
+struct scx_exit_task_args {
+ /* Whether the task exited before running on sched_ext. */
+ bool cancelled;
+};
+
+/**
+ * struct sched_ext_ops - Operation table for BPF scheduler implementation
+ *
+ * Userland can implement an arbitrary scheduling policy by implementing and
+ * loading operations in this table.
+ */
+struct sched_ext_ops {
+ /**
+ * select_cpu - Pick the target CPU for a task which is being woken up
+ * @p: task being woken up
+ * @prev_cpu: the cpu @p was on before sleeping
+ * @wake_flags: SCX_WAKE_*
+ *
+ * Decision made here isn't final. @p may be moved to any CPU while it
+ * is getting dispatched for execution later. However, as @p is not on
+ * the rq at this point, getting the eventual execution CPU right here
+ * saves a small bit of overhead down the line.
+ *
+ * If an idle CPU is returned, the CPU is kicked and will try to
+ * dispatch. While an explicit custom mechanism can be added,
+ * select_cpu() serves as the default way to wake up idle CPUs.
+ *
+ * @p may be dispatched directly by calling scx_bpf_dispatch(). If @p
+ * is dispatched, the ops.enqueue() callback will be skipped. Finally,
+ * if @p is dispatched to SCX_DSQ_LOCAL, it will be dispatched to the
+ * local DSQ of whatever CPU is returned by this callback.
+ */
+ s32 (*select_cpu)(struct task_struct *p, s32 prev_cpu, u64 wake_flags);
+
+ /**
+ * enqueue - Enqueue a task on the BPF scheduler
+ * @p: task being enqueued
+ * @enq_flags: %SCX_ENQ_*
+ *
+ * @p is ready to run. Dispatch directly by calling scx_bpf_dispatch()
+ * or enqueue on the BPF scheduler. If not directly dispatched, the bpf
+ * scheduler owns @p and if it fails to dispatch @p, the task will
+ * stall.
+ *
+ * If @p was dispatched from ops.select_cpu(), this callback is
+ * skipped.
+ */
+ void (*enqueue)(struct task_struct *p, u64 enq_flags);
+
+ /**
+ * dequeue - Remove a task from the BPF scheduler
+ * @p: task being dequeued
+ * @deq_flags: %SCX_DEQ_*
+ *
+ * Remove @p from the BPF scheduler. This is usually called to isolate
+ * the task while updating its scheduling properties (e.g. priority).
+ *
+ * The ext core keeps track of whether the BPF side owns a given task or
+ * not and can gracefully ignore spurious dispatches from BPF side,
+ * which makes it safe to not implement this method. However, depending
+ * on the scheduling logic, this can lead to confusing behaviors - e.g.
+ * scheduling position not being updated across a priority change.
+ */
+ void (*dequeue)(struct task_struct *p, u64 deq_flags);
+
+ /**
+ * dispatch - Dispatch tasks from the BPF scheduler and/or consume DSQs
+ * @cpu: CPU to dispatch tasks for
+ * @prev: previous task being switched out
+ *
+ * Called when a CPU's local dsq is empty. The operation should dispatch
+ * one or more tasks from the BPF scheduler into the DSQs using
+ * scx_bpf_dispatch() and/or consume user DSQs into the local DSQ using
+ * scx_bpf_consume().
+ *
+ * The maximum number of times scx_bpf_dispatch() can be called without
+ * an intervening scx_bpf_consume() is specified by
+ * ops.dispatch_max_batch. See the comments on top of the two functions
+ * for more details.
+ *
+ * When not %NULL, @prev is an SCX task with its slice depleted. If
+ * @prev is still runnable as indicated by set %SCX_TASK_QUEUED in
+ * @prev->scx.flags, it is not enqueued yet and will be enqueued after
+ * ops.dispatch() returns. To keep executing @prev, return without
+ * dispatching or consuming any tasks. Also see %SCX_OPS_ENQ_LAST.
+ */
+ void (*dispatch)(s32 cpu, struct task_struct *prev);
+
+ /**
+ * tick - Periodic tick
+ * @p: task running currently
+ *
+ * This operation is called every 1/HZ seconds on CPUs which are
+ * executing an SCX task. Setting @p->scx.slice to 0 will trigger an
+ * immediate dispatch cycle on the CPU.
+ */
+ void (*tick)(struct task_struct *p);
+
+ /**
+ * yield - Yield CPU
+ * @from: yielding task
+ * @to: optional yield target task
+ *
+ * If @to is NULL, @from is yielding the CPU to other runnable tasks.
+ * The BPF scheduler should ensure that other available tasks are
+ * dispatched before the yielding task. Return value is ignored in this
+ * case.
+ *
+ * If @to is not-NULL, @from wants to yield the CPU to @to. If the bpf
+ * scheduler can implement the request, return %true; otherwise, %false.
+ */
+ bool (*yield)(struct task_struct *from, struct task_struct *to);
+
+ /**
+ * set_weight - Set task weight
+ * @p: task to set weight for
+ * @weight: new eight [1..10000]
+ *
+ * Update @p's weight to @weight.
+ */
+ void (*set_weight)(struct task_struct *p, u32 weight);
+
+ /**
+ * set_cpumask - Set CPU affinity
+ * @p: task to set CPU affinity for
+ * @cpumask: cpumask of cpus that @p can run on
+ *
+ * Update @p's CPU affinity to @cpumask.
+ */
+ void (*set_cpumask)(struct task_struct *p,
+ const struct cpumask *cpumask);
+
+ /**
+ * update_idle - Update the idle state of a CPU
+ * @cpu: CPU to udpate the idle state for
+ * @idle: whether entering or exiting the idle state
+ *
+ * This operation is called when @rq's CPU goes or leaves the idle
+ * state. By default, implementing this operation disables the built-in
+ * idle CPU tracking and the following helpers become unavailable:
+ *
+ * - scx_bpf_select_cpu_dfl()
+ * - scx_bpf_test_and_clear_cpu_idle()
+ * - scx_bpf_pick_idle_cpu()
+ *
+ * The user also must implement ops.select_cpu() as the default
+ * implementation relies on scx_bpf_select_cpu_dfl().
+ *
+ * Specify the %SCX_OPS_KEEP_BUILTIN_IDLE flag to keep the built-in idle
+ * tracking.
+ */
+ void (*update_idle)(s32 cpu, bool idle);
+
+ /**
+ * init_task - Initialize a task to run in a BPF scheduler
+ * @p: task to initialize for BPF scheduling
+ * @args: init arguments, see the struct definition
+ *
+ * Either we're loading a BPF scheduler or a new task is being forked.
+ * Initialize @p for BPF scheduling. This operation may block and can
+ * be used for allocations, and is called exactly once for a task.
+ *
+ * Return 0 for success, -errno for failure. An error return while
+ * loading will abort loading of the BPF scheduler. During a fork, it
+ * will abort that specific fork.
+ */
+ s32 (*init_task)(struct task_struct *p, struct scx_init_task_args *args);
+
+ /**
+ * exit_task - Exit a previously-running task from the system
+ * @p: task to exit
+ *
+ * @p is exiting or the BPF scheduler is being unloaded. Perform any
+ * necessary cleanup for @p.
+ */
+ void (*exit_task)(struct task_struct *p, struct scx_exit_task_args *args);
+
+ /**
+ * enable - Enable BPF scheduling for a task
+ * @p: task to enable BPF scheduling for
+ *
+ * Enable @p for BPF scheduling. enable() is called on @p any time it
+ * enters SCX, and is always paired with a matching disable().
+ */
+ void (*enable)(struct task_struct *p);
+
+ /**
+ * disable - Disable BPF scheduling for a task
+ * @p: task to disable BPF scheduling for
+ *
+ * @p is exiting, leaving SCX or the BPF scheduler is being unloaded.
+ * Disable BPF scheduling for @p. A disable() call is always matched
+ * with a prior enable() call.
+ */
+ void (*disable)(struct task_struct *p);
+
+ /*
+ * All online ops must come before ops.init().
+ */
+
+ /**
+ * init - Initialize the BPF scheduler
+ */
+ s32 (*init)(void);
+
+ /**
+ * exit - Clean up after the BPF scheduler
+ * @info: Exit info
+ */
+ void (*exit)(struct scx_exit_info *info);
+
+ /**
+ * dispatch_max_batch - Max nr of tasks that dispatch() can dispatch
+ */
+ u32 dispatch_max_batch;
+
+ /**
+ * flags - %SCX_OPS_* flags
+ */
+ u64 flags;
+
+ /**
+ * name - BPF scheduler's name
+ *
+ * Must be a non-zero valid BPF object name including only isalnum(),
+ * '_' and '.' chars. Shows up in kernel.sched_ext_ops sysctl while the
+ * BPF scheduler is enabled.
+ */
+ char name[SCX_OPS_NAME_LEN];
+};
+
+enum scx_opi {
+ SCX_OPI_BEGIN = 0,
+ SCX_OPI_NORMAL_BEGIN = 0,
+ SCX_OPI_NORMAL_END = SCX_OP_IDX(init),
+ SCX_OPI_END = SCX_OP_IDX(init),
+};
+
+enum scx_wake_flags {
+ /* expose select WF_* flags as enums */
+ SCX_WAKE_FORK = WF_FORK,
+ SCX_WAKE_TTWU = WF_TTWU,
+ SCX_WAKE_SYNC = WF_SYNC,
+};
+
+enum scx_enq_flags {
+ /* expose select ENQUEUE_* flags as enums */
+ SCX_ENQ_WAKEUP = ENQUEUE_WAKEUP,
+ SCX_ENQ_HEAD = ENQUEUE_HEAD,
+
+ /* high 32bits are SCX specific */
+
+ /*
+ * The task being enqueued is the only task available for the cpu. By
+ * default, ext core keeps executing such tasks but when
+ * %SCX_OPS_ENQ_LAST is specified, they're ops.enqueue()'d with the
+ * %SCX_ENQ_LAST flag set.
+ *
+ * If the BPF scheduler wants to continue executing the task,
+ * ops.enqueue() should dispatch the task to %SCX_DSQ_LOCAL immediately.
+ * If the task gets queued on a different dsq or the BPF side, the BPF
+ * scheduler is responsible for triggering a follow-up scheduling event.
+ * Otherwise, Execution may stall.
+ */
+ SCX_ENQ_LAST = 1LLU << 41,
+
+ /* high 8 bits are internal */
+ __SCX_ENQ_INTERNAL_MASK = 0xffLLU << 56,
+
+ SCX_ENQ_CLEAR_OPSS = 1LLU << 56,
+};
+
+enum scx_deq_flags {
+ /* expose select DEQUEUE_* flags as enums */
+ SCX_DEQ_SLEEP = DEQUEUE_SLEEP,
+};
+
+enum scx_pick_idle_cpu_flags {
+ SCX_PICK_IDLE_CORE = 1LLU << 0, /* pick a CPU whose SMT siblings are also idle */
+};
+
+enum scx_ops_enable_state {
+ SCX_OPS_PREPPING,
+ SCX_OPS_ENABLING,
+ SCX_OPS_ENABLED,
+ SCX_OPS_DISABLING,
+ SCX_OPS_DISABLED,
+};
+
+static const char *scx_ops_enable_state_str[] = {
+ [SCX_OPS_PREPPING] = "prepping",
+ [SCX_OPS_ENABLING] = "enabling",
+ [SCX_OPS_ENABLED] = "enabled",
+ [SCX_OPS_DISABLING] = "disabling",
+ [SCX_OPS_DISABLED] = "disabled",
+};
+
+/*
+ * sched_ext_entity->ops_state
+ *
+ * Used to track the task ownership between the SCX core and the BPF scheduler.
+ * State transitions look as follows:
+ *
+ * NONE -> QUEUEING -> QUEUED -> DISPATCHING
+ * ^ | |
+ * | v v
+ * \-------------------------------/
+ *
+ * QUEUEING and DISPATCHING states can be waited upon. See wait_ops_state() call
+ * sites for explanations on the conditions being waited upon and why they are
+ * safe. Transitions out of them into NONE or QUEUED must store_release and the
+ * waiters should load_acquire.
+ *
+ * Tracking scx_ops_state enables sched_ext core to reliably determine whether
+ * any given task can be dispatched by the BPF scheduler at all times and thus
+ * relaxes the requirements on the BPF scheduler. This allows the BPF scheduler
+ * to try to dispatch any task anytime regardless of its state as the SCX core
+ * can safely reject invalid dispatches.
+ */
+enum scx_ops_state {
+ SCX_OPSS_NONE, /* owned by the SCX core */
+ SCX_OPSS_QUEUEING, /* in transit to the BPF scheduler */
+ SCX_OPSS_QUEUED, /* owned by the BPF scheduler */
+ SCX_OPSS_DISPATCHING, /* in transit back to the SCX core */
+
+ /*
+ * QSEQ brands each QUEUED instance so that, when dispatch races
+ * dequeue/requeue, the dispatcher can tell whether it still has a claim
+ * on the task being dispatched.
+ *
+ * As some 32bit archs can't do 64bit store_release/load_acquire,
+ * p->scx.ops_state is atomic_long_t which leaves 30 bits for QSEQ on
+ * 32bit machines. The dispatch race window QSEQ protects is very narrow
+ * and runs with IRQ disabled. 30 bits should be sufficient.
+ */
+ SCX_OPSS_QSEQ_SHIFT = 2,
+};
+
+/* Use macros to ensure that the type is unsigned long for the masks */
+#define SCX_OPSS_STATE_MASK ((1LU << SCX_OPSS_QSEQ_SHIFT) - 1)
+#define SCX_OPSS_QSEQ_MASK (~SCX_OPSS_STATE_MASK)
+
+/*
+ * During exit, a task may schedule after losing its PIDs. When disabling the
+ * BPF scheduler, we need to be able to iterate tasks in every state to
+ * guarantee system safety. Maintain a dedicated task list which contains every
+ * task between its fork and eventual free.
+ */
+static DEFINE_SPINLOCK(scx_tasks_lock);
+static LIST_HEAD(scx_tasks);
+
+/* ops enable/disable */
+static struct kthread_worker *scx_ops_helper;
+static DEFINE_MUTEX(scx_ops_enable_mutex);
+DEFINE_STATIC_KEY_FALSE(__scx_ops_enabled);
+DEFINE_STATIC_PERCPU_RWSEM(scx_fork_rwsem);
+static atomic_t scx_ops_enable_state_var = ATOMIC_INIT(SCX_OPS_DISABLED);
+static atomic_t scx_ops_bypass_depth = ATOMIC_INIT(0);
+static bool scx_switching_all;
+DEFINE_STATIC_KEY_FALSE(__scx_switched_all);
+
+static struct sched_ext_ops scx_ops;
+static bool scx_warned_zero_slice;
+
+static DEFINE_STATIC_KEY_FALSE(scx_ops_enq_last);
+static DEFINE_STATIC_KEY_FALSE(scx_ops_enq_exiting);
+static DEFINE_STATIC_KEY_FALSE(scx_builtin_idle_enabled);
+
+struct static_key_false scx_has_op[SCX_OPI_END] =
+ { [0 ... SCX_OPI_END-1] = STATIC_KEY_FALSE_INIT };
+
+static atomic_t scx_exit_kind = ATOMIC_INIT(SCX_EXIT_DONE);
+static struct scx_exit_info *scx_exit_info;
+
+/* idle tracking */
+#ifdef CONFIG_SMP
+#ifdef CONFIG_CPUMASK_OFFSTACK
+#define CL_ALIGNED_IF_ONSTACK
+#else
+#define CL_ALIGNED_IF_ONSTACK __cacheline_aligned_in_smp
+#endif
+
+static struct {
+ cpumask_var_t cpu;
+ cpumask_var_t smt;
+} idle_masks CL_ALIGNED_IF_ONSTACK;
+
+#endif /* CONFIG_SMP */
+
+/*
+ * Direct dispatch marker.
+ *
+ * Non-NULL values are used for direct dispatch from enqueue path. A valid
+ * pointer points to the task currently being enqueued. An ERR_PTR value is used
+ * to indicate that direct dispatch has already happened.
+ */
+static DEFINE_PER_CPU(struct task_struct *, direct_dispatch_task);
+
+/* dispatch queues */
+static struct scx_dispatch_q __cacheline_aligned_in_smp scx_dsq_global;
+
+static const struct rhashtable_params dsq_hash_params = {
+ .key_len = 8,
+ .key_offset = offsetof(struct scx_dispatch_q, id),
+ .head_offset = offsetof(struct scx_dispatch_q, hash_node),
+};
+
+static struct rhashtable dsq_hash;
+static LLIST_HEAD(dsqs_to_free);
+
+/* dispatch buf */
+struct scx_dsp_buf_ent {
+ struct task_struct *task;
+ unsigned long qseq;
+ u64 dsq_id;
+ u64 enq_flags;
+};
+
+static u32 scx_dsp_max_batch;
+
+struct scx_dsp_ctx {
+ struct rq *rq;
+ struct rq_flags *rf;
+ u32 cursor;
+ u32 nr_tasks;
+ struct scx_dsp_buf_ent buf[];
+};
+
+static struct scx_dsp_ctx __percpu *scx_dsp_ctx;
+
+/* string formatting from BPF */
+struct scx_bstr_buf {
+ u64 data[MAX_BPRINTF_VARARGS];
+ char line[SCX_EXIT_MSG_LEN];
+};
+
+static DEFINE_RAW_SPINLOCK(scx_exit_bstr_buf_lock);
+static struct scx_bstr_buf scx_exit_bstr_buf;
+
+/* /sys/kernel/sched_ext interface */
+static struct kset *scx_kset;
+static struct kobject *scx_root_kobj;
+
+static __printf(3, 4) void scx_ops_exit_kind(enum scx_exit_kind kind,
+ s64 exit_code,
+ const char *fmt, ...);
+
+#define scx_ops_error_kind(err, fmt, args...) \
+ scx_ops_exit_kind((err), 0, fmt, ##args)
+
+#define scx_ops_exit(code, fmt, args...) \
+ scx_ops_exit_kind(SCX_EXIT_UNREG_KERN, (code), fmt, ##args)
+
+#define scx_ops_error(fmt, args...) \
+ scx_ops_error_kind(SCX_EXIT_ERROR, fmt, ##args)
+
+#define SCX_HAS_OP(op) static_branch_likely(&scx_has_op[SCX_OP_IDX(op)])
+
+/* if the highest set bit is N, return a mask with bits [N+1, 31] set */
+static u32 higher_bits(u32 flags)
+{
+ return ~((1 << fls(flags)) - 1);
+}
+
+/* return the mask with only the highest bit set */
+static u32 highest_bit(u32 flags)
+{
+ int bit = fls(flags);
+ return ((u64)1 << bit) >> 1;
+}
+
+/*
+ * scx_kf_mask enforcement. Some kfuncs can only be called from specific SCX
+ * ops. When invoking SCX ops, SCX_CALL_OP[_RET]() should be used to indicate
+ * the allowed kfuncs and those kfuncs should use scx_kf_allowed() to check
+ * whether it's running from an allowed context.
+ *
+ * @mask is constant, always inline to cull the mask calculations.
+ */
+static __always_inline void scx_kf_allow(u32 mask)
+{
+ /* nesting is allowed only in increasing scx_kf_mask order */
+ WARN_ONCE((mask | higher_bits(mask)) & current->scx.kf_mask,
+ "invalid nesting current->scx.kf_mask=0x%x mask=0x%x\n",
+ current->scx.kf_mask, mask);
+ current->scx.kf_mask |= mask;
+ barrier();
+}
+
+static void scx_kf_disallow(u32 mask)
+{
+ barrier();
+ current->scx.kf_mask &= ~mask;
+}
+
+#define SCX_CALL_OP(mask, op, args...) \
+do { \
+ if (mask) { \
+ scx_kf_allow(mask); \
+ scx_ops.op(args); \
+ scx_kf_disallow(mask); \
+ } else { \
+ scx_ops.op(args); \
+ } \
+} while (0)
+
+#define SCX_CALL_OP_RET(mask, op, args...) \
+({ \
+ __typeof__(scx_ops.op(args)) __ret; \
+ if (mask) { \
+ scx_kf_allow(mask); \
+ __ret = scx_ops.op(args); \
+ scx_kf_disallow(mask); \
+ } else { \
+ __ret = scx_ops.op(args); \
+ } \
+ __ret; \
+})
+
+/* @mask is constant, always inline to cull unnecessary branches */
+static __always_inline bool scx_kf_allowed(u32 mask)
+{
+ if (unlikely(!(current->scx.kf_mask & mask))) {
+ scx_ops_error("kfunc with mask 0x%x called from an operation only allowing 0x%x",
+ mask, current->scx.kf_mask);
+ return false;
+ }
+
+ if (unlikely((mask & SCX_KF_SLEEPABLE) && in_interrupt())) {
+ scx_ops_error("sleepable kfunc called from non-sleepable context");
+ return false;
+ }
+
+ /*
+ * Enforce nesting boundaries. e.g. A kfunc which can be called from
+ * DISPATCH must not be called if we're running DEQUEUE which is nested
+ * inside ops.dispatch(). We don't need to check the SCX_KF_SLEEPABLE
+ * boundary thanks to the above in_interrupt() check.
+ */
+ if (unlikely(highest_bit(mask) == SCX_KF_DISPATCH &&
+ (current->scx.kf_mask & higher_bits(SCX_KF_DISPATCH)))) {
+ scx_ops_error("dispatch kfunc called from a nested operation");
+ return false;
+ }
+
+ return true;
+}
+
+
+/*
+ * SCX task iterator.
+ */
+struct scx_task_iter {
+ struct sched_ext_entity cursor;
+ struct task_struct *locked;
+ struct rq *rq;
+ struct rq_flags rf;
+};
+
+/**
+ * scx_task_iter_init - Initialize a task iterator
+ * @iter: iterator to init
+ *
+ * Initialize @iter. Must be called with scx_tasks_lock held. Once initialized,
+ * @iter must eventually be exited with scx_task_iter_exit().
+ *
+ * scx_tasks_lock may be released between this and the first next() call or
+ * between any two next() calls. If scx_tasks_lock is released between two
+ * next() calls, the caller is responsible for ensuring that the task being
+ * iterated remains accessible either through RCU read lock or obtaining a
+ * reference count.
+ *
+ * All tasks which existed when the iteration started are guaranteed to be
+ * visited as long as they still exist.
+ */
+static void scx_task_iter_init(struct scx_task_iter *iter)
+{
+ lockdep_assert_held(&scx_tasks_lock);
+
+ iter->cursor = (struct sched_ext_entity){ .flags = SCX_TASK_CURSOR };
+ list_add(&iter->cursor.tasks_node, &scx_tasks);
+ iter->locked = NULL;
+}
+
+/**
+ * scx_task_iter_rq_unlock - Unlock rq locked by a task iterator
+ * @iter: iterator to unlock rq for
+ *
+ * If @iter is in the middle of a locked iteration, it may be locking the rq of
+ * the task currently being visited. Unlock the rq if so. This function can be
+ * safely called anytime during an iteration.
+ *
+ * Returns %true if the rq @iter was locking is unlocked. %false if @iter was
+ * not locking an rq.
+ */
+static bool scx_task_iter_rq_unlock(struct scx_task_iter *iter)
+{
+ if (iter->locked) {
+ task_rq_unlock(iter->rq, iter->locked, &iter->rf);
+ iter->locked = NULL;
+ return true;
+ } else {
+ return false;
+ }
+}
+
+/**
+ * scx_task_iter_exit - Exit a task iterator
+ * @iter: iterator to exit
+ *
+ * Exit a previously initialized @iter. Must be called with scx_tasks_lock held.
+ * If the iterator holds a task's rq lock, that rq lock is released. See
+ * scx_task_iter_init() for details.
+ */
+static void scx_task_iter_exit(struct scx_task_iter *iter)
+{
+ lockdep_assert_held(&scx_tasks_lock);
+
+ scx_task_iter_rq_unlock(iter);
+ list_del_init(&iter->cursor.tasks_node);
+}
+
+/**
+ * scx_task_iter_next - Next task
+ * @iter: iterator to walk
+ *
+ * Visit the next task. See scx_task_iter_init() for details.
+ */
+static struct task_struct *scx_task_iter_next(struct scx_task_iter *iter)
+{
+ struct list_head *cursor = &iter->cursor.tasks_node;
+ struct sched_ext_entity *pos;
+
+ lockdep_assert_held(&scx_tasks_lock);
+
+ list_for_each_entry(pos, cursor, tasks_node) {
+ if (&pos->tasks_node == &scx_tasks)
+ return NULL;
+ if (!(pos->flags & SCX_TASK_CURSOR)) {
+ list_move(cursor, &pos->tasks_node);
+ return container_of(pos, struct task_struct, scx);
+ }
+ }
+
+ /* can't happen, should always terminate at scx_tasks above */
+ BUG();
+}
+
+/**
+ * scx_task_iter_next_locked - Next non-idle task with its rq locked
+ * @iter: iterator to walk
+ * @include_dead: Whether we should include dead tasks in the iteration
+ *
+ * Visit the non-idle task with its rq lock held. Allows callers to specify
+ * whether they would like to filter out dead tasks. See scx_task_iter_init()
+ * for details.
+ */
+static struct task_struct *
+scx_task_iter_next_locked(struct scx_task_iter *iter, bool include_dead)
+{
+ struct task_struct *p;
+retry:
+ scx_task_iter_rq_unlock(iter);
+
+ while ((p = scx_task_iter_next(iter))) {
+ /*
+ * is_idle_task() tests %PF_IDLE which may not be set for CPUs
+ * which haven't yet been onlined. Test sched_class directly.
+ */
+ if (p->sched_class != &idle_sched_class)
+ break;
+ }
+ if (!p)
+ return NULL;
+
+ iter->rq = task_rq_lock(p, &iter->rf);
+ iter->locked = p;
+
+ /*
+ * If we see %TASK_DEAD, @p already disabled preemption, is about to do
+ * the final __schedule(), won't ever need to be scheduled again and can
+ * thus be safely ignored. If we don't see %TASK_DEAD, @p can't enter
+ * the final __schedle() while we're locking its rq and thus will stay
+ * alive until the rq is unlocked.
+ */
+ if (!include_dead && READ_ONCE(p->__state) == TASK_DEAD)
+ goto retry;
+
+ return p;
+}
+
+static enum scx_ops_enable_state scx_ops_enable_state(void)
+{
+ return atomic_read(&scx_ops_enable_state_var);
+}
+
+static enum scx_ops_enable_state
+scx_ops_set_enable_state(enum scx_ops_enable_state to)
+{
+ return atomic_xchg(&scx_ops_enable_state_var, to);
+}
+
+static bool scx_ops_tryset_enable_state(enum scx_ops_enable_state to,
+ enum scx_ops_enable_state from)
+{
+ int from_v = from;
+
+ return atomic_try_cmpxchg(&scx_ops_enable_state_var, &from_v, to);
+}
+
+static bool scx_ops_bypassing(void)
+{
+ return unlikely(atomic_read(&scx_ops_bypass_depth));
+}
+
+/**
+ * wait_ops_state - Busy-wait the specified ops state to end
+ * @p: target task
+ * @opss: state to wait the end of
+ *
+ * Busy-wait for @p to transition out of @opss. This can only be used when the
+ * state part of @opss is %SCX_QUEUEING or %SCX_DISPATCHING. This function also
+ * has load_acquire semantics to ensure that the caller can see the updates made
+ * in the enqueueing and dispatching paths.
+ */
+static void wait_ops_state(struct task_struct *p, unsigned long opss)
+{
+ do {
+ cpu_relax();
+ } while (atomic_long_read_acquire(&p->scx.ops_state) == opss);
+}
+
+/**
+ * ops_cpu_valid - Verify a cpu number
+ * @cpu: cpu number which came from a BPF ops
+ * @where: extra information reported on error
+ *
+ * @cpu is a cpu number which came from the BPF scheduler and can be any value.
+ * Verify that it is in range and one of the possible cpus. If invalid, trigger
+ * an ops error.
+ */
+static bool ops_cpu_valid(s32 cpu, const char *where)
+{
+ if (likely(cpu >= 0 && cpu < nr_cpu_ids && cpu_possible(cpu))) {
+ return true;
+ } else {
+ scx_ops_error("invalid CPU %d%s%s", cpu,
+ where ? " " : "", where ?: "");
+ return false;
+ }
+}
+
+/**
+ * ops_sanitize_err - Sanitize a -errno value
+ * @ops_name: operation to blame on failure
+ * @err: -errno value to sanitize
+ *
+ * Verify @err is a valid -errno. If not, trigger scx_ops_error() and return
+ * -%EPROTO. This is necessary because returning a rogue -errno up the chain can
+ * cause misbehaviors. For an example, a large negative return from
+ * ops.init_task() triggers an oops when passed up the call chain because the
+ * value fails IS_ERR() test after being encoded with ERR_PTR() and then is
+ * handled as a pointer.
+ */
+static int ops_sanitize_err(const char *ops_name, s32 err)
+{
+ if (err < 0 && err >= -MAX_ERRNO)
+ return err;
+
+ scx_ops_error("ops.%s() returned an invalid errno %d", ops_name, err);
+ return -EPROTO;
+}
+
+static void update_curr_scx(struct rq *rq)
+{
+ struct task_struct *curr = rq->curr;
+ u64 now = rq_clock_task(rq);
+ u64 delta_exec;
+
+ if (time_before_eq64(now, curr->se.exec_start))
+ return;
+
+ delta_exec = now - curr->se.exec_start;
+ curr->se.exec_start = now;
+ curr->se.sum_exec_runtime += delta_exec;
+ account_group_exec_runtime(curr, delta_exec);
+ cgroup_account_cputime(curr, delta_exec);
+
+ curr->scx.slice -= min(curr->scx.slice, delta_exec);
+}
+
+static void dsq_mod_nr(struct scx_dispatch_q *dsq, s32 delta)
+{
+ /* scx_bpf_dsq_nr_queued() reads ->nr without locking, use WRITE_ONCE() */
+ WRITE_ONCE(dsq->nr, dsq->nr + delta);
+}
+
+static void dispatch_enqueue(struct scx_dispatch_q *dsq, struct task_struct *p,
+ u64 enq_flags)
+{
+ bool is_local = dsq->id == SCX_DSQ_LOCAL;
+
+ WARN_ON_ONCE(p->scx.dsq || !list_empty(&p->scx.dsq_node));
+
+ if (!is_local) {
+ raw_spin_lock(&dsq->lock);
+ if (unlikely(dsq->id == SCX_DSQ_INVALID)) {
+ scx_ops_error("attempting to dispatch to a destroyed dsq");
+ /* fall back to the global dsq */
+ raw_spin_unlock(&dsq->lock);
+ dsq = &scx_dsq_global;
+ raw_spin_lock(&dsq->lock);
+ }
+ }
+
+ if (enq_flags & SCX_ENQ_HEAD)
+ list_add(&p->scx.dsq_node, &dsq->list);
+ else
+ list_add_tail(&p->scx.dsq_node, &dsq->list);
+
+ dsq_mod_nr(dsq, 1);
+ p->scx.dsq = dsq;
+
+ /*
+ * scx.ddsp_dsq_id and scx.ddsp_enq_flags are only relevant on the
+ * direct dispatch path, but we clear them here because the direct
+ * dispatch verdict may be overridden on the enqueue path during e.g.
+ * bypass.
+ */
+ p->scx.ddsp_dsq_id = SCX_DSQ_INVALID;
+ p->scx.ddsp_enq_flags = 0;
+
+ /*
+ * We're transitioning out of QUEUEING or DISPATCHING. store_release to
+ * match waiters' load_acquire.
+ */
+ if (enq_flags & SCX_ENQ_CLEAR_OPSS)
+ atomic_long_set_release(&p->scx.ops_state, SCX_OPSS_NONE);
+
+ if (is_local) {
+ struct rq *rq = container_of(dsq, struct rq, scx.local_dsq);
+
+ if (sched_class_above(&ext_sched_class, rq->curr->sched_class))
+ resched_curr(rq);
+ } else {
+ raw_spin_unlock(&dsq->lock);
+ }
+}
+
+static void dispatch_dequeue(struct rq *rq, struct task_struct *p)
+{
+ struct scx_dispatch_q *dsq = p->scx.dsq;
+ bool is_local = dsq == &rq->scx.local_dsq;
+
+ if (!dsq) {
+ WARN_ON_ONCE(!list_empty(&p->scx.dsq_node));
+ /*
+ * When dispatching directly from the BPF scheduler to a local
+ * DSQ, the task isn't associated with any DSQ but
+ * @p->scx.holding_cpu may be set under the protection of
+ * %SCX_OPSS_DISPATCHING.
+ */
+ if (p->scx.holding_cpu >= 0)
+ p->scx.holding_cpu = -1;
+ return;
+ }
+
+ if (!is_local)
+ raw_spin_lock(&dsq->lock);
+
+ /*
+ * Now that we hold @dsq->lock, @p->holding_cpu and @p->scx.dsq_node
+ * can't change underneath us.
+ */
+ if (p->scx.holding_cpu < 0) {
+ /* @p must still be on @dsq, dequeue */
+ WARN_ON_ONCE(list_empty(&p->scx.dsq_node));
+ list_del_init(&p->scx.dsq_node);
+ dsq_mod_nr(dsq, -1);
+ } else {
+ /*
+ * We're racing against dispatch_to_local_dsq() which already
+ * removed @p from @dsq and set @p->scx.holding_cpu. Clear the
+ * holding_cpu which tells dispatch_to_local_dsq() that it lost
+ * the race.
+ */
+ WARN_ON_ONCE(!list_empty(&p->scx.dsq_node));
+ p->scx.holding_cpu = -1;
+ }
+ p->scx.dsq = NULL;
+
+ if (!is_local)
+ raw_spin_unlock(&dsq->lock);
+}
+
+static struct scx_dispatch_q *find_user_dsq(u64 dsq_id)
+{
+ return rhashtable_lookup_fast(&dsq_hash, &dsq_id, dsq_hash_params);
+}
+
+static struct scx_dispatch_q *find_non_local_dsq(u64 dsq_id)
+{
+ lockdep_assert(rcu_read_lock_any_held());
+
+ if (dsq_id == SCX_DSQ_GLOBAL)
+ return &scx_dsq_global;
+ else
+ return find_user_dsq(dsq_id);
+}
+
+static struct scx_dispatch_q *find_dsq_for_dispatch(struct rq *rq, u64 dsq_id,
+ struct task_struct *p)
+{
+ struct scx_dispatch_q *dsq;
+
+ if (dsq_id == SCX_DSQ_LOCAL)
+ return &rq->scx.local_dsq;
+
+ dsq = find_non_local_dsq(dsq_id);
+ if (unlikely(!dsq)) {
+ scx_ops_error("non-existent DSQ 0x%llx for %s[%d]",
+ dsq_id, p->comm, p->pid);
+ return &scx_dsq_global;
+ }
+
+ return dsq;
+}
+
+static void mark_direct_dispatch(struct task_struct *ddsp_task,
+ struct task_struct *p, u64 dsq_id,
+ u64 enq_flags)
+{
+ /*
+ * Mark that dispatch already happened from ops.select_cpu() or
+ * ops.enqueue() by spoiling direct_dispatch_task with a non-NULL value
+ * which can never match a valid task pointer.
+ */
+ __this_cpu_write(direct_dispatch_task, ERR_PTR(-ESRCH));
+
+ /* @p must match the task on the enqueue path */
+ if (unlikely(p != ddsp_task)) {
+ if (IS_ERR(ddsp_task))
+ scx_ops_error("%s[%d] already direct-dispatched",
+ p->comm, p->pid);
+ else
+ scx_ops_error("scheduling for %s[%d] but trying to direct-dispatch %s[%d]",
+ ddsp_task->comm, ddsp_task->pid,
+ p->comm, p->pid);
+ return;
+ }
+
+ /*
+ * %SCX_DSQ_LOCAL_ON is not supported during direct dispatch because
+ * dispatching to the local DSQ of a different CPU requires unlocking
+ * the current rq which isn't allowed in the enqueue path. Use
+ * ops.select_cpu() to be on the target CPU and then %SCX_DSQ_LOCAL.
+ */
+ if (unlikely((dsq_id & SCX_DSQ_LOCAL_ON) == SCX_DSQ_LOCAL_ON)) {
+ scx_ops_error("SCX_DSQ_LOCAL_ON can't be used for direct-dispatch");
+ return;
+ }
+
+ WARN_ON_ONCE(p->scx.ddsp_dsq_id != SCX_DSQ_INVALID);
+ WARN_ON_ONCE(p->scx.ddsp_enq_flags);
+
+ p->scx.ddsp_dsq_id = dsq_id;
+ p->scx.ddsp_enq_flags = enq_flags;
+}
+
+static void direct_dispatch(struct task_struct *p, u64 enq_flags)
+{
+ struct scx_dispatch_q *dsq;
+
+ enq_flags |= (p->scx.ddsp_enq_flags | SCX_ENQ_CLEAR_OPSS);
+ dsq = find_dsq_for_dispatch(task_rq(p), p->scx.ddsp_dsq_id, p);
+ dispatch_enqueue(dsq, p, enq_flags);
+}
+
+static bool scx_rq_online(struct rq *rq)
+{
+#ifdef CONFIG_SMP
+ return likely(rq->online);
+#else
+ return true;
+#endif
+}
+
+static void do_enqueue_task(struct rq *rq, struct task_struct *p, u64 enq_flags,
+ int sticky_cpu)
+{
+ struct task_struct **ddsp_taskp;
+ unsigned long qseq;
+
+ WARN_ON_ONCE(!(p->scx.flags & SCX_TASK_QUEUED));
+
+ /* rq migration */
+ if (sticky_cpu == cpu_of(rq))
+ goto local_norefill;
+
+ if (!scx_rq_online(rq))
+ goto local;
+
+ if (scx_ops_bypassing()) {
+ if (enq_flags & SCX_ENQ_LAST)
+ goto local;
+ else
+ goto global;
+ }
+
+ if (p->scx.ddsp_dsq_id != SCX_DSQ_INVALID)
+ goto direct;
+
+ /* see %SCX_OPS_ENQ_EXITING */
+ if (!static_branch_unlikely(&scx_ops_enq_exiting) &&
+ unlikely(p->flags & PF_EXITING))
+ goto local;
+
+ /* see %SCX_OPS_ENQ_LAST */
+ if (!static_branch_unlikely(&scx_ops_enq_last) &&
+ (enq_flags & SCX_ENQ_LAST))
+ goto local;
+
+ if (!SCX_HAS_OP(enqueue))
+ goto global;
+
+ /* DSQ bypass didn't trigger, enqueue on the BPF scheduler */
+ qseq = rq->scx.ops_qseq++ << SCX_OPSS_QSEQ_SHIFT;
+
+ WARN_ON_ONCE(atomic_long_read(&p->scx.ops_state) != SCX_OPSS_NONE);
+ atomic_long_set(&p->scx.ops_state, SCX_OPSS_QUEUEING | qseq);
+
+ ddsp_taskp = this_cpu_ptr(&direct_dispatch_task);
+ WARN_ON_ONCE(*ddsp_taskp);
+ *ddsp_taskp = p;
+
+ SCX_CALL_OP(SCX_KF_ENQUEUE, enqueue, p, enq_flags);
+
+ *ddsp_taskp = NULL;
+ if (p->scx.ddsp_dsq_id != SCX_DSQ_INVALID)
+ goto direct;
+
+ /*
+ * If not directly dispatched, QUEUEING isn't clear yet and dispatch or
+ * dequeue may be waiting. The store_release matches their load_acquire.
+ */
+ atomic_long_set_release(&p->scx.ops_state, SCX_OPSS_QUEUED | qseq);
+ return;
+
+direct:
+ direct_dispatch(p, enq_flags);
+ return;
+
+local:
+ p->scx.slice = SCX_SLICE_DFL;
+local_norefill:
+ dispatch_enqueue(&rq->scx.local_dsq, p, enq_flags);
+ return;
+
+global:
+ p->scx.slice = SCX_SLICE_DFL;
+ dispatch_enqueue(&scx_dsq_global, p, enq_flags);
+}
+
+static bool task_runnable(const struct task_struct *p)
+{
+ return !list_empty(&p->scx.runnable_node);
+}
+
+static void set_task_runnable(struct rq *rq, struct task_struct *p)
+{
+ lockdep_assert_rq_held(rq);
+
+ /*
+ * list_add_tail() must be used. scx_ops_bypass() depends on tasks being
+ * appened to the runnable_list.
+ */
+ list_add_tail(&p->scx.runnable_node, &rq->scx.runnable_list);
+}
+
+static void clr_task_runnable(struct task_struct *p)
+{
+ list_del_init(&p->scx.runnable_node);
+}
+
+static void enqueue_task_scx(struct rq *rq, struct task_struct *p, int enq_flags)
+{
+ int sticky_cpu = p->scx.sticky_cpu;
+
+ enq_flags |= rq->scx.extra_enq_flags;
+
+ if (sticky_cpu >= 0)
+ p->scx.sticky_cpu = -1;
+
+ /*
+ * Restoring a running task will be immediately followed by
+ * set_next_task_scx() which expects the task to not be on the BPF
+ * scheduler as tasks can only start running through local DSQs. Force
+ * direct-dispatch into the local DSQ by setting the sticky_cpu.
+ */
+ if (unlikely(enq_flags & ENQUEUE_RESTORE) && task_current(rq, p))
+ sticky_cpu = cpu_of(rq);
+
+ if (p->scx.flags & SCX_TASK_QUEUED) {
+ WARN_ON_ONCE(!task_runnable(p));
+ return;
+ }
+
+ set_task_runnable(rq, p);
+ p->scx.flags |= SCX_TASK_QUEUED;
+ rq->scx.nr_running++;
+ add_nr_running(rq, 1);
+
+ do_enqueue_task(rq, p, enq_flags, sticky_cpu);
+}
+
+static void ops_dequeue(struct task_struct *p, u64 deq_flags)
+{
+ unsigned long opss;
+
+ clr_task_runnable(p);
+
+ /* acquire ensures that we see the preceding updates on QUEUED */
+ opss = atomic_long_read_acquire(&p->scx.ops_state);
+
+ switch (opss & SCX_OPSS_STATE_MASK) {
+ case SCX_OPSS_NONE:
+ break;
+ case SCX_OPSS_QUEUEING:
+ /*
+ * QUEUEING is started and finished while holding @p's rq lock.
+ * As we're holding the rq lock now, we shouldn't see QUEUEING.
+ */
+ BUG();
+ case SCX_OPSS_QUEUED:
+ if (SCX_HAS_OP(dequeue))
+ SCX_CALL_OP(SCX_KF_REST, dequeue, p, deq_flags);
+
+ if (atomic_long_try_cmpxchg(&p->scx.ops_state, &opss,
+ SCX_OPSS_NONE))
+ break;
+ fallthrough;
+ case SCX_OPSS_DISPATCHING:
+ /*
+ * If @p is being dispatched from the BPF scheduler to a DSQ,
+ * wait for the transfer to complete so that @p doesn't get
+ * added to its DSQ after dequeueing is complete.
+ *
+ * As we're waiting on DISPATCHING with the rq locked, the
+ * dispatching side shouldn't try to lock the rq while
+ * DISPATCHING is set. See dispatch_to_local_dsq().
+ *
+ * DISPATCHING shouldn't have qseq set and control can reach
+ * here with NONE @opss from the above QUEUED case block.
+ * Explicitly wait on %SCX_OPSS_DISPATCHING instead of @opss.
+ */
+ wait_ops_state(p, SCX_OPSS_DISPATCHING);
+ BUG_ON(atomic_long_read(&p->scx.ops_state) != SCX_OPSS_NONE);
+ break;
+ }
+}
+
+static void dequeue_task_scx(struct rq *rq, struct task_struct *p, int deq_flags)
+{
+ if (!(p->scx.flags & SCX_TASK_QUEUED)) {
+ WARN_ON_ONCE(task_runnable(p));
+ return;
+ }
+
+ ops_dequeue(p, deq_flags);
+
+ if (deq_flags & SCX_DEQ_SLEEP)
+ p->scx.flags |= SCX_TASK_DEQD_FOR_SLEEP;
+ else
+ p->scx.flags &= ~SCX_TASK_DEQD_FOR_SLEEP;
+
+ p->scx.flags &= ~SCX_TASK_QUEUED;
+ rq->scx.nr_running--;
+ sub_nr_running(rq, 1);
+
+ dispatch_dequeue(rq, p);
+}
+
+static void yield_task_scx(struct rq *rq)
+{
+ struct task_struct *p = rq->curr;
+
+ if (SCX_HAS_OP(yield))
+ SCX_CALL_OP_RET(SCX_KF_REST, yield, p, NULL);
+ else
+ p->scx.slice = 0;
+}
+
+static bool yield_to_task_scx(struct rq *rq, struct task_struct *to)
+{
+ struct task_struct *from = rq->curr;
+
+ if (SCX_HAS_OP(yield))
+ return SCX_CALL_OP_RET(SCX_KF_REST, yield, from, to);
+ else
+ return false;
+}
+
+#ifdef CONFIG_SMP
+/**
+ * move_task_to_local_dsq - Move a task from a different rq to a local DSQ
+ * @rq: rq to move the task into, currently locked
+ * @p: task to move
+ * @enq_flags: %SCX_ENQ_*
+ *
+ * Move @p which is currently on a different rq to @rq's local DSQ. The caller
+ * must:
+ *
+ * 1. Start with exclusive access to @p either through its DSQ lock or
+ * %SCX_OPSS_DISPATCHING flag.
+ *
+ * 2. Set @p->scx.holding_cpu to raw_smp_processor_id().
+ *
+ * 3. Remember task_rq(@p). Release the exclusive access so that we don't
+ * deadlock with dequeue.
+ *
+ * 4. Lock @rq and the task_rq from #3.
+ *
+ * 5. Call this function.
+ *
+ * Returns %true if @p was successfully moved. %false after racing dequeue and
+ * losing.
+ */
+static bool move_task_to_local_dsq(struct rq *rq, struct task_struct *p,
+ u64 enq_flags)
+{
+ struct rq *task_rq;
+
+ lockdep_assert_rq_held(rq);
+
+ /*
+ * If dequeue got to @p while we were trying to lock both rq's, it'd
+ * have cleared @p->scx.holding_cpu to -1. While other cpus may have
+ * updated it to different values afterwards, as this operation can't be
+ * preempted or recurse, @p->scx.holding_cpu can never become
+ * raw_smp_processor_id() again before we're done. Thus, we can tell
+ * whether we lost to dequeue by testing whether @p->scx.holding_cpu is
+ * still raw_smp_processor_id().
+ *
+ * See dispatch_dequeue() for the counterpart.
+ */
+ if (unlikely(p->scx.holding_cpu != raw_smp_processor_id()))
+ return false;
+
+ /* @p->rq couldn't have changed if we're still the holding cpu */
+ task_rq = task_rq(p);
+ lockdep_assert_rq_held(task_rq);
+
+ WARN_ON_ONCE(!cpumask_test_cpu(cpu_of(rq), p->cpus_ptr));
+ deactivate_task(task_rq, p, 0);
+ set_task_cpu(p, cpu_of(rq));
+ p->scx.sticky_cpu = cpu_of(rq);
+
+ /*
+ * We want to pass scx-specific enq_flags but activate_task() will
+ * truncate the upper 32 bit. As we own @rq, we can pass them through
+ * @rq->scx.extra_enq_flags instead.
+ */
+ WARN_ON_ONCE(rq->scx.extra_enq_flags);
+ rq->scx.extra_enq_flags = enq_flags;
+ activate_task(rq, p, 0);
+ rq->scx.extra_enq_flags = 0;
+
+ return true;
+}
+
+/**
+ * dispatch_to_local_dsq_lock - Ensure source and destination rq's are locked
+ * @rq: current rq which is locked
+ * @rf: rq_flags to use when unlocking @rq
+ * @src_rq: rq to move task from
+ * @dst_rq: rq to move task to
+ *
+ * We're holding @rq lock and trying to dispatch a task from @src_rq to
+ * @dst_rq's local DSQ and thus need to lock both @src_rq and @dst_rq. Whether
+ * @rq stays locked isn't important as long as the state is restored after
+ * dispatch_to_local_dsq_unlock().
+ */
+static void dispatch_to_local_dsq_lock(struct rq *rq, struct rq_flags *rf,
+ struct rq *src_rq, struct rq *dst_rq)
+{
+ rq_unpin_lock(rq, rf);
+
+ if (src_rq == dst_rq) {
+ raw_spin_rq_unlock(rq);
+ raw_spin_rq_lock(dst_rq);
+ } else if (rq == src_rq) {
+ double_lock_balance(rq, dst_rq);
+ rq_repin_lock(rq, rf);
+ } else if (rq == dst_rq) {
+ double_lock_balance(rq, src_rq);
+ rq_repin_lock(rq, rf);
+ } else {
+ raw_spin_rq_unlock(rq);
+ double_rq_lock(src_rq, dst_rq);
+ }
+}
+
+/**
+ * dispatch_to_local_dsq_unlock - Undo dispatch_to_local_dsq_lock()
+ * @rq: current rq which is locked
+ * @rf: rq_flags to use when unlocking @rq
+ * @src_rq: rq to move task from
+ * @dst_rq: rq to move task to
+ *
+ * Unlock @src_rq and @dst_rq and ensure that @rq is locked on return.
+ */
+static void dispatch_to_local_dsq_unlock(struct rq *rq, struct rq_flags *rf,
+ struct rq *src_rq, struct rq *dst_rq)
+{
+ if (src_rq == dst_rq) {
+ raw_spin_rq_unlock(dst_rq);
+ raw_spin_rq_lock(rq);
+ rq_repin_lock(rq, rf);
+ } else if (rq == src_rq) {
+ double_unlock_balance(rq, dst_rq);
+ } else if (rq == dst_rq) {
+ double_unlock_balance(rq, src_rq);
+ } else {
+ double_rq_unlock(src_rq, dst_rq);
+ raw_spin_rq_lock(rq);
+ rq_repin_lock(rq, rf);
+ }
+}
+#endif /* CONFIG_SMP */
+
+static void consume_local_task(struct rq *rq, struct scx_dispatch_q *dsq,
+ struct task_struct *p)
+{
+ lockdep_assert_held(&dsq->lock); /* released on return */
+
+ /* @dsq is locked and @p is on this rq */
+ WARN_ON_ONCE(p->scx.holding_cpu >= 0);
+ list_move_tail(&p->scx.dsq_node, &rq->scx.local_dsq.list);
+ dsq_mod_nr(dsq, -1);
+ dsq_mod_nr(&rq->scx.local_dsq, 1);
+ p->scx.dsq = &rq->scx.local_dsq;
+ raw_spin_unlock(&dsq->lock);
+}
+
+#ifdef CONFIG_SMP
+/*
+ * Similar to kernel/sched/core.c::is_cpu_allowed() but we're testing whether @p
+ * can be pulled to @rq.
+ */
+static bool task_can_run_on_remote_rq(struct task_struct *p, struct rq *rq)
+{
+ int cpu = cpu_of(rq);
+
+ if (!cpumask_test_cpu(cpu, p->cpus_ptr))
+ return false;
+ if (unlikely(is_migration_disabled(p)))
+ return false;
+ if (!(p->flags & PF_KTHREAD) && unlikely(!task_cpu_possible(cpu, p)))
+ return false;
+ if (!scx_rq_online(rq))
+ return false;
+ return true;
+}
+
+static bool consume_remote_task(struct rq *rq, struct rq_flags *rf,
+ struct scx_dispatch_q *dsq,
+ struct task_struct *p, struct rq *task_rq)
+{
+ bool moved = false;
+
+ lockdep_assert_held(&dsq->lock); /* released on return */
+
+ /*
+ * @dsq is locked and @p is on a remote rq. @p is currently protected by
+ * @dsq->lock. We want to pull @p to @rq but may deadlock if we grab
+ * @task_rq while holding @dsq and @rq locks. As dequeue can't drop the
+ * rq lock or fail, do a little dancing from our side. See
+ * move_task_to_local_dsq().
+ */
+ WARN_ON_ONCE(p->scx.holding_cpu >= 0);
+ list_del_init(&p->scx.dsq_node);
+ dsq_mod_nr(dsq, -1);
+ p->scx.holding_cpu = raw_smp_processor_id();
+ raw_spin_unlock(&dsq->lock);
+
+ rq_unpin_lock(rq, rf);
+ double_lock_balance(rq, task_rq);
+ rq_repin_lock(rq, rf);
+
+ moved = move_task_to_local_dsq(rq, p, 0);
+
+ double_unlock_balance(rq, task_rq);
+
+ return moved;
+}
+#else /* CONFIG_SMP */
+static bool task_can_run_on_remote_rq(struct task_struct *p, struct rq *rq) { return false; }
+static bool consume_remote_task(struct rq *rq, struct rq_flags *rf,
+ struct scx_dispatch_q *dsq,
+ struct task_struct *p, struct rq *task_rq) { return false; }
+#endif /* CONFIG_SMP */
+
+static bool consume_dispatch_q(struct rq *rq, struct rq_flags *rf,
+ struct scx_dispatch_q *dsq)
+{
+ struct task_struct *p;
+retry:
+ if (list_empty(&dsq->list))
+ return false;
+
+ raw_spin_lock(&dsq->lock);
+
+ list_for_each_entry(p, &dsq->list, scx.dsq_node) {
+ struct rq *task_rq = task_rq(p);
+
+ if (rq == task_rq) {
+ consume_local_task(rq, dsq, p);
+ return true;
+ }
+
+ if (task_can_run_on_remote_rq(p, rq)) {
+ if (likely(consume_remote_task(rq, rf, dsq, p, task_rq)))
+ return true;
+ goto retry;
+ }
+ }
+
+ raw_spin_unlock(&dsq->lock);
+ return false;
+}
+
+enum dispatch_to_local_dsq_ret {
+ DTL_DISPATCHED, /* successfully dispatched */
+ DTL_LOST, /* lost race to dequeue */
+ DTL_NOT_LOCAL, /* destination is not a local DSQ */
+ DTL_INVALID, /* invalid local dsq_id */
+};
+
+/**
+ * dispatch_to_local_dsq - Dispatch a task to a local dsq
+ * @rq: current rq which is locked
+ * @rf: rq_flags to use when unlocking @rq
+ * @dsq_id: destination dsq ID
+ * @p: task to dispatch
+ * @enq_flags: %SCX_ENQ_*
+ *
+ * We're holding @rq lock and want to dispatch @p to the local DSQ identified by
+ * @dsq_id. This function performs all the synchronization dancing needed
+ * because local DSQs are protected with rq locks.
+ *
+ * The caller must have exclusive ownership of @p (e.g. through
+ * %SCX_OPSS_DISPATCHING).
+ */
+static enum dispatch_to_local_dsq_ret
+dispatch_to_local_dsq(struct rq *rq, struct rq_flags *rf, u64 dsq_id,
+ struct task_struct *p, u64 enq_flags)
+{
+ struct rq *src_rq = task_rq(p);
+ struct rq *dst_rq;
+
+ /*
+ * We're synchronized against dequeue through DISPATCHING. As @p can't
+ * be dequeued, its task_rq and cpus_allowed are stable too.
+ */
+ if (dsq_id == SCX_DSQ_LOCAL) {
+ dst_rq = rq;
+ } else if ((dsq_id & SCX_DSQ_LOCAL_ON) == SCX_DSQ_LOCAL_ON) {
+ s32 cpu = dsq_id & SCX_DSQ_LOCAL_CPU_MASK;
+
+ if (!ops_cpu_valid(cpu, "in SCX_DSQ_LOCAL_ON dispatch verdict"))
+ return DTL_INVALID;
+ dst_rq = cpu_rq(cpu);
+ } else {
+ return DTL_NOT_LOCAL;
+ }
+
+ /* if dispatching to @rq that @p is already on, no lock dancing needed */
+ if (rq == src_rq && rq == dst_rq) {
+ dispatch_enqueue(&dst_rq->scx.local_dsq, p,
+ enq_flags | SCX_ENQ_CLEAR_OPSS);
+ return DTL_DISPATCHED;
+ }
+
+#ifdef CONFIG_SMP
+ if (cpumask_test_cpu(cpu_of(dst_rq), p->cpus_ptr)) {
+ struct rq *locked_dst_rq = dst_rq;
+ bool dsp;
+
+ /*
+ * @p is on a possibly remote @src_rq which we need to lock to
+ * move the task. If dequeue is in progress, it'd be locking
+ * @src_rq and waiting on DISPATCHING, so we can't grab @src_rq
+ * lock while holding DISPATCHING.
+ *
+ * As DISPATCHING guarantees that @p is wholly ours, we can
+ * pretend that we're moving from a DSQ and use the same
+ * mechanism - mark the task under transfer with holding_cpu,
+ * release DISPATCHING and then follow the same protocol.
+ */
+ p->scx.holding_cpu = raw_smp_processor_id();
+
+ /* store_release ensures that dequeue sees the above */
+ atomic_long_set_release(&p->scx.ops_state, SCX_OPSS_NONE);
+
+ dispatch_to_local_dsq_lock(rq, rf, src_rq, locked_dst_rq);
+
+ /*
+ * We don't require the BPF scheduler to avoid dispatching to
+ * offline CPUs mostly for convenience but also because CPUs can
+ * go offline between scx_bpf_dispatch() calls and here. If @p
+ * is destined to an offline CPU, queue it on its current CPU
+ * instead, which should always be safe. As this is an allowed
+ * behavior, don't trigger an ops error.
+ */
+ if (!scx_rq_online(dst_rq))
+ dst_rq = src_rq;
+
+ if (src_rq == dst_rq) {
+ /*
+ * As @p is staying on the same rq, there's no need to
+ * go through the full deactivate/activate cycle.
+ * Optimize by abbreviating the operations in
+ * move_task_to_local_dsq().
+ */
+ dsp = p->scx.holding_cpu == raw_smp_processor_id();
+ if (likely(dsp)) {
+ p->scx.holding_cpu = -1;
+ dispatch_enqueue(&dst_rq->scx.local_dsq, p,
+ enq_flags);
+ }
+ } else {
+ dsp = move_task_to_local_dsq(dst_rq, p, enq_flags);
+ }
+
+ /* if the destination CPU is idle, wake it up */
+ if (dsp && sched_class_above(p->sched_class,
+ dst_rq->curr->sched_class))
+ resched_curr(dst_rq);
+
+ dispatch_to_local_dsq_unlock(rq, rf, src_rq, locked_dst_rq);
+
+ return dsp ? DTL_DISPATCHED : DTL_LOST;
+ }
+#endif /* CONFIG_SMP */
+
+ scx_ops_error("SCX_DSQ_LOCAL[_ON] verdict target cpu %d not allowed for %s[%d]",
+ cpu_of(dst_rq), p->comm, p->pid);
+ return DTL_INVALID;
+}
+
+/**
+ * finish_dispatch - Asynchronously finish dispatching a task
+ * @rq: current rq which is locked
+ * @rf: rq_flags to use when unlocking @rq
+ * @p: task to finish dispatching
+ * @qseq_at_dispatch: qseq when @p started getting dispatched
+ * @dsq_id: destination DSQ ID
+ * @enq_flags: %SCX_ENQ_*
+ *
+ * Dispatching to local DSQs may need to wait for queueing to complete or
+ * require rq lock dancing. As we don't wanna do either while inside
+ * ops.dispatch() to avoid locking order inversion, we split dispatching into
+ * two parts. scx_bpf_dispatch() which is called by ops.dispatch() records the
+ * task and its qseq. Once ops.dispatch() returns, this function is called to
+ * finish up.
+ *
+ * There is no guarantee that @p is still valid for dispatching or even that it
+ * was valid in the first place. Make sure that the task is still owned by the
+ * BPF scheduler and claim the ownership before dispatching.
+ */
+static void finish_dispatch(struct rq *rq, struct rq_flags *rf,
+ struct task_struct *p,
+ unsigned long qseq_at_dispatch,
+ u64 dsq_id, u64 enq_flags)
+{
+ struct scx_dispatch_q *dsq;
+ unsigned long opss;
+
+retry:
+ /*
+ * No need for _acquire here. @p is accessed only after a successful
+ * try_cmpxchg to DISPATCHING.
+ */
+ opss = atomic_long_read(&p->scx.ops_state);
+
+ switch (opss & SCX_OPSS_STATE_MASK) {
+ case SCX_OPSS_DISPATCHING:
+ case SCX_OPSS_NONE:
+ /* someone else already got to it */
+ return;
+ case SCX_OPSS_QUEUED:
+ /*
+ * If qseq doesn't match, @p has gone through at least one
+ * dispatch/dequeue and re-enqueue cycle between
+ * scx_bpf_dispatch() and here and we have no claim on it.
+ */
+ if ((opss & SCX_OPSS_QSEQ_MASK) != qseq_at_dispatch)
+ return;
+
+ /*
+ * While we know @p is accessible, we don't yet have a claim on
+ * it - the BPF scheduler is allowed to dispatch tasks
+ * spuriously and there can be a racing dequeue attempt. Let's
+ * claim @p by atomically transitioning it from QUEUED to
+ * DISPATCHING.
+ */
+ if (likely(atomic_long_try_cmpxchg(&p->scx.ops_state, &opss,
+ SCX_OPSS_DISPATCHING)))
+ break;
+ goto retry;
+ case SCX_OPSS_QUEUEING:
+ /*
+ * do_enqueue_task() is in the process of transferring the task
+ * to the BPF scheduler while holding @p's rq lock. As we aren't
+ * holding any kernel or BPF resource that the enqueue path may
+ * depend upon, it's safe to wait.
+ */
+ wait_ops_state(p, opss);
+ goto retry;
+ }
+
+ BUG_ON(!(p->scx.flags & SCX_TASK_QUEUED));
+
+ switch (dispatch_to_local_dsq(rq, rf, dsq_id, p, enq_flags)) {
+ case DTL_DISPATCHED:
+ break;
+ case DTL_LOST:
+ break;
+ case DTL_INVALID:
+ dsq_id = SCX_DSQ_GLOBAL;
+ fallthrough;
+ case DTL_NOT_LOCAL:
+ dsq = find_dsq_for_dispatch(cpu_rq(raw_smp_processor_id()),
+ dsq_id, p);
+ dispatch_enqueue(dsq, p, enq_flags | SCX_ENQ_CLEAR_OPSS);
+ break;
+ }
+}
+
+static void flush_dispatch_buf(struct rq *rq, struct rq_flags *rf)
+{
+ struct scx_dsp_ctx *dspc = this_cpu_ptr(scx_dsp_ctx);
+ u32 u;
+
+ for (u = 0; u < dspc->cursor; u++) {
+ struct scx_dsp_buf_ent *ent = &dspc->buf[u];
+
+ finish_dispatch(rq, rf, ent->task, ent->qseq, ent->dsq_id,
+ ent->enq_flags);
+ }
+
+ dspc->nr_tasks += dspc->cursor;
+ dspc->cursor = 0;
+}
+
+static int balance_scx(struct rq *rq, struct task_struct *prev,
+ struct rq_flags *rf)
+{
+ struct scx_dsp_ctx *dspc = this_cpu_ptr(scx_dsp_ctx);
+ bool prev_on_scx = prev->sched_class == &ext_sched_class;
+
+ lockdep_assert_rq_held(rq);
+
+ if (prev_on_scx) {
+ WARN_ON_ONCE(prev->scx.flags & SCX_TASK_BAL_KEEP);
+ update_curr_scx(rq);
+
+ /*
+ * If @prev is runnable & has slice left, it has priority and
+ * fetching more just increases latency for the fetched tasks.
+ * Tell put_prev_task_scx() to put @prev on local_dsq.
+ *
+ * See scx_ops_disable_workfn() for the explanation on the
+ * bypassing test.
+ */
+ if ((prev->scx.flags & SCX_TASK_QUEUED) &&
+ prev->scx.slice && !scx_ops_bypassing()) {
+ prev->scx.flags |= SCX_TASK_BAL_KEEP;
+ return 1;
+ }
+ }
+
+ /* if there already are tasks to run, nothing to do */
+ if (rq->scx.local_dsq.nr)
+ return 1;
+
+ if (consume_dispatch_q(rq, rf, &scx_dsq_global))
+ return 1;
+
+ if (!SCX_HAS_OP(dispatch) || scx_ops_bypassing() || !scx_rq_online(rq))
+ return 0;
+
+ dspc->rq = rq;
+ dspc->rf = rf;
+
+ /*
+ * The dispatch loop. Because flush_dispatch_buf() may drop the rq lock,
+ * the local DSQ might still end up empty after a successful
+ * ops.dispatch(). If the local DSQ is empty even after ops.dispatch()
+ * produced some tasks, retry. The BPF scheduler may depend on this
+ * looping behavior to simplify its implementation.
+ */
+ do {
+ dspc->nr_tasks = 0;
+
+ SCX_CALL_OP(SCX_KF_DISPATCH, dispatch, cpu_of(rq),
+ prev_on_scx ? prev : NULL);
+
+ flush_dispatch_buf(rq, rf);
+
+ if (rq->scx.local_dsq.nr)
+ return 1;
+ if (consume_dispatch_q(rq, rf, &scx_dsq_global))
+ return 1;
+ } while (dspc->nr_tasks);
+
+ return 0;
+}
+
+static void set_next_task_scx(struct rq *rq, struct task_struct *p, bool first)
+{
+ if (p->scx.flags & SCX_TASK_QUEUED) {
+ WARN_ON_ONCE(atomic_long_read(&p->scx.ops_state) != SCX_OPSS_NONE);
+ dispatch_dequeue(rq, p);
+ }
+
+ p->se.exec_start = rq_clock_task(rq);
+
+ clr_task_runnable(p);
+}
+
+static void put_prev_task_scx(struct rq *rq, struct task_struct *p)
+{
+#ifndef CONFIG_SMP
+ /*
+ * UP workaround.
+ *
+ * Because SCX may transfer tasks across CPUs during dispatch, dispatch
+ * is performed from its balance operation which isn't called in UP.
+ * Let's work around by calling it from the operations which come right
+ * after.
+ *
+ * 1. If the prev task is on SCX, pick_next_task() calls
+ * .put_prev_task() right after. As .put_prev_task() is also called
+ * from other places, we need to distinguish the calls which can be
+ * done by looking at the previous task's state - if still queued or
+ * dequeued with %SCX_DEQ_SLEEP, the caller must be pick_next_task().
+ * This case is handled here.
+ *
+ * 2. If the prev task is not on SCX, the first following call into SCX
+ * will be .pick_next_task(), which is covered by calling
+ * balance_scx() from pick_next_task_scx().
+ *
+ * Note that we can't merge the first case into the second as
+ * balance_scx() must be called before the previous SCX task goes
+ * through put_prev_task_scx().
+ *
+ * As UP doesn't transfer tasks around, balance_scx() doesn't need @rf.
+ * Pass in %NULL.
+ */
+ if (p->scx.flags & (SCX_TASK_QUEUED | SCX_TASK_DEQD_FOR_SLEEP))
+ balance_scx(rq, p, NULL);
+#endif
+
+ update_curr_scx(rq);
+
+ /*
+ * If we're being called from put_prev_task_balance(), balance_scx() may
+ * have decided that @p should keep running.
+ */
+ if (p->scx.flags & SCX_TASK_BAL_KEEP) {
+ p->scx.flags &= ~SCX_TASK_BAL_KEEP;
+ set_task_runnable(rq, p);
+ dispatch_enqueue(&rq->scx.local_dsq, p, SCX_ENQ_HEAD);
+ return;
+ }
+
+ if (p->scx.flags & SCX_TASK_QUEUED) {
+ set_task_runnable(rq, p);
+
+ /*
+ * If @p has slice left and balance_scx() didn't tag it for
+ * keeping, @p is getting preempted by a higher priority
+ * scheduler class. Leave it at the head of the local DSQ.
+ */
+ if (p->scx.slice && !scx_ops_bypassing()) {
+ dispatch_enqueue(&rq->scx.local_dsq, p, SCX_ENQ_HEAD);
+ return;
+ }
+
+ /*
+ * If we're in the pick_next_task path, balance_scx() should
+ * have already populated the local DSQ if there are any other
+ * available tasks. If empty, tell ops.enqueue() that @p is the
+ * only one available for this cpu. ops.enqueue() should put it
+ * on the local DSQ so that the subsequent pick_next_task_scx()
+ * can find the task unless it wants to trigger a separate
+ * follow-up scheduling event.
+ */
+ if (list_empty(&rq->scx.local_dsq.list))
+ do_enqueue_task(rq, p, SCX_ENQ_LAST, -1);
+ else
+ do_enqueue_task(rq, p, 0, -1);
+ }
+}
+
+static struct task_struct *first_local_task(struct rq *rq)
+{
+ return list_first_entry_or_null(&rq->scx.local_dsq.list,
+ struct task_struct, scx.dsq_node);
+}
+
+static struct task_struct *pick_next_task_scx(struct rq *rq)
+{
+ struct task_struct *p;
+
+#ifndef CONFIG_SMP
+ /* UP workaround - see the comment at the head of put_prev_task_scx() */
+ if (unlikely(rq->curr->sched_class != &ext_sched_class))
+ balance_scx(rq, rq->curr, NULL);
+#endif
+
+ p = first_local_task(rq);
+ if (!p)
+ return NULL;
+
+ set_next_task_scx(rq, p, true);
+
+ if (unlikely(!p->scx.slice)) {
+ if (!scx_ops_bypassing() && !scx_warned_zero_slice) {
+ printk_deferred(KERN_WARNING "sched_ext: %s[%d] has zero slice in pick_next_task_scx()\n",
+ p->comm, p->pid);
+ scx_warned_zero_slice = true;
+ }
+ p->scx.slice = SCX_SLICE_DFL;
+ }
+
+ return p;
+}
+
+#ifdef CONFIG_SMP
+
+static bool test_and_clear_cpu_idle(int cpu)
+{
+#ifdef CONFIG_SCHED_SMT
+ /*
+ * SMT mask should be cleared whether we can claim @cpu or not. The SMT
+ * cluster is not wholly idle either way. This also prevents
+ * scx_pick_idle_cpu() from getting caught in an infinite loop.
+ */
+ if (sched_smt_active()) {
+ const struct cpumask *smt = cpu_smt_mask(cpu);
+
+ /*
+ * If offline, @cpu is not its own sibling and
+ * scx_pick_idle_cpu() can get caught in an infinite loop as
+ * @cpu is never cleared from idle_masks.smt. Ensure that @cpu
+ * is eventually cleared.
+ */
+ if (cpumask_intersects(smt, idle_masks.smt))
+ cpumask_andnot(idle_masks.smt, idle_masks.smt, smt);
+ else if (cpumask_test_cpu(cpu, idle_masks.smt))
+ __cpumask_clear_cpu(cpu, idle_masks.smt);
+ }
+#endif
+ return cpumask_test_and_clear_cpu(cpu, idle_masks.cpu);
+}
+
+static s32 scx_pick_idle_cpu(const struct cpumask *cpus_allowed, u64 flags)
+{
+ int cpu;
+
+retry:
+ if (sched_smt_active()) {
+ cpu = cpumask_any_and_distribute(idle_masks.smt, cpus_allowed);
+ if (cpu < nr_cpu_ids)
+ goto found;
+
+ if (flags & SCX_PICK_IDLE_CORE)
+ return -EBUSY;
+ }
+
+ cpu = cpumask_any_and_distribute(idle_masks.cpu, cpus_allowed);
+ if (cpu >= nr_cpu_ids)
+ return -EBUSY;
+
+found:
+ if (test_and_clear_cpu_idle(cpu))
+ return cpu;
+ else
+ goto retry;
+}
+
+static s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu,
+ u64 wake_flags, bool *found)
+{
+ s32 cpu;
+
+ *found = false;
+
+ if (!static_branch_likely(&scx_builtin_idle_enabled)) {
+ scx_ops_error("built-in idle tracking is disabled");
+ return prev_cpu;
+ }
+
+ /*
+ * If WAKE_SYNC, the waker's local DSQ is empty, and the system is
+ * under utilized, wake up @p to the local DSQ of the waker. Checking
+ * only for an empty local DSQ is insufficient as it could give the
+ * wakee an unfair advantage when the system is oversaturated.
+ * Checking only for the presence of idle CPUs is also insufficient as
+ * the local DSQ of the waker could have tasks piled up on it even if
+ * there is an idle core elsewhere on the system.
+ */
+ cpu = smp_processor_id();
+ if ((wake_flags & SCX_WAKE_SYNC) && p->nr_cpus_allowed > 1 &&
+ !cpumask_empty(idle_masks.cpu) && !(current->flags & PF_EXITING) &&
+ cpu_rq(cpu)->scx.local_dsq.nr == 0) {
+ if (cpumask_test_cpu(cpu, p->cpus_ptr))
+ goto cpu_found;
+ }
+
+ if (p->nr_cpus_allowed == 1) {
+ if (test_and_clear_cpu_idle(prev_cpu)) {
+ cpu = prev_cpu;
+ goto cpu_found;
+ } else {
+ return prev_cpu;
+ }
+ }
+
+ /*
+ * If CPU has SMT, any wholly idle CPU is likely a better pick than
+ * partially idle @prev_cpu.
+ */
+ if (sched_smt_active()) {
+ if (cpumask_test_cpu(prev_cpu, idle_masks.smt) &&
+ test_and_clear_cpu_idle(prev_cpu)) {
+ cpu = prev_cpu;
+ goto cpu_found;
+ }
+
+ cpu = scx_pick_idle_cpu(p->cpus_ptr, SCX_PICK_IDLE_CORE);
+ if (cpu >= 0)
+ goto cpu_found;
+ }
+
+ if (test_and_clear_cpu_idle(prev_cpu)) {
+ cpu = prev_cpu;
+ goto cpu_found;
+ }
+
+ cpu = scx_pick_idle_cpu(p->cpus_ptr, 0);
+ if (cpu >= 0)
+ goto cpu_found;
+
+ return prev_cpu;
+
+cpu_found:
+ *found = true;
+ return cpu;
+}
+
+static int select_task_rq_scx(struct task_struct *p, int prev_cpu, int wake_flags)
+{
+ /*
+ * sched_exec() calls with %WF_EXEC when @p is about to exec(2) as it
+ * can be a good migration opportunity with low cache and memory
+ * footprint. Returning a CPU different than @prev_cpu triggers
+ * immediate rq migration. However, for SCX, as the current rq
+ * association doesn't dictate where the task is going to run, this
+ * doesn't fit well. If necessary, we can later add a dedicated method
+ * which can decide to preempt self to force it through the regular
+ * scheduling path.
+ */
+ if (unlikely(wake_flags & WF_EXEC))
+ return prev_cpu;
+
+ if (SCX_HAS_OP(select_cpu)) {
+ s32 cpu;
+ struct task_struct **ddsp_taskp;
+
+ ddsp_taskp = this_cpu_ptr(&direct_dispatch_task);
+ WARN_ON_ONCE(*ddsp_taskp);
+ *ddsp_taskp = p;
+
+ cpu = SCX_CALL_OP_RET(SCX_KF_ENQUEUE | SCX_KF_SELECT_CPU,
+ select_cpu, p, prev_cpu, wake_flags);
+ *ddsp_taskp = NULL;
+ if (ops_cpu_valid(cpu, "from ops.select_cpu()"))
+ return cpu;
+ else
+ return prev_cpu;
+ } else {
+ bool found;
+ s32 cpu;
+
+ cpu = scx_select_cpu_dfl(p, prev_cpu, wake_flags, &found);
+ if (found) {
+ p->scx.slice = SCX_SLICE_DFL;
+ p->scx.ddsp_dsq_id = SCX_DSQ_LOCAL;
+ }
+ return cpu;
+ }
+}
+
+static void set_cpus_allowed_scx(struct task_struct *p,
+ struct affinity_context *ac)
+{
+ set_cpus_allowed_common(p, ac);
+
+ /*
+ * The effective cpumask is stored in @p->cpus_ptr which may temporarily
+ * differ from the configured one in @p->cpus_mask. Always tell the bpf
+ * scheduler the effective one.
+ *
+ * Fine-grained memory write control is enforced by BPF making the const
+ * designation pointless. Cast it away when calling the operation.
+ */
+ if (SCX_HAS_OP(set_cpumask))
+ SCX_CALL_OP(SCX_KF_REST, set_cpumask, p,
+ (struct cpumask *)p->cpus_ptr);
+}
+
+static void reset_idle_masks(void)
+{
+ /*
+ * Consider all online cpus idle. Should converge to the actual state
+ * quickly.
+ */
+ cpumask_copy(idle_masks.cpu, cpu_online_mask);
+ cpumask_copy(idle_masks.smt, cpu_online_mask);
+}
+
+void __scx_update_idle(struct rq *rq, bool idle)
+{
+ int cpu = cpu_of(rq);
+
+ if (SCX_HAS_OP(update_idle)) {
+ SCX_CALL_OP(SCX_KF_REST, update_idle, cpu_of(rq), idle);
+ if (!static_branch_unlikely(&scx_builtin_idle_enabled))
+ return;
+ }
+
+ if (idle)
+ cpumask_set_cpu(cpu, idle_masks.cpu);
+ else
+ cpumask_clear_cpu(cpu, idle_masks.cpu);
+
+#ifdef CONFIG_SCHED_SMT
+ if (sched_smt_active()) {
+ const struct cpumask *smt = cpu_smt_mask(cpu);
+
+ if (idle) {
+ /*
+ * idle_masks.smt handling is racy but that's fine as
+ * it's only for optimization and self-correcting.
+ */
+ for_each_cpu(cpu, smt) {
+ if (!cpumask_test_cpu(cpu, idle_masks.cpu))
+ return;
+ }
+ cpumask_or(idle_masks.smt, idle_masks.smt, smt);
+ } else {
+ cpumask_andnot(idle_masks.smt, idle_masks.smt, smt);
+ }
+ }
+#endif
+}
+
+#else /* CONFIG_SMP */
+
+static bool test_and_clear_cpu_idle(int cpu) { return false; }
+static s32 scx_pick_idle_cpu(const struct cpumask *cpus_allowed, u64 flags) { return -EBUSY; }
+static void reset_idle_masks(void) {}
+
+#endif /* CONFIG_SMP */
+
+static void task_tick_scx(struct rq *rq, struct task_struct *curr, int queued)
+{
+ update_other_load_avgs(rq);
+ update_curr_scx(rq);
+
+ /*
+ * While bypassing, always resched as we can't trust the slice
+ * management.
+ */
+ if (scx_ops_bypassing())
+ curr->scx.slice = 0;
+ else if (SCX_HAS_OP(tick))
+ SCX_CALL_OP(SCX_KF_REST, tick, curr);
+
+ if (!curr->scx.slice)
+ resched_curr(rq);
+}
+
+static enum scx_task_state scx_get_task_state(const struct task_struct *p)
+{
+ return (p->scx.flags & SCX_TASK_STATE_MASK) >> SCX_TASK_STATE_SHIFT;
+}
+
+static void scx_set_task_state(struct task_struct *p, enum scx_task_state state)
+{
+ enum scx_task_state prev_state = scx_get_task_state(p);
+ bool warn = false;
+
+ BUILD_BUG_ON(SCX_TASK_NR_STATES > (1 << SCX_TASK_STATE_BITS));
+
+ switch (state) {
+ case SCX_TASK_NONE:
+ break;
+ case SCX_TASK_INIT:
+ warn = prev_state != SCX_TASK_NONE;
+ break;
+ case SCX_TASK_READY:
+ warn = prev_state == SCX_TASK_NONE;
+ break;
+ case SCX_TASK_ENABLED:
+ warn = prev_state != SCX_TASK_READY;
+ break;
+ default:
+ warn = true;
+ return;
+ }
+
+ WARN_ONCE(warn, "sched_ext: Invalid task state transition %d -> %d for %s[%d]",
+ prev_state, state, p->comm, p->pid);
+
+ p->scx.flags &= ~SCX_TASK_STATE_MASK;
+ p->scx.flags |= state << SCX_TASK_STATE_SHIFT;
+}
+
+static int scx_ops_init_task(struct task_struct *p, struct task_group *tg, bool fork)
+{
+ int ret;
+
+ if (SCX_HAS_OP(init_task)) {
+ struct scx_init_task_args args = {
+ .fork = fork,
+ };
+
+ ret = SCX_CALL_OP_RET(SCX_KF_SLEEPABLE, init_task, p, &args);
+ if (unlikely(ret)) {
+ ret = ops_sanitize_err("init_task", ret);
+ return ret;
+ }
+ }
+
+ scx_set_task_state(p, SCX_TASK_INIT);
+
+ return 0;
+}
+
+static void set_task_scx_weight(struct task_struct *p)
+{
+ u32 weight = sched_prio_to_weight[p->static_prio - MAX_RT_PRIO];
+
+ p->scx.weight = sched_weight_to_cgroup(weight);
+}
+
+static void scx_ops_enable_task(struct task_struct *p)
+{
+ lockdep_assert_rq_held(task_rq(p));
+
+ /*
+ * Set the weight before calling ops.enable() so that the scheduler
+ * doesn't see a stale value if they inspect the task struct.
+ */
+ set_task_scx_weight(p);
+ if (SCX_HAS_OP(enable))
+ SCX_CALL_OP(SCX_KF_REST, enable, p);
+ scx_set_task_state(p, SCX_TASK_ENABLED);
+
+ if (SCX_HAS_OP(set_weight))
+ SCX_CALL_OP(SCX_KF_REST, set_weight, p, p->scx.weight);
+}
+
+static void scx_ops_disable_task(struct task_struct *p)
+{
+ lockdep_assert_rq_held(task_rq(p));
+ WARN_ON_ONCE(scx_get_task_state(p) != SCX_TASK_ENABLED);
+
+ if (SCX_HAS_OP(disable))
+ SCX_CALL_OP(SCX_KF_REST, disable, p);
+ scx_set_task_state(p, SCX_TASK_READY);
+}
+
+static void scx_ops_exit_task(struct task_struct *p)
+{
+ struct scx_exit_task_args args = {
+ .cancelled = false,
+ };
+
+ lockdep_assert_rq_held(task_rq(p));
+
+ switch (scx_get_task_state(p)) {
+ case SCX_TASK_NONE:
+ return;
+ case SCX_TASK_INIT:
+ args.cancelled = true;
+ break;
+ case SCX_TASK_READY:
+ break;
+ case SCX_TASK_ENABLED:
+ scx_ops_disable_task(p);
+ break;
+ default:
+ WARN_ON_ONCE(true);
+ return;
+ }
+
+ if (SCX_HAS_OP(exit_task))
+ SCX_CALL_OP(SCX_KF_REST, exit_task, p, &args);
+ scx_set_task_state(p, SCX_TASK_NONE);
+}
+
+void init_scx_entity(struct sched_ext_entity *scx)
+{
+ /*
+ * init_idle() calls this function again after fork sequence is
+ * complete. Don't touch ->tasks_node as it's already linked.
+ */
+ memset(scx, 0, offsetof(struct sched_ext_entity, tasks_node));
+
+ INIT_LIST_HEAD(&scx->dsq_node);
+ scx->sticky_cpu = -1;
+ scx->holding_cpu = -1;
+ INIT_LIST_HEAD(&scx->runnable_node);
+ scx->ddsp_dsq_id = SCX_DSQ_INVALID;
+ scx->slice = SCX_SLICE_DFL;
+}
+
+void scx_pre_fork(struct task_struct *p)
+{
+ /*
+ * BPF scheduler enable/disable paths want to be able to iterate and
+ * update all tasks which can become complex when racing forks. As
+ * enable/disable are very cold paths, let's use a percpu_rwsem to
+ * exclude forks.
+ */
+ percpu_down_read(&scx_fork_rwsem);
+}
+
+int scx_fork(struct task_struct *p)
+{
+ percpu_rwsem_assert_held(&scx_fork_rwsem);
+
+ if (scx_enabled())
+ return scx_ops_init_task(p, task_group(p), true);
+ else
+ return 0;
+}
+
+void scx_post_fork(struct task_struct *p)
+{
+ if (scx_enabled()) {
+ scx_set_task_state(p, SCX_TASK_READY);
+
+ /*
+ * Enable the task immediately if it's running on sched_ext.
+ * Otherwise, it'll be enabled in switching_to_scx() if and
+ * when it's ever configured to run with a SCHED_EXT policy.
+ */
+ if (p->sched_class == &ext_sched_class) {
+ struct rq_flags rf;
+ struct rq *rq;
+
+ rq = task_rq_lock(p, &rf);
+ scx_ops_enable_task(p);
+ task_rq_unlock(rq, p, &rf);
+ }
+ }
+
+ spin_lock_irq(&scx_tasks_lock);
+ list_add_tail(&p->scx.tasks_node, &scx_tasks);
+ spin_unlock_irq(&scx_tasks_lock);
+
+ percpu_up_read(&scx_fork_rwsem);
+}
+
+void scx_cancel_fork(struct task_struct *p)
+{
+ if (scx_enabled()) {
+ struct rq *rq;
+ struct rq_flags rf;
+
+ rq = task_rq_lock(p, &rf);
+ WARN_ON_ONCE(scx_get_task_state(p) >= SCX_TASK_READY);
+ scx_ops_exit_task(p);
+ task_rq_unlock(rq, p, &rf);
+ }
+
+ percpu_up_read(&scx_fork_rwsem);
+}
+
+void sched_ext_free(struct task_struct *p)
+{
+ unsigned long flags;
+
+ spin_lock_irqsave(&scx_tasks_lock, flags);
+ list_del_init(&p->scx.tasks_node);
+ spin_unlock_irqrestore(&scx_tasks_lock, flags);
+
+ /*
+ * @p is off scx_tasks and wholly ours. scx_ops_enable()'s READY ->
+ * ENABLED transitions can't race us. Disable ops for @p.
+ */
+ if (scx_get_task_state(p) != SCX_TASK_NONE) {
+ struct rq_flags rf;
+ struct rq *rq;
+
+ rq = task_rq_lock(p, &rf);
+ scx_ops_exit_task(p);
+ task_rq_unlock(rq, p, &rf);
+ }
+}
+
+static void reweight_task_scx(struct rq *rq, struct task_struct *p, int newprio)
+{
+ lockdep_assert_rq_held(task_rq(p));
+
+ set_task_scx_weight(p);
+ if (SCX_HAS_OP(set_weight))
+ SCX_CALL_OP(SCX_KF_REST, set_weight, p, p->scx.weight);
+}
+
+static void prio_changed_scx(struct rq *rq, struct task_struct *p, int oldprio)
+{
+}
+
+static void switching_to_scx(struct rq *rq, struct task_struct *p)
+{
+ scx_ops_enable_task(p);
+
+ /*
+ * set_cpus_allowed_scx() is not called while @p is associated with a
+ * different scheduler class. Keep the BPF scheduler up-to-date.
+ */
+ if (SCX_HAS_OP(set_cpumask))
+ SCX_CALL_OP(SCX_KF_REST, set_cpumask, p,
+ (struct cpumask *)p->cpus_ptr);
+}
+
+static void switched_from_scx(struct rq *rq, struct task_struct *p)
+{
+ scx_ops_disable_task(p);
+}
+
+static void wakeup_preempt_scx(struct rq *rq, struct task_struct *p,int wake_flags) {}
+static void switched_to_scx(struct rq *rq, struct task_struct *p) {}
+
+/*
+ * Omitted operations:
+ *
+ * - wakeup_preempt: NOOP as it isn't useful in the wakeup path because the task
+ * isn't tied to the CPU at that point.
+ *
+ * - migrate_task_rq: Unnecessary as task to cpu mapping is transient.
+ *
+ * - task_fork/dead: We need fork/dead notifications for all tasks regardless of
+ * their current sched_class. Call them directly from sched core instead.
+ *
+ * - task_woken: Unnecessary.
+ */
+DEFINE_SCHED_CLASS(ext) = {
+ .enqueue_task = enqueue_task_scx,
+ .dequeue_task = dequeue_task_scx,
+ .yield_task = yield_task_scx,
+ .yield_to_task = yield_to_task_scx,
+
+ .wakeup_preempt = wakeup_preempt_scx,
+
+ .pick_next_task = pick_next_task_scx,
+
+ .put_prev_task = put_prev_task_scx,
+ .set_next_task = set_next_task_scx,
+
+#ifdef CONFIG_SMP
+ .balance = balance_scx,
+ .select_task_rq = select_task_rq_scx,
+ .set_cpus_allowed = set_cpus_allowed_scx,
+#endif
+
+ .task_tick = task_tick_scx,
+
+ .switching_to = switching_to_scx,
+ .switched_from = switched_from_scx,
+ .switched_to = switched_to_scx,
+ .reweight_task = reweight_task_scx,
+ .prio_changed = prio_changed_scx,
+
+ .update_curr = update_curr_scx,
+
+#ifdef CONFIG_UCLAMP_TASK
+ .uclamp_enabled = 0,
+#endif
+};
+
+static void init_dsq(struct scx_dispatch_q *dsq, u64 dsq_id)
+{
+ memset(dsq, 0, sizeof(*dsq));
+
+ raw_spin_lock_init(&dsq->lock);
+ INIT_LIST_HEAD(&dsq->list);
+ dsq->id = dsq_id;
+}
+
+static struct scx_dispatch_q *create_dsq(u64 dsq_id, int node)
+{
+ struct scx_dispatch_q *dsq;
+ int ret;
+
+ if (dsq_id & SCX_DSQ_FLAG_BUILTIN)
+ return ERR_PTR(-EINVAL);
+
+ dsq = kmalloc_node(sizeof(*dsq), GFP_KERNEL, node);
+ if (!dsq)
+ return ERR_PTR(-ENOMEM);
+
+ init_dsq(dsq, dsq_id);
+
+ ret = rhashtable_insert_fast(&dsq_hash, &dsq->hash_node,
+ dsq_hash_params);
+ if (ret) {
+ kfree(dsq);
+ return ERR_PTR(ret);
+ }
+ return dsq;
+}
+
+static void free_dsq_irq_workfn(struct irq_work *irq_work)
+{
+ struct llist_node *to_free = llist_del_all(&dsqs_to_free);
+ struct scx_dispatch_q *dsq, *tmp_dsq;
+
+ llist_for_each_entry_safe(dsq, tmp_dsq, to_free, free_node)
+ kfree_rcu(dsq, rcu);
+}
+
+static DEFINE_IRQ_WORK(free_dsq_irq_work, free_dsq_irq_workfn);
+
+static void destroy_dsq(u64 dsq_id)
+{
+ struct scx_dispatch_q *dsq;
+ unsigned long flags;
+
+ rcu_read_lock();
+
+ dsq = find_user_dsq(dsq_id);
+ if (!dsq)
+ goto out_unlock_rcu;
+
+ raw_spin_lock_irqsave(&dsq->lock, flags);
+
+ if (dsq->nr) {
+ scx_ops_error("attempting to destroy in-use dsq 0x%016llx (nr=%u)",
+ dsq->id, dsq->nr);
+ goto out_unlock_dsq;
+ }
+
+ if (rhashtable_remove_fast(&dsq_hash, &dsq->hash_node, dsq_hash_params))
+ goto out_unlock_dsq;
+
+ /*
+ * Mark dead by invalidating ->id to prevent dispatch_enqueue() from
+ * queueing more tasks. As this function can be called from anywhere,
+ * freeing is bounced through an irq work to avoid nesting RCU
+ * operations inside scheduler locks.
+ */
+ dsq->id = SCX_DSQ_INVALID;
+ llist_add(&dsq->free_node, &dsqs_to_free);
+ irq_work_queue(&free_dsq_irq_work);
+
+out_unlock_dsq:
+ raw_spin_unlock_irqrestore(&dsq->lock, flags);
+out_unlock_rcu:
+ rcu_read_unlock();
+}
+
+
+/********************************************************************************
+ * Sysfs interface and ops enable/disable.
+ */
+
+#define SCX_ATTR(_name) \
+ static struct kobj_attribute scx_attr_##_name = { \
+ .attr = { .name = __stringify(_name), .mode = 0444 }, \
+ .show = scx_attr_##_name##_show, \
+ }
+
+static ssize_t scx_attr_state_show(struct kobject *kobj,
+ struct kobj_attribute *ka, char *buf)
+{
+ return sysfs_emit(buf, "%s\n",
+ scx_ops_enable_state_str[scx_ops_enable_state()]);
+}
+SCX_ATTR(state);
+
+static ssize_t scx_attr_switch_all_show(struct kobject *kobj,
+ struct kobj_attribute *ka, char *buf)
+{
+ return sysfs_emit(buf, "%d\n", READ_ONCE(scx_switching_all));
+}
+SCX_ATTR(switch_all);
+
+static struct attribute *scx_global_attrs[] = {
+ &scx_attr_state.attr,
+ &scx_attr_switch_all.attr,
+ NULL,
+};
+
+static const struct attribute_group scx_global_attr_group = {
+ .attrs = scx_global_attrs,
+};
+
+static void scx_kobj_release(struct kobject *kobj)
+{
+ kfree(kobj);
+}
+
+static ssize_t scx_attr_ops_show(struct kobject *kobj,
+ struct kobj_attribute *ka, char *buf)
+{
+ return sysfs_emit(buf, "%s\n", scx_ops.name);
+}
+SCX_ATTR(ops);
+
+static struct attribute *scx_sched_attrs[] = {
+ &scx_attr_ops.attr,
+ NULL,
+};
+ATTRIBUTE_GROUPS(scx_sched);
+
+static const struct kobj_type scx_ktype = {
+ .release = scx_kobj_release,
+ .sysfs_ops = &kobj_sysfs_ops,
+ .default_groups = scx_sched_groups,
+};
+
+static int scx_uevent(const struct kobject *kobj, struct kobj_uevent_env *env)
+{
+ return add_uevent_var(env, "SCXOPS=%s", scx_ops.name);
+}
+
+static const struct kset_uevent_ops scx_uevent_ops = {
+ .uevent = scx_uevent,
+};
+
+/*
+ * Used by sched_fork() and __setscheduler_prio() to pick the matching
+ * sched_class. dl/rt are already handled.
+ */
+bool task_should_scx(struct task_struct *p)
+{
+ if (!scx_enabled() ||
+ unlikely(scx_ops_enable_state() == SCX_OPS_DISABLING))
+ return false;
+ if (READ_ONCE(scx_switching_all))
+ return true;
+ return p->policy == SCHED_EXT;
+}
+
+/**
+ * scx_ops_bypass - [Un]bypass scx_ops and guarantee forward progress
+ *
+ * Bypassing guarantees that all runnable tasks make forward progress without
+ * trusting the BPF scheduler. We can't grab any mutexes or rwsems as they might
+ * be held by tasks that the BPF scheduler is forgetting to run, which
+ * unfortunately also excludes toggling the static branches.
+ *
+ * Let's work around by overriding a couple ops and modifying behaviors based on
+ * the DISABLING state and then cycling the queued tasks through dequeue/enqueue
+ * to force global FIFO scheduling.
+ *
+ * a. ops.enqueue() is ignored and tasks are queued in simple global FIFO order.
+ *
+ * b. ops.dispatch() is ignored.
+ *
+ * c. balance_scx() never sets %SCX_TASK_BAL_KEEP as the slice value can't be
+ * trusted. Whenever a tick triggers, the running task is rotated to the tail
+ * of the queue.
+ *
+ * d. pick_next_task() suppresses zero slice warning.
+ */
+static void scx_ops_bypass(bool bypass)
+{
+ int depth, cpu;
+
+ if (bypass) {
+ depth = atomic_inc_return(&scx_ops_bypass_depth);
+ WARN_ON_ONCE(depth <= 0);
+ if (depth != 1)
+ return;
+ } else {
+ depth = atomic_dec_return(&scx_ops_bypass_depth);
+ WARN_ON_ONCE(depth < 0);
+ if (depth != 0)
+ return;
+ }
+
+ /*
+ * We need to guarantee that no tasks are on the BPF scheduler while
+ * bypassing. Either we see enabled or the enable path sees the
+ * increased bypass_depth before moving tasks to SCX.
+ */
+ if (!scx_enabled())
+ return;
+
+ /*
+ * No task property is changing. We just need to make sure all currently
+ * queued tasks are re-queued according to the new scx_ops_bypassing()
+ * state. As an optimization, walk each rq's runnable_list instead of
+ * the scx_tasks list.
+ *
+ * This function can't trust the scheduler and thus can't use
+ * cpus_read_lock(). Walk all possible CPUs instead of online.
+ */
+ for_each_possible_cpu(cpu) {
+ struct rq *rq = cpu_rq(cpu);
+ struct rq_flags rf;
+ struct task_struct *p, *n;
+
+ rq_lock_irqsave(rq, &rf);
+
+ /*
+ * The use of list_for_each_entry_safe_reverse() is required
+ * because each task is going to be removed from and added back
+ * to the runnable_list during iteration. Because they're added
+ * to the tail of the list, safe reverse iteration can still
+ * visit all nodes.
+ */
+ list_for_each_entry_safe_reverse(p, n, &rq->scx.runnable_list,
+ scx.runnable_node) {
+ struct sched_enq_and_set_ctx ctx;
+
+ /* cycling deq/enq is enough, see the function comment */
+ sched_deq_and_put_task(p, DEQUEUE_SAVE | DEQUEUE_MOVE, &ctx);
+ sched_enq_and_set_task(&ctx);
+ }
+
+ rq_unlock_irqrestore(rq, &rf);
+ }
+}
+
+static void free_exit_info(struct scx_exit_info *ei)
+{
+ kfree(ei->msg);
+ kfree(ei->bt);
+ kfree(ei);
+}
+
+static struct scx_exit_info *alloc_exit_info(void)
+{
+ struct scx_exit_info *ei;
+
+ ei = kzalloc(sizeof(*ei), GFP_KERNEL);
+ if (!ei)
+ return NULL;
+
+ ei->bt = kcalloc(sizeof(ei->bt[0]), SCX_EXIT_BT_LEN, GFP_KERNEL);
+ ei->msg = kzalloc(SCX_EXIT_MSG_LEN, GFP_KERNEL);
+
+ if (!ei->bt || !ei->msg) {
+ free_exit_info(ei);
+ return NULL;
+ }
+
+ return ei;
+}
+
+static const char *scx_exit_reason(enum scx_exit_kind kind)
+{
+ switch (kind) {
+ case SCX_EXIT_UNREG:
+ return "Scheduler unregistered from user space";
+ case SCX_EXIT_UNREG_BPF:
+ return "Scheduler unregistered from BPF";
+ case SCX_EXIT_UNREG_KERN:
+ return "Scheduler unregistered from the main kernel";
+ case SCX_EXIT_ERROR:
+ return "runtime error";
+ case SCX_EXIT_ERROR_BPF:
+ return "scx_bpf_error";
+ default:
+ return "<UNKNOWN>";
+ }
+}
+
+static void scx_ops_disable_workfn(struct kthread_work *work)
+{
+ struct scx_exit_info *ei = scx_exit_info;
+ struct scx_task_iter sti;
+ struct task_struct *p;
+ struct rhashtable_iter rht_iter;
+ struct scx_dispatch_q *dsq;
+ int i, kind;
+
+ kind = atomic_read(&scx_exit_kind);
+ while (true) {
+ /*
+ * NONE indicates that a new scx_ops has been registered since
+ * disable was scheduled - don't kill the new ops. DONE
+ * indicates that the ops has already been disabled.
+ */
+ if (kind == SCX_EXIT_NONE || kind == SCX_EXIT_DONE)
+ return;
+ if (atomic_try_cmpxchg(&scx_exit_kind, &kind, SCX_EXIT_DONE))
+ break;
+ }
+ ei->kind = kind;
+ ei->reason = scx_exit_reason(ei->kind);
+
+ /* guarantee forward progress by bypassing scx_ops */
+ scx_ops_bypass(true);
+
+ switch (scx_ops_set_enable_state(SCX_OPS_DISABLING)) {
+ case SCX_OPS_DISABLING:
+ WARN_ONCE(true, "sched_ext: duplicate disabling instance?");
+ break;
+ case SCX_OPS_DISABLED:
+ pr_warn("sched_ext: ops error detected without ops (%s)\n",
+ scx_exit_info->msg);
+ WARN_ON_ONCE(scx_ops_set_enable_state(SCX_OPS_DISABLED) !=
+ SCX_OPS_DISABLING);
+ goto done;
+ default:
+ break;
+ }
+
+ /*
+ * Here, every runnable task is guaranteed to make forward progress and
+ * we can safely use blocking synchronization constructs. Actually
+ * disable ops.
+ */
+ mutex_lock(&scx_ops_enable_mutex);
+
+ static_branch_disable(&__scx_switched_all);
+ WRITE_ONCE(scx_switching_all, false);
+
+ /*
+ * Avoid racing against fork. See scx_ops_enable() for explanation on
+ * the locking order.
+ */
+ percpu_down_write(&scx_fork_rwsem);
+ cpus_read_lock();
+
+ spin_lock_irq(&scx_tasks_lock);
+ scx_task_iter_init(&sti);
+ /*
+ * Invoke scx_ops_exit_task() on all non-idle tasks, including
+ * TASK_DEAD tasks. Because dead tasks may have a nonzero refcount,
+ * we may not have invoked sched_ext_free() on them by the time a
+ * scheduler is disabled. We must therefore exit the task here, or we'd
+ * fail to invoke ops.exit_task(), as the scheduler will have been
+ * unloaded by the time the task is subsequently exited on the
+ * sched_ext_free() path.
+ */
+ while ((p = scx_task_iter_next_locked(&sti, true))) {
+ const struct sched_class *old_class = p->sched_class;
+ struct sched_enq_and_set_ctx ctx;
+
+ if (READ_ONCE(p->__state) != TASK_DEAD) {
+ sched_deq_and_put_task(p, DEQUEUE_SAVE | DEQUEUE_MOVE,
+ &ctx);
+
+ p->scx.slice = min_t(u64, p->scx.slice, SCX_SLICE_DFL);
+ __setscheduler_prio(p, p->prio);
+ check_class_changing(task_rq(p), p, old_class);
+
+ sched_enq_and_set_task(&ctx);
+
+ check_class_changed(task_rq(p), p, old_class, p->prio);
+ }
+ scx_ops_exit_task(p);
+ }
+ scx_task_iter_exit(&sti);
+ spin_unlock_irq(&scx_tasks_lock);
+
+ /* no task is on scx, turn off all the switches and flush in-progress calls */
+ static_branch_disable_cpuslocked(&__scx_ops_enabled);
+ for (i = SCX_OPI_BEGIN; i < SCX_OPI_END; i++)
+ static_branch_disable_cpuslocked(&scx_has_op[i]);
+ static_branch_disable_cpuslocked(&scx_ops_enq_last);
+ static_branch_disable_cpuslocked(&scx_ops_enq_exiting);
+ static_branch_disable_cpuslocked(&scx_builtin_idle_enabled);
+ synchronize_rcu();
+
+ cpus_read_unlock();
+ percpu_up_write(&scx_fork_rwsem);
+
+ if (ei->kind >= SCX_EXIT_ERROR) {
+ printk(KERN_ERR "sched_ext: BPF scheduler \"%s\" errored, disabling\n", scx_ops.name);
+
+ if (ei->msg[0] == '\0')
+ printk(KERN_ERR "sched_ext: %s\n", ei->reason);
+ else
+ printk(KERN_ERR "sched_ext: %s (%s)\n", ei->reason, ei->msg);
+
+ stack_trace_print(ei->bt, ei->bt_len, 2);
+ }
+
+ if (scx_ops.exit)
+ SCX_CALL_OP(SCX_KF_UNLOCKED, exit, ei);
+
+ /*
+ * Delete the kobject from the hierarchy eagerly in addition to just
+ * dropping a reference. Otherwise, if the object is deleted
+ * asynchronously, sysfs could observe an object of the same name still
+ * in the hierarchy when another scheduler is loaded.
+ */
+ kobject_del(scx_root_kobj);
+ kobject_put(scx_root_kobj);
+ scx_root_kobj = NULL;
+
+ memset(&scx_ops, 0, sizeof(scx_ops));
+
+ rhashtable_walk_enter(&dsq_hash, &rht_iter);
+ do {
+ rhashtable_walk_start(&rht_iter);
+
+ while ((dsq = rhashtable_walk_next(&rht_iter)) && !IS_ERR(dsq))
+ destroy_dsq(dsq->id);
+
+ rhashtable_walk_stop(&rht_iter);
+ } while (dsq == ERR_PTR(-EAGAIN));
+ rhashtable_walk_exit(&rht_iter);
+
+ free_percpu(scx_dsp_ctx);
+ scx_dsp_ctx = NULL;
+ scx_dsp_max_batch = 0;
+
+ free_exit_info(scx_exit_info);
+ scx_exit_info = NULL;
+
+ mutex_unlock(&scx_ops_enable_mutex);
+
+ WARN_ON_ONCE(scx_ops_set_enable_state(SCX_OPS_DISABLED) !=
+ SCX_OPS_DISABLING);
+done:
+ scx_ops_bypass(false);
+}
+
+static DEFINE_KTHREAD_WORK(scx_ops_disable_work, scx_ops_disable_workfn);
+
+static void schedule_scx_ops_disable_work(void)
+{
+ struct kthread_worker *helper = READ_ONCE(scx_ops_helper);
+
+ /*
+ * We may be called spuriously before the first bpf_sched_ext_reg(). If
+ * scx_ops_helper isn't set up yet, there's nothing to do.
+ */
+ if (helper)
+ kthread_queue_work(helper, &scx_ops_disable_work);
+}
+
+static void scx_ops_disable(enum scx_exit_kind kind)
+{
+ int none = SCX_EXIT_NONE;
+
+ if (WARN_ON_ONCE(kind == SCX_EXIT_NONE || kind == SCX_EXIT_DONE))
+ kind = SCX_EXIT_ERROR;
+
+ atomic_try_cmpxchg(&scx_exit_kind, &none, kind);
+
+ schedule_scx_ops_disable_work();
+}
+
+static void scx_ops_error_irq_workfn(struct irq_work *irq_work)
+{
+ schedule_scx_ops_disable_work();
+}
+
+static DEFINE_IRQ_WORK(scx_ops_error_irq_work, scx_ops_error_irq_workfn);
+
+static __printf(3, 4) void scx_ops_exit_kind(enum scx_exit_kind kind,
+ s64 exit_code,
+ const char *fmt, ...)
+{
+ struct scx_exit_info *ei = scx_exit_info;
+ int none = SCX_EXIT_NONE;
+ va_list args;
+
+ if (!atomic_try_cmpxchg(&scx_exit_kind, &none, kind))
+ return;
+
+ ei->exit_code = exit_code;
+
+ if (kind >= SCX_EXIT_ERROR)
+ ei->bt_len = stack_trace_save(ei->bt, SCX_EXIT_BT_LEN, 1);
+
+ va_start(args, fmt);
+ vscnprintf(ei->msg, SCX_EXIT_MSG_LEN, fmt, args);
+ va_end(args);
+
+ irq_work_queue(&scx_ops_error_irq_work);
+}
+
+static struct kthread_worker *scx_create_rt_helper(const char *name)
+{
+ struct kthread_worker *helper;
+
+ helper = kthread_create_worker(0, name);
+ if (helper)
+ sched_set_fifo(helper->task);
+ return helper;
+}
+
+static int validate_ops(const struct sched_ext_ops *ops)
+{
+ /*
+ * It doesn't make sense to specify the SCX_OPS_ENQ_LAST flag if the
+ * ops.enqueue() callback isn't implemented.
+ */
+ if ((ops->flags & SCX_OPS_ENQ_LAST) && !ops->enqueue) {
+ scx_ops_error("SCX_OPS_ENQ_LAST requires ops.enqueue() to be implemented");
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static int scx_ops_enable(struct sched_ext_ops *ops, struct bpf_link *link)
+{
+ struct scx_task_iter sti;
+ struct task_struct *p;
+ int i, ret;
+
+ mutex_lock(&scx_ops_enable_mutex);
+
+ if (!scx_ops_helper) {
+ WRITE_ONCE(scx_ops_helper,
+ scx_create_rt_helper("sched_ext_ops_helper"));
+ if (!scx_ops_helper) {
+ ret = -ENOMEM;
+ goto err_unlock;
+ }
+ }
+
+ if (scx_ops_enable_state() != SCX_OPS_DISABLED) {
+ ret = -EBUSY;
+ goto err_unlock;
+ }
+
+ scx_root_kobj = kzalloc(sizeof(*scx_root_kobj), GFP_KERNEL);
+ if (!scx_root_kobj) {
+ ret = -ENOMEM;
+ goto err_unlock;
+ }
+
+ scx_root_kobj->kset = scx_kset;
+ ret = kobject_init_and_add(scx_root_kobj, &scx_ktype, NULL, "root");
+ if (ret < 0)
+ goto err;
+
+ scx_exit_info = alloc_exit_info();
+ if (!scx_exit_info) {
+ ret = -ENOMEM;
+ goto err_del;
+ }
+
+ /*
+ * Set scx_ops, transition to PREPPING and clear exit info to arm the
+ * disable path. Failure triggers full disabling from here on.
+ */
+ scx_ops = *ops;
+
+ WARN_ON_ONCE(scx_ops_set_enable_state(SCX_OPS_PREPPING) !=
+ SCX_OPS_DISABLED);
+
+ atomic_set(&scx_exit_kind, SCX_EXIT_NONE);
+ scx_warned_zero_slice = false;
+
+ /*
+ * Keep CPUs stable during enable so that the BPF scheduler can track
+ * online CPUs by watching ->on/offline_cpu() after ->init().
+ */
+ cpus_read_lock();
+
+ if (scx_ops.init) {
+ ret = SCX_CALL_OP_RET(SCX_KF_SLEEPABLE, init);
+ if (ret) {
+ ret = ops_sanitize_err("init", ret);
+ goto err_disable_unlock_cpus;
+ }
+ }
+
+ cpus_read_unlock();
+
+ ret = validate_ops(ops);
+ if (ret)
+ goto err_disable;
+
+ WARN_ON_ONCE(scx_dsp_ctx);
+ scx_dsp_max_batch = ops->dispatch_max_batch ?: SCX_DSP_DFL_MAX_BATCH;
+ scx_dsp_ctx = __alloc_percpu(struct_size_t(struct scx_dsp_ctx, buf,
+ scx_dsp_max_batch),
+ __alignof__(struct scx_dsp_ctx));
+ if (!scx_dsp_ctx) {
+ ret = -ENOMEM;
+ goto err_disable;
+ }
+
+ /*
+ * Lock out forks before opening the floodgate so that they don't wander
+ * into the operations prematurely.
+ *
+ * We don't need to keep the CPUs stable but grab cpus_read_lock() to
+ * ease future locking changes for cgroup suport.
+ *
+ * Note that cpu_hotplug_lock must nest inside scx_fork_rwsem due to the
+ * following dependency chain:
+ *
+ * scx_fork_rwsem --> pernet_ops_rwsem --> cpu_hotplug_lock
+ */
+ percpu_down_write(&scx_fork_rwsem);
+ cpus_read_lock();
+
+ for (i = SCX_OPI_NORMAL_BEGIN; i < SCX_OPI_NORMAL_END; i++)
+ if (((void (**)(void))ops)[i])
+ static_branch_enable_cpuslocked(&scx_has_op[i]);
+
+ if (ops->flags & SCX_OPS_ENQ_LAST)
+ static_branch_enable_cpuslocked(&scx_ops_enq_last);
+
+ if (ops->flags & SCX_OPS_ENQ_EXITING)
+ static_branch_enable_cpuslocked(&scx_ops_enq_exiting);
+
+ if (!ops->update_idle || (ops->flags & SCX_OPS_KEEP_BUILTIN_IDLE)) {
+ reset_idle_masks();
+ static_branch_enable_cpuslocked(&scx_builtin_idle_enabled);
+ } else {
+ static_branch_disable_cpuslocked(&scx_builtin_idle_enabled);
+ }
+
+ static_branch_enable_cpuslocked(&__scx_ops_enabled);
+
+ /*
+ * Enable ops for every task. Fork is excluded by scx_fork_rwsem
+ * preventing new tasks from being added. No need to exclude tasks
+ * leaving as sched_ext_free() can handle both prepped and enabled
+ * tasks. Prep all tasks first and then enable them with preemption
+ * disabled.
+ */
+ spin_lock_irq(&scx_tasks_lock);
+
+ scx_task_iter_init(&sti);
+ while ((p = scx_task_iter_next_locked(&sti, false))) {
+ get_task_struct(p);
+ scx_task_iter_rq_unlock(&sti);
+ spin_unlock_irq(&scx_tasks_lock);
+
+ ret = scx_ops_init_task(p, task_group(p), false);
+ if (ret) {
+ put_task_struct(p);
+ spin_lock_irq(&scx_tasks_lock);
+ scx_task_iter_exit(&sti);
+ spin_unlock_irq(&scx_tasks_lock);
+ pr_err("sched_ext: ops.init_task() failed (%d) for %s[%d] while loading\n",
+ ret, p->comm, p->pid);
+ goto err_disable_unlock_all;
+ }
+
+ put_task_struct(p);
+ spin_lock_irq(&scx_tasks_lock);
+ }
+ scx_task_iter_exit(&sti);
+
+ /*
+ * All tasks are prepped but are still ops-disabled. Ensure that
+ * %current can't be scheduled out and switch everyone.
+ * preempt_disable() is necessary because we can't guarantee that
+ * %current won't be starved if scheduled out while switching.
+ */
+ preempt_disable();
+
+ /*
+ * From here on, the disable path must assume that tasks have ops
+ * enabled and need to be recovered.
+ *
+ * Transition to ENABLING fails iff the BPF scheduler has already
+ * triggered scx_bpf_error(). Returning an error code here would lose
+ * the recorded error information. Exit indicating success so that the
+ * error is notified through ops.exit() with all the details.
+ */
+ if (!scx_ops_tryset_enable_state(SCX_OPS_ENABLING, SCX_OPS_PREPPING)) {
+ preempt_enable();
+ spin_unlock_irq(&scx_tasks_lock);
+ WARN_ON_ONCE(atomic_read(&scx_exit_kind) == SCX_EXIT_NONE);
+ ret = 0;
+ goto err_disable_unlock_all;
+ }
+
+ /*
+ * We're fully committed and can't fail. The PREPPED -> ENABLED
+ * transitions here are synchronized against sched_ext_free() through
+ * scx_tasks_lock.
+ */
+ WRITE_ONCE(scx_switching_all, !(ops->flags & SCX_OPS_SWITCH_PARTIAL));
+
+ scx_task_iter_init(&sti);
+ while ((p = scx_task_iter_next_locked(&sti, false))) {
+ const struct sched_class *old_class = p->sched_class;
+ struct sched_enq_and_set_ctx ctx;
+
+ sched_deq_and_put_task(p, DEQUEUE_SAVE | DEQUEUE_MOVE, &ctx);
+
+ scx_set_task_state(p, SCX_TASK_READY);
+ __setscheduler_prio(p, p->prio);
+ check_class_changing(task_rq(p), p, old_class);
+
+ sched_enq_and_set_task(&ctx);
+
+ check_class_changed(task_rq(p), p, old_class, p->prio);
+ }
+ scx_task_iter_exit(&sti);
+
+ spin_unlock_irq(&scx_tasks_lock);
+ preempt_enable();
+ cpus_read_unlock();
+ percpu_up_write(&scx_fork_rwsem);
+
+ /* see above ENABLING transition for the explanation on exiting with 0 */
+ if (!scx_ops_tryset_enable_state(SCX_OPS_ENABLED, SCX_OPS_ENABLING)) {
+ WARN_ON_ONCE(atomic_read(&scx_exit_kind) == SCX_EXIT_NONE);
+ ret = 0;
+ goto err_disable;
+ }
+
+ if (!(ops->flags & SCX_OPS_SWITCH_PARTIAL))
+ static_branch_enable(&__scx_switched_all);
+
+ kobject_uevent(scx_root_kobj, KOBJ_ADD);
+ mutex_unlock(&scx_ops_enable_mutex);
+
+ return 0;
+
+err_del:
+ kobject_del(scx_root_kobj);
+err:
+ kobject_put(scx_root_kobj);
+ scx_root_kobj = NULL;
+ if (scx_exit_info) {
+ free_exit_info(scx_exit_info);
+ scx_exit_info = NULL;
+ }
+err_unlock:
+ mutex_unlock(&scx_ops_enable_mutex);
+ return ret;
+
+err_disable_unlock_all:
+ percpu_up_write(&scx_fork_rwsem);
+err_disable_unlock_cpus:
+ cpus_read_unlock();
+err_disable:
+ mutex_unlock(&scx_ops_enable_mutex);
+ /* must be fully disabled before returning */
+ scx_ops_disable(SCX_EXIT_ERROR);
+ kthread_flush_work(&scx_ops_disable_work);
+ return ret;
+}
+
+
+/********************************************************************************
+ * bpf_struct_ops plumbing.
+ */
+#include <linux/bpf_verifier.h>
+#include <linux/bpf.h>
+#include <linux/btf.h>
+
+extern struct btf *btf_vmlinux;
+static const struct btf_type *task_struct_type;
+static u32 task_struct_type_id;
+
+static bool set_arg_maybe_null(const char *op, int arg_n, int off, int size,
+ enum bpf_access_type type,
+ const struct bpf_prog *prog,
+ struct bpf_insn_access_aux *info)
+{
+ struct btf *btf = bpf_get_btf_vmlinux();
+ const struct bpf_struct_ops_desc *st_ops_desc;
+ const struct btf_member *member;
+ const struct btf_type *t;
+ u32 btf_id, member_idx;
+ const char *mname;
+
+ /* struct_ops op args are all sequential, 64-bit numbers */
+ if (off != arg_n * sizeof(__u64))
+ return false;
+
+ /* btf_id should be the type id of struct sched_ext_ops */
+ btf_id = prog->aux->attach_btf_id;
+ st_ops_desc = bpf_struct_ops_find(btf, btf_id);
+ if (!st_ops_desc)
+ return false;
+
+ /* BTF type of struct sched_ext_ops */
+ t = st_ops_desc->type;
+
+ member_idx = prog->expected_attach_type;
+ if (member_idx >= btf_type_vlen(t))
+ return false;
+
+ /*
+ * Get the member name of this struct_ops program, which corresponds to
+ * a field in struct sched_ext_ops. For example, the member name of the
+ * dispatch struct_ops program (callback) is "dispatch".
+ */
+ member = &btf_type_member(t)[member_idx];
+ mname = btf_name_by_offset(btf_vmlinux, member->name_off);
+
+ if (!strcmp(mname, op)) {
+ /*
+ * The value is a pointer to a type (struct task_struct) given
+ * by a BTF ID (PTR_TO_BTF_ID). It is trusted (PTR_TRUSTED),
+ * however, can be a NULL (PTR_MAYBE_NULL). The BPF program
+ * should check the pointer to make sure it is not NULL before
+ * using it, or the verifier will reject the program.
+ *
+ * Longer term, this is something that should be addressed by
+ * BTF, and be fully contained within the verifier.
+ */
+ info->reg_type = PTR_MAYBE_NULL | PTR_TO_BTF_ID | PTR_TRUSTED;
+ info->btf = btf_vmlinux;
+ info->btf_id = task_struct_type_id;
+
+ return true;
+ }
+
+ return false;
+}
+
+static bool bpf_scx_is_valid_access(int off, int size,
+ enum bpf_access_type type,
+ const struct bpf_prog *prog,
+ struct bpf_insn_access_aux *info)
+{
+ if (type != BPF_READ)
+ return false;
+ if (set_arg_maybe_null("dispatch", 1, off, size, type, prog, info) ||
+ set_arg_maybe_null("yield", 1, off, size, type, prog, info))
+ return true;
+ if (off < 0 || off >= sizeof(__u64) * MAX_BPF_FUNC_ARGS)
+ return false;
+ if (off % size != 0)
+ return false;
+
+ return btf_ctx_access(off, size, type, prog, info);
+}
+
+static int bpf_scx_btf_struct_access(struct bpf_verifier_log *log,
+ const struct bpf_reg_state *reg, int off,
+ int size)
+{
+ const struct btf_type *t;
+
+ t = btf_type_by_id(reg->btf, reg->btf_id);
+ if (t == task_struct_type) {
+ if (off >= offsetof(struct task_struct, scx.slice) &&
+ off + size <= offsetofend(struct task_struct, scx.slice))
+ return SCALAR_VALUE;
+ }
+
+ return -EACCES;
+}
+
+static const struct bpf_func_proto *
+bpf_scx_get_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
+{
+ switch (func_id) {
+ case BPF_FUNC_task_storage_get:
+ return &bpf_task_storage_get_proto;
+ case BPF_FUNC_task_storage_delete:
+ return &bpf_task_storage_delete_proto;
+ default:
+ return bpf_base_func_proto(func_id, prog);
+ }
+}
+
+static const struct bpf_verifier_ops bpf_scx_verifier_ops = {
+ .get_func_proto = bpf_scx_get_func_proto,
+ .is_valid_access = bpf_scx_is_valid_access,
+ .btf_struct_access = bpf_scx_btf_struct_access,
+};
+
+static int bpf_scx_init_member(const struct btf_type *t,
+ const struct btf_member *member,
+ void *kdata, const void *udata)
+{
+ const struct sched_ext_ops *uops = udata;
+ struct sched_ext_ops *ops = kdata;
+ u32 moff = __btf_member_bit_offset(t, member) / 8;
+ int ret;
+
+ switch (moff) {
+ case offsetof(struct sched_ext_ops, dispatch_max_batch):
+ if (*(u32 *)(udata + moff) > INT_MAX)
+ return -E2BIG;
+ ops->dispatch_max_batch = *(u32 *)(udata + moff);
+ return 1;
+ case offsetof(struct sched_ext_ops, flags):
+ if (*(u64 *)(udata + moff) & ~SCX_OPS_ALL_FLAGS)
+ return -EINVAL;
+ ops->flags = *(u64 *)(udata + moff);
+ return 1;
+ case offsetof(struct sched_ext_ops, name):
+ ret = bpf_obj_name_cpy(ops->name, uops->name,
+ sizeof(ops->name));
+ if (ret < 0)
+ return ret;
+ if (ret == 0)
+ return -EINVAL;
+ return 1;
+ }
+
+ return 0;
+}
+
+static int bpf_scx_check_member(const struct btf_type *t,
+ const struct btf_member *member,
+ const struct bpf_prog *prog)
+{
+ u32 moff = __btf_member_bit_offset(t, member) / 8;
+
+ switch (moff) {
+ case offsetof(struct sched_ext_ops, init_task):
+ case offsetof(struct sched_ext_ops, init):
+ case offsetof(struct sched_ext_ops, exit):
+ break;
+ default:
+ if (prog->sleepable)
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static int bpf_scx_reg(void *kdata, struct bpf_link *link)
+{
+ return scx_ops_enable(kdata, link);
+}
+
+static void bpf_scx_unreg(void *kdata, struct bpf_link *link)
+{
+ scx_ops_disable(SCX_EXIT_UNREG);
+ kthread_flush_work(&scx_ops_disable_work);
+}
+
+static int bpf_scx_init(struct btf *btf)
+{
+ u32 type_id;
+
+ type_id = btf_find_by_name_kind(btf, "task_struct", BTF_KIND_STRUCT);
+ if (type_id < 0)
+ return -EINVAL;
+ task_struct_type = btf_type_by_id(btf, type_id);
+ task_struct_type_id = type_id;
+
+ return 0;
+}
+
+static int bpf_scx_update(void *kdata, void *old_kdata, struct bpf_link *link)
+{
+ /*
+ * sched_ext does not support updating the actively-loaded BPF
+ * scheduler, as registering a BPF scheduler can always fail if the
+ * scheduler returns an error code for e.g. ops.init(), ops.init_task(),
+ * etc. Similarly, we can always race with unregistration happening
+ * elsewhere, such as with sysrq.
+ */
+ return -EOPNOTSUPP;
+}
+
+static int bpf_scx_validate(void *kdata)
+{
+ return 0;
+}
+
+static s32 select_cpu_stub(struct task_struct *p, s32 prev_cpu, u64 wake_flags) { return -EINVAL; }
+static void enqueue_stub(struct task_struct *p, u64 enq_flags) {}
+static void dequeue_stub(struct task_struct *p, u64 enq_flags) {}
+static void dispatch_stub(s32 prev_cpu, struct task_struct *p) {}
+static bool yield_stub(struct task_struct *from, struct task_struct *to) { return false; }
+static void set_weight_stub(struct task_struct *p, u32 weight) {}
+static void set_cpumask_stub(struct task_struct *p, const struct cpumask *mask) {}
+static void update_idle_stub(s32 cpu, bool idle) {}
+static s32 init_task_stub(struct task_struct *p, struct scx_init_task_args *args) { return -EINVAL; }
+static void exit_task_stub(struct task_struct *p, struct scx_exit_task_args *args) {}
+static void enable_stub(struct task_struct *p) {}
+static void disable_stub(struct task_struct *p) {}
+static s32 init_stub(void) { return -EINVAL; }
+static void exit_stub(struct scx_exit_info *info) {}
+
+static struct sched_ext_ops __bpf_ops_sched_ext_ops = {
+ .select_cpu = select_cpu_stub,
+ .enqueue = enqueue_stub,
+ .dequeue = dequeue_stub,
+ .dispatch = dispatch_stub,
+ .yield = yield_stub,
+ .set_weight = set_weight_stub,
+ .set_cpumask = set_cpumask_stub,
+ .update_idle = update_idle_stub,
+ .init_task = init_task_stub,
+ .exit_task = exit_task_stub,
+ .enable = enable_stub,
+ .disable = disable_stub,
+ .init = init_stub,
+ .exit = exit_stub,
+};
+
+static struct bpf_struct_ops bpf_sched_ext_ops = {
+ .verifier_ops = &bpf_scx_verifier_ops,
+ .reg = bpf_scx_reg,
+ .unreg = bpf_scx_unreg,
+ .check_member = bpf_scx_check_member,
+ .init_member = bpf_scx_init_member,
+ .init = bpf_scx_init,
+ .update = bpf_scx_update,
+ .validate = bpf_scx_validate,
+ .name = "sched_ext_ops",
+ .owner = THIS_MODULE,
+ .cfi_stubs = &__bpf_ops_sched_ext_ops
+};
+
+
+/********************************************************************************
+ * System integration and init.
+ */
+
+void __init init_sched_ext_class(void)
+{
+ s32 cpu, v;
+
+ /*
+ * The following is to prevent the compiler from optimizing out the enum
+ * definitions so that BPF scheduler implementations can use them
+ * through the generated vmlinux.h.
+ */
+ WRITE_ONCE(v, SCX_ENQ_WAKEUP | SCX_DEQ_SLEEP);
+
+ BUG_ON(rhashtable_init(&dsq_hash, &dsq_hash_params));
+ init_dsq(&scx_dsq_global, SCX_DSQ_GLOBAL);
+#ifdef CONFIG_SMP
+ BUG_ON(!alloc_cpumask_var(&idle_masks.cpu, GFP_KERNEL));
+ BUG_ON(!alloc_cpumask_var(&idle_masks.smt, GFP_KERNEL));
+#endif
+ for_each_possible_cpu(cpu) {
+ struct rq *rq = cpu_rq(cpu);
+
+ init_dsq(&rq->scx.local_dsq, SCX_DSQ_LOCAL);
+ INIT_LIST_HEAD(&rq->scx.runnable_list);
+ }
+}
+
+
+/********************************************************************************
+ * Helpers that can be called from the BPF scheduler.
+ */
+#include <linux/btf_ids.h>
+
+__bpf_kfunc_start_defs();
+
+/**
+ * scx_bpf_create_dsq - Create a custom DSQ
+ * @dsq_id: DSQ to create
+ * @node: NUMA node to allocate from
+ *
+ * Create a custom DSQ identified by @dsq_id. Can be called from ops.init() and
+ * ops.init_task().
+ */
+__bpf_kfunc s32 scx_bpf_create_dsq(u64 dsq_id, s32 node)
+{
+ if (!scx_kf_allowed(SCX_KF_SLEEPABLE))
+ return -EINVAL;
+
+ if (unlikely(node >= (int)nr_node_ids ||
+ (node < 0 && node != NUMA_NO_NODE)))
+ return -EINVAL;
+ return PTR_ERR_OR_ZERO(create_dsq(dsq_id, node));
+}
+
+__bpf_kfunc_end_defs();
+
+BTF_KFUNCS_START(scx_kfunc_ids_sleepable)
+BTF_ID_FLAGS(func, scx_bpf_create_dsq, KF_SLEEPABLE)
+BTF_KFUNCS_END(scx_kfunc_ids_sleepable)
+
+static const struct btf_kfunc_id_set scx_kfunc_set_sleepable = {
+ .owner = THIS_MODULE,
+ .set = &scx_kfunc_ids_sleepable,
+};
+
+__bpf_kfunc_start_defs();
+
+/**
+ * scx_bpf_select_cpu_dfl - The default implementation of ops.select_cpu()
+ * @p: task_struct to select a CPU for
+ * @prev_cpu: CPU @p was on previously
+ * @wake_flags: %SCX_WAKE_* flags
+ * @is_idle: out parameter indicating whether the returned CPU is idle
+ *
+ * Can only be called from ops.select_cpu() if the built-in CPU selection is
+ * enabled - ops.update_idle() is missing or %SCX_OPS_KEEP_BUILTIN_IDLE is set.
+ * @p, @prev_cpu and @wake_flags match ops.select_cpu().
+ *
+ * Returns the picked CPU with *@is_idle indicating whether the picked CPU is
+ * currently idle and thus a good candidate for direct dispatching.
+ */
+__bpf_kfunc s32 scx_bpf_select_cpu_dfl(struct task_struct *p, s32 prev_cpu,
+ u64 wake_flags, bool *is_idle)
+{
+ if (!scx_kf_allowed(SCX_KF_SELECT_CPU)) {
+ *is_idle = false;
+ return prev_cpu;
+ }
+#ifdef CONFIG_SMP
+ return scx_select_cpu_dfl(p, prev_cpu, wake_flags, is_idle);
+#else
+ *is_idle = false;
+ return prev_cpu;
+#endif
+}
+
+__bpf_kfunc_end_defs();
+
+BTF_KFUNCS_START(scx_kfunc_ids_select_cpu)
+BTF_ID_FLAGS(func, scx_bpf_select_cpu_dfl, KF_RCU)
+BTF_KFUNCS_END(scx_kfunc_ids_select_cpu)
+
+static const struct btf_kfunc_id_set scx_kfunc_set_select_cpu = {
+ .owner = THIS_MODULE,
+ .set = &scx_kfunc_ids_select_cpu,
+};
+
+static bool scx_dispatch_preamble(struct task_struct *p, u64 enq_flags)
+{
+ if (!scx_kf_allowed(SCX_KF_ENQUEUE | SCX_KF_DISPATCH))
+ return false;
+
+ lockdep_assert_irqs_disabled();
+
+ if (unlikely(!p)) {
+ scx_ops_error("called with NULL task");
+ return false;
+ }
+
+ if (unlikely(enq_flags & __SCX_ENQ_INTERNAL_MASK)) {
+ scx_ops_error("invalid enq_flags 0x%llx", enq_flags);
+ return false;
+ }
+
+ return true;
+}
+
+static void scx_dispatch_commit(struct task_struct *p, u64 dsq_id, u64 enq_flags)
+{
+ struct scx_dsp_ctx *dspc = this_cpu_ptr(scx_dsp_ctx);
+ struct task_struct *ddsp_task;
+
+ ddsp_task = __this_cpu_read(direct_dispatch_task);
+ if (ddsp_task) {
+ mark_direct_dispatch(ddsp_task, p, dsq_id, enq_flags);
+ return;
+ }
+
+ if (unlikely(dspc->cursor >= scx_dsp_max_batch)) {
+ scx_ops_error("dispatch buffer overflow");
+ return;
+ }
+
+ dspc->buf[dspc->cursor++] = (struct scx_dsp_buf_ent){
+ .task = p,
+ .qseq = atomic_long_read(&p->scx.ops_state) & SCX_OPSS_QSEQ_MASK,
+ .dsq_id = dsq_id,
+ .enq_flags = enq_flags,
+ };
+}
+
+__bpf_kfunc_start_defs();
+
+/**
+ * scx_bpf_dispatch - Dispatch a task into the FIFO queue of a DSQ
+ * @p: task_struct to dispatch
+ * @dsq_id: DSQ to dispatch to
+ * @slice: duration @p can run for in nsecs
+ * @enq_flags: SCX_ENQ_*
+ *
+ * Dispatch @p into the FIFO queue of the DSQ identified by @dsq_id. It is safe
+ * to call this function spuriously. Can be called from ops.enqueue(),
+ * ops.select_cpu(), and ops.dispatch().
+ *
+ * When called from ops.select_cpu() or ops.enqueue(), it's for direct dispatch
+ * and @p must match the task being enqueued. Also, %SCX_DSQ_LOCAL_ON can't be
+ * used to target the local DSQ of a CPU other than the enqueueing one. Use
+ * ops.select_cpu() to be on the target CPU in the first place.
+ *
+ * When called from ops.select_cpu(), @enq_flags and @dsp_id are stored, and @p
+ * will be directly dispatched to the corresponding dispatch queue after
+ * ops.select_cpu() returns. If @p is dispatched to SCX_DSQ_LOCAL, it will be
+ * dispatched to the local DSQ of the CPU returned by ops.select_cpu().
+ * @enq_flags are OR'd with the enqueue flags on the enqueue path before the
+ * task is dispatched.
+ *
+ * When called from ops.dispatch(), there are no restrictions on @p or @dsq_id
+ * and this function can be called upto ops.dispatch_max_batch times to dispatch
+ * multiple tasks. scx_bpf_dispatch_nr_slots() returns the number of the
+ * remaining slots. scx_bpf_consume() flushes the batch and resets the counter.
+ *
+ * This function doesn't have any locking restrictions and may be called under
+ * BPF locks (in the future when BPF introduces more flexible locking).
+ *
+ * @p is allowed to run for @slice. The scheduling path is triggered on slice
+ * exhaustion. If zero, the current residual slice is maintained.
+ */
+__bpf_kfunc void scx_bpf_dispatch(struct task_struct *p, u64 dsq_id, u64 slice,
+ u64 enq_flags)
+{
+ if (!scx_dispatch_preamble(p, enq_flags))
+ return;
+
+ if (slice)
+ p->scx.slice = slice;
+ else
+ p->scx.slice = p->scx.slice ?: 1;
+
+ scx_dispatch_commit(p, dsq_id, enq_flags);
+}
+
+__bpf_kfunc_end_defs();
+
+BTF_KFUNCS_START(scx_kfunc_ids_enqueue_dispatch)
+BTF_ID_FLAGS(func, scx_bpf_dispatch, KF_RCU)
+BTF_KFUNCS_END(scx_kfunc_ids_enqueue_dispatch)
+
+static const struct btf_kfunc_id_set scx_kfunc_set_enqueue_dispatch = {
+ .owner = THIS_MODULE,
+ .set = &scx_kfunc_ids_enqueue_dispatch,
+};
+
+__bpf_kfunc_start_defs();
+
+/**
+ * scx_bpf_dispatch_nr_slots - Return the number of remaining dispatch slots
+ *
+ * Can only be called from ops.dispatch().
+ */
+__bpf_kfunc u32 scx_bpf_dispatch_nr_slots(void)
+{
+ if (!scx_kf_allowed(SCX_KF_DISPATCH))
+ return 0;
+
+ return scx_dsp_max_batch - __this_cpu_read(scx_dsp_ctx->cursor);
+}
+
+/**
+ * scx_bpf_dispatch_cancel - Cancel the latest dispatch
+ *
+ * Cancel the latest dispatch. Can be called multiple times to cancel further
+ * dispatches. Can only be called from ops.dispatch().
+ */
+__bpf_kfunc void scx_bpf_dispatch_cancel(void)
+{
+ struct scx_dsp_ctx *dspc = this_cpu_ptr(scx_dsp_ctx);
+
+ if (!scx_kf_allowed(SCX_KF_DISPATCH))
+ return;
+
+ if (dspc->cursor > 0)
+ dspc->cursor--;
+ else
+ scx_ops_error("dispatch buffer underflow");
+}
+
+/**
+ * scx_bpf_consume - Transfer a task from a DSQ to the current CPU's local DSQ
+ * @dsq_id: DSQ to consume
+ *
+ * Consume a task from the non-local DSQ identified by @dsq_id and transfer it
+ * to the current CPU's local DSQ for execution. Can only be called from
+ * ops.dispatch().
+ *
+ * This function flushes the in-flight dispatches from scx_bpf_dispatch() before
+ * trying to consume the specified DSQ. It may also grab rq locks and thus can't
+ * be called under any BPF locks.
+ *
+ * Returns %true if a task has been consumed, %false if there isn't any task to
+ * consume.
+ */
+__bpf_kfunc bool scx_bpf_consume(u64 dsq_id)
+{
+ struct scx_dsp_ctx *dspc = this_cpu_ptr(scx_dsp_ctx);
+ struct scx_dispatch_q *dsq;
+
+ if (!scx_kf_allowed(SCX_KF_DISPATCH))
+ return false;
+
+ flush_dispatch_buf(dspc->rq, dspc->rf);
+
+ dsq = find_non_local_dsq(dsq_id);
+ if (unlikely(!dsq)) {
+ scx_ops_error("invalid DSQ ID 0x%016llx", dsq_id);
+ return false;
+ }
+
+ if (consume_dispatch_q(dspc->rq, dspc->rf, dsq)) {
+ /*
+ * A successfully consumed task can be dequeued before it starts
+ * running while the CPU is trying to migrate other dispatched
+ * tasks. Bump nr_tasks to tell balance_scx() to retry on empty
+ * local DSQ.
+ */
+ dspc->nr_tasks++;
+ return true;
+ } else {
+ return false;
+ }
+}
+
+__bpf_kfunc_end_defs();
+
+BTF_KFUNCS_START(scx_kfunc_ids_dispatch)
+BTF_ID_FLAGS(func, scx_bpf_dispatch_nr_slots)
+BTF_ID_FLAGS(func, scx_bpf_dispatch_cancel)
+BTF_ID_FLAGS(func, scx_bpf_consume)
+BTF_KFUNCS_END(scx_kfunc_ids_dispatch)
+
+static const struct btf_kfunc_id_set scx_kfunc_set_dispatch = {
+ .owner = THIS_MODULE,
+ .set = &scx_kfunc_ids_dispatch,
+};
+
+__bpf_kfunc_start_defs();
+
+/**
+ * scx_bpf_dsq_nr_queued - Return the number of queued tasks
+ * @dsq_id: id of the DSQ
+ *
+ * Return the number of tasks in the DSQ matching @dsq_id. If not found,
+ * -%ENOENT is returned.
+ */
+__bpf_kfunc s32 scx_bpf_dsq_nr_queued(u64 dsq_id)
+{
+ struct scx_dispatch_q *dsq;
+ s32 ret;
+
+ preempt_disable();
+
+ if (dsq_id == SCX_DSQ_LOCAL) {
+ ret = READ_ONCE(this_rq()->scx.local_dsq.nr);
+ goto out;
+ } else if ((dsq_id & SCX_DSQ_LOCAL_ON) == SCX_DSQ_LOCAL_ON) {
+ s32 cpu = dsq_id & SCX_DSQ_LOCAL_CPU_MASK;
+
+ if (ops_cpu_valid(cpu, NULL)) {
+ ret = READ_ONCE(cpu_rq(cpu)->scx.local_dsq.nr);
+ goto out;
+ }
+ } else {
+ dsq = find_non_local_dsq(dsq_id);
+ if (dsq) {
+ ret = READ_ONCE(dsq->nr);
+ goto out;
+ }
+ }
+ ret = -ENOENT;
+out:
+ preempt_enable();
+ return ret;
+}
+
+/**
+ * scx_bpf_destroy_dsq - Destroy a custom DSQ
+ * @dsq_id: DSQ to destroy
+ *
+ * Destroy the custom DSQ identified by @dsq_id. Only DSQs created with
+ * scx_bpf_create_dsq() can be destroyed. The caller must ensure that the DSQ is
+ * empty and no further tasks are dispatched to it. Ignored if called on a DSQ
+ * which doesn't exist. Can be called from any online scx_ops operations.
+ */
+__bpf_kfunc void scx_bpf_destroy_dsq(u64 dsq_id)
+{
+ destroy_dsq(dsq_id);
+}
+
+__bpf_kfunc_end_defs();
+
+static s32 __bstr_format(u64 *data_buf, char *line_buf, size_t line_size,
+ char *fmt, unsigned long long *data, u32 data__sz)
+{
+ struct bpf_bprintf_data bprintf_data = { .get_bin_args = true };
+ s32 ret;
+
+ if (data__sz % 8 || data__sz > MAX_BPRINTF_VARARGS * 8 ||
+ (data__sz && !data)) {
+ scx_ops_error("invalid data=%p and data__sz=%u",
+ (void *)data, data__sz);
+ return -EINVAL;
+ }
+
+ ret = copy_from_kernel_nofault(data_buf, data, data__sz);
+ if (ret < 0) {
+ scx_ops_error("failed to read data fields (%d)", ret);
+ return ret;
+ }
+
+ ret = bpf_bprintf_prepare(fmt, UINT_MAX, data_buf, data__sz / 8,
+ &bprintf_data);
+ if (ret < 0) {
+ scx_ops_error("format preparation failed (%d)", ret);
+ return ret;
+ }
+
+ ret = bstr_printf(line_buf, line_size, fmt,
+ bprintf_data.bin_args);
+ bpf_bprintf_cleanup(&bprintf_data);
+ if (ret < 0) {
+ scx_ops_error("(\"%s\", %p, %u) failed to format",
+ fmt, data, data__sz);
+ return ret;
+ }
+
+ return ret;
+}
+
+static s32 bstr_format(struct scx_bstr_buf *buf,
+ char *fmt, unsigned long long *data, u32 data__sz)
+{
+ return __bstr_format(buf->data, buf->line, sizeof(buf->line),
+ fmt, data, data__sz);
+}
+
+__bpf_kfunc_start_defs();
+
+/**
+ * scx_bpf_exit_bstr - Gracefully exit the BPF scheduler.
+ * @exit_code: Exit value to pass to user space via struct scx_exit_info.
+ * @fmt: error message format string
+ * @data: format string parameters packaged using ___bpf_fill() macro
+ * @data__sz: @data len, must end in '__sz' for the verifier
+ *
+ * Indicate that the BPF scheduler wants to exit gracefully, and initiate ops
+ * disabling.
+ */
+__bpf_kfunc void scx_bpf_exit_bstr(s64 exit_code, char *fmt,
+ unsigned long long *data, u32 data__sz)
+{
+ unsigned long flags;
+
+ raw_spin_lock_irqsave(&scx_exit_bstr_buf_lock, flags);
+ if (bstr_format(&scx_exit_bstr_buf, fmt, data, data__sz) >= 0)
+ scx_ops_exit_kind(SCX_EXIT_UNREG_BPF, exit_code, "%s",
+ scx_exit_bstr_buf.line);
+ raw_spin_unlock_irqrestore(&scx_exit_bstr_buf_lock, flags);
+}
+
+/**
+ * scx_bpf_error_bstr - Indicate fatal error
+ * @fmt: error message format string
+ * @data: format string parameters packaged using ___bpf_fill() macro
+ * @data__sz: @data len, must end in '__sz' for the verifier
+ *
+ * Indicate that the BPF scheduler encountered a fatal error and initiate ops
+ * disabling.
+ */
+__bpf_kfunc void scx_bpf_error_bstr(char *fmt, unsigned long long *data,
+ u32 data__sz)
+{
+ unsigned long flags;
+
+ raw_spin_lock_irqsave(&scx_exit_bstr_buf_lock, flags);
+ if (bstr_format(&scx_exit_bstr_buf, fmt, data, data__sz) >= 0)
+ scx_ops_exit_kind(SCX_EXIT_ERROR_BPF, 0, "%s",
+ scx_exit_bstr_buf.line);
+ raw_spin_unlock_irqrestore(&scx_exit_bstr_buf_lock, flags);
+}
+
+/**
+ * scx_bpf_nr_cpu_ids - Return the number of possible CPU IDs
+ *
+ * All valid CPU IDs in the system are smaller than the returned value.
+ */
+__bpf_kfunc u32 scx_bpf_nr_cpu_ids(void)
+{
+ return nr_cpu_ids;
+}
+
+/**
+ * scx_bpf_get_possible_cpumask - Get a referenced kptr to cpu_possible_mask
+ */
+__bpf_kfunc const struct cpumask *scx_bpf_get_possible_cpumask(void)
+{
+ return cpu_possible_mask;
+}
+
+/**
+ * scx_bpf_get_online_cpumask - Get a referenced kptr to cpu_online_mask
+ */
+__bpf_kfunc const struct cpumask *scx_bpf_get_online_cpumask(void)
+{
+ return cpu_online_mask;
+}
+
+/**
+ * scx_bpf_put_cpumask - Release a possible/online cpumask
+ * @cpumask: cpumask to release
+ */
+__bpf_kfunc void scx_bpf_put_cpumask(const struct cpumask *cpumask)
+{
+ /*
+ * Empty function body because we aren't actually acquiring or releasing
+ * a reference to a global cpumask, which is read-only in the caller and
+ * is never released. The acquire / release semantics here are just used
+ * to make the cpumask is a trusted pointer in the caller.
+ */
+}
+
+/**
+ * scx_bpf_get_idle_cpumask - Get a referenced kptr to the idle-tracking
+ * per-CPU cpumask.
+ *
+ * Returns NULL if idle tracking is not enabled, or running on a UP kernel.
+ */
+__bpf_kfunc const struct cpumask *scx_bpf_get_idle_cpumask(void)
+{
+ if (!static_branch_likely(&scx_builtin_idle_enabled)) {
+ scx_ops_error("built-in idle tracking is disabled");
+ return cpu_none_mask;
+ }
+
+#ifdef CONFIG_SMP
+ return idle_masks.cpu;
+#else
+ return cpu_none_mask;
+#endif
+}
+
+/**
+ * scx_bpf_get_idle_smtmask - Get a referenced kptr to the idle-tracking,
+ * per-physical-core cpumask. Can be used to determine if an entire physical
+ * core is free.
+ *
+ * Returns NULL if idle tracking is not enabled, or running on a UP kernel.
+ */
+__bpf_kfunc const struct cpumask *scx_bpf_get_idle_smtmask(void)
+{
+ if (!static_branch_likely(&scx_builtin_idle_enabled)) {
+ scx_ops_error("built-in idle tracking is disabled");
+ return cpu_none_mask;
+ }
+
+#ifdef CONFIG_SMP
+ if (sched_smt_active())
+ return idle_masks.smt;
+ else
+ return idle_masks.cpu;
+#else
+ return cpu_none_mask;
+#endif
+}
+
+/**
+ * scx_bpf_put_idle_cpumask - Release a previously acquired referenced kptr to
+ * either the percpu, or SMT idle-tracking cpumask.
+ */
+__bpf_kfunc void scx_bpf_put_idle_cpumask(const struct cpumask *idle_mask)
+{
+ /*
+ * Empty function body because we aren't actually acquiring or releasing
+ * a reference to a global idle cpumask, which is read-only in the
+ * caller and is never released. The acquire / release semantics here
+ * are just used to make the cpumask a trusted pointer in the caller.
+ */
+}
+
+/**
+ * scx_bpf_test_and_clear_cpu_idle - Test and clear @cpu's idle state
+ * @cpu: cpu to test and clear idle for
+ *
+ * Returns %true if @cpu was idle and its idle state was successfully cleared.
+ * %false otherwise.
+ *
+ * Unavailable if ops.update_idle() is implemented and
+ * %SCX_OPS_KEEP_BUILTIN_IDLE is not set.
+ */
+__bpf_kfunc bool scx_bpf_test_and_clear_cpu_idle(s32 cpu)
+{
+ if (!static_branch_likely(&scx_builtin_idle_enabled)) {
+ scx_ops_error("built-in idle tracking is disabled");
+ return false;
+ }
+
+ if (ops_cpu_valid(cpu, NULL))
+ return test_and_clear_cpu_idle(cpu);
+ else
+ return false;
+}
+
+/**
+ * scx_bpf_pick_idle_cpu - Pick and claim an idle cpu
+ * @cpus_allowed: Allowed cpumask
+ * @flags: %SCX_PICK_IDLE_CPU_* flags
+ *
+ * Pick and claim an idle cpu in @cpus_allowed. Returns the picked idle cpu
+ * number on success. -%EBUSY if no matching cpu was found.
+ *
+ * Idle CPU tracking may race against CPU scheduling state transitions. For
+ * example, this function may return -%EBUSY as CPUs are transitioning into the
+ * idle state. If the caller then assumes that there will be dispatch events on
+ * the CPUs as they were all busy, the scheduler may end up stalling with CPUs
+ * idling while there are pending tasks. Use scx_bpf_pick_any_cpu() and
+ * scx_bpf_kick_cpu() to guarantee that there will be at least one dispatch
+ * event in the near future.
+ *
+ * Unavailable if ops.update_idle() is implemented and
+ * %SCX_OPS_KEEP_BUILTIN_IDLE is not set.
+ */
+__bpf_kfunc s32 scx_bpf_pick_idle_cpu(const struct cpumask *cpus_allowed,
+ u64 flags)
+{
+ if (!static_branch_likely(&scx_builtin_idle_enabled)) {
+ scx_ops_error("built-in idle tracking is disabled");
+ return -EBUSY;
+ }
+
+ return scx_pick_idle_cpu(cpus_allowed, flags);
+}
+
+/**
+ * scx_bpf_pick_any_cpu - Pick and claim an idle cpu if available or pick any CPU
+ * @cpus_allowed: Allowed cpumask
+ * @flags: %SCX_PICK_IDLE_CPU_* flags
+ *
+ * Pick and claim an idle cpu in @cpus_allowed. If none is available, pick any
+ * CPU in @cpus_allowed. Guaranteed to succeed and returns the picked idle cpu
+ * number if @cpus_allowed is not empty. -%EBUSY is returned if @cpus_allowed is
+ * empty.
+ *
+ * If ops.update_idle() is implemented and %SCX_OPS_KEEP_BUILTIN_IDLE is not
+ * set, this function can't tell which CPUs are idle and will always pick any
+ * CPU.
+ */
+__bpf_kfunc s32 scx_bpf_pick_any_cpu(const struct cpumask *cpus_allowed,
+ u64 flags)
+{
+ s32 cpu;
+
+ if (static_branch_likely(&scx_builtin_idle_enabled)) {
+ cpu = scx_pick_idle_cpu(cpus_allowed, flags);
+ if (cpu >= 0)
+ return cpu;
+ }
+
+ cpu = cpumask_any_distribute(cpus_allowed);
+ if (cpu < nr_cpu_ids)
+ return cpu;
+ else
+ return -EBUSY;
+}
+
+/**
+ * scx_bpf_task_running - Is task currently running?
+ * @p: task of interest
+ */
+__bpf_kfunc bool scx_bpf_task_running(const struct task_struct *p)
+{
+ return task_rq(p)->curr == p;
+}
+
+/**
+ * scx_bpf_task_cpu - CPU a task is currently associated with
+ * @p: task of interest
+ */
+__bpf_kfunc s32 scx_bpf_task_cpu(const struct task_struct *p)
+{
+ return task_cpu(p);
+}
+
+__bpf_kfunc_end_defs();
+
+BTF_KFUNCS_START(scx_kfunc_ids_any)
+BTF_ID_FLAGS(func, scx_bpf_dsq_nr_queued)
+BTF_ID_FLAGS(func, scx_bpf_destroy_dsq)
+BTF_ID_FLAGS(func, scx_bpf_exit_bstr, KF_TRUSTED_ARGS)
+BTF_ID_FLAGS(func, scx_bpf_error_bstr, KF_TRUSTED_ARGS)
+BTF_ID_FLAGS(func, scx_bpf_nr_cpu_ids)
+BTF_ID_FLAGS(func, scx_bpf_get_possible_cpumask, KF_ACQUIRE)
+BTF_ID_FLAGS(func, scx_bpf_get_online_cpumask, KF_ACQUIRE)
+BTF_ID_FLAGS(func, scx_bpf_put_cpumask, KF_RELEASE)
+BTF_ID_FLAGS(func, scx_bpf_get_idle_cpumask, KF_ACQUIRE)
+BTF_ID_FLAGS(func, scx_bpf_get_idle_smtmask, KF_ACQUIRE)
+BTF_ID_FLAGS(func, scx_bpf_put_idle_cpumask, KF_RELEASE)
+BTF_ID_FLAGS(func, scx_bpf_test_and_clear_cpu_idle)
+BTF_ID_FLAGS(func, scx_bpf_pick_idle_cpu, KF_RCU)
+BTF_ID_FLAGS(func, scx_bpf_pick_any_cpu, KF_RCU)
+BTF_ID_FLAGS(func, scx_bpf_task_running, KF_RCU)
+BTF_ID_FLAGS(func, scx_bpf_task_cpu, KF_RCU)
+BTF_KFUNCS_END(scx_kfunc_ids_any)
+
+static const struct btf_kfunc_id_set scx_kfunc_set_any = {
+ .owner = THIS_MODULE,
+ .set = &scx_kfunc_ids_any,
+};
+
+static int __init scx_init(void)
+{
+ int ret;
+
+ /*
+ * kfunc registration can't be done from init_sched_ext_class() as
+ * register_btf_kfunc_id_set() needs most of the system to be up.
+ *
+ * Some kfuncs are context-sensitive and can only be called from
+ * specific SCX ops. They are grouped into BTF sets accordingly.
+ * Unfortunately, BPF currently doesn't have a way of enforcing such
+ * restrictions. Eventually, the verifier should be able to enforce
+ * them. For now, register them the same and make each kfunc explicitly
+ * check using scx_kf_allowed().
+ */
+ if ((ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_STRUCT_OPS,
+ &scx_kfunc_set_sleepable)) ||
+ (ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_STRUCT_OPS,
+ &scx_kfunc_set_select_cpu)) ||
+ (ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_STRUCT_OPS,
+ &scx_kfunc_set_enqueue_dispatch)) ||
+ (ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_STRUCT_OPS,
+ &scx_kfunc_set_dispatch)) ||
+ (ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_STRUCT_OPS,
+ &scx_kfunc_set_any)) ||
+ (ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_TRACING,
+ &scx_kfunc_set_any)) ||
+ (ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_SYSCALL,
+ &scx_kfunc_set_any))) {
+ pr_err("sched_ext: Failed to register kfunc sets (%d)\n", ret);
+ return ret;
+ }
+
+ ret = register_bpf_struct_ops(&bpf_sched_ext_ops, sched_ext_ops);
+ if (ret) {
+ pr_err("sched_ext: Failed to register struct_ops (%d)\n", ret);
+ return ret;
+ }
+
+ scx_kset = kset_create_and_add("sched_ext", &scx_uevent_ops, kernel_kobj);
+ if (!scx_kset) {
+ pr_err("sched_ext: Failed to create /sys/kernel/sched_ext\n");
+ return -ENOMEM;
+ }
+
+ ret = sysfs_create_group(&scx_kset->kobj, &scx_global_attr_group);
+ if (ret < 0) {
+ pr_err("sched_ext: Failed to add global attributes\n");
+ return ret;
+ }
+
+ return 0;
+}
+__initcall(scx_init);
diff --git a/kernel/sched/ext.h b/kernel/sched/ext.h
index 6a93c4825339..9c5a2d928281 100644
--- a/kernel/sched/ext.h
+++ b/kernel/sched/ext.h
@@ -1,15 +1,76 @@
/* SPDX-License-Identifier: GPL-2.0 */
-
+/*
+ * Copyright (c) 2022 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2022 Tejun Heo <tj@kernel.org>
+ * Copyright (c) 2022 David Vernet <dvernet@meta.com>
+ */
#ifdef CONFIG_SCHED_CLASS_EXT
-#error "NOT IMPLEMENTED YET"
+
+struct sched_enq_and_set_ctx {
+ struct task_struct *p;
+ int queue_flags;
+ bool queued;
+ bool running;
+};
+
+void sched_deq_and_put_task(struct task_struct *p, int queue_flags,
+ struct sched_enq_and_set_ctx *ctx);
+void sched_enq_and_set_task(struct sched_enq_and_set_ctx *ctx);
+
+extern const struct sched_class ext_sched_class;
+
+DECLARE_STATIC_KEY_FALSE(__scx_ops_enabled);
+DECLARE_STATIC_KEY_FALSE(__scx_switched_all);
+#define scx_enabled() static_branch_unlikely(&__scx_ops_enabled)
+#define scx_switched_all() static_branch_unlikely(&__scx_switched_all)
+
+static inline bool task_on_scx(const struct task_struct *p)
+{
+ return scx_enabled() && p->sched_class == &ext_sched_class;
+}
+
+void init_scx_entity(struct sched_ext_entity *scx);
+void scx_pre_fork(struct task_struct *p);
+int scx_fork(struct task_struct *p);
+void scx_post_fork(struct task_struct *p);
+void scx_cancel_fork(struct task_struct *p);
+bool task_should_scx(struct task_struct *p);
+void init_sched_ext_class(void);
+
+static inline const struct sched_class *next_active_class(const struct sched_class *class)
+{
+ class++;
+ if (scx_switched_all() && class == &fair_sched_class)
+ class++;
+ if (!scx_enabled() && class == &ext_sched_class)
+ class++;
+ return class;
+}
+
+#define for_active_class_range(class, _from, _to) \
+ for (class = (_from); class != (_to); class = next_active_class(class))
+
+#define for_each_active_class(class) \
+ for_active_class_range(class, __sched_class_highest, __sched_class_lowest)
+
+/*
+ * SCX requires a balance() call before every pick_next_task() call including
+ * when waking up from idle.
+ */
+#define for_balance_class_range(class, prev_class, end_class) \
+ for_active_class_range(class, (prev_class) > &ext_sched_class ? \
+ &ext_sched_class : (prev_class), (end_class))
+
#else /* CONFIG_SCHED_CLASS_EXT */
#define scx_enabled() false
+#define scx_switched_all() false
static inline void scx_pre_fork(struct task_struct *p) {}
static inline int scx_fork(struct task_struct *p) { return 0; }
static inline void scx_post_fork(struct task_struct *p) {}
static inline void scx_cancel_fork(struct task_struct *p) {}
+static inline bool task_on_scx(const struct task_struct *p) { return false; }
static inline void init_sched_ext_class(void) {}
#define for_each_active_class for_each_class
@@ -18,7 +79,13 @@ static inline void init_sched_ext_class(void) {}
#endif /* CONFIG_SCHED_CLASS_EXT */
#if defined(CONFIG_SCHED_CLASS_EXT) && defined(CONFIG_SMP)
-#error "NOT IMPLEMENTED YET"
+void __scx_update_idle(struct rq *rq, bool idle);
+
+static inline void scx_update_idle(struct rq *rq, bool idle)
+{
+ if (scx_enabled())
+ __scx_update_idle(rq, idle);
+}
#else
static inline void scx_update_idle(struct rq *rq, bool idle) {}
#endif
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index c52ad5fdd096..2960e153c3a7 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -194,6 +194,10 @@ static inline int idle_policy(int policy)
static inline int normal_policy(int policy)
{
+#ifdef CONFIG_SCHED_CLASS_EXT
+ if (policy == SCHED_EXT)
+ return true;
+#endif
return policy == SCHED_NORMAL;
}
@@ -719,6 +723,16 @@ struct cfs_rq {
#endif /* CONFIG_FAIR_GROUP_SCHED */
};
+#ifdef CONFIG_SCHED_CLASS_EXT
+struct scx_rq {
+ struct scx_dispatch_q local_dsq;
+ struct list_head runnable_list; /* runnable tasks on this rq */
+ unsigned long ops_qseq;
+ u64 extra_enq_flags; /* see move_task_to_local_dsq() */
+ u32 nr_running;
+};
+#endif /* CONFIG_SCHED_CLASS_EXT */
+
static inline int rt_bandwidth_enabled(void)
{
return sysctl_sched_rt_runtime >= 0;
@@ -1066,6 +1080,9 @@ struct rq {
struct cfs_rq cfs;
struct rt_rq rt;
struct dl_rq dl;
+#ifdef CONFIG_SCHED_CLASS_EXT
+ struct scx_rq scx;
+#endif
#ifdef CONFIG_FAIR_GROUP_SCHED
/* list of leaf cfs_rq on this CPU: */
diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
index 050215ef8fa4..18d44d180db1 100644
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -1622,6 +1622,7 @@ SYSCALL_DEFINE1(sched_get_priority_max, int, policy)
case SCHED_NORMAL:
case SCHED_BATCH:
case SCHED_IDLE:
+ case SCHED_EXT:
ret = 0;
break;
}
@@ -1649,6 +1650,7 @@ SYSCALL_DEFINE1(sched_get_priority_min, int, policy)
case SCHED_NORMAL:
case SCHED_BATCH:
case SCHED_IDLE:
+ case SCHED_EXT:
ret = 0;
}
return ret;
--
2.45.2
Diff
No diff found.
Implementation Analysis
Overview
PATCH 09/30 is the heart of the sched_ext series. It adds 4,256 lines to create kernel/sched/ext.c — the full implementation of a new scheduler class that allows scheduling policies to be written as BPF programs. This is the patch that makes sched_ext real: before it, the earlier patches in the series were preparatory groundwork; after it, a working BPF-programmable scheduler class exists in the kernel.
The patch creates ext_sched_class, implements all required sched_class callbacks, defines the sched_ext_ops BPF interface, implements the Dispatch Queue (DSQ) abstraction, and handles the complex enable/disable lifecycle. It touches 14 files and creates the 4,256-line kernel/sched/ext.c from scratch.
The sched_ext_ops Interface: The BPF Contract
struct sched_ext_ops (defined in include/linux/sched/ext.h) is the primary interface between the kernel and a BPF scheduler. It is a BPF struct_ops type, meaning BPF programs implement it by defining a global variable of this type and loading it with bpf_prog_load. The kernel calls the BPF-implemented functions through this structure.
Key design principle: every field except .name is optional. If a BPF scheduler doesn't implement a callback, the kernel provides a correct default. This means a minimal BPF scheduler can be just a name and an empty struct — it will behave identically to the global FIFO fallback.
CPU selection and task placement callbacks:
-
ops.select_cpu(p, prev_cpu, wake_flags): Called when a task wakes up. BPF returns a preferred CPU (or -1 to let the kernel decide). Ifscx_bpf_dispatch()is called from within this callback, the task is dispatched directly toSCX_DSQ_LOCALof the selected CPU, bypassingops.enqueue(). This optimization (added in v6) avoids the overhead of re-examining the task in enqueue after the CPU has already been chosen. -
ops.enqueue(p, enq_flags): Called when a task becomes runnable and needs to be placed somewhere. BPF must callscx_bpf_dispatch(p, dsq_id, slice, enq_flags)to put the task in a DSQ. Ifops.enqueueis not implemented, all tasks go toSCX_DSQ_GLOBAL. -
ops.dequeue(p, deq_flags): Called when a task is being dequeued (e.g., because it's being moved to a different class, or the BPF scheduler is being disabled). BPF should remove the task from any internal data structures. -
ops.dispatch(cpu, prev): Called when a CPU needs a task to run. BPF callsscx_bpf_consume(dsq_id)to move tasks from a custom DSQ into the CPU's local queue, or callsscx_bpf_dispatch()directly.previs the previously running task (may be NULL on first dispatch after enable).
Task state notification callbacks:
-
ops.runnable(p, enq_flags)/ops.quiescent(p, deq_flags): Inform the BPF scheduler when a task transitions between runnable and not-runnable states. Useful for load tracking. -
ops.running(p)/ops.stopping(p, runnable): Inform the BPF scheduler when a task actually begins and ends execution on a CPU. -
ops.set_weight(p, weight): Notifies BPF when a task's weight changes (due to nice value changes). Push model avoids polling. -
ops.set_cpumask(p, cpumask): Notifies BPF when a task's allowed CPU set changes.
Task lifecycle callbacks:
-
ops.init_task(p, args): Called when a task is about to join SCX. BPF allocates per-task state and returns 0 on success or a negative errno to reject the task (which keeps it on CFS).argscarries initial state (weight, cpumask). -
ops.exit_task(p, ctx): Called when a task leaves SCX. BPF frees per-task state. -
ops.enable(p)/ops.disable(p): Called when a task starts and stops being scheduled by SCX. Unlikeinit/exit_task, these are called whenever the task transitions in or out of active SCX scheduling (e.g., if the BPF scheduler temporarily bypasses and then resumes).
Scheduler lifecycle callbacks:
-
ops.init(): Called when the BPF scheduler loads. Runs sleepable (can allocate memory). BPF sets up global data structures here. -
ops.exit(info): Called when the BPF scheduler unloads or is forcibly disabled.infocarries the exit reason, code, and a debug message string.
The Dispatch Queue (DSQ) Architecture
DSQs are the central abstraction that bridges the gap between the BPF scheduler's view (global scheduling decisions) and the kernel's view (per-CPU runqueues). The fundamental insight: a BPF scheduler should not need to manage per-CPU runqueues directly. Instead, it places tasks in DSQs, and the kernel pulls tasks from DSQs onto CPUs.
Built-in DSQs:
-
SCX_DSQ_GLOBAL(id=0): A single global FIFO. Any CPU can dequeue from it viascx_bpf_consume(SCX_DSQ_GLOBAL)inops.dispatch(). Provides a trivial "just run everything" fallback. Ifops.enqueueis not implemented, all tasks go here. -
SCX_DSQ_LOCAL(id=1): Per-CPU local FIFO. Tasks dispatched here run on the specific CPU. The scheduler core pulls from the local DSQ automatically inpick_task_scx()— BPF does not need to explicitly consume from it.
Custom DSQs:
BPF schedulers create custom DSQs with scx_bpf_create_dsq(dsq_id, node) where node is a NUMA node (or -1 for any). Custom DSQs are the mechanism for implementing sophisticated policies:
- A priority scheduler creates one DSQ per priority level
- A NUMA-aware scheduler creates per-node DSQs
- A per-cgroup scheduler creates one DSQ per cgroup
Custom DSQs support two ordering modes:
- FIFO (default):
scx_bpf_dispatch(p, dsq_id, slice, 0)— tasks run in dispatch order - Vtime-ordered:
scx_bpf_dispatch_vtime(p, dsq_id, slice, vtime, 0)— tasks ordered by virtual time (for weighted fair queueing)
In ops.dispatch(), BPF calls scx_bpf_consume(dsq_id) to move tasks from a custom DSQ to the calling CPU's local queue. Multiple custom DSQs can be consumed in priority order. The dispatch loop continues until the local queue has tasks or all DSQs are exhausted.
ext_sched_class: Implementing sched_class
ext_sched_class is defined at the bottom of kernel/sched/ext.c with all required sched_class function pointers filled in. Key implementations:
enqueue_task_scx(): The entry point when a task becomes runnable. Calls ops.enqueue() if implemented, or dispatches directly to SCX_DSQ_GLOBAL. Sets SCX_TASK_QUEUED in p->scx.flags. Handles the case where the task was running on another CPU and is being re-enqueued after preemption.
dequeue_task_scx(): Removes a task from its current DSQ. Calls ops.dequeue() if implemented. Must handle the case where the task might be mid-dispatch (the ops_qseq mechanism handles this — see below).
pick_task_scx() and balance_scx(): These two functions together implement the CPU-local scheduling decision. balance_scx() calls ops.dispatch() to populate the local DSQ; pick_task_scx() then picks the next task from the local DSQ. On UP (uniprocessor) systems, balance_scx() is called from within these paths as needed.
put_prev_task_scx(): Called when the current task is being preempted or voluntarily yields. Re-enqueues the task if it's still runnable. The SCX_TASK_DEQD_FOR_SLEEP flag (added in v2) handles the UP case where this function must decide whether to call balance_scx().
set_cpus_allowed_scx(): Updates CPU affinity. Calls ops.set_cpumask() if the cpumask changed.
task_woken_scx(): Called when a sleeping task wakes up. Calls ops.select_cpu() to potentially dispatch directly to a local DSQ.
The Task Ownership Model: p->scx.ops_state
A core design challenge: when the BPF scheduler is being disabled, all tasks must be atomically moved back to CFS. But tasks can be in various states: queued in a DSQ, running, sleeping, mid-dispatch. The ops_state field tracks ownership through these transitions:
SCX_OPS_STATE_NONE: Task is not under SCX controlSCX_OPS_STATE_QUEUEING: Task is being enqueued (ops.enqueue() is executing)SCX_OPS_STATE_QUEUED: Task is queued in a DSQSCX_OPS_STATE_DISPATCHING: Task is being moved from a DSQ to a local queue
Transitions are atomic. The disable path can iterate scx_tasks (the global list of all SCX tasks) and, for each task, atomically claim it from whatever state it's in and move it to CFS. The ops_state ensures no task is claimed twice or missed.
The ops_qseq Dequeue Race
A subtle correctness problem: a task can be in a custom DSQ, and simultaneously dequeue_task_scx() is called (e.g., the task is being killed or moved to a different class) while ops.dispatch() is trying to consume from that same DSQ. These execute on different CPUs with the rq lock held for their respective CPUs — they can run concurrently.
The ops_qseq (operation sequence number) mechanism resolves this. When dequeue_task_scx() starts, it increments the task's ops_qseq. When the dispatch path is about to move a task from a DSQ to a local queue, it checks that ops_qseq hasn't changed since it found the task. If it has, the task is being dequeued concurrently and the dispatch path backs off.
The sticky_cpu field complements this: when a task is being transferred between runqueues (during dispatch_to_local_dsq()), it is "stuck" to its source CPU to prevent it from being migrated during the transfer.
Enable and Disable Paths
These are explicitly called out in the commit message as "a bit complicated."
scx_ops_enable(): Runs in a work queue (sleepable context). Steps:
- Calls
ops.init()(sleepable, can allocate) - Iterates all tasks and calls
ops.init_task()for each - Atomically switches all
SCHED_EXTtasks from CFS to SCX (without blocking, to avoid starvation) - Enables the static branch that makes hot paths enter SCX code
- Calls
ops.enable()for each task now on SCX
The "without blocking" requirement for step 3 is critical: if the enable path blocked, the tasks it had already switched but not yet stabilized could be in a partially-switched state. The task currently running on the enabling CPU (which is doing this work) could itself be starved. The entire switch is done under brief rq locks with no sleepable operations in the switching loop.
scx_ops_disable_workfn(): Runs in a work queue. Steps:
- Enters bypass mode (all scheduling decisions go to a simple FIFO, bypassing BPF)
- Iterates all SCX tasks and moves them back to CFS
- Calls
ops.exit_task()for each task - Calls
ops.exit()on the BPF scheduler - Frees all DSQs
- Disables the static branch
The disable path cannot trust the BPF scheduler at all — it might be in a broken state (that's why it's being disabled). Bypass mode ensures forward progress even if the BPF scheduler has corrupted its own state. All disable-path operations must complete without blocking on BPF.
Bypass mode (added in v6) is a lightweight "safe mode": it replaces the BPF dispatch path with a simple global FIFO, allowing all tasks to continue running while the disable cleanup happens in the background.
Static Branch Optimization
When no BPF scheduler is loaded, sched_ext has zero overhead on the scheduling hot paths. This is achieved through Linux's static branch mechanism (also called "jump labels"):
DEFINE_STATIC_KEY_FALSE(scx_ops_enabled);
All hot-path entry points (enqueue_task_scx(), etc.) are guarded by if (static_branch_unlikely(&scx_ops_enabled)). When no BPF scheduler is loaded, this compiles down to a single NOP instruction. When a BPF scheduler loads, the static branch is flipped and all NOP sites are patched to unconditional jumps.
This means enabling CONFIG_SCHED_CLASS_EXT in your kernel has zero performance cost when no BPF scheduler is loaded.
The scx_entity in task_struct
Every task has a struct scx_entity scx embedded in task_struct. Key fields:
ops_state: Current ownership state (see above)ops_qseq: Dequeue sequence number for race detectionflags: Bitmask ofSCX_TASK_*flags (QUEUED,RUNNABLE,DEQD_FOR_SLEEP, etc.)dsq: Pointer to the DSQ the task is currently in (NULL if not queued)dsq_node: List or rbtree node for DSQ membership (FIFO vs. vtime)dsq_vtime: Virtual time for vtime-ordered DSQsslice: Current time slice assigned by the BPF scheduler (default:SCX_SLICE_DFL)sticky_cpu: CPU the task is pinned to during cross-rq transferswatchdog_node: List node for the watchdog (later patch adds stall detection)kf_mask: Bitmask of which kfunc helpers are currently callable (enforced by BPF verifier integration)
Locking Model
The sched_ext locking model is complex because BPF schedulers call back into the kernel through kfunc helpers, and those helpers have different locking requirements depending on which callback they're called from:
-
rq->lock: The per-runqueue spinlock. Held whenops.enqueue(),ops.dispatch(),ops.dequeue(), and the state-change callbacks are called. -
scx_tasks_lock: Protects thescx_taskslinked list of all SCX tasks. Taken during enable/disable iteration and during task fork/exit. -
scx_ops_enable_mutex: A regular mutex protecting the BPF scheduler loading/unloading state. Only taken in the enable/disable paths (not hot paths).
Key invariant noted in the commit: ops.enqueue() and ops.dispatch() are never called with the same CPU's rq lock held simultaneously. ops.dispatch() is called to populate the local queue of CPU X while holding rq[X]->lock. ops.enqueue() is called when a task becomes runnable, holding the rq lock of whatever CPU the task is on (which might be X, but enqueue and dispatch are not concurrent for the same rq).
The kf_mask field enforces which kfuncs are callable from which context. For example, scx_bpf_dispatch() can be called from ops.enqueue(), ops.dispatch(), and ops.select_cpu() — but not from ops.running(). The mask is set before each callback invocation and cleared after, and the BPF verifier checks it at load time.
Error Handling and Graceful Degradation
scx_ops_error(fmt, ...) is the primary error reporting mechanism. BPF code can call scx_bpf_error() (which maps to this), and kernel code can call it directly. When called:
- The error message is stored in
scx_exit_info scx_ops_disable()is scheduled asynchronously (cannot be called from interrupt context)- Bypass mode engages immediately
- The disable work runs, moves all tasks back to CFS, and calls
ops.exit()
The scx_exit_info structure carries:
kind:SCX_EXIT_DONE(clean exit),SCX_EXIT_UNREG(unregistered),SCX_EXIT_ERROR(BPF error),SCX_EXIT_ERROR_BPF(BPF verifier trapped), etc.exit_code: Numeric code for non-error exits (used for CPU hotplug handling)reason: String description of the exitmsg: BPF-provided debug message
The entire error path is designed so the kernel remains stable regardless of what the BPF scheduler does. The BPF scheduler can call scx_bpf_error() with any string, can panic its own state, and can fail to respond — the kernel will always recover.
Naming Convention
The commit message documents the naming convention explicitly: both "sched_ext" and "scx" are used. The rule is: if the identifier already has "sched" in it, use "ext" (e.g., sched_ext_ops, SCHED_EXT, sched_ext_entity). Otherwise use "scx" (e.g., scx_bpf_dispatch, scx_ops_enable, SCX_DSQ_GLOBAL). This convention is consistently followed throughout the implementation and is worth knowing when searching the codebase.
Evolutionary History
The commit message's version history (v2 through v7) reveals how the design evolved:
- v2: Dropped a UP-only function, added
SCX_TASK_DEQD_FOR_SLEEPflag for correct UP behavior - v3: Added
ops.set_weight(), fixedmove_task_to_local_dsq()flag propagation, addedSCX_CALL_OP*()helper, switched toKF_RCUfor kfuncs - v4: Renamed
task_on_scx()totask_should_scx()(the old name was confusing), replacedscx_has_idle_cpuswith direct cpumask checks - v5: Fixed 32-bit compatibility (
atomic_long_tinstead ofatomic64_t), fixed BPF struct access security bug, added kfunc context enforcement - v6: Major: added bypass mode, replaced
SCX_OPS_DISABLINGstate machine, merged partial-switching patch, moved monitoring from debugfs to/sys/sched_ext, fixedWAKE_SYNChandling, changedops.select_cpu()to allow direct dispatch - v7: Fixed
scx_ops_bypass()deadlock, fixed use-after-free in enable path'sTASK_DEADhandling, various cleanups
The v6 changes are particularly significant — bypass mode replaced a simpler but less flexible "disabling state" approach, and the ops.select_cpu() direct dispatch optimization removed the need for BPF schedulers to communicate their CPU selection to ops.enqueue() via per-task storage.
What Maintainers Should Know
This patch is the reference implementation for everything sched_ext does. When reviewing future patches:
-
DSQ changes: Any change to DSQ ordering, creation, or destruction must maintain the invariant that all tasks are accounted for (no task lost during transitions).
-
ops_state changes: The ownership state machine is a critical correctness mechanism. New states or transitions require careful analysis of the enable/disable paths.
-
New callbacks: Must be called through
SCX_CALL_OP*()macros to ensure correct kf_mask setup. Must be handled in the bypass mode path. Must be safe to call (or skip) during the disable path. -
Locking: New code that touches
scx_entityfields must respect the locking rules. Fields modified in hot paths (enqueue, dispatch) require rq lock. Fields modified in lifecycle paths (init, exit) require appropriate serialization. -
Static branch: Any new entry points from the scheduler core into sched_ext code must be guarded by the
scx_ops_enabledstatic branch (or an equivalent) to maintain zero-overhead when no BPF scheduler is loaded.
The commit message's note that "subsequent patches will gradually add functionalities including safety guarantee mechanisms, nohz and cgroup support" accurately describes the rest of the patch series: patches 10-30 build on this foundation to add the watchdog, CPU hotplug handling, cgroup integration, core-sched support, and the example schedulers.
[PATCH 10/30] sched_ext: Add scx_simple and scx_example_qmap example schedulers
View on Lore: https://lore.kernel.org/all/20240618212056.2833381-11-tj@kernel.org
Commit Message
Add two simple example BPF schedulers - simple and qmap.
* simple: In terms of scheduling, it behaves identical to not having any
operation implemented at all. The two operations it implements are only to
improve visibility and exit handling. On certain homogeneous
configurations, this actually can perform pretty well.
* qmap: A fixed five level priority scheduler to demonstrate queueing PIDs
on BPF maps for scheduling. While not very practical, this is useful as a
simple example and will be used to demonstrate different features.
v7: - Compat helpers stripped out in prepartion of upstreaming as the
upstreamed patchset will be the baselinfe. Utility macros that can be
used to implement compat features are kept.
- Explicitly disable map autoattach on struct_ops to avoid trying to
attach twice while maintaining compatbility with older libbpf.
v6: - Common header files reorganized and cleaned up. Compat helpers are
added to demonstrate how schedulers can maintain backward
compatibility with older kernels while making use of newly added
features.
- simple_select_cpu() added to keep track of the number of local
dispatches. This is needed because the default ops.select_cpu()
implementation is updated to dispatch directly and won't call
ops.enqueue().
- Updated to reflect the sched_ext API changes. Switching all tasks is
the default behavior now and scx_qmap supports partial switching when
`-p` is specified.
- tools/sched_ext/Kconfig dropped. This will be included in the doc
instead.
v5: - Improve Makefile. Build artifects are now collected into a separate
dir which change be changed. Install and help targets are added and
clean actually cleans everything.
- MEMBER_VPTR() improved to improve access to structs. ARRAY_ELEM_PTR()
and RESIZEABLE_ARRAY() are added to support resizable arrays in .bss.
- Add scx_common.h which provides common utilities to user code such as
SCX_BUG[_ON]() and RESIZE_ARRAY().
- Use SCX_BUG[_ON]() to simplify error handling.
v4: - Dropped _example prefix from scheduler names.
v3: - Rename scx_example_dummy to scx_example_simple and restructure a bit
to ease later additions. Comment updates.
- Added declarations for BPF inline iterators. In the future, hopefully,
these will be consolidated into a generic BPF header so that they
don't need to be replicated here.
v2: - Updated with the generic BPF cpumask helpers.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: David Vernet <dvernet@meta.com>
Acked-by: Josh Don <joshdon@google.com>
Acked-by: Hao Luo <haoluo@google.com>
Acked-by: Barret Rhoden <brho@google.com>
---
Makefile | 8 +-
tools/Makefile | 10 +-
tools/sched_ext/.gitignore | 2 +
tools/sched_ext/Makefile | 246 ++++++++++++
.../sched_ext/include/bpf-compat/gnu/stubs.h | 11 +
tools/sched_ext/include/scx/common.bpf.h | 379 ++++++++++++++++++
tools/sched_ext/include/scx/common.h | 75 ++++
tools/sched_ext/include/scx/compat.bpf.h | 28 ++
tools/sched_ext/include/scx/compat.h | 153 +++++++
tools/sched_ext/include/scx/user_exit_info.h | 64 +++
tools/sched_ext/scx_qmap.bpf.c | 264 ++++++++++++
tools/sched_ext/scx_qmap.c | 99 +++++
tools/sched_ext/scx_simple.bpf.c | 63 +++
tools/sched_ext/scx_simple.c | 99 +++++
14 files changed, 1499 insertions(+), 2 deletions(-)
create mode 100644 tools/sched_ext/.gitignore
create mode 100644 tools/sched_ext/Makefile
create mode 100644 tools/sched_ext/include/bpf-compat/gnu/stubs.h
create mode 100644 tools/sched_ext/include/scx/common.bpf.h
create mode 100644 tools/sched_ext/include/scx/common.h
create mode 100644 tools/sched_ext/include/scx/compat.bpf.h
create mode 100644 tools/sched_ext/include/scx/compat.h
create mode 100644 tools/sched_ext/include/scx/user_exit_info.h
create mode 100644 tools/sched_ext/scx_qmap.bpf.c
create mode 100644 tools/sched_ext/scx_qmap.c
create mode 100644 tools/sched_ext/scx_simple.bpf.c
create mode 100644 tools/sched_ext/scx_simple.c
diff --git a/Makefile b/Makefile
index 7f921ae547f1..b2e57dfe8c7a 100644
--- a/Makefile
+++ b/Makefile
@@ -1355,6 +1355,12 @@ ifneq ($(wildcard $(resolve_btfids_O)),)
$(Q)$(MAKE) -sC $(srctree)/tools/bpf/resolve_btfids O=$(resolve_btfids_O) clean
endif
+tools-clean-targets := sched_ext
+PHONY += $(tools-clean-targets)
+$(tools-clean-targets):
+ $(Q)$(MAKE) -sC tools $@_clean
+tools_clean: $(tools-clean-targets)
+
# Clear a bunch of variables before executing the submake
ifeq ($(quiet),silent_)
tools_silent=s
@@ -1527,7 +1533,7 @@ PHONY += $(mrproper-dirs) mrproper
$(mrproper-dirs):
$(Q)$(MAKE) $(clean)=$(patsubst _mrproper_%,%,$@)
-mrproper: clean $(mrproper-dirs)
+mrproper: clean $(mrproper-dirs) tools_clean
$(call cmd,rmfiles)
@find . $(RCS_FIND_IGNORE) \
\( -name '*.rmeta' \) \
diff --git a/tools/Makefile b/tools/Makefile
index 276f5d0d53a4..278d24723b74 100644
--- a/tools/Makefile
+++ b/tools/Makefile
@@ -28,6 +28,7 @@ include scripts/Makefile.include
@echo ' pci - PCI tools'
@echo ' perf - Linux performance measurement and analysis tool'
@echo ' selftests - various kernel selftests'
+ @echo ' sched_ext - sched_ext example schedulers'
@echo ' bootconfig - boot config tool'
@echo ' spi - spi tools'
@echo ' tmon - thermal monitoring and tuning tool'
@@ -91,6 +92,9 @@ perf: FORCE
$(Q)mkdir -p $(PERF_O) .
$(Q)$(MAKE) --no-print-directory -C perf O=$(PERF_O) subdir=
+sched_ext: FORCE
+ $(call descend,sched_ext)
+
selftests: FORCE
$(call descend,testing/$@)
@@ -184,6 +188,9 @@ install: acpi_install counter_install cpupower_install gpio_install \
$(Q)mkdir -p $(PERF_O) .
$(Q)$(MAKE) --no-print-directory -C perf O=$(PERF_O) subdir= clean
+sched_ext_clean:
+ $(call descend,sched_ext,clean)
+
selftests_clean:
$(call descend,testing/$(@:_clean=),clean)
@@ -213,6 +220,7 @@ clean: acpi_clean counter_clean cpupower_clean hv_clean firewire_clean \
mm_clean bpf_clean iio_clean x86_energy_perf_policy_clean tmon_clean \
freefall_clean build_clean libbpf_clean libsubcmd_clean \
gpio_clean objtool_clean leds_clean wmi_clean pci_clean firmware_clean debugging_clean \
- intel-speed-select_clean tracing_clean thermal_clean thermometer_clean thermal-engine_clean
+ intel-speed-select_clean tracing_clean thermal_clean thermometer_clean thermal-engine_clean \
+ sched_ext_clean
.PHONY: FORCE
diff --git a/tools/sched_ext/.gitignore b/tools/sched_ext/.gitignore
new file mode 100644
index 000000000000..d6264fe1c8cd
--- /dev/null
+++ b/tools/sched_ext/.gitignore
@@ -0,0 +1,2 @@
+tools/
+build/
diff --git a/tools/sched_ext/Makefile b/tools/sched_ext/Makefile
new file mode 100644
index 000000000000..626782a21375
--- /dev/null
+++ b/tools/sched_ext/Makefile
@@ -0,0 +1,246 @@
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2022 Meta Platforms, Inc. and affiliates.
+include ../build/Build.include
+include ../scripts/Makefile.arch
+include ../scripts/Makefile.include
+
+all: all_targets
+
+ifneq ($(LLVM),)
+ifneq ($(filter %/,$(LLVM)),)
+LLVM_PREFIX := $(LLVM)
+else ifneq ($(filter -%,$(LLVM)),)
+LLVM_SUFFIX := $(LLVM)
+endif
+
+CLANG_TARGET_FLAGS_arm := arm-linux-gnueabi
+CLANG_TARGET_FLAGS_arm64 := aarch64-linux-gnu
+CLANG_TARGET_FLAGS_hexagon := hexagon-linux-musl
+CLANG_TARGET_FLAGS_m68k := m68k-linux-gnu
+CLANG_TARGET_FLAGS_mips := mipsel-linux-gnu
+CLANG_TARGET_FLAGS_powerpc := powerpc64le-linux-gnu
+CLANG_TARGET_FLAGS_riscv := riscv64-linux-gnu
+CLANG_TARGET_FLAGS_s390 := s390x-linux-gnu
+CLANG_TARGET_FLAGS_x86 := x86_64-linux-gnu
+CLANG_TARGET_FLAGS := $(CLANG_TARGET_FLAGS_$(ARCH))
+
+ifeq ($(CROSS_COMPILE),)
+ifeq ($(CLANG_TARGET_FLAGS),)
+$(error Specify CROSS_COMPILE or add '--target=' option to lib.mk)
+else
+CLANG_FLAGS += --target=$(CLANG_TARGET_FLAGS)
+endif # CLANG_TARGET_FLAGS
+else
+CLANG_FLAGS += --target=$(notdir $(CROSS_COMPILE:%-=%))
+endif # CROSS_COMPILE
+
+CC := $(LLVM_PREFIX)clang$(LLVM_SUFFIX) $(CLANG_FLAGS) -fintegrated-as
+else
+CC := $(CROSS_COMPILE)gcc
+endif # LLVM
+
+CURDIR := $(abspath .)
+TOOLSDIR := $(abspath ..)
+LIBDIR := $(TOOLSDIR)/lib
+BPFDIR := $(LIBDIR)/bpf
+TOOLSINCDIR := $(TOOLSDIR)/include
+BPFTOOLDIR := $(TOOLSDIR)/bpf/bpftool
+APIDIR := $(TOOLSINCDIR)/uapi
+GENDIR := $(abspath ../../include/generated)
+GENHDR := $(GENDIR)/autoconf.h
+
+ifeq ($(O),)
+OUTPUT_DIR := $(CURDIR)/build
+else
+OUTPUT_DIR := $(O)/build
+endif # O
+OBJ_DIR := $(OUTPUT_DIR)/obj
+INCLUDE_DIR := $(OUTPUT_DIR)/include
+BPFOBJ_DIR := $(OBJ_DIR)/libbpf
+SCXOBJ_DIR := $(OBJ_DIR)/sched_ext
+BINDIR := $(OUTPUT_DIR)/bin
+BPFOBJ := $(BPFOBJ_DIR)/libbpf.a
+ifneq ($(CROSS_COMPILE),)
+HOST_BUILD_DIR := $(OBJ_DIR)/host
+HOST_OUTPUT_DIR := host-tools
+HOST_INCLUDE_DIR := $(HOST_OUTPUT_DIR)/include
+else
+HOST_BUILD_DIR := $(OBJ_DIR)
+HOST_OUTPUT_DIR := $(OUTPUT_DIR)
+HOST_INCLUDE_DIR := $(INCLUDE_DIR)
+endif
+HOST_BPFOBJ := $(HOST_BUILD_DIR)/libbpf/libbpf.a
+RESOLVE_BTFIDS := $(HOST_BUILD_DIR)/resolve_btfids/resolve_btfids
+DEFAULT_BPFTOOL := $(HOST_OUTPUT_DIR)/sbin/bpftool
+
+VMLINUX_BTF_PATHS ?= $(if $(O),$(O)/vmlinux) \
+ $(if $(KBUILD_OUTPUT),$(KBUILD_OUTPUT)/vmlinux) \
+ ../../vmlinux \
+ /sys/kernel/btf/vmlinux \
+ /boot/vmlinux-$(shell uname -r)
+VMLINUX_BTF ?= $(abspath $(firstword $(wildcard $(VMLINUX_BTF_PATHS))))
+ifeq ($(VMLINUX_BTF),)
+$(error Cannot find a vmlinux for VMLINUX_BTF at any of "$(VMLINUX_BTF_PATHS)")
+endif
+
+BPFTOOL ?= $(DEFAULT_BPFTOOL)
+
+ifneq ($(wildcard $(GENHDR)),)
+ GENFLAGS := -DHAVE_GENHDR
+endif
+
+CFLAGS += -g -O2 -rdynamic -pthread -Wall -Werror $(GENFLAGS) \
+ -I$(INCLUDE_DIR) -I$(GENDIR) -I$(LIBDIR) \
+ -I$(TOOLSINCDIR) -I$(APIDIR) -I$(CURDIR)/include
+
+# Silence some warnings when compiled with clang
+ifneq ($(LLVM),)
+CFLAGS += -Wno-unused-command-line-argument
+endif
+
+LDFLAGS = -lelf -lz -lpthread
+
+IS_LITTLE_ENDIAN = $(shell $(CC) -dM -E - </dev/null | \
+ grep 'define __BYTE_ORDER__ __ORDER_LITTLE_ENDIAN__')
+
+# Get Clang's default includes on this system, as opposed to those seen by
+# '-target bpf'. This fixes "missing" files on some architectures/distros,
+# such as asm/byteorder.h, asm/socket.h, asm/sockios.h, sys/cdefs.h etc.
+#
+# Use '-idirafter': Don't interfere with include mechanics except where the
+# build would have failed anyways.
+define get_sys_includes
+$(shell $(1) -v -E - </dev/null 2>&1 \
+ | sed -n '/<...> search starts here:/,/End of search list./{ s| \(/.*\)|-idirafter \1|p }') \
+$(shell $(1) -dM -E - </dev/null | grep '__riscv_xlen ' | awk '{printf("-D__riscv_xlen=%d -D__BITS_PER_LONG=%d", $$3, $$3)}')
+endef
+
+BPF_CFLAGS = -g -D__TARGET_ARCH_$(SRCARCH) \
+ $(if $(IS_LITTLE_ENDIAN),-mlittle-endian,-mbig-endian) \
+ -I$(CURDIR)/include -I$(CURDIR)/include/bpf-compat \
+ -I$(INCLUDE_DIR) -I$(APIDIR) \
+ -I../../include \
+ $(call get_sys_includes,$(CLANG)) \
+ -Wall -Wno-compare-distinct-pointer-types \
+ -O2 -mcpu=v3
+
+# sort removes libbpf duplicates when not cross-building
+MAKE_DIRS := $(sort $(OBJ_DIR)/libbpf $(HOST_BUILD_DIR)/libbpf \
+ $(HOST_BUILD_DIR)/bpftool $(HOST_BUILD_DIR)/resolve_btfids \
+ $(INCLUDE_DIR) $(SCXOBJ_DIR) $(BINDIR))
+
+$(MAKE_DIRS):
+ $(call msg,MKDIR,,$@)
+ $(Q)mkdir -p $@
+
+$(BPFOBJ): $(wildcard $(BPFDIR)/*.[ch] $(BPFDIR)/Makefile) \
+ $(APIDIR)/linux/bpf.h \
+ | $(OBJ_DIR)/libbpf
+ $(Q)$(MAKE) $(submake_extras) -C $(BPFDIR) OUTPUT=$(OBJ_DIR)/libbpf/ \
+ EXTRA_CFLAGS='-g -O0 -fPIC' \
+ DESTDIR=$(OUTPUT_DIR) prefix= all install_headers
+
+$(DEFAULT_BPFTOOL): $(wildcard $(BPFTOOLDIR)/*.[ch] $(BPFTOOLDIR)/Makefile) \
+ $(HOST_BPFOBJ) | $(HOST_BUILD_DIR)/bpftool
+ $(Q)$(MAKE) $(submake_extras) -C $(BPFTOOLDIR) \
+ ARCH= CROSS_COMPILE= CC=$(HOSTCC) LD=$(HOSTLD) \
+ EXTRA_CFLAGS='-g -O0' \
+ OUTPUT=$(HOST_BUILD_DIR)/bpftool/ \
+ LIBBPF_OUTPUT=$(HOST_BUILD_DIR)/libbpf/ \
+ LIBBPF_DESTDIR=$(HOST_OUTPUT_DIR)/ \
+ prefix= DESTDIR=$(HOST_OUTPUT_DIR)/ install-bin
+
+$(INCLUDE_DIR)/vmlinux.h: $(VMLINUX_BTF) $(BPFTOOL) | $(INCLUDE_DIR)
+ifeq ($(VMLINUX_H),)
+ $(call msg,GEN,,$@)
+ $(Q)$(BPFTOOL) btf dump file $(VMLINUX_BTF) format c > $@
+else
+ $(call msg,CP,,$@)
+ $(Q)cp "$(VMLINUX_H)" $@
+endif
+
+$(SCXOBJ_DIR)/%.bpf.o: %.bpf.c $(INCLUDE_DIR)/vmlinux.h include/scx/*.h \
+ | $(BPFOBJ) $(SCXOBJ_DIR)
+ $(call msg,CLNG-BPF,,$(notdir $@))
+ $(Q)$(CLANG) $(BPF_CFLAGS) -target bpf -c $< -o $@
+
+$(INCLUDE_DIR)/%.bpf.skel.h: $(SCXOBJ_DIR)/%.bpf.o $(INCLUDE_DIR)/vmlinux.h $(BPFTOOL)
+ $(eval sched=$(notdir $@))
+ $(call msg,GEN-SKEL,,$(sched))
+ $(Q)$(BPFTOOL) gen object $(<:.o=.linked1.o) $<
+ $(Q)$(BPFTOOL) gen object $(<:.o=.linked2.o) $(<:.o=.linked1.o)
+ $(Q)$(BPFTOOL) gen object $(<:.o=.linked3.o) $(<:.o=.linked2.o)
+ $(Q)diff $(<:.o=.linked2.o) $(<:.o=.linked3.o)
+ $(Q)$(BPFTOOL) gen skeleton $(<:.o=.linked3.o) name $(subst .bpf.skel.h,,$(sched)) > $@
+ $(Q)$(BPFTOOL) gen subskeleton $(<:.o=.linked3.o) name $(subst .bpf.skel.h,,$(sched)) > $(@:.skel.h=.subskel.h)
+
+SCX_COMMON_DEPS := include/scx/common.h include/scx/user_exit_info.h | $(BINDIR)
+
+c-sched-targets = scx_simple scx_qmap
+
+$(addprefix $(BINDIR)/,$(c-sched-targets)): \
+ $(BINDIR)/%: \
+ $(filter-out %.bpf.c,%.c) \
+ $(INCLUDE_DIR)/%.bpf.skel.h \
+ $(SCX_COMMON_DEPS)
+ $(eval sched=$(notdir $@))
+ $(CC) $(CFLAGS) -c $(sched).c -o $(SCXOBJ_DIR)/$(sched).o
+ $(CC) -o $@ $(SCXOBJ_DIR)/$(sched).o $(HOST_BPFOBJ) $(LDFLAGS)
+
+$(c-sched-targets): %: $(BINDIR)/%
+
+install: all
+ $(Q)mkdir -p $(DESTDIR)/usr/local/bin/
+ $(Q)cp $(BINDIR)/* $(DESTDIR)/usr/local/bin/
+
+clean:
+ rm -rf $(OUTPUT_DIR) $(HOST_OUTPUT_DIR)
+ rm -f *.o *.bpf.o *.bpf.skel.h *.bpf.subskel.h
+ rm -f $(c-sched-targets)
+
+help:
+ @echo 'Building targets'
+ @echo '================'
+ @echo ''
+ @echo ' all - Compile all schedulers'
+ @echo ''
+ @echo 'Alternatively, you may compile individual schedulers:'
+ @echo ''
+ @printf ' %s\n' $(c-sched-targets)
+ @echo ''
+ @echo 'For any scheduler build target, you may specify an alternative'
+ @echo 'build output path with the O= environment variable. For example:'
+ @echo ''
+ @echo ' O=/tmp/sched_ext make all'
+ @echo ''
+ @echo 'will compile all schedulers, and emit the build artifacts to'
+ @echo '/tmp/sched_ext/build.'
+ @echo ''
+ @echo ''
+ @echo 'Installing targets'
+ @echo '=================='
+ @echo ''
+ @echo ' install - Compile and install all schedulers to /usr/bin.'
+ @echo ' You may specify the DESTDIR= environment variable'
+ @echo ' to indicate a prefix for /usr/bin. For example:'
+ @echo ''
+ @echo ' DESTDIR=/tmp/sched_ext make install'
+ @echo ''
+ @echo ' will build the schedulers in CWD/build, and'
+ @echo ' install the schedulers to /tmp/sched_ext/usr/bin.'
+ @echo ''
+ @echo ''
+ @echo 'Cleaning targets'
+ @echo '================'
+ @echo ''
+ @echo ' clean - Remove all generated files'
+
+all_targets: $(c-sched-targets)
+
+.PHONY: all all_targets $(c-sched-targets) clean help
+
+# delete failed targets
+.DELETE_ON_ERROR:
+
+# keep intermediate (.bpf.skel.h, .bpf.o, etc) targets
+.SECONDARY:
diff --git a/tools/sched_ext/include/bpf-compat/gnu/stubs.h b/tools/sched_ext/include/bpf-compat/gnu/stubs.h
new file mode 100644
index 000000000000..ad7d139ce907
--- /dev/null
+++ b/tools/sched_ext/include/bpf-compat/gnu/stubs.h
@@ -0,0 +1,11 @@
+/*
+ * Dummy gnu/stubs.h. clang can end up including /usr/include/gnu/stubs.h when
+ * compiling BPF files although its content doesn't play any role. The file in
+ * turn includes stubs-64.h or stubs-32.h depending on whether __x86_64__ is
+ * defined. When compiling a BPF source, __x86_64__ isn't set and thus
+ * stubs-32.h is selected. However, the file is not there if the system doesn't
+ * have 32bit glibc devel package installed leading to a build failure.
+ *
+ * The problem is worked around by making this file available in the include
+ * search paths before the system one when building BPF.
+ */
diff --git a/tools/sched_ext/include/scx/common.bpf.h b/tools/sched_ext/include/scx/common.bpf.h
new file mode 100644
index 000000000000..833fe1bdccf9
--- /dev/null
+++ b/tools/sched_ext/include/scx/common.bpf.h
@@ -0,0 +1,379 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2022 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2022 Tejun Heo <tj@kernel.org>
+ * Copyright (c) 2022 David Vernet <dvernet@meta.com>
+ */
+#ifndef __SCX_COMMON_BPF_H
+#define __SCX_COMMON_BPF_H
+
+#include "vmlinux.h"
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+#include <asm-generic/errno.h>
+#include "user_exit_info.h"
+
+#define PF_WQ_WORKER 0x00000020 /* I'm a workqueue worker */
+#define PF_KTHREAD 0x00200000 /* I am a kernel thread */
+#define PF_EXITING 0x00000004
+#define CLOCK_MONOTONIC 1
+
+/*
+ * Earlier versions of clang/pahole lost upper 32bits in 64bit enums which can
+ * lead to really confusing misbehaviors. Let's trigger a build failure.
+ */
+static inline void ___vmlinux_h_sanity_check___(void)
+{
+ _Static_assert(SCX_DSQ_FLAG_BUILTIN,
+ "bpftool generated vmlinux.h is missing high bits for 64bit enums, upgrade clang and pahole");
+}
+
+s32 scx_bpf_create_dsq(u64 dsq_id, s32 node) __ksym;
+s32 scx_bpf_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags, bool *is_idle) __ksym;
+void scx_bpf_dispatch(struct task_struct *p, u64 dsq_id, u64 slice, u64 enq_flags) __ksym;
+u32 scx_bpf_dispatch_nr_slots(void) __ksym;
+void scx_bpf_dispatch_cancel(void) __ksym;
+bool scx_bpf_consume(u64 dsq_id) __ksym;
+s32 scx_bpf_dsq_nr_queued(u64 dsq_id) __ksym;
+void scx_bpf_destroy_dsq(u64 dsq_id) __ksym;
+void scx_bpf_exit_bstr(s64 exit_code, char *fmt, unsigned long long *data, u32 data__sz) __ksym __weak;
+void scx_bpf_error_bstr(char *fmt, unsigned long long *data, u32 data_len) __ksym;
+u32 scx_bpf_nr_cpu_ids(void) __ksym __weak;
+const struct cpumask *scx_bpf_get_possible_cpumask(void) __ksym __weak;
+const struct cpumask *scx_bpf_get_online_cpumask(void) __ksym __weak;
+void scx_bpf_put_cpumask(const struct cpumask *cpumask) __ksym __weak;
+const struct cpumask *scx_bpf_get_idle_cpumask(void) __ksym;
+const struct cpumask *scx_bpf_get_idle_smtmask(void) __ksym;
+void scx_bpf_put_idle_cpumask(const struct cpumask *cpumask) __ksym;
+bool scx_bpf_test_and_clear_cpu_idle(s32 cpu) __ksym;
+s32 scx_bpf_pick_idle_cpu(const cpumask_t *cpus_allowed, u64 flags) __ksym;
+s32 scx_bpf_pick_any_cpu(const cpumask_t *cpus_allowed, u64 flags) __ksym;
+bool scx_bpf_task_running(const struct task_struct *p) __ksym;
+s32 scx_bpf_task_cpu(const struct task_struct *p) __ksym;
+
+static inline __attribute__((format(printf, 1, 2)))
+void ___scx_bpf_bstr_format_checker(const char *fmt, ...) {}
+
+/*
+ * Helper macro for initializing the fmt and variadic argument inputs to both
+ * bstr exit kfuncs. Callers to this function should use ___fmt and ___param to
+ * refer to the initialized list of inputs to the bstr kfunc.
+ */
+#define scx_bpf_bstr_preamble(fmt, args...) \
+ static char ___fmt[] = fmt; \
+ /* \
+ * Note that __param[] must have at least one \
+ * element to keep the verifier happy. \
+ */ \
+ unsigned long long ___param[___bpf_narg(args) ?: 1] = {}; \
+ \
+ _Pragma("GCC diagnostic push") \
+ _Pragma("GCC diagnostic ignored \"-Wint-conversion\"") \
+ ___bpf_fill(___param, args); \
+ _Pragma("GCC diagnostic pop") \
+
+/*
+ * scx_bpf_exit() wraps the scx_bpf_exit_bstr() kfunc with variadic arguments
+ * instead of an array of u64. Using this macro will cause the scheduler to
+ * exit cleanly with the specified exit code being passed to user space.
+ */
+#define scx_bpf_exit(code, fmt, args...) \
+({ \
+ scx_bpf_bstr_preamble(fmt, args) \
+ scx_bpf_exit_bstr(code, ___fmt, ___param, sizeof(___param)); \
+ ___scx_bpf_bstr_format_checker(fmt, ##args); \
+})
+
+/*
+ * scx_bpf_error() wraps the scx_bpf_error_bstr() kfunc with variadic arguments
+ * instead of an array of u64. Invoking this macro will cause the scheduler to
+ * exit in an erroneous state, with diagnostic information being passed to the
+ * user.
+ */
+#define scx_bpf_error(fmt, args...) \
+({ \
+ scx_bpf_bstr_preamble(fmt, args) \
+ scx_bpf_error_bstr(___fmt, ___param, sizeof(___param)); \
+ ___scx_bpf_bstr_format_checker(fmt, ##args); \
+})
+
+#define BPF_STRUCT_OPS(name, args...) \
+SEC("struct_ops/"#name) \
+BPF_PROG(name, ##args)
+
+#define BPF_STRUCT_OPS_SLEEPABLE(name, args...) \
+SEC("struct_ops.s/"#name) \
+BPF_PROG(name, ##args)
+
+/**
+ * RESIZABLE_ARRAY - Generates annotations for an array that may be resized
+ * @elfsec: the data section of the BPF program in which to place the array
+ * @arr: the name of the array
+ *
+ * libbpf has an API for setting map value sizes. Since data sections (i.e.
+ * bss, data, rodata) themselves are maps, a data section can be resized. If
+ * a data section has an array as its last element, the BTF info for that
+ * array will be adjusted so that length of the array is extended to meet the
+ * new length of the data section. This macro annotates an array to have an
+ * element count of one with the assumption that this array can be resized
+ * within the userspace program. It also annotates the section specifier so
+ * this array exists in a custom sub data section which can be resized
+ * independently.
+ *
+ * See RESIZE_ARRAY() for the userspace convenience macro for resizing an
+ * array declared with RESIZABLE_ARRAY().
+ */
+#define RESIZABLE_ARRAY(elfsec, arr) arr[1] SEC("."#elfsec"."#arr)
+
+/**
+ * MEMBER_VPTR - Obtain the verified pointer to a struct or array member
+ * @base: struct or array to index
+ * @member: dereferenced member (e.g. .field, [idx0][idx1], .field[idx0] ...)
+ *
+ * The verifier often gets confused by the instruction sequence the compiler
+ * generates for indexing struct fields or arrays. This macro forces the
+ * compiler to generate a code sequence which first calculates the byte offset,
+ * checks it against the struct or array size and add that byte offset to
+ * generate the pointer to the member to help the verifier.
+ *
+ * Ideally, we want to abort if the calculated offset is out-of-bounds. However,
+ * BPF currently doesn't support abort, so evaluate to %NULL instead. The caller
+ * must check for %NULL and take appropriate action to appease the verifier. To
+ * avoid confusing the verifier, it's best to check for %NULL and dereference
+ * immediately.
+ *
+ * vptr = MEMBER_VPTR(my_array, [i][j]);
+ * if (!vptr)
+ * return error;
+ * *vptr = new_value;
+ *
+ * sizeof(@base) should encompass the memory area to be accessed and thus can't
+ * be a pointer to the area. Use `MEMBER_VPTR(*ptr, .member)` instead of
+ * `MEMBER_VPTR(ptr, ->member)`.
+ */
+#define MEMBER_VPTR(base, member) (typeof((base) member) *) \
+({ \
+ u64 __base = (u64)&(base); \
+ u64 __addr = (u64)&((base) member) - __base; \
+ _Static_assert(sizeof(base) >= sizeof((base) member), \
+ "@base is smaller than @member, is @base a pointer?"); \
+ asm volatile ( \
+ "if %0 <= %[max] goto +2\n" \
+ "%0 = 0\n" \
+ "goto +1\n" \
+ "%0 += %1\n" \
+ : "+r"(__addr) \
+ : "r"(__base), \
+ [max]"i"(sizeof(base) - sizeof((base) member))); \
+ __addr; \
+})
+
+/**
+ * ARRAY_ELEM_PTR - Obtain the verified pointer to an array element
+ * @arr: array to index into
+ * @i: array index
+ * @n: number of elements in array
+ *
+ * Similar to MEMBER_VPTR() but is intended for use with arrays where the
+ * element count needs to be explicit.
+ * It can be used in cases where a global array is defined with an initial
+ * size but is intended to be be resized before loading the BPF program.
+ * Without this version of the macro, MEMBER_VPTR() will use the compile time
+ * size of the array to compute the max, which will result in rejection by
+ * the verifier.
+ */
+#define ARRAY_ELEM_PTR(arr, i, n) (typeof(arr[i]) *) \
+({ \
+ u64 __base = (u64)arr; \
+ u64 __addr = (u64)&(arr[i]) - __base; \
+ asm volatile ( \
+ "if %0 <= %[max] goto +2\n" \
+ "%0 = 0\n" \
+ "goto +1\n" \
+ "%0 += %1\n" \
+ : "+r"(__addr) \
+ : "r"(__base), \
+ [max]"r"(sizeof(arr[0]) * ((n) - 1))); \
+ __addr; \
+})
+
+
+/*
+ * BPF declarations and helpers
+ */
+
+/* list and rbtree */
+#define __contains(name, node) __attribute__((btf_decl_tag("contains:" #name ":" #node)))
+#define private(name) SEC(".data." #name) __hidden __attribute__((aligned(8)))
+
+void *bpf_obj_new_impl(__u64 local_type_id, void *meta) __ksym;
+void bpf_obj_drop_impl(void *kptr, void *meta) __ksym;
+
+#define bpf_obj_new(type) ((type *)bpf_obj_new_impl(bpf_core_type_id_local(type), NULL))
+#define bpf_obj_drop(kptr) bpf_obj_drop_impl(kptr, NULL)
+
+void bpf_list_push_front(struct bpf_list_head *head, struct bpf_list_node *node) __ksym;
+void bpf_list_push_back(struct bpf_list_head *head, struct bpf_list_node *node) __ksym;
+struct bpf_list_node *bpf_list_pop_front(struct bpf_list_head *head) __ksym;
+struct bpf_list_node *bpf_list_pop_back(struct bpf_list_head *head) __ksym;
+struct bpf_rb_node *bpf_rbtree_remove(struct bpf_rb_root *root,
+ struct bpf_rb_node *node) __ksym;
+int bpf_rbtree_add_impl(struct bpf_rb_root *root, struct bpf_rb_node *node,
+ bool (less)(struct bpf_rb_node *a, const struct bpf_rb_node *b),
+ void *meta, __u64 off) __ksym;
+#define bpf_rbtree_add(head, node, less) bpf_rbtree_add_impl(head, node, less, NULL, 0)
+
+struct bpf_rb_node *bpf_rbtree_first(struct bpf_rb_root *root) __ksym;
+
+void *bpf_refcount_acquire_impl(void *kptr, void *meta) __ksym;
+#define bpf_refcount_acquire(kptr) bpf_refcount_acquire_impl(kptr, NULL)
+
+/* task */
+struct task_struct *bpf_task_from_pid(s32 pid) __ksym;
+struct task_struct *bpf_task_acquire(struct task_struct *p) __ksym;
+void bpf_task_release(struct task_struct *p) __ksym;
+
+/* cgroup */
+struct cgroup *bpf_cgroup_ancestor(struct cgroup *cgrp, int level) __ksym;
+void bpf_cgroup_release(struct cgroup *cgrp) __ksym;
+struct cgroup *bpf_cgroup_from_id(u64 cgid) __ksym;
+
+/* css iteration */
+struct bpf_iter_css;
+struct cgroup_subsys_state;
+extern int bpf_iter_css_new(struct bpf_iter_css *it,
+ struct cgroup_subsys_state *start,
+ unsigned int flags) __weak __ksym;
+extern struct cgroup_subsys_state *
+bpf_iter_css_next(struct bpf_iter_css *it) __weak __ksym;
+extern void bpf_iter_css_destroy(struct bpf_iter_css *it) __weak __ksym;
+
+/* cpumask */
+struct bpf_cpumask *bpf_cpumask_create(void) __ksym;
+struct bpf_cpumask *bpf_cpumask_acquire(struct bpf_cpumask *cpumask) __ksym;
+void bpf_cpumask_release(struct bpf_cpumask *cpumask) __ksym;
+u32 bpf_cpumask_first(const struct cpumask *cpumask) __ksym;
+u32 bpf_cpumask_first_zero(const struct cpumask *cpumask) __ksym;
+void bpf_cpumask_set_cpu(u32 cpu, struct bpf_cpumask *cpumask) __ksym;
+void bpf_cpumask_clear_cpu(u32 cpu, struct bpf_cpumask *cpumask) __ksym;
+bool bpf_cpumask_test_cpu(u32 cpu, const struct cpumask *cpumask) __ksym;
+bool bpf_cpumask_test_and_set_cpu(u32 cpu, struct bpf_cpumask *cpumask) __ksym;
+bool bpf_cpumask_test_and_clear_cpu(u32 cpu, struct bpf_cpumask *cpumask) __ksym;
+void bpf_cpumask_setall(struct bpf_cpumask *cpumask) __ksym;
+void bpf_cpumask_clear(struct bpf_cpumask *cpumask) __ksym;
+bool bpf_cpumask_and(struct bpf_cpumask *dst, const struct cpumask *src1,
+ const struct cpumask *src2) __ksym;
+void bpf_cpumask_or(struct bpf_cpumask *dst, const struct cpumask *src1,
+ const struct cpumask *src2) __ksym;
+void bpf_cpumask_xor(struct bpf_cpumask *dst, const struct cpumask *src1,
+ const struct cpumask *src2) __ksym;
+bool bpf_cpumask_equal(const struct cpumask *src1, const struct cpumask *src2) __ksym;
+bool bpf_cpumask_intersects(const struct cpumask *src1, const struct cpumask *src2) __ksym;
+bool bpf_cpumask_subset(const struct cpumask *src1, const struct cpumask *src2) __ksym;
+bool bpf_cpumask_empty(const struct cpumask *cpumask) __ksym;
+bool bpf_cpumask_full(const struct cpumask *cpumask) __ksym;
+void bpf_cpumask_copy(struct bpf_cpumask *dst, const struct cpumask *src) __ksym;
+u32 bpf_cpumask_any_distribute(const struct cpumask *cpumask) __ksym;
+u32 bpf_cpumask_any_and_distribute(const struct cpumask *src1,
+ const struct cpumask *src2) __ksym;
+
+/* rcu */
+void bpf_rcu_read_lock(void) __ksym;
+void bpf_rcu_read_unlock(void) __ksym;
+
+
+/*
+ * Other helpers
+ */
+
+/* useful compiler attributes */
+#define likely(x) __builtin_expect(!!(x), 1)
+#define unlikely(x) __builtin_expect(!!(x), 0)
+#define __maybe_unused __attribute__((__unused__))
+
+/*
+ * READ/WRITE_ONCE() are from kernel (include/asm-generic/rwonce.h). They
+ * prevent compiler from caching, redoing or reordering reads or writes.
+ */
+typedef __u8 __attribute__((__may_alias__)) __u8_alias_t;
+typedef __u16 __attribute__((__may_alias__)) __u16_alias_t;
+typedef __u32 __attribute__((__may_alias__)) __u32_alias_t;
+typedef __u64 __attribute__((__may_alias__)) __u64_alias_t;
+
+static __always_inline void __read_once_size(const volatile void *p, void *res, int size)
+{
+ switch (size) {
+ case 1: *(__u8_alias_t *) res = *(volatile __u8_alias_t *) p; break;
+ case 2: *(__u16_alias_t *) res = *(volatile __u16_alias_t *) p; break;
+ case 4: *(__u32_alias_t *) res = *(volatile __u32_alias_t *) p; break;
+ case 8: *(__u64_alias_t *) res = *(volatile __u64_alias_t *) p; break;
+ default:
+ barrier();
+ __builtin_memcpy((void *)res, (const void *)p, size);
+ barrier();
+ }
+}
+
+static __always_inline void __write_once_size(volatile void *p, void *res, int size)
+{
+ switch (size) {
+ case 1: *(volatile __u8_alias_t *) p = *(__u8_alias_t *) res; break;
+ case 2: *(volatile __u16_alias_t *) p = *(__u16_alias_t *) res; break;
+ case 4: *(volatile __u32_alias_t *) p = *(__u32_alias_t *) res; break;
+ case 8: *(volatile __u64_alias_t *) p = *(__u64_alias_t *) res; break;
+ default:
+ barrier();
+ __builtin_memcpy((void *)p, (const void *)res, size);
+ barrier();
+ }
+}
+
+#define READ_ONCE(x) \
+({ \
+ union { typeof(x) __val; char __c[1]; } __u = \
+ { .__c = { 0 } }; \
+ __read_once_size(&(x), __u.__c, sizeof(x)); \
+ __u.__val; \
+})
+
+#define WRITE_ONCE(x, val) \
+({ \
+ union { typeof(x) __val; char __c[1]; } __u = \
+ { .__val = (val) }; \
+ __write_once_size(&(x), __u.__c, sizeof(x)); \
+ __u.__val; \
+})
+
+/*
+ * log2_u32 - Compute the base 2 logarithm of a 32-bit exponential value.
+ * @v: The value for which we're computing the base 2 logarithm.
+ */
+static inline u32 log2_u32(u32 v)
+{
+ u32 r;
+ u32 shift;
+
+ r = (v > 0xFFFF) << 4; v >>= r;
+ shift = (v > 0xFF) << 3; v >>= shift; r |= shift;
+ shift = (v > 0xF) << 2; v >>= shift; r |= shift;
+ shift = (v > 0x3) << 1; v >>= shift; r |= shift;
+ r |= (v >> 1);
+ return r;
+}
+
+/*
+ * log2_u64 - Compute the base 2 logarithm of a 64-bit exponential value.
+ * @v: The value for which we're computing the base 2 logarithm.
+ */
+static inline u32 log2_u64(u64 v)
+{
+ u32 hi = v >> 32;
+ if (hi)
+ return log2_u32(hi) + 32 + 1;
+ else
+ return log2_u32(v) + 1;
+}
+
+#include "compat.bpf.h"
+
+#endif /* __SCX_COMMON_BPF_H */
diff --git a/tools/sched_ext/include/scx/common.h b/tools/sched_ext/include/scx/common.h
new file mode 100644
index 000000000000..5b0f90152152
--- /dev/null
+++ b/tools/sched_ext/include/scx/common.h
@@ -0,0 +1,75 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2023 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2023 Tejun Heo <tj@kernel.org>
+ * Copyright (c) 2023 David Vernet <dvernet@meta.com>
+ */
+#ifndef __SCHED_EXT_COMMON_H
+#define __SCHED_EXT_COMMON_H
+
+#ifdef __KERNEL__
+#error "Should not be included by BPF programs"
+#endif
+
+#include <stdarg.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <stdint.h>
+#include <errno.h>
+
+typedef uint8_t u8;
+typedef uint16_t u16;
+typedef uint32_t u32;
+typedef uint64_t u64;
+typedef int8_t s8;
+typedef int16_t s16;
+typedef int32_t s32;
+typedef int64_t s64;
+
+#define SCX_BUG(__fmt, ...) \
+ do { \
+ fprintf(stderr, "[SCX_BUG] %s:%d", __FILE__, __LINE__); \
+ if (errno) \
+ fprintf(stderr, " (%s)\n", strerror(errno)); \
+ else \
+ fprintf(stderr, "\n"); \
+ fprintf(stderr, __fmt __VA_OPT__(,) __VA_ARGS__); \
+ fprintf(stderr, "\n"); \
+ \
+ exit(EXIT_FAILURE); \
+ } while (0)
+
+#define SCX_BUG_ON(__cond, __fmt, ...) \
+ do { \
+ if (__cond) \
+ SCX_BUG((__fmt) __VA_OPT__(,) __VA_ARGS__); \
+ } while (0)
+
+/**
+ * RESIZE_ARRAY - Convenience macro for resizing a BPF array
+ * @__skel: the skeleton containing the array
+ * @elfsec: the data section of the BPF program in which the array exists
+ * @arr: the name of the array
+ * @n: the desired array element count
+ *
+ * For BPF arrays declared with RESIZABLE_ARRAY(), this macro performs two
+ * operations. It resizes the map which corresponds to the custom data
+ * section that contains the target array. As a side effect, the BTF info for
+ * the array is adjusted so that the array length is sized to cover the new
+ * data section size. The second operation is reassigning the skeleton pointer
+ * for that custom data section so that it points to the newly memory mapped
+ * region.
+ */
+#define RESIZE_ARRAY(__skel, elfsec, arr, n) \
+ do { \
+ size_t __sz; \
+ bpf_map__set_value_size((__skel)->maps.elfsec##_##arr, \
+ sizeof((__skel)->elfsec##_##arr->arr[0]) * (n)); \
+ (__skel)->elfsec##_##arr = \
+ bpf_map__initial_value((__skel)->maps.elfsec##_##arr, &__sz); \
+ } while (0)
+
+#include "user_exit_info.h"
+#include "compat.h"
+
+#endif /* __SCHED_EXT_COMMON_H */
diff --git a/tools/sched_ext/include/scx/compat.bpf.h b/tools/sched_ext/include/scx/compat.bpf.h
new file mode 100644
index 000000000000..3d2fe1208900
--- /dev/null
+++ b/tools/sched_ext/include/scx/compat.bpf.h
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2024 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2024 Tejun Heo <tj@kernel.org>
+ * Copyright (c) 2024 David Vernet <dvernet@meta.com>
+ */
+#ifndef __SCX_COMPAT_BPF_H
+#define __SCX_COMPAT_BPF_H
+
+#define __COMPAT_ENUM_OR_ZERO(__type, __ent) \
+({ \
+ __type __ret = 0; \
+ if (bpf_core_enum_value_exists(__type, __ent)) \
+ __ret = __ent; \
+ __ret; \
+})
+
+/*
+ * Define sched_ext_ops. This may be expanded to define multiple variants for
+ * backward compatibility. See compat.h::SCX_OPS_LOAD/ATTACH().
+ */
+#define SCX_OPS_DEFINE(__name, ...) \
+ SEC(".struct_ops.link") \
+ struct sched_ext_ops __name = { \
+ __VA_ARGS__, \
+ };
+
+#endif /* __SCX_COMPAT_BPF_H */
diff --git a/tools/sched_ext/include/scx/compat.h b/tools/sched_ext/include/scx/compat.h
new file mode 100644
index 000000000000..a7fdaf8a858e
--- /dev/null
+++ b/tools/sched_ext/include/scx/compat.h
@@ -0,0 +1,153 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2024 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2024 Tejun Heo <tj@kernel.org>
+ * Copyright (c) 2024 David Vernet <dvernet@meta.com>
+ */
+#ifndef __SCX_COMPAT_H
+#define __SCX_COMPAT_H
+
+#include <bpf/btf.h>
+
+struct btf *__COMPAT_vmlinux_btf __attribute__((weak));
+
+static inline void __COMPAT_load_vmlinux_btf(void)
+{
+ if (!__COMPAT_vmlinux_btf) {
+ __COMPAT_vmlinux_btf = btf__load_vmlinux_btf();
+ SCX_BUG_ON(!__COMPAT_vmlinux_btf, "btf__load_vmlinux_btf()");
+ }
+}
+
+static inline bool __COMPAT_read_enum(const char *type, const char *name, u64 *v)
+{
+ const struct btf_type *t;
+ const char *n;
+ s32 tid;
+ int i;
+
+ __COMPAT_load_vmlinux_btf();
+
+ tid = btf__find_by_name(__COMPAT_vmlinux_btf, type);
+ if (tid < 0)
+ return false;
+
+ t = btf__type_by_id(__COMPAT_vmlinux_btf, tid);
+ SCX_BUG_ON(!t, "btf__type_by_id(%d)", tid);
+
+ if (btf_is_enum(t)) {
+ struct btf_enum *e = btf_enum(t);
+
+ for (i = 0; i < BTF_INFO_VLEN(t->info); i++) {
+ n = btf__name_by_offset(__COMPAT_vmlinux_btf, e[i].name_off);
+ SCX_BUG_ON(!n, "btf__name_by_offset()");
+ if (!strcmp(n, name)) {
+ *v = e[i].val;
+ return true;
+ }
+ }
+ } else if (btf_is_enum64(t)) {
+ struct btf_enum64 *e = btf_enum64(t);
+
+ for (i = 0; i < BTF_INFO_VLEN(t->info); i++) {
+ n = btf__name_by_offset(__COMPAT_vmlinux_btf, e[i].name_off);
+ SCX_BUG_ON(!n, "btf__name_by_offset()");
+ if (!strcmp(n, name)) {
+ *v = btf_enum64_value(&e[i]);
+ return true;
+ }
+ }
+ }
+
+ return false;
+}
+
+#define __COMPAT_ENUM_OR_ZERO(__type, __ent) \
+({ \
+ u64 __val = 0; \
+ __COMPAT_read_enum(__type, __ent, &__val); \
+ __val; \
+})
+
+static inline bool __COMPAT_has_ksym(const char *ksym)
+{
+ __COMPAT_load_vmlinux_btf();
+ return btf__find_by_name(__COMPAT_vmlinux_btf, ksym) >= 0;
+}
+
+static inline bool __COMPAT_struct_has_field(const char *type, const char *field)
+{
+ const struct btf_type *t;
+ const struct btf_member *m;
+ const char *n;
+ s32 tid;
+ int i;
+
+ __COMPAT_load_vmlinux_btf();
+ tid = btf__find_by_name_kind(__COMPAT_vmlinux_btf, type, BTF_KIND_STRUCT);
+ if (tid < 0)
+ return false;
+
+ t = btf__type_by_id(__COMPAT_vmlinux_btf, tid);
+ SCX_BUG_ON(!t, "btf__type_by_id(%d)", tid);
+
+ m = btf_members(t);
+
+ for (i = 0; i < BTF_INFO_VLEN(t->info); i++) {
+ n = btf__name_by_offset(__COMPAT_vmlinux_btf, m[i].name_off);
+ SCX_BUG_ON(!n, "btf__name_by_offset()");
+ if (!strcmp(n, field))
+ return true;
+ }
+
+ return false;
+}
+
+#define SCX_OPS_SWITCH_PARTIAL \
+ __COMPAT_ENUM_OR_ZERO("scx_ops_flags", "SCX_OPS_SWITCH_PARTIAL")
+
+/*
+ * struct sched_ext_ops can change over time. If compat.bpf.h::SCX_OPS_DEFINE()
+ * is used to define ops and compat.h::SCX_OPS_LOAD/ATTACH() are used to load
+ * and attach it, backward compatibility is automatically maintained where
+ * reasonable.
+ */
+#define SCX_OPS_OPEN(__ops_name, __scx_name) ({ \
+ struct __scx_name *__skel; \
+ \
+ __skel = __scx_name##__open(); \
+ SCX_BUG_ON(!__skel, "Could not open " #__scx_name); \
+ __skel; \
+})
+
+#define SCX_OPS_LOAD(__skel, __ops_name, __scx_name) ({ \
+ SCX_BUG_ON(__scx_name##__load((__skel)), "Failed to load skel"); \
+})
+
+/*
+ * New versions of bpftool now emit additional link placeholders for BPF maps,
+ * and set up BPF skeleton in such a way that libbpf will auto-attach BPF maps
+ * automatically, assumming libbpf is recent enough (v1.5+). Old libbpf will do
+ * nothing with those links and won't attempt to auto-attach maps.
+ *
+ * To maintain compatibility with older libbpf while avoiding trying to attach
+ * twice, disable the autoattach feature on newer libbpf.
+ */
+#if LIBBPF_MAJOR_VERSION > 1 || \
+ (LIBBPF_MAJOR_VERSION == 1 && LIBBPF_MINOR_VERSION >= 5)
+#define __SCX_OPS_DISABLE_AUTOATTACH(__skel, __ops_name) \
+ bpf_map__set_autoattach((__skel)->maps.__ops_name, false)
+#else
+#define __SCX_OPS_DISABLE_AUTOATTACH(__skel, __ops_name) do {} while (0)
+#endif
+
+#define SCX_OPS_ATTACH(__skel, __ops_name, __scx_name) ({ \
+ struct bpf_link *__link; \
+ __SCX_OPS_DISABLE_AUTOATTACH(__skel, __ops_name); \
+ SCX_BUG_ON(__scx_name##__attach((__skel)), "Failed to attach skel"); \
+ __link = bpf_map__attach_struct_ops((__skel)->maps.__ops_name); \
+ SCX_BUG_ON(!__link, "Failed to attach struct_ops"); \
+ __link; \
+})
+
+#endif /* __SCX_COMPAT_H */
diff --git a/tools/sched_ext/include/scx/user_exit_info.h b/tools/sched_ext/include/scx/user_exit_info.h
new file mode 100644
index 000000000000..8c3b7fac4d05
--- /dev/null
+++ b/tools/sched_ext/include/scx/user_exit_info.h
@@ -0,0 +1,64 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Define struct user_exit_info which is shared between BPF and userspace parts
+ * to communicate exit status and other information.
+ *
+ * Copyright (c) 2022 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2022 Tejun Heo <tj@kernel.org>
+ * Copyright (c) 2022 David Vernet <dvernet@meta.com>
+ */
+#ifndef __USER_EXIT_INFO_H
+#define __USER_EXIT_INFO_H
+
+enum uei_sizes {
+ UEI_REASON_LEN = 128,
+ UEI_MSG_LEN = 1024,
+};
+
+struct user_exit_info {
+ int kind;
+ s64 exit_code;
+ char reason[UEI_REASON_LEN];
+ char msg[UEI_MSG_LEN];
+};
+
+#ifdef __bpf__
+
+#include "vmlinux.h"
+#include <bpf/bpf_core_read.h>
+
+#define UEI_DEFINE(__name) \
+ struct user_exit_info __name SEC(".data")
+
+#define UEI_RECORD(__uei_name, __ei) ({ \
+ bpf_probe_read_kernel_str(__uei_name.reason, \
+ sizeof(__uei_name.reason), (__ei)->reason); \
+ bpf_probe_read_kernel_str(__uei_name.msg, \
+ sizeof(__uei_name.msg), (__ei)->msg); \
+ if (bpf_core_field_exists((__ei)->exit_code)) \
+ __uei_name.exit_code = (__ei)->exit_code; \
+ /* use __sync to force memory barrier */ \
+ __sync_val_compare_and_swap(&__uei_name.kind, __uei_name.kind, \
+ (__ei)->kind); \
+})
+
+#else /* !__bpf__ */
+
+#include <stdio.h>
+#include <stdbool.h>
+
+#define UEI_EXITED(__skel, __uei_name) ({ \
+ /* use __sync to force memory barrier */ \
+ __sync_val_compare_and_swap(&(__skel)->data->__uei_name.kind, -1, -1); \
+})
+
+#define UEI_REPORT(__skel, __uei_name) ({ \
+ struct user_exit_info *__uei = &(__skel)->data->__uei_name; \
+ fprintf(stderr, "EXIT: %s", __uei->reason); \
+ if (__uei->msg[0] != '\0') \
+ fprintf(stderr, " (%s)", __uei->msg); \
+ fputs("\n", stderr); \
+})
+
+#endif /* __bpf__ */
+#endif /* __USER_EXIT_INFO_H */
diff --git a/tools/sched_ext/scx_qmap.bpf.c b/tools/sched_ext/scx_qmap.bpf.c
new file mode 100644
index 000000000000..976a2693da71
--- /dev/null
+++ b/tools/sched_ext/scx_qmap.bpf.c
@@ -0,0 +1,264 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * A simple five-level FIFO queue scheduler.
+ *
+ * There are five FIFOs implemented using BPF_MAP_TYPE_QUEUE. A task gets
+ * assigned to one depending on its compound weight. Each CPU round robins
+ * through the FIFOs and dispatches more from FIFOs with higher indices - 1 from
+ * queue0, 2 from queue1, 4 from queue2 and so on.
+ *
+ * This scheduler demonstrates:
+ *
+ * - BPF-side queueing using PIDs.
+ * - Sleepable per-task storage allocation using ops.prep_enable().
+ *
+ * This scheduler is primarily for demonstration and testing of sched_ext
+ * features and unlikely to be useful for actual workloads.
+ *
+ * Copyright (c) 2022 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2022 Tejun Heo <tj@kernel.org>
+ * Copyright (c) 2022 David Vernet <dvernet@meta.com>
+ */
+#include <scx/common.bpf.h>
+
+enum consts {
+ ONE_SEC_IN_NS = 1000000000,
+ SHARED_DSQ = 0,
+};
+
+char _license[] SEC("license") = "GPL";
+
+const volatile u64 slice_ns = SCX_SLICE_DFL;
+const volatile u32 dsp_batch;
+
+u32 test_error_cnt;
+
+UEI_DEFINE(uei);
+
+struct qmap {
+ __uint(type, BPF_MAP_TYPE_QUEUE);
+ __uint(max_entries, 4096);
+ __type(value, u32);
+} queue0 SEC(".maps"),
+ queue1 SEC(".maps"),
+ queue2 SEC(".maps"),
+ queue3 SEC(".maps"),
+ queue4 SEC(".maps");
+
+struct {
+ __uint(type, BPF_MAP_TYPE_ARRAY_OF_MAPS);
+ __uint(max_entries, 5);
+ __type(key, int);
+ __array(values, struct qmap);
+} queue_arr SEC(".maps") = {
+ .values = {
+ [0] = &queue0,
+ [1] = &queue1,
+ [2] = &queue2,
+ [3] = &queue3,
+ [4] = &queue4,
+ },
+};
+
+/* Per-task scheduling context */
+struct task_ctx {
+ bool force_local; /* Dispatch directly to local_dsq */
+};
+
+struct {
+ __uint(type, BPF_MAP_TYPE_TASK_STORAGE);
+ __uint(map_flags, BPF_F_NO_PREALLOC);
+ __type(key, int);
+ __type(value, struct task_ctx);
+} task_ctx_stor SEC(".maps");
+
+struct cpu_ctx {
+ u64 dsp_idx; /* dispatch index */
+ u64 dsp_cnt; /* remaining count */
+};
+
+struct {
+ __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY);
+ __uint(max_entries, 1);
+ __type(key, u32);
+ __type(value, struct cpu_ctx);
+} cpu_ctx_stor SEC(".maps");
+
+/* Statistics */
+u64 nr_enqueued, nr_dispatched, nr_dequeued;
+
+s32 BPF_STRUCT_OPS(qmap_select_cpu, struct task_struct *p,
+ s32 prev_cpu, u64 wake_flags)
+{
+ struct task_ctx *tctx;
+ s32 cpu;
+
+ tctx = bpf_task_storage_get(&task_ctx_stor, p, 0, 0);
+ if (!tctx) {
+ scx_bpf_error("task_ctx lookup failed");
+ return -ESRCH;
+ }
+
+ if (p->nr_cpus_allowed == 1 ||
+ scx_bpf_test_and_clear_cpu_idle(prev_cpu)) {
+ tctx->force_local = true;
+ return prev_cpu;
+ }
+
+ cpu = scx_bpf_pick_idle_cpu(p->cpus_ptr, 0);
+ if (cpu >= 0)
+ return cpu;
+
+ return prev_cpu;
+}
+
+static int weight_to_idx(u32 weight)
+{
+ /* Coarsely map the compound weight to a FIFO. */
+ if (weight <= 25)
+ return 0;
+ else if (weight <= 50)
+ return 1;
+ else if (weight < 200)
+ return 2;
+ else if (weight < 400)
+ return 3;
+ else
+ return 4;
+}
+
+void BPF_STRUCT_OPS(qmap_enqueue, struct task_struct *p, u64 enq_flags)
+{
+ struct task_ctx *tctx;
+ u32 pid = p->pid;
+ int idx = weight_to_idx(p->scx.weight);
+ void *ring;
+
+ if (test_error_cnt && !--test_error_cnt)
+ scx_bpf_error("test triggering error");
+
+ tctx = bpf_task_storage_get(&task_ctx_stor, p, 0, 0);
+ if (!tctx) {
+ scx_bpf_error("task_ctx lookup failed");
+ return;
+ }
+
+ /* Is select_cpu() is telling us to enqueue locally? */
+ if (tctx->force_local) {
+ tctx->force_local = false;
+ scx_bpf_dispatch(p, SCX_DSQ_LOCAL, slice_ns, enq_flags);
+ return;
+ }
+
+ ring = bpf_map_lookup_elem(&queue_arr, &idx);
+ if (!ring) {
+ scx_bpf_error("failed to find ring %d", idx);
+ return;
+ }
+
+ /* Queue on the selected FIFO. If the FIFO overflows, punt to global. */
+ if (bpf_map_push_elem(ring, &pid, 0)) {
+ scx_bpf_dispatch(p, SHARED_DSQ, slice_ns, enq_flags);
+ return;
+ }
+
+ __sync_fetch_and_add(&nr_enqueued, 1);
+}
+
+/*
+ * The BPF queue map doesn't support removal and sched_ext can handle spurious
+ * dispatches. qmap_dequeue() is only used to collect statistics.
+ */
+void BPF_STRUCT_OPS(qmap_dequeue, struct task_struct *p, u64 deq_flags)
+{
+ __sync_fetch_and_add(&nr_dequeued, 1);
+}
+
+void BPF_STRUCT_OPS(qmap_dispatch, s32 cpu, struct task_struct *prev)
+{
+ struct task_struct *p;
+ struct cpu_ctx *cpuc;
+ u32 zero = 0, batch = dsp_batch ?: 1;
+ void *fifo;
+ s32 i, pid;
+
+ if (scx_bpf_consume(SHARED_DSQ))
+ return;
+
+ if (!(cpuc = bpf_map_lookup_elem(&cpu_ctx_stor, &zero))) {
+ scx_bpf_error("failed to look up cpu_ctx");
+ return;
+ }
+
+ for (i = 0; i < 5; i++) {
+ /* Advance the dispatch cursor and pick the fifo. */
+ if (!cpuc->dsp_cnt) {
+ cpuc->dsp_idx = (cpuc->dsp_idx + 1) % 5;
+ cpuc->dsp_cnt = 1 << cpuc->dsp_idx;
+ }
+
+ fifo = bpf_map_lookup_elem(&queue_arr, &cpuc->dsp_idx);
+ if (!fifo) {
+ scx_bpf_error("failed to find ring %llu", cpuc->dsp_idx);
+ return;
+ }
+
+ /* Dispatch or advance. */
+ bpf_repeat(BPF_MAX_LOOPS) {
+ if (bpf_map_pop_elem(fifo, &pid))
+ break;
+
+ p = bpf_task_from_pid(pid);
+ if (!p)
+ continue;
+
+ __sync_fetch_and_add(&nr_dispatched, 1);
+ scx_bpf_dispatch(p, SHARED_DSQ, slice_ns, 0);
+ bpf_task_release(p);
+ batch--;
+ cpuc->dsp_cnt--;
+ if (!batch || !scx_bpf_dispatch_nr_slots()) {
+ scx_bpf_consume(SHARED_DSQ);
+ return;
+ }
+ if (!cpuc->dsp_cnt)
+ break;
+ }
+
+ cpuc->dsp_cnt = 0;
+ }
+}
+
+s32 BPF_STRUCT_OPS(qmap_init_task, struct task_struct *p,
+ struct scx_init_task_args *args)
+{
+ /*
+ * @p is new. Let's ensure that its task_ctx is available. We can sleep
+ * in this function and the following will automatically use GFP_KERNEL.
+ */
+ if (bpf_task_storage_get(&task_ctx_stor, p, 0,
+ BPF_LOCAL_STORAGE_GET_F_CREATE))
+ return 0;
+ else
+ return -ENOMEM;
+}
+
+s32 BPF_STRUCT_OPS_SLEEPABLE(qmap_init)
+{
+ return scx_bpf_create_dsq(SHARED_DSQ, -1);
+}
+
+void BPF_STRUCT_OPS(qmap_exit, struct scx_exit_info *ei)
+{
+ UEI_RECORD(uei, ei);
+}
+
+SCX_OPS_DEFINE(qmap_ops,
+ .select_cpu = (void *)qmap_select_cpu,
+ .enqueue = (void *)qmap_enqueue,
+ .dequeue = (void *)qmap_dequeue,
+ .dispatch = (void *)qmap_dispatch,
+ .init_task = (void *)qmap_init_task,
+ .init = (void *)qmap_init,
+ .exit = (void *)qmap_exit,
+ .name = "qmap");
diff --git a/tools/sched_ext/scx_qmap.c b/tools/sched_ext/scx_qmap.c
new file mode 100644
index 000000000000..7c84ade7ecfb
--- /dev/null
+++ b/tools/sched_ext/scx_qmap.c
@@ -0,0 +1,99 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2022 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2022 Tejun Heo <tj@kernel.org>
+ * Copyright (c) 2022 David Vernet <dvernet@meta.com>
+ */
+#include <stdio.h>
+#include <stdlib.h>
+#include <unistd.h>
+#include <inttypes.h>
+#include <signal.h>
+#include <libgen.h>
+#include <bpf/bpf.h>
+#include <scx/common.h>
+#include "scx_qmap.bpf.skel.h"
+
+const char help_fmt[] =
+"A simple five-level FIFO queue sched_ext scheduler.\n"
+"\n"
+"See the top-level comment in .bpf.c for more details.\n"
+"\n"
+"Usage: %s [-s SLICE_US] [-e COUNT] [-b COUNT] [-p] [-v]\n"
+"\n"
+" -s SLICE_US Override slice duration\n"
+" -e COUNT Trigger scx_bpf_error() after COUNT enqueues\n"
+" -b COUNT Dispatch upto COUNT tasks together\n"
+" -p Switch only tasks on SCHED_EXT policy intead of all\n"
+" -v Print libbpf debug messages\n"
+" -h Display this help and exit\n";
+
+static bool verbose;
+static volatile int exit_req;
+
+static int libbpf_print_fn(enum libbpf_print_level level, const char *format, va_list args)
+{
+ if (level == LIBBPF_DEBUG && !verbose)
+ return 0;
+ return vfprintf(stderr, format, args);
+}
+
+static void sigint_handler(int dummy)
+{
+ exit_req = 1;
+}
+
+int main(int argc, char **argv)
+{
+ struct scx_qmap *skel;
+ struct bpf_link *link;
+ int opt;
+
+ libbpf_set_print(libbpf_print_fn);
+ signal(SIGINT, sigint_handler);
+ signal(SIGTERM, sigint_handler);
+
+ skel = SCX_OPS_OPEN(qmap_ops, scx_qmap);
+
+ while ((opt = getopt(argc, argv, "s:e:b:pvh")) != -1) {
+ switch (opt) {
+ case 's':
+ skel->rodata->slice_ns = strtoull(optarg, NULL, 0) * 1000;
+ break;
+ case 'e':
+ skel->bss->test_error_cnt = strtoul(optarg, NULL, 0);
+ break;
+ case 'b':
+ skel->rodata->dsp_batch = strtoul(optarg, NULL, 0);
+ break;
+ case 'p':
+ skel->struct_ops.qmap_ops->flags |= SCX_OPS_SWITCH_PARTIAL;
+ break;
+ case 'v':
+ verbose = true;
+ break;
+ default:
+ fprintf(stderr, help_fmt, basename(argv[0]));
+ return opt != 'h';
+ }
+ }
+
+ SCX_OPS_LOAD(skel, qmap_ops, scx_qmap);
+ link = SCX_OPS_ATTACH(skel, qmap_ops, scx_qmap);
+
+ while (!exit_req && !UEI_EXITED(skel, uei)) {
+ long nr_enqueued = skel->bss->nr_enqueued;
+ long nr_dispatched = skel->bss->nr_dispatched;
+
+ printf("stats : enq=%lu dsp=%lu delta=%ld deq=%"PRIu64"\n",
+ nr_enqueued, nr_dispatched, nr_enqueued - nr_dispatched,
+ skel->bss->nr_dequeued);
+ fflush(stdout);
+ sleep(1);
+ }
+
+ bpf_link__destroy(link);
+ UEI_REPORT(skel, uei);
+ scx_qmap__destroy(skel);
+ return 0;
+}
diff --git a/tools/sched_ext/scx_simple.bpf.c b/tools/sched_ext/scx_simple.bpf.c
new file mode 100644
index 000000000000..6bb13a3c801b
--- /dev/null
+++ b/tools/sched_ext/scx_simple.bpf.c
@@ -0,0 +1,63 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * A simple scheduler.
+ *
+ * A simple global FIFO scheduler. It also demonstrates the following niceties.
+ *
+ * - Statistics tracking how many tasks are queued to local and global dsq's.
+ * - Termination notification for userspace.
+ *
+ * Copyright (c) 2022 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2022 Tejun Heo <tj@kernel.org>
+ * Copyright (c) 2022 David Vernet <dvernet@meta.com>
+ */
+#include <scx/common.bpf.h>
+
+char _license[] SEC("license") = "GPL";
+
+UEI_DEFINE(uei);
+
+struct {
+ __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY);
+ __uint(key_size, sizeof(u32));
+ __uint(value_size, sizeof(u64));
+ __uint(max_entries, 2); /* [local, global] */
+} stats SEC(".maps");
+
+static void stat_inc(u32 idx)
+{
+ u64 *cnt_p = bpf_map_lookup_elem(&stats, &idx);
+ if (cnt_p)
+ (*cnt_p)++;
+}
+
+s32 BPF_STRUCT_OPS(simple_select_cpu, struct task_struct *p, s32 prev_cpu, u64 wake_flags)
+{
+ bool is_idle = false;
+ s32 cpu;
+
+ cpu = scx_bpf_select_cpu_dfl(p, prev_cpu, wake_flags, &is_idle);
+ if (is_idle) {
+ stat_inc(0); /* count local queueing */
+ scx_bpf_dispatch(p, SCX_DSQ_LOCAL, SCX_SLICE_DFL, 0);
+ }
+
+ return cpu;
+}
+
+void BPF_STRUCT_OPS(simple_enqueue, struct task_struct *p, u64 enq_flags)
+{
+ stat_inc(1); /* count global queueing */
+ scx_bpf_dispatch(p, SCX_DSQ_GLOBAL, SCX_SLICE_DFL, enq_flags);
+}
+
+void BPF_STRUCT_OPS(simple_exit, struct scx_exit_info *ei)
+{
+ UEI_RECORD(uei, ei);
+}
+
+SCX_OPS_DEFINE(simple_ops,
+ .select_cpu = (void *)simple_select_cpu,
+ .enqueue = (void *)simple_enqueue,
+ .exit = (void *)simple_exit,
+ .name = "simple");
diff --git a/tools/sched_ext/scx_simple.c b/tools/sched_ext/scx_simple.c
new file mode 100644
index 000000000000..789ac62fea8e
--- /dev/null
+++ b/tools/sched_ext/scx_simple.c
@@ -0,0 +1,99 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2022 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2022 Tejun Heo <tj@kernel.org>
+ * Copyright (c) 2022 David Vernet <dvernet@meta.com>
+ */
+#include <stdio.h>
+#include <unistd.h>
+#include <signal.h>
+#include <libgen.h>
+#include <bpf/bpf.h>
+#include <scx/common.h>
+#include "scx_simple.bpf.skel.h"
+
+const char help_fmt[] =
+"A simple sched_ext scheduler.\n"
+"\n"
+"See the top-level comment in .bpf.c for more details.\n"
+"\n"
+"Usage: %s [-v]\n"
+"\n"
+" -v Print libbpf debug messages\n"
+" -h Display this help and exit\n";
+
+static bool verbose;
+static volatile int exit_req;
+
+static int libbpf_print_fn(enum libbpf_print_level level, const char *format, va_list args)
+{
+ if (level == LIBBPF_DEBUG && !verbose)
+ return 0;
+ return vfprintf(stderr, format, args);
+}
+
+static void sigint_handler(int simple)
+{
+ exit_req = 1;
+}
+
+static void read_stats(struct scx_simple *skel, __u64 *stats)
+{
+ int nr_cpus = libbpf_num_possible_cpus();
+ __u64 cnts[2][nr_cpus];
+ __u32 idx;
+
+ memset(stats, 0, sizeof(stats[0]) * 2);
+
+ for (idx = 0; idx < 2; idx++) {
+ int ret, cpu;
+
+ ret = bpf_map_lookup_elem(bpf_map__fd(skel->maps.stats),
+ &idx, cnts[idx]);
+ if (ret < 0)
+ continue;
+ for (cpu = 0; cpu < nr_cpus; cpu++)
+ stats[idx] += cnts[idx][cpu];
+ }
+}
+
+int main(int argc, char **argv)
+{
+ struct scx_simple *skel;
+ struct bpf_link *link;
+ __u32 opt;
+
+ libbpf_set_print(libbpf_print_fn);
+ signal(SIGINT, sigint_handler);
+ signal(SIGTERM, sigint_handler);
+
+ skel = SCX_OPS_OPEN(simple_ops, scx_simple);
+
+ while ((opt = getopt(argc, argv, "vh")) != -1) {
+ switch (opt) {
+ case 'v':
+ verbose = true;
+ break;
+ default:
+ fprintf(stderr, help_fmt, basename(argv[0]));
+ return opt != 'h';
+ }
+ }
+
+ SCX_OPS_LOAD(skel, simple_ops, scx_simple);
+ link = SCX_OPS_ATTACH(skel, simple_ops, scx_simple);
+
+ while (!exit_req && !UEI_EXITED(skel, uei)) {
+ __u64 stats[2];
+
+ read_stats(skel, stats);
+ printf("local=%llu global=%llu\n", stats[0], stats[1]);
+ fflush(stdout);
+ sleep(1);
+ }
+
+ bpf_link__destroy(link);
+ UEI_REPORT(skel, uei);
+ scx_simple__destroy(skel);
+ return 0;
+}
--
2.45.2
Diff
---
Makefile | 8 +-
tools/Makefile | 10 +-
tools/sched_ext/.gitignore | 2 +
tools/sched_ext/Makefile | 246 ++++++++++++
.../sched_ext/include/bpf-compat/gnu/stubs.h | 11 +
tools/sched_ext/include/scx/common.bpf.h | 379 ++++++++++++++++++
tools/sched_ext/include/scx/common.h | 75 ++++
tools/sched_ext/include/scx/compat.bpf.h | 28 ++
tools/sched_ext/include/scx/compat.h | 153 +++++++
tools/sched_ext/include/scx/user_exit_info.h | 64 +++
tools/sched_ext/scx_qmap.bpf.c | 264 ++++++++++++
tools/sched_ext/scx_qmap.c | 99 +++++
tools/sched_ext/scx_simple.bpf.c | 63 +++
tools/sched_ext/scx_simple.c | 99 +++++
14 files changed, 1499 insertions(+), 2 deletions(-)
create mode 100644 tools/sched_ext/.gitignore
create mode 100644 tools/sched_ext/Makefile
create mode 100644 tools/sched_ext/include/bpf-compat/gnu/stubs.h
create mode 100644 tools/sched_ext/include/scx/common.bpf.h
create mode 100644 tools/sched_ext/include/scx/common.h
create mode 100644 tools/sched_ext/include/scx/compat.bpf.h
create mode 100644 tools/sched_ext/include/scx/compat.h
create mode 100644 tools/sched_ext/include/scx/user_exit_info.h
create mode 100644 tools/sched_ext/scx_qmap.bpf.c
create mode 100644 tools/sched_ext/scx_qmap.c
create mode 100644 tools/sched_ext/scx_simple.bpf.c
create mode 100644 tools/sched_ext/scx_simple.c
diff --git a/Makefile b/Makefile
index 7f921ae547f1..b2e57dfe8c7a 100644
--- a/Makefile
+++ b/Makefile
@@ -1355,6 +1355,12 @@ ifneq ($(wildcard $(resolve_btfids_O)),)
$(Q)$(MAKE) -sC $(srctree)/tools/bpf/resolve_btfids O=$(resolve_btfids_O) clean
endif
+tools-clean-targets := sched_ext
+PHONY += $(tools-clean-targets)
+$(tools-clean-targets):
+ $(Q)$(MAKE) -sC tools $@_clean
+tools_clean: $(tools-clean-targets)
+
# Clear a bunch of variables before executing the submake
ifeq ($(quiet),silent_)
tools_silent=s
@@ -1527,7 +1533,7 @@ PHONY += $(mrproper-dirs) mrproper
$(mrproper-dirs):
$(Q)$(MAKE) $(clean)=$(patsubst _mrproper_%,%,$@)
-mrproper: clean $(mrproper-dirs)
+mrproper: clean $(mrproper-dirs) tools_clean
$(call cmd,rmfiles)
@find . $(RCS_FIND_IGNORE) \
\( -name '*.rmeta' \) \
diff --git a/tools/Makefile b/tools/Makefile
index 276f5d0d53a4..278d24723b74 100644
--- a/tools/Makefile
+++ b/tools/Makefile
@@ -28,6 +28,7 @@ include scripts/Makefile.include
@echo ' pci - PCI tools'
@echo ' perf - Linux performance measurement and analysis tool'
@echo ' selftests - various kernel selftests'
+ @echo ' sched_ext - sched_ext example schedulers'
@echo ' bootconfig - boot config tool'
@echo ' spi - spi tools'
@echo ' tmon - thermal monitoring and tuning tool'
@@ -91,6 +92,9 @@ perf: FORCE
$(Q)mkdir -p $(PERF_O) .
$(Q)$(MAKE) --no-print-directory -C perf O=$(PERF_O) subdir=
+sched_ext: FORCE
+ $(call descend,sched_ext)
+
selftests: FORCE
$(call descend,testing/$@)
@@ -184,6 +188,9 @@ install: acpi_install counter_install cpupower_install gpio_install \
$(Q)mkdir -p $(PERF_O) .
$(Q)$(MAKE) --no-print-directory -C perf O=$(PERF_O) subdir= clean
+sched_ext_clean:
+ $(call descend,sched_ext,clean)
+
selftests_clean:
$(call descend,testing/$(@:_clean=),clean)
@@ -213,6 +220,7 @@ clean: acpi_clean counter_clean cpupower_clean hv_clean firewire_clean \
mm_clean bpf_clean iio_clean x86_energy_perf_policy_clean tmon_clean \
freefall_clean build_clean libbpf_clean libsubcmd_clean \
gpio_clean objtool_clean leds_clean wmi_clean pci_clean firmware_clean debugging_clean \
- intel-speed-select_clean tracing_clean thermal_clean thermometer_clean thermal-engine_clean
+ intel-speed-select_clean tracing_clean thermal_clean thermometer_clean thermal-engine_clean \
+ sched_ext_clean
.PHONY: FORCE
diff --git a/tools/sched_ext/.gitignore b/tools/sched_ext/.gitignore
new file mode 100644
index 000000000000..d6264fe1c8cd
--- /dev/null
+++ b/tools/sched_ext/.gitignore
@@ -0,0 +1,2 @@
+tools/
+build/
diff --git a/tools/sched_ext/Makefile b/tools/sched_ext/Makefile
new file mode 100644
index 000000000000..626782a21375
--- /dev/null
+++ b/tools/sched_ext/Makefile
@@ -0,0 +1,246 @@
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2022 Meta Platforms, Inc. and affiliates.
+include ../build/Build.include
+include ../scripts/Makefile.arch
+include ../scripts/Makefile.include
+
+all: all_targets
+
+ifneq ($(LLVM),)
+ifneq ($(filter %/,$(LLVM)),)
+LLVM_PREFIX := $(LLVM)
+else ifneq ($(filter -%,$(LLVM)),)
+LLVM_SUFFIX := $(LLVM)
+endif
+
+CLANG_TARGET_FLAGS_arm := arm-linux-gnueabi
+CLANG_TARGET_FLAGS_arm64 := aarch64-linux-gnu
+CLANG_TARGET_FLAGS_hexagon := hexagon-linux-musl
+CLANG_TARGET_FLAGS_m68k := m68k-linux-gnu
+CLANG_TARGET_FLAGS_mips := mipsel-linux-gnu
+CLANG_TARGET_FLAGS_powerpc := powerpc64le-linux-gnu
+CLANG_TARGET_FLAGS_riscv := riscv64-linux-gnu
+CLANG_TARGET_FLAGS_s390 := s390x-linux-gnu
+CLANG_TARGET_FLAGS_x86 := x86_64-linux-gnu
+CLANG_TARGET_FLAGS := $(CLANG_TARGET_FLAGS_$(ARCH))
+
+ifeq ($(CROSS_COMPILE),)
+ifeq ($(CLANG_TARGET_FLAGS),)
+$(error Specify CROSS_COMPILE or add '--target=' option to lib.mk)
+else
+CLANG_FLAGS += --target=$(CLANG_TARGET_FLAGS)
+endif # CLANG_TARGET_FLAGS
+else
+CLANG_FLAGS += --target=$(notdir $(CROSS_COMPILE:%-=%))
+endif # CROSS_COMPILE
+
+CC := $(LLVM_PREFIX)clang$(LLVM_SUFFIX) $(CLANG_FLAGS) -fintegrated-as
+else
+CC := $(CROSS_COMPILE)gcc
+endif # LLVM
+
+CURDIR := $(abspath .)
+TOOLSDIR := $(abspath ..)
+LIBDIR := $(TOOLSDIR)/lib
+BPFDIR := $(LIBDIR)/bpf
+TOOLSINCDIR := $(TOOLSDIR)/include
+BPFTOOLDIR := $(TOOLSDIR)/bpf/bpftool
+APIDIR := $(TOOLSINCDIR)/uapi
+GENDIR := $(abspath ../../include/generated)
+GENHDR := $(GENDIR)/autoconf.h
+
+ifeq ($(O),)
+OUTPUT_DIR := $(CURDIR)/build
+else
+OUTPUT_DIR := $(O)/build
+endif # O
+OBJ_DIR := $(OUTPUT_DIR)/obj
+INCLUDE_DIR := $(OUTPUT_DIR)/include
+BPFOBJ_DIR := $(OBJ_DIR)/libbpf
+SCXOBJ_DIR := $(OBJ_DIR)/sched_ext
+BINDIR := $(OUTPUT_DIR)/bin
+BPFOBJ := $(BPFOBJ_DIR)/libbpf.a
+ifneq ($(CROSS_COMPILE),)
+HOST_BUILD_DIR := $(OBJ_DIR)/host
+HOST_OUTPUT_DIR := host-tools
+HOST_INCLUDE_DIR := $(HOST_OUTPUT_DIR)/include
+else
+HOST_BUILD_DIR := $(OBJ_DIR)
+HOST_OUTPUT_DIR := $(OUTPUT_DIR)
+HOST_INCLUDE_DIR := $(INCLUDE_DIR)
+endif
+HOST_BPFOBJ := $(HOST_BUILD_DIR)/libbpf/libbpf.a
+RESOLVE_BTFIDS := $(HOST_BUILD_DIR)/resolve_btfids/resolve_btfids
+DEFAULT_BPFTOOL := $(HOST_OUTPUT_DIR)/sbin/bpftool
+
+VMLINUX_BTF_PATHS ?= $(if $(O),$(O)/vmlinux) \
+ $(if $(KBUILD_OUTPUT),$(KBUILD_OUTPUT)/vmlinux) \
+ ../../vmlinux \
+ /sys/kernel/btf/vmlinux \
+ /boot/vmlinux-$(shell uname -r)
+VMLINUX_BTF ?= $(abspath $(firstword $(wildcard $(VMLINUX_BTF_PATHS))))
+ifeq ($(VMLINUX_BTF),)
+$(error Cannot find a vmlinux for VMLINUX_BTF at any of "$(VMLINUX_BTF_PATHS)")
+endif
+
+BPFTOOL ?= $(DEFAULT_BPFTOOL)
+
+ifneq ($(wildcard $(GENHDR)),)
+ GENFLAGS := -DHAVE_GENHDR
+endif
+
+CFLAGS += -g -O2 -rdynamic -pthread -Wall -Werror $(GENFLAGS) \
+ -I$(INCLUDE_DIR) -I$(GENDIR) -I$(LIBDIR) \
+ -I$(TOOLSINCDIR) -I$(APIDIR) -I$(CURDIR)/include
+
+# Silence some warnings when compiled with clang
+ifneq ($(LLVM),)
+CFLAGS += -Wno-unused-command-line-argument
+endif
+
+LDFLAGS = -lelf -lz -lpthread
+
+IS_LITTLE_ENDIAN = $(shell $(CC) -dM -E - </dev/null | \
+ grep 'define __BYTE_ORDER__ __ORDER_LITTLE_ENDIAN__')
+
+# Get Clang's default includes on this system, as opposed to those seen by
+# '-target bpf'. This fixes "missing" files on some architectures/distros,
+# such as asm/byteorder.h, asm/socket.h, asm/sockios.h, sys/cdefs.h etc.
+#
+# Use '-idirafter': Don't interfere with include mechanics except where the
+# build would have failed anyways.
+define get_sys_includes
+$(shell $(1) -v -E - </dev/null 2>&1 \
+ | sed -n '/<...> search starts here:/,/End of search list./{ s| \(/.*\)|-idirafter \1|p }') \
+$(shell $(1) -dM -E - </dev/null | grep '__riscv_xlen ' | awk '{printf("-D__riscv_xlen=%d -D__BITS_PER_LONG=%d", $$3, $$3)}')
+endef
+
+BPF_CFLAGS = -g -D__TARGET_ARCH_$(SRCARCH) \
+ $(if $(IS_LITTLE_ENDIAN),-mlittle-endian,-mbig-endian) \
+ -I$(CURDIR)/include -I$(CURDIR)/include/bpf-compat \
+ -I$(INCLUDE_DIR) -I$(APIDIR) \
+ -I../../include \
+ $(call get_sys_includes,$(CLANG)) \
+ -Wall -Wno-compare-distinct-pointer-types \
+ -O2 -mcpu=v3
+
+# sort removes libbpf duplicates when not cross-building
+MAKE_DIRS := $(sort $(OBJ_DIR)/libbpf $(HOST_BUILD_DIR)/libbpf \
+ $(HOST_BUILD_DIR)/bpftool $(HOST_BUILD_DIR)/resolve_btfids \
+ $(INCLUDE_DIR) $(SCXOBJ_DIR) $(BINDIR))
+
+$(MAKE_DIRS):
+ $(call msg,MKDIR,,$@)
+ $(Q)mkdir -p $@
+
+$(BPFOBJ): $(wildcard $(BPFDIR)/*.[ch] $(BPFDIR)/Makefile) \
+ $(APIDIR)/linux/bpf.h \
+ | $(OBJ_DIR)/libbpf
+ $(Q)$(MAKE) $(submake_extras) -C $(BPFDIR) OUTPUT=$(OBJ_DIR)/libbpf/ \
+ EXTRA_CFLAGS='-g -O0 -fPIC' \
+ DESTDIR=$(OUTPUT_DIR) prefix= all install_headers
+
+$(DEFAULT_BPFTOOL): $(wildcard $(BPFTOOLDIR)/*.[ch] $(BPFTOOLDIR)/Makefile) \
+ $(HOST_BPFOBJ) | $(HOST_BUILD_DIR)/bpftool
+ $(Q)$(MAKE) $(submake_extras) -C $(BPFTOOLDIR) \
+ ARCH= CROSS_COMPILE= CC=$(HOSTCC) LD=$(HOSTLD) \
+ EXTRA_CFLAGS='-g -O0' \
+ OUTPUT=$(HOST_BUILD_DIR)/bpftool/ \
+ LIBBPF_OUTPUT=$(HOST_BUILD_DIR)/libbpf/ \
+ LIBBPF_DESTDIR=$(HOST_OUTPUT_DIR)/ \
+ prefix= DESTDIR=$(HOST_OUTPUT_DIR)/ install-bin
+
+$(INCLUDE_DIR)/vmlinux.h: $(VMLINUX_BTF) $(BPFTOOL) | $(INCLUDE_DIR)
+ifeq ($(VMLINUX_H),)
+ $(call msg,GEN,,$@)
+ $(Q)$(BPFTOOL) btf dump file $(VMLINUX_BTF) format c > $@
+else
+ $(call msg,CP,,$@)
+ $(Q)cp "$(VMLINUX_H)" $@
+endif
+
+$(SCXOBJ_DIR)/%.bpf.o: %.bpf.c $(INCLUDE_DIR)/vmlinux.h include/scx/*.h \
+ | $(BPFOBJ) $(SCXOBJ_DIR)
+ $(call msg,CLNG-BPF,,$(notdir $@))
+ $(Q)$(CLANG) $(BPF_CFLAGS) -target bpf -c $< -o $@
+
+$(INCLUDE_DIR)/%.bpf.skel.h: $(SCXOBJ_DIR)/%.bpf.o $(INCLUDE_DIR)/vmlinux.h $(BPFTOOL)
+ $(eval sched=$(notdir $@))
+ $(call msg,GEN-SKEL,,$(sched))
+ $(Q)$(BPFTOOL) gen object $(<:.o=.linked1.o) $<
+ $(Q)$(BPFTOOL) gen object $(<:.o=.linked2.o) $(<:.o=.linked1.o)
+ $(Q)$(BPFTOOL) gen object $(<:.o=.linked3.o) $(<:.o=.linked2.o)
+ $(Q)diff $(<:.o=.linked2.o) $(<:.o=.linked3.o)
+ $(Q)$(BPFTOOL) gen skeleton $(<:.o=.linked3.o) name $(subst .bpf.skel.h,,$(sched)) > $@
+ $(Q)$(BPFTOOL) gen subskeleton $(<:.o=.linked3.o) name $(subst .bpf.skel.h,,$(sched)) > $(@:.skel.h=.subskel.h)
+
+SCX_COMMON_DEPS := include/scx/common.h include/scx/user_exit_info.h | $(BINDIR)
+
+c-sched-targets = scx_simple scx_qmap
+
+$(addprefix $(BINDIR)/,$(c-sched-targets)): \
+ $(BINDIR)/%: \
+ $(filter-out %.bpf.c,%.c) \
+ $(INCLUDE_DIR)/%.bpf.skel.h \
+ $(SCX_COMMON_DEPS)
+ $(eval sched=$(notdir $@))
+ $(CC) $(CFLAGS) -c $(sched).c -o $(SCXOBJ_DIR)/$(sched).o
+ $(CC) -o $@ $(SCXOBJ_DIR)/$(sched).o $(HOST_BPFOBJ) $(LDFLAGS)
+
+$(c-sched-targets): %: $(BINDIR)/%
+
+install: all
+ $(Q)mkdir -p $(DESTDIR)/usr/local/bin/
+ $(Q)cp $(BINDIR)/* $(DESTDIR)/usr/local/bin/
+
+clean:
+ rm -rf $(OUTPUT_DIR) $(HOST_OUTPUT_DIR)
+ rm -f *.o *.bpf.o *.bpf.skel.h *.bpf.subskel.h
+ rm -f $(c-sched-targets)
+
+help:
+ @echo 'Building targets'
+ @echo '================'
+ @echo ''
+ @echo ' all - Compile all schedulers'
+ @echo ''
+ @echo 'Alternatively, you may compile individual schedulers:'
+ @echo ''
+ @printf ' %s\n' $(c-sched-targets)
+ @echo ''
+ @echo 'For any scheduler build target, you may specify an alternative'
+ @echo 'build output path with the O= environment variable. For example:'
+ @echo ''
+ @echo ' O=/tmp/sched_ext make all'
+ @echo ''
+ @echo 'will compile all schedulers, and emit the build artifacts to'
+ @echo '/tmp/sched_ext/build.'
+ @echo ''
+ @echo ''
+ @echo 'Installing targets'
+ @echo '=================='
+ @echo ''
+ @echo ' install - Compile and install all schedulers to /usr/bin.'
+ @echo ' You may specify the DESTDIR= environment variable'
+ @echo ' to indicate a prefix for /usr/bin. For example:'
+ @echo ''
+ @echo ' DESTDIR=/tmp/sched_ext make install'
+ @echo ''
+ @echo ' will build the schedulers in CWD/build, and'
+ @echo ' install the schedulers to /tmp/sched_ext/usr/bin.'
+ @echo ''
+ @echo ''
+ @echo 'Cleaning targets'
+ @echo '================'
+ @echo ''
+ @echo ' clean - Remove all generated files'
+
+all_targets: $(c-sched-targets)
+
+.PHONY: all all_targets $(c-sched-targets) clean help
+
+# delete failed targets
+.DELETE_ON_ERROR:
+
+# keep intermediate (.bpf.skel.h, .bpf.o, etc) targets
+.SECONDARY:
diff --git a/tools/sched_ext/include/bpf-compat/gnu/stubs.h b/tools/sched_ext/include/bpf-compat/gnu/stubs.h
new file mode 100644
index 000000000000..ad7d139ce907
--- /dev/null
+++ b/tools/sched_ext/include/bpf-compat/gnu/stubs.h
@@ -0,0 +1,11 @@
+/*
+ * Dummy gnu/stubs.h. clang can end up including /usr/include/gnu/stubs.h when
+ * compiling BPF files although its content doesn't play any role. The file in
+ * turn includes stubs-64.h or stubs-32.h depending on whether __x86_64__ is
+ * defined. When compiling a BPF source, __x86_64__ isn't set and thus
+ * stubs-32.h is selected. However, the file is not there if the system doesn't
+ * have 32bit glibc devel package installed leading to a build failure.
+ *
+ * The problem is worked around by making this file available in the include
+ * search paths before the system one when building BPF.
+ */
diff --git a/tools/sched_ext/include/scx/common.bpf.h b/tools/sched_ext/include/scx/common.bpf.h
new file mode 100644
index 000000000000..833fe1bdccf9
--- /dev/null
+++ b/tools/sched_ext/include/scx/common.bpf.h
@@ -0,0 +1,379 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2022 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2022 Tejun Heo <tj@kernel.org>
+ * Copyright (c) 2022 David Vernet <dvernet@meta.com>
+ */
+#ifndef __SCX_COMMON_BPF_H
+#define __SCX_COMMON_BPF_H
+
+#include "vmlinux.h"
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+#include <asm-generic/errno.h>
+#include "user_exit_info.h"
+
+#define PF_WQ_WORKER 0x00000020 /* I'm a workqueue worker */
+#define PF_KTHREAD 0x00200000 /* I am a kernel thread */
+#define PF_EXITING 0x00000004
+#define CLOCK_MONOTONIC 1
+
+/*
+ * Earlier versions of clang/pahole lost upper 32bits in 64bit enums which can
+ * lead to really confusing misbehaviors. Let's trigger a build failure.
+ */
+static inline void ___vmlinux_h_sanity_check___(void)
+{
+ _Static_assert(SCX_DSQ_FLAG_BUILTIN,
+ "bpftool generated vmlinux.h is missing high bits for 64bit enums, upgrade clang and pahole");
+}
+
+s32 scx_bpf_create_dsq(u64 dsq_id, s32 node) __ksym;
+s32 scx_bpf_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags, bool *is_idle) __ksym;
+void scx_bpf_dispatch(struct task_struct *p, u64 dsq_id, u64 slice, u64 enq_flags) __ksym;
+u32 scx_bpf_dispatch_nr_slots(void) __ksym;
+void scx_bpf_dispatch_cancel(void) __ksym;
+bool scx_bpf_consume(u64 dsq_id) __ksym;
+s32 scx_bpf_dsq_nr_queued(u64 dsq_id) __ksym;
+void scx_bpf_destroy_dsq(u64 dsq_id) __ksym;
+void scx_bpf_exit_bstr(s64 exit_code, char *fmt, unsigned long long *data, u32 data__sz) __ksym __weak;
+void scx_bpf_error_bstr(char *fmt, unsigned long long *data, u32 data_len) __ksym;
+u32 scx_bpf_nr_cpu_ids(void) __ksym __weak;
+const struct cpumask *scx_bpf_get_possible_cpumask(void) __ksym __weak;
+const struct cpumask *scx_bpf_get_online_cpumask(void) __ksym __weak;
+void scx_bpf_put_cpumask(const struct cpumask *cpumask) __ksym __weak;
+const struct cpumask *scx_bpf_get_idle_cpumask(void) __ksym;
+const struct cpumask *scx_bpf_get_idle_smtmask(void) __ksym;
+void scx_bpf_put_idle_cpumask(const struct cpumask *cpumask) __ksym;
+bool scx_bpf_test_and_clear_cpu_idle(s32 cpu) __ksym;
+s32 scx_bpf_pick_idle_cpu(const cpumask_t *cpus_allowed, u64 flags) __ksym;
+s32 scx_bpf_pick_any_cpu(const cpumask_t *cpus_allowed, u64 flags) __ksym;
+bool scx_bpf_task_running(const struct task_struct *p) __ksym;
+s32 scx_bpf_task_cpu(const struct task_struct *p) __ksym;
+
+static inline __attribute__((format(printf, 1, 2)))
+void ___scx_bpf_bstr_format_checker(const char *fmt, ...) {}
+
+/*
+ * Helper macro for initializing the fmt and variadic argument inputs to both
+ * bstr exit kfuncs. Callers to this function should use ___fmt and ___param to
+ * refer to the initialized list of inputs to the bstr kfunc.
+ */
+#define scx_bpf_bstr_preamble(fmt, args...) \
+ static char ___fmt[] = fmt; \
+ /* \
+ * Note that __param[] must have at least one \
+ * element to keep the verifier happy. \
+ */ \
+ unsigned long long ___param[___bpf_narg(args) ?: 1] = {}; \
+ \
+ _Pragma("GCC diagnostic push") \
+ _Pragma("GCC diagnostic ignored \"-Wint-conversion\"") \
+ ___bpf_fill(___param, args); \
+ _Pragma("GCC diagnostic pop") \
+
+/*
+ * scx_bpf_exit() wraps the scx_bpf_exit_bstr() kfunc with variadic arguments
+ * instead of an array of u64. Using this macro will cause the scheduler to
+ * exit cleanly with the specified exit code being passed to user space.
+ */
+#define scx_bpf_exit(code, fmt, args...) \
+({ \
+ scx_bpf_bstr_preamble(fmt, args) \
+ scx_bpf_exit_bstr(code, ___fmt, ___param, sizeof(___param)); \
+ ___scx_bpf_bstr_format_checker(fmt, ##args); \
+})
+
+/*
+ * scx_bpf_error() wraps the scx_bpf_error_bstr() kfunc with variadic arguments
+ * instead of an array of u64. Invoking this macro will cause the scheduler to
+ * exit in an erroneous state, with diagnostic information being passed to the
+ * user.
+ */
+#define scx_bpf_error(fmt, args...) \
+({ \
+ scx_bpf_bstr_preamble(fmt, args) \
+ scx_bpf_error_bstr(___fmt, ___param, sizeof(___param)); \
+ ___scx_bpf_bstr_format_checker(fmt, ##args); \
+})
+
+#define BPF_STRUCT_OPS(name, args...) \
+SEC("struct_ops/"#name) \
+BPF_PROG(name, ##args)
+
+#define BPF_STRUCT_OPS_SLEEPABLE(name, args...) \
+SEC("struct_ops.s/"#name) \
+BPF_PROG(name, ##args)
+
+/**
+ * RESIZABLE_ARRAY - Generates annotations for an array that may be resized
+ * @elfsec: the data section of the BPF program in which to place the array
+ * @arr: the name of the array
+ *
+ * libbpf has an API for setting map value sizes. Since data sections (i.e.
+ * bss, data, rodata) themselves are maps, a data section can be resized. If
+ * a data section has an array as its last element, the BTF info for that
+ * array will be adjusted so that length of the array is extended to meet the
+ * new length of the data section. This macro annotates an array to have an
+ * element count of one with the assumption that this array can be resized
+ * within the userspace program. It also annotates the section specifier so
+ * this array exists in a custom sub data section which can be resized
+ * independently.
+ *
+ * See RESIZE_ARRAY() for the userspace convenience macro for resizing an
+ * array declared with RESIZABLE_ARRAY().
+ */
+#define RESIZABLE_ARRAY(elfsec, arr) arr[1] SEC("."#elfsec"."#arr)
+
+/**
+ * MEMBER_VPTR - Obtain the verified pointer to a struct or array member
+ * @base: struct or array to index
+ * @member: dereferenced member (e.g. .field, [idx0][idx1], .field[idx0] ...)
+ *
+ * The verifier often gets confused by the instruction sequence the compiler
+ * generates for indexing struct fields or arrays. This macro forces the
+ * compiler to generate a code sequence which first calculates the byte offset,
+ * checks it against the struct or array size and add that byte offset to
+ * generate the pointer to the member to help the verifier.
+ *
+ * Ideally, we want to abort if the calculated offset is out-of-bounds. However,
+ * BPF currently doesn't support abort, so evaluate to %NULL instead. The caller
+ * must check for %NULL and take appropriate action to appease the verifier. To
+ * avoid confusing the verifier, it's best to check for %NULL and dereference
+ * immediately.
+ *
+ * vptr = MEMBER_VPTR(my_array, [i][j]);
+ * if (!vptr)
+ * return error;
+ * *vptr = new_value;
+ *
+ * sizeof(@base) should encompass the memory area to be accessed and thus can't
+ * be a pointer to the area. Use `MEMBER_VPTR(*ptr, .member)` instead of
+ * `MEMBER_VPTR(ptr, ->member)`.
+ */
+#define MEMBER_VPTR(base, member) (typeof((base) member) *) \
+({ \
+ u64 __base = (u64)&(base); \
+ u64 __addr = (u64)&((base) member) - __base; \
+ _Static_assert(sizeof(base) >= sizeof((base) member), \
+ "@base is smaller than @member, is @base a pointer?"); \
+ asm volatile ( \
+ "if %0 <= %[max] goto +2\n" \
+ "%0 = 0\n" \
+ "goto +1\n" \
+ "%0 += %1\n" \
+ : "+r"(__addr) \
+ : "r"(__base), \
+ [max]"i"(sizeof(base) - sizeof((base) member))); \
+ __addr; \
+})
+
+/**
+ * ARRAY_ELEM_PTR - Obtain the verified pointer to an array element
+ * @arr: array to index into
+ * @i: array index
+ * @n: number of elements in array
+ *
+ * Similar to MEMBER_VPTR() but is intended for use with arrays where the
+ * element count needs to be explicit.
+ * It can be used in cases where a global array is defined with an initial
+ * size but is intended to be be resized before loading the BPF program.
+ * Without this version of the macro, MEMBER_VPTR() will use the compile time
+ * size of the array to compute the max, which will result in rejection by
+ * the verifier.
+ */
+#define ARRAY_ELEM_PTR(arr, i, n) (typeof(arr[i]) *) \
+({ \
+ u64 __base = (u64)arr; \
+ u64 __addr = (u64)&(arr[i]) - __base; \
+ asm volatile ( \
+ "if %0 <= %[max] goto +2\n" \
+ "%0 = 0\n" \
+ "goto +1\n" \
+ "%0 += %1\n" \
+ : "+r"(__addr) \
+ : "r"(__base), \
+ [max]"r"(sizeof(arr[0]) * ((n) - 1))); \
+ __addr; \
+})
+
+
+/*
+ * BPF declarations and helpers
+ */
+
+/* list and rbtree */
+#define __contains(name, node) __attribute__((btf_decl_tag("contains:" #name ":" #node)))
+#define private(name) SEC(".data." #name) __hidden __attribute__((aligned(8)))
+
+void *bpf_obj_new_impl(__u64 local_type_id, void *meta) __ksym;
+void bpf_obj_drop_impl(void *kptr, void *meta) __ksym;
+
+#define bpf_obj_new(type) ((type *)bpf_obj_new_impl(bpf_core_type_id_local(type), NULL))
+#define bpf_obj_drop(kptr) bpf_obj_drop_impl(kptr, NULL)
+
+void bpf_list_push_front(struct bpf_list_head *head, struct bpf_list_node *node) __ksym;
+void bpf_list_push_back(struct bpf_list_head *head, struct bpf_list_node *node) __ksym;
+struct bpf_list_node *bpf_list_pop_front(struct bpf_list_head *head) __ksym;
+struct bpf_list_node *bpf_list_pop_back(struct bpf_list_head *head) __ksym;
+struct bpf_rb_node *bpf_rbtree_remove(struct bpf_rb_root *root,
+ struct bpf_rb_node *node) __ksym;
+int bpf_rbtree_add_impl(struct bpf_rb_root *root, struct bpf_rb_node *node,
+ bool (less)(struct bpf_rb_node *a, const struct bpf_rb_node *b),
+ void *meta, __u64 off) __ksym;
+#define bpf_rbtree_add(head, node, less) bpf_rbtree_add_impl(head, node, less, NULL, 0)
+
+struct bpf_rb_node *bpf_rbtree_first(struct bpf_rb_root *root) __ksym;
+
+void *bpf_refcount_acquire_impl(void *kptr, void *meta) __ksym;
+#define bpf_refcount_acquire(kptr) bpf_refcount_acquire_impl(kptr, NULL)
+
+/* task */
+struct task_struct *bpf_task_from_pid(s32 pid) __ksym;
+struct task_struct *bpf_task_acquire(struct task_struct *p) __ksym;
+void bpf_task_release(struct task_struct *p) __ksym;
+
+/* cgroup */
+struct cgroup *bpf_cgroup_ancestor(struct cgroup *cgrp, int level) __ksym;
+void bpf_cgroup_release(struct cgroup *cgrp) __ksym;
+struct cgroup *bpf_cgroup_from_id(u64 cgid) __ksym;
+
+/* css iteration */
+struct bpf_iter_css;
+struct cgroup_subsys_state;
+extern int bpf_iter_css_new(struct bpf_iter_css *it,
+ struct cgroup_subsys_state *start,
+ unsigned int flags) __weak __ksym;
+extern struct cgroup_subsys_state *
+bpf_iter_css_next(struct bpf_iter_css *it) __weak __ksym;
+extern void bpf_iter_css_destroy(struct bpf_iter_css *it) __weak __ksym;
+
+/* cpumask */
+struct bpf_cpumask *bpf_cpumask_create(void) __ksym;
+struct bpf_cpumask *bpf_cpumask_acquire(struct bpf_cpumask *cpumask) __ksym;
+void bpf_cpumask_release(struct bpf_cpumask *cpumask) __ksym;
+u32 bpf_cpumask_first(const struct cpumask *cpumask) __ksym;
+u32 bpf_cpumask_first_zero(const struct cpumask *cpumask) __ksym;
+void bpf_cpumask_set_cpu(u32 cpu, struct bpf_cpumask *cpumask) __ksym;
+void bpf_cpumask_clear_cpu(u32 cpu, struct bpf_cpumask *cpumask) __ksym;
+bool bpf_cpumask_test_cpu(u32 cpu, const struct cpumask *cpumask) __ksym;
+bool bpf_cpumask_test_and_set_cpu(u32 cpu, struct bpf_cpumask *cpumask) __ksym;
+bool bpf_cpumask_test_and_clear_cpu(u32 cpu, struct bpf_cpumask *cpumask) __ksym;
+void bpf_cpumask_setall(struct bpf_cpumask *cpumask) __ksym;
+void bpf_cpumask_clear(struct bpf_cpumask *cpumask) __ksym;
+bool bpf_cpumask_and(struct bpf_cpumask *dst, const struct cpumask *src1,
+ const struct cpumask *src2) __ksym;
+void bpf_cpumask_or(struct bpf_cpumask *dst, const struct cpumask *src1,
+ const struct cpumask *src2) __ksym;
+void bpf_cpumask_xor(struct bpf_cpumask *dst, const struct cpumask *src1,
+ const struct cpumask *src2) __ksym;
+bool bpf_cpumask_equal(const struct cpumask *src1, const struct cpumask *src2) __ksym;
+bool bpf_cpumask_intersects(const struct cpumask *src1, const struct cpumask *src2) __ksym;
+bool bpf_cpumask_subset(const struct cpumask *src1, const struct cpumask *src2) __ksym;
+bool bpf_cpumask_empty(const struct cpumask *cpumask) __ksym;
+bool bpf_cpumask_full(const struct cpumask *cpumask) __ksym;
+void bpf_cpumask_copy(struct bpf_cpumask *dst, const struct cpumask *src) __ksym;
+u32 bpf_cpumask_any_distribute(const struct cpumask *cpumask) __ksym;
+u32 bpf_cpumask_any_and_distribute(const struct cpumask *src1,
+ const struct cpumask *src2) __ksym;
+
+/* rcu */
+void bpf_rcu_read_lock(void) __ksym;
+void bpf_rcu_read_unlock(void) __ksym;
+
+
+/*
+ * Other helpers
+ */
+
+/* useful compiler attributes */
+#define likely(x) __builtin_expect(!!(x), 1)
+#define unlikely(x) __builtin_expect(!!(x), 0)
+#define __maybe_unused __attribute__((__unused__))
+
+/*
+ * READ/WRITE_ONCE() are from kernel (include/asm-generic/rwonce.h). They
+ * prevent compiler from caching, redoing or reordering reads or writes.
+ */
+typedef __u8 __attribute__((__may_alias__)) __u8_alias_t;
+typedef __u16 __attribute__((__may_alias__)) __u16_alias_t;
+typedef __u32 __attribute__((__may_alias__)) __u32_alias_t;
+typedef __u64 __attribute__((__may_alias__)) __u64_alias_t;
+
+static __always_inline void __read_once_size(const volatile void *p, void *res, int size)
+{
+ switch (size) {
+ case 1: *(__u8_alias_t *) res = *(volatile __u8_alias_t *) p; break;
+ case 2: *(__u16_alias_t *) res = *(volatile __u16_alias_t *) p; break;
+ case 4: *(__u32_alias_t *) res = *(volatile __u32_alias_t *) p; break;
+ case 8: *(__u64_alias_t *) res = *(volatile __u64_alias_t *) p; break;
+ default:
+ barrier();
+ __builtin_memcpy((void *)res, (const void *)p, size);
+ barrier();
+ }
+}
+
+static __always_inline void __write_once_size(volatile void *p, void *res, int size)
+{
+ switch (size) {
+ case 1: *(volatile __u8_alias_t *) p = *(__u8_alias_t *) res; break;
+ case 2: *(volatile __u16_alias_t *) p = *(__u16_alias_t *) res; break;
+ case 4: *(volatile __u32_alias_t *) p = *(__u32_alias_t *) res; break;
+ case 8: *(volatile __u64_alias_t *) p = *(__u64_alias_t *) res; break;
+ default:
+ barrier();
+ __builtin_memcpy((void *)p, (const void *)res, size);
+ barrier();
+ }
+}
+
+#define READ_ONCE(x) \
+({ \
+ union { typeof(x) __val; char __c[1]; } __u = \
+ { .__c = { 0 } }; \
+ __read_once_size(&(x), __u.__c, sizeof(x)); \
+ __u.__val; \
+})
+
+#define WRITE_ONCE(x, val) \
+({ \
+ union { typeof(x) __val; char __c[1]; } __u = \
+ { .__val = (val) }; \
+ __write_once_size(&(x), __u.__c, sizeof(x)); \
+ __u.__val; \
+})
+
+/*
+ * log2_u32 - Compute the base 2 logarithm of a 32-bit exponential value.
+ * @v: The value for which we're computing the base 2 logarithm.
+ */
+static inline u32 log2_u32(u32 v)
+{
+ u32 r;
+ u32 shift;
+
+ r = (v > 0xFFFF) << 4; v >>= r;
+ shift = (v > 0xFF) << 3; v >>= shift; r |= shift;
+ shift = (v > 0xF) << 2; v >>= shift; r |= shift;
+ shift = (v > 0x3) << 1; v >>= shift; r |= shift;
+ r |= (v >> 1);
+ return r;
+}
+
+/*
+ * log2_u64 - Compute the base 2 logarithm of a 64-bit exponential value.
+ * @v: The value for which we're computing the base 2 logarithm.
+ */
+static inline u32 log2_u64(u64 v)
+{
+ u32 hi = v >> 32;
+ if (hi)
+ return log2_u32(hi) + 32 + 1;
+ else
+ return log2_u32(v) + 1;
+}
+
+#include "compat.bpf.h"
+
+#endif /* __SCX_COMMON_BPF_H */
diff --git a/tools/sched_ext/include/scx/common.h b/tools/sched_ext/include/scx/common.h
new file mode 100644
index 000000000000..5b0f90152152
--- /dev/null
+++ b/tools/sched_ext/include/scx/common.h
@@ -0,0 +1,75 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2023 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2023 Tejun Heo <tj@kernel.org>
+ * Copyright (c) 2023 David Vernet <dvernet@meta.com>
+ */
+#ifndef __SCHED_EXT_COMMON_H
+#define __SCHED_EXT_COMMON_H
+
+#ifdef __KERNEL__
+#error "Should not be included by BPF programs"
+#endif
+
+#include <stdarg.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <stdint.h>
+#include <errno.h>
+
+typedef uint8_t u8;
+typedef uint16_t u16;
+typedef uint32_t u32;
+typedef uint64_t u64;
+typedef int8_t s8;
+typedef int16_t s16;
+typedef int32_t s32;
+typedef int64_t s64;
+
+#define SCX_BUG(__fmt, ...) \
+ do { \
+ fprintf(stderr, "[SCX_BUG] %s:%d", __FILE__, __LINE__); \
+ if (errno) \
+ fprintf(stderr, " (%s)\n", strerror(errno)); \
+ else \
+ fprintf(stderr, "\n"); \
+ fprintf(stderr, __fmt __VA_OPT__(,) __VA_ARGS__); \
+ fprintf(stderr, "\n"); \
+ \
+ exit(EXIT_FAILURE); \
+ } while (0)
+
+#define SCX_BUG_ON(__cond, __fmt, ...) \
+ do { \
+ if (__cond) \
+ SCX_BUG((__fmt) __VA_OPT__(,) __VA_ARGS__); \
+ } while (0)
+
+/**
+ * RESIZE_ARRAY - Convenience macro for resizing a BPF array
+ * @__skel: the skeleton containing the array
+ * @elfsec: the data section of the BPF program in which the array exists
+ * @arr: the name of the array
+ * @n: the desired array element count
+ *
+ * For BPF arrays declared with RESIZABLE_ARRAY(), this macro performs two
+ * operations. It resizes the map which corresponds to the custom data
+ * section that contains the target array. As a side effect, the BTF info for
+ * the array is adjusted so that the array length is sized to cover the new
+ * data section size. The second operation is reassigning the skeleton pointer
+ * for that custom data section so that it points to the newly memory mapped
+ * region.
+ */
+#define RESIZE_ARRAY(__skel, elfsec, arr, n) \
+ do { \
+ size_t __sz; \
+ bpf_map__set_value_size((__skel)->maps.elfsec##_##arr, \
+ sizeof((__skel)->elfsec##_##arr->arr[0]) * (n)); \
+ (__skel)->elfsec##_##arr = \
+ bpf_map__initial_value((__skel)->maps.elfsec##_##arr, &__sz); \
+ } while (0)
+
+#include "user_exit_info.h"
+#include "compat.h"
+
+#endif /* __SCHED_EXT_COMMON_H */
diff --git a/tools/sched_ext/include/scx/compat.bpf.h b/tools/sched_ext/include/scx/compat.bpf.h
new file mode 100644
index 000000000000..3d2fe1208900
--- /dev/null
+++ b/tools/sched_ext/include/scx/compat.bpf.h
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2024 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2024 Tejun Heo <tj@kernel.org>
+ * Copyright (c) 2024 David Vernet <dvernet@meta.com>
+ */
+#ifndef __SCX_COMPAT_BPF_H
+#define __SCX_COMPAT_BPF_H
+
+#define __COMPAT_ENUM_OR_ZERO(__type, __ent) \
+({ \
+ __type __ret = 0; \
+ if (bpf_core_enum_value_exists(__type, __ent)) \
+ __ret = __ent; \
+ __ret; \
+})
+
+/*
+ * Define sched_ext_ops. This may be expanded to define multiple variants for
+ * backward compatibility. See compat.h::SCX_OPS_LOAD/ATTACH().
+ */
+#define SCX_OPS_DEFINE(__name, ...) \
+ SEC(".struct_ops.link") \
+ struct sched_ext_ops __name = { \
+ __VA_ARGS__, \
+ };
+
+#endif /* __SCX_COMPAT_BPF_H */
diff --git a/tools/sched_ext/include/scx/compat.h b/tools/sched_ext/include/scx/compat.h
new file mode 100644
index 000000000000..a7fdaf8a858e
--- /dev/null
+++ b/tools/sched_ext/include/scx/compat.h
@@ -0,0 +1,153 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2024 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2024 Tejun Heo <tj@kernel.org>
+ * Copyright (c) 2024 David Vernet <dvernet@meta.com>
+ */
+#ifndef __SCX_COMPAT_H
+#define __SCX_COMPAT_H
+
+#include <bpf/btf.h>
+
+struct btf *__COMPAT_vmlinux_btf __attribute__((weak));
+
+static inline void __COMPAT_load_vmlinux_btf(void)
+{
+ if (!__COMPAT_vmlinux_btf) {
+ __COMPAT_vmlinux_btf = btf__load_vmlinux_btf();
+ SCX_BUG_ON(!__COMPAT_vmlinux_btf, "btf__load_vmlinux_btf()");
+ }
+}
+
+static inline bool __COMPAT_read_enum(const char *type, const char *name, u64 *v)
+{
+ const struct btf_type *t;
+ const char *n;
+ s32 tid;
+ int i;
+
+ __COMPAT_load_vmlinux_btf();
+
+ tid = btf__find_by_name(__COMPAT_vmlinux_btf, type);
+ if (tid < 0)
+ return false;
+
+ t = btf__type_by_id(__COMPAT_vmlinux_btf, tid);
+ SCX_BUG_ON(!t, "btf__type_by_id(%d)", tid);
+
+ if (btf_is_enum(t)) {
+ struct btf_enum *e = btf_enum(t);
+
+ for (i = 0; i < BTF_INFO_VLEN(t->info); i++) {
+ n = btf__name_by_offset(__COMPAT_vmlinux_btf, e[i].name_off);
+ SCX_BUG_ON(!n, "btf__name_by_offset()");
+ if (!strcmp(n, name)) {
+ *v = e[i].val;
+ return true;
+ }
+ }
+ } else if (btf_is_enum64(t)) {
+ struct btf_enum64 *e = btf_enum64(t);
+
+ for (i = 0; i < BTF_INFO_VLEN(t->info); i++) {
+ n = btf__name_by_offset(__COMPAT_vmlinux_btf, e[i].name_off);
+ SCX_BUG_ON(!n, "btf__name_by_offset()");
+ if (!strcmp(n, name)) {
+ *v = btf_enum64_value(&e[i]);
+ return true;
+ }
+ }
+ }
+
+ return false;
+}
+
+#define __COMPAT_ENUM_OR_ZERO(__type, __ent) \
+({ \
+ u64 __val = 0; \
+ __COMPAT_read_enum(__type, __ent, &__val); \
+ __val; \
+})
+
+static inline bool __COMPAT_has_ksym(const char *ksym)
+{
+ __COMPAT_load_vmlinux_btf();
+ return btf__find_by_name(__COMPAT_vmlinux_btf, ksym) >= 0;
+}
+
+static inline bool __COMPAT_struct_has_field(const char *type, const char *field)
+{
+ const struct btf_type *t;
+ const struct btf_member *m;
+ const char *n;
+ s32 tid;
+ int i;
+
+ __COMPAT_load_vmlinux_btf();
+ tid = btf__find_by_name_kind(__COMPAT_vmlinux_btf, type, BTF_KIND_STRUCT);
+ if (tid < 0)
+ return false;
+
+ t = btf__type_by_id(__COMPAT_vmlinux_btf, tid);
+ SCX_BUG_ON(!t, "btf__type_by_id(%d)", tid);
+
+ m = btf_members(t);
+
+ for (i = 0; i < BTF_INFO_VLEN(t->info); i++) {
+ n = btf__name_by_offset(__COMPAT_vmlinux_btf, m[i].name_off);
+ SCX_BUG_ON(!n, "btf__name_by_offset()");
+ if (!strcmp(n, field))
+ return true;
+ }
+
+ return false;
+}
+
+#define SCX_OPS_SWITCH_PARTIAL \
+ __COMPAT_ENUM_OR_ZERO("scx_ops_flags", "SCX_OPS_SWITCH_PARTIAL")
+
+/*
+ * struct sched_ext_ops can change over time. If compat.bpf.h::SCX_OPS_DEFINE()
+ * is used to define ops and compat.h::SCX_OPS_LOAD/ATTACH() are used to load
+ * and attach it, backward compatibility is automatically maintained where
+ * reasonable.
+ */
+#define SCX_OPS_OPEN(__ops_name, __scx_name) ({ \
+ struct __scx_name *__skel; \
+ \
+ __skel = __scx_name##__open(); \
+ SCX_BUG_ON(!__skel, "Could not open " #__scx_name); \
+ __skel; \
+})
+
+#define SCX_OPS_LOAD(__skel, __ops_name, __scx_name) ({ \
+ SCX_BUG_ON(__scx_name##__load((__skel)), "Failed to load skel"); \
+})
+
+/*
+ * New versions of bpftool now emit additional link placeholders for BPF maps,
+ * and set up BPF skeleton in such a way that libbpf will auto-attach BPF maps
+ * automatically, assumming libbpf is recent enough (v1.5+). Old libbpf will do
+ * nothing with those links and won't attempt to auto-attach maps.
+ *
+ * To maintain compatibility with older libbpf while avoiding trying to attach
+ * twice, disable the autoattach feature on newer libbpf.
+ */
+#if LIBBPF_MAJOR_VERSION > 1 || \
+ (LIBBPF_MAJOR_VERSION == 1 && LIBBPF_MINOR_VERSION >= 5)
+#define __SCX_OPS_DISABLE_AUTOATTACH(__skel, __ops_name) \
+ bpf_map__set_autoattach((__skel)->maps.__ops_name, false)
+#else
+#define __SCX_OPS_DISABLE_AUTOATTACH(__skel, __ops_name) do {} while (0)
+#endif
+
+#define SCX_OPS_ATTACH(__skel, __ops_name, __scx_name) ({ \
+ struct bpf_link *__link; \
+ __SCX_OPS_DISABLE_AUTOATTACH(__skel, __ops_name); \
+ SCX_BUG_ON(__scx_name##__attach((__skel)), "Failed to attach skel"); \
+ __link = bpf_map__attach_struct_ops((__skel)->maps.__ops_name); \
+ SCX_BUG_ON(!__link, "Failed to attach struct_ops"); \
+ __link; \
+})
+
+#endif /* __SCX_COMPAT_H */
diff --git a/tools/sched_ext/include/scx/user_exit_info.h b/tools/sched_ext/include/scx/user_exit_info.h
new file mode 100644
index 000000000000..8c3b7fac4d05
--- /dev/null
+++ b/tools/sched_ext/include/scx/user_exit_info.h
@@ -0,0 +1,64 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Define struct user_exit_info which is shared between BPF and userspace parts
+ * to communicate exit status and other information.
+ *
+ * Copyright (c) 2022 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2022 Tejun Heo <tj@kernel.org>
+ * Copyright (c) 2022 David Vernet <dvernet@meta.com>
+ */
+#ifndef __USER_EXIT_INFO_H
+#define __USER_EXIT_INFO_H
+
+enum uei_sizes {
+ UEI_REASON_LEN = 128,
+ UEI_MSG_LEN = 1024,
+};
+
+struct user_exit_info {
+ int kind;
+ s64 exit_code;
+ char reason[UEI_REASON_LEN];
+ char msg[UEI_MSG_LEN];
+};
+
+#ifdef __bpf__
+
+#include "vmlinux.h"
+#include <bpf/bpf_core_read.h>
+
+#define UEI_DEFINE(__name) \
+ struct user_exit_info __name SEC(".data")
+
+#define UEI_RECORD(__uei_name, __ei) ({ \
+ bpf_probe_read_kernel_str(__uei_name.reason, \
+ sizeof(__uei_name.reason), (__ei)->reason); \
+ bpf_probe_read_kernel_str(__uei_name.msg, \
+ sizeof(__uei_name.msg), (__ei)->msg); \
+ if (bpf_core_field_exists((__ei)->exit_code)) \
+ __uei_name.exit_code = (__ei)->exit_code; \
+ /* use __sync to force memory barrier */ \
+ __sync_val_compare_and_swap(&__uei_name.kind, __uei_name.kind, \
+ (__ei)->kind); \
+})
+
+#else /* !__bpf__ */
+
+#include <stdio.h>
+#include <stdbool.h>
+
+#define UEI_EXITED(__skel, __uei_name) ({ \
+ /* use __sync to force memory barrier */ \
+ __sync_val_compare_and_swap(&(__skel)->data->__uei_name.kind, -1, -1); \
+})
+
+#define UEI_REPORT(__skel, __uei_name) ({ \
+ struct user_exit_info *__uei = &(__skel)->data->__uei_name; \
+ fprintf(stderr, "EXIT: %s", __uei->reason); \
+ if (__uei->msg[0] != '\0') \
+ fprintf(stderr, " (%s)", __uei->msg); \
+ fputs("\n", stderr); \
+})
+
+#endif /* __bpf__ */
+#endif /* __USER_EXIT_INFO_H */
diff --git a/tools/sched_ext/scx_qmap.bpf.c b/tools/sched_ext/scx_qmap.bpf.c
new file mode 100644
index 000000000000..976a2693da71
--- /dev/null
+++ b/tools/sched_ext/scx_qmap.bpf.c
@@ -0,0 +1,264 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * A simple five-level FIFO queue scheduler.
+ *
+ * There are five FIFOs implemented using BPF_MAP_TYPE_QUEUE. A task gets
+ * assigned to one depending on its compound weight. Each CPU round robins
+ * through the FIFOs and dispatches more from FIFOs with higher indices - 1 from
+ * queue0, 2 from queue1, 4 from queue2 and so on.
+ *
+ * This scheduler demonstrates:
+ *
+ * - BPF-side queueing using PIDs.
+ * - Sleepable per-task storage allocation using ops.prep_enable().
+ *
+ * This scheduler is primarily for demonstration and testing of sched_ext
+ * features and unlikely to be useful for actual workloads.
+ *
+ * Copyright (c) 2022 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2022 Tejun Heo <tj@kernel.org>
+ * Copyright (c) 2022 David Vernet <dvernet@meta.com>
+ */
+#include <scx/common.bpf.h>
+
+enum consts {
+ ONE_SEC_IN_NS = 1000000000,
+ SHARED_DSQ = 0,
+};
+
+char _license[] SEC("license") = "GPL";
+
+const volatile u64 slice_ns = SCX_SLICE_DFL;
+const volatile u32 dsp_batch;
+
+u32 test_error_cnt;
+
+UEI_DEFINE(uei);
+
+struct qmap {
+ __uint(type, BPF_MAP_TYPE_QUEUE);
+ __uint(max_entries, 4096);
+ __type(value, u32);
+} queue0 SEC(".maps"),
+ queue1 SEC(".maps"),
+ queue2 SEC(".maps"),
+ queue3 SEC(".maps"),
+ queue4 SEC(".maps");
+
+struct {
+ __uint(type, BPF_MAP_TYPE_ARRAY_OF_MAPS);
+ __uint(max_entries, 5);
+ __type(key, int);
+ __array(values, struct qmap);
+} queue_arr SEC(".maps") = {
+ .values = {
+ [0] = &queue0,
+ [1] = &queue1,
+ [2] = &queue2,
+ [3] = &queue3,
+ [4] = &queue4,
+ },
+};
+
+/* Per-task scheduling context */
+struct task_ctx {
+ bool force_local; /* Dispatch directly to local_dsq */
+};
+
+struct {
+ __uint(type, BPF_MAP_TYPE_TASK_STORAGE);
+ __uint(map_flags, BPF_F_NO_PREALLOC);
+ __type(key, int);
+ __type(value, struct task_ctx);
+} task_ctx_stor SEC(".maps");
+
+struct cpu_ctx {
+ u64 dsp_idx; /* dispatch index */
+ u64 dsp_cnt; /* remaining count */
+};
+
+struct {
+ __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY);
+ __uint(max_entries, 1);
+ __type(key, u32);
+ __type(value, struct cpu_ctx);
+} cpu_ctx_stor SEC(".maps");
+
+/* Statistics */
+u64 nr_enqueued, nr_dispatched, nr_dequeued;
+
+s32 BPF_STRUCT_OPS(qmap_select_cpu, struct task_struct *p,
+ s32 prev_cpu, u64 wake_flags)
+{
+ struct task_ctx *tctx;
+ s32 cpu;
+
+ tctx = bpf_task_storage_get(&task_ctx_stor, p, 0, 0);
+ if (!tctx) {
+ scx_bpf_error("task_ctx lookup failed");
+ return -ESRCH;
+ }
+
+ if (p->nr_cpus_allowed == 1 ||
+ scx_bpf_test_and_clear_cpu_idle(prev_cpu)) {
+ tctx->force_local = true;
+ return prev_cpu;
+ }
+
+ cpu = scx_bpf_pick_idle_cpu(p->cpus_ptr, 0);
+ if (cpu >= 0)
+ return cpu;
+
+ return prev_cpu;
+}
+
+static int weight_to_idx(u32 weight)
+{
+ /* Coarsely map the compound weight to a FIFO. */
+ if (weight <= 25)
+ return 0;
+ else if (weight <= 50)
+ return 1;
+ else if (weight < 200)
+ return 2;
+ else if (weight < 400)
+ return 3;
+ else
+ return 4;
+}
+
+void BPF_STRUCT_OPS(qmap_enqueue, struct task_struct *p, u64 enq_flags)
+{
+ struct task_ctx *tctx;
+ u32 pid = p->pid;
+ int idx = weight_to_idx(p->scx.weight);
+ void *ring;
+
+ if (test_error_cnt && !--test_error_cnt)
+ scx_bpf_error("test triggering error");
+
+ tctx = bpf_task_storage_get(&task_ctx_stor, p, 0, 0);
+ if (!tctx) {
+ scx_bpf_error("task_ctx lookup failed");
+ return;
+ }
+
+ /* Is select_cpu() is telling us to enqueue locally? */
+ if (tctx->force_local) {
+ tctx->force_local = false;
+ scx_bpf_dispatch(p, SCX_DSQ_LOCAL, slice_ns, enq_flags);
+ return;
+ }
+
+ ring = bpf_map_lookup_elem(&queue_arr, &idx);
+ if (!ring) {
+ scx_bpf_error("failed to find ring %d", idx);
+ return;
+ }
+
+ /* Queue on the selected FIFO. If the FIFO overflows, punt to global. */
+ if (bpf_map_push_elem(ring, &pid, 0)) {
+ scx_bpf_dispatch(p, SHARED_DSQ, slice_ns, enq_flags);
+ return;
+ }
+
+ __sync_fetch_and_add(&nr_enqueued, 1);
+}
+
+/*
+ * The BPF queue map doesn't support removal and sched_ext can handle spurious
+ * dispatches. qmap_dequeue() is only used to collect statistics.
+ */
+void BPF_STRUCT_OPS(qmap_dequeue, struct task_struct *p, u64 deq_flags)
+{
+ __sync_fetch_and_add(&nr_dequeued, 1);
+}
+
+void BPF_STRUCT_OPS(qmap_dispatch, s32 cpu, struct task_struct *prev)
+{
+ struct task_struct *p;
+ struct cpu_ctx *cpuc;
+ u32 zero = 0, batch = dsp_batch ?: 1;
+ void *fifo;
+ s32 i, pid;
+
+ if (scx_bpf_consume(SHARED_DSQ))
+ return;
+
+ if (!(cpuc = bpf_map_lookup_elem(&cpu_ctx_stor, &zero))) {
+ scx_bpf_error("failed to look up cpu_ctx");
+ return;
+ }
+
+ for (i = 0; i < 5; i++) {
+ /* Advance the dispatch cursor and pick the fifo. */
+ if (!cpuc->dsp_cnt) {
+ cpuc->dsp_idx = (cpuc->dsp_idx + 1) % 5;
+ cpuc->dsp_cnt = 1 << cpuc->dsp_idx;
+ }
+
+ fifo = bpf_map_lookup_elem(&queue_arr, &cpuc->dsp_idx);
+ if (!fifo) {
+ scx_bpf_error("failed to find ring %llu", cpuc->dsp_idx);
+ return;
+ }
+
+ /* Dispatch or advance. */
+ bpf_repeat(BPF_MAX_LOOPS) {
+ if (bpf_map_pop_elem(fifo, &pid))
+ break;
+
+ p = bpf_task_from_pid(pid);
+ if (!p)
+ continue;
+
+ __sync_fetch_and_add(&nr_dispatched, 1);
+ scx_bpf_dispatch(p, SHARED_DSQ, slice_ns, 0);
+ bpf_task_release(p);
+ batch--;
+ cpuc->dsp_cnt--;
+ if (!batch || !scx_bpf_dispatch_nr_slots()) {
+ scx_bpf_consume(SHARED_DSQ);
+ return;
+ }
+ if (!cpuc->dsp_cnt)
+ break;
+ }
+
+ cpuc->dsp_cnt = 0;
+ }
+}
+
+s32 BPF_STRUCT_OPS(qmap_init_task, struct task_struct *p,
+ struct scx_init_task_args *args)
+{
+ /*
+ * @p is new. Let's ensure that its task_ctx is available. We can sleep
+ * in this function and the following will automatically use GFP_KERNEL.
+ */
+ if (bpf_task_storage_get(&task_ctx_stor, p, 0,
+ BPF_LOCAL_STORAGE_GET_F_CREATE))
+ return 0;
+ else
+ return -ENOMEM;
+}
+
+s32 BPF_STRUCT_OPS_SLEEPABLE(qmap_init)
+{
+ return scx_bpf_create_dsq(SHARED_DSQ, -1);
+}
+
+void BPF_STRUCT_OPS(qmap_exit, struct scx_exit_info *ei)
+{
+ UEI_RECORD(uei, ei);
+}
+
+SCX_OPS_DEFINE(qmap_ops,
+ .select_cpu = (void *)qmap_select_cpu,
+ .enqueue = (void *)qmap_enqueue,
+ .dequeue = (void *)qmap_dequeue,
+ .dispatch = (void *)qmap_dispatch,
+ .init_task = (void *)qmap_init_task,
+ .init = (void *)qmap_init,
+ .exit = (void *)qmap_exit,
+ .name = "qmap");
diff --git a/tools/sched_ext/scx_qmap.c b/tools/sched_ext/scx_qmap.c
new file mode 100644
index 000000000000..7c84ade7ecfb
--- /dev/null
+++ b/tools/sched_ext/scx_qmap.c
@@ -0,0 +1,99 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2022 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2022 Tejun Heo <tj@kernel.org>
+ * Copyright (c) 2022 David Vernet <dvernet@meta.com>
+ */
+#include <stdio.h>
+#include <stdlib.h>
+#include <unistd.h>
+#include <inttypes.h>
+#include <signal.h>
+#include <libgen.h>
+#include <bpf/bpf.h>
+#include <scx/common.h>
+#include "scx_qmap.bpf.skel.h"
+
+const char help_fmt[] =
+"A simple five-level FIFO queue sched_ext scheduler.\n"
+"\n"
+"See the top-level comment in .bpf.c for more details.\n"
+"\n"
+"Usage: %s [-s SLICE_US] [-e COUNT] [-b COUNT] [-p] [-v]\n"
+"\n"
+" -s SLICE_US Override slice duration\n"
+" -e COUNT Trigger scx_bpf_error() after COUNT enqueues\n"
+" -b COUNT Dispatch upto COUNT tasks together\n"
+" -p Switch only tasks on SCHED_EXT policy intead of all\n"
+" -v Print libbpf debug messages\n"
+" -h Display this help and exit\n";
+
+static bool verbose;
+static volatile int exit_req;
+
+static int libbpf_print_fn(enum libbpf_print_level level, const char *format, va_list args)
+{
+ if (level == LIBBPF_DEBUG && !verbose)
+ return 0;
+ return vfprintf(stderr, format, args);
+}
+
+static void sigint_handler(int dummy)
+{
+ exit_req = 1;
+}
+
+int main(int argc, char **argv)
+{
+ struct scx_qmap *skel;
+ struct bpf_link *link;
+ int opt;
+
+ libbpf_set_print(libbpf_print_fn);
+ signal(SIGINT, sigint_handler);
+ signal(SIGTERM, sigint_handler);
+
+ skel = SCX_OPS_OPEN(qmap_ops, scx_qmap);
+
+ while ((opt = getopt(argc, argv, "s:e:b:pvh")) != -1) {
+ switch (opt) {
+ case 's':
+ skel->rodata->slice_ns = strtoull(optarg, NULL, 0) * 1000;
+ break;
+ case 'e':
+ skel->bss->test_error_cnt = strtoul(optarg, NULL, 0);
+ break;
+ case 'b':
+ skel->rodata->dsp_batch = strtoul(optarg, NULL, 0);
+ break;
+ case 'p':
+ skel->struct_ops.qmap_ops->flags |= SCX_OPS_SWITCH_PARTIAL;
+ break;
+ case 'v':
+ verbose = true;
+ break;
+ default:
+ fprintf(stderr, help_fmt, basename(argv[0]));
+ return opt != 'h';
+ }
+ }
+
+ SCX_OPS_LOAD(skel, qmap_ops, scx_qmap);
+ link = SCX_OPS_ATTACH(skel, qmap_ops, scx_qmap);
+
+ while (!exit_req && !UEI_EXITED(skel, uei)) {
+ long nr_enqueued = skel->bss->nr_enqueued;
+ long nr_dispatched = skel->bss->nr_dispatched;
+
+ printf("stats : enq=%lu dsp=%lu delta=%ld deq=%"PRIu64"\n",
+ nr_enqueued, nr_dispatched, nr_enqueued - nr_dispatched,
+ skel->bss->nr_dequeued);
+ fflush(stdout);
+ sleep(1);
+ }
+
+ bpf_link__destroy(link);
+ UEI_REPORT(skel, uei);
+ scx_qmap__destroy(skel);
+ return 0;
+}
diff --git a/tools/sched_ext/scx_simple.bpf.c b/tools/sched_ext/scx_simple.bpf.c
new file mode 100644
index 000000000000..6bb13a3c801b
--- /dev/null
+++ b/tools/sched_ext/scx_simple.bpf.c
@@ -0,0 +1,63 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * A simple scheduler.
+ *
+ * A simple global FIFO scheduler. It also demonstrates the following niceties.
+ *
+ * - Statistics tracking how many tasks are queued to local and global dsq's.
+ * - Termination notification for userspace.
+ *
+ * Copyright (c) 2022 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2022 Tejun Heo <tj@kernel.org>
+ * Copyright (c) 2022 David Vernet <dvernet@meta.com>
+ */
+#include <scx/common.bpf.h>
+
+char _license[] SEC("license") = "GPL";
+
+UEI_DEFINE(uei);
+
+struct {
+ __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY);
+ __uint(key_size, sizeof(u32));
+ __uint(value_size, sizeof(u64));
+ __uint(max_entries, 2); /* [local, global] */
+} stats SEC(".maps");
+
+static void stat_inc(u32 idx)
+{
+ u64 *cnt_p = bpf_map_lookup_elem(&stats, &idx);
+ if (cnt_p)
+ (*cnt_p)++;
+}
+
+s32 BPF_STRUCT_OPS(simple_select_cpu, struct task_struct *p, s32 prev_cpu, u64 wake_flags)
+{
+ bool is_idle = false;
+ s32 cpu;
+
+ cpu = scx_bpf_select_cpu_dfl(p, prev_cpu, wake_flags, &is_idle);
+ if (is_idle) {
+ stat_inc(0); /* count local queueing */
+ scx_bpf_dispatch(p, SCX_DSQ_LOCAL, SCX_SLICE_DFL, 0);
+ }
+
+ return cpu;
+}
+
+void BPF_STRUCT_OPS(simple_enqueue, struct task_struct *p, u64 enq_flags)
+{
+ stat_inc(1); /* count global queueing */
+ scx_bpf_dispatch(p, SCX_DSQ_GLOBAL, SCX_SLICE_DFL, enq_flags);
+}
+
+void BPF_STRUCT_OPS(simple_exit, struct scx_exit_info *ei)
+{
+ UEI_RECORD(uei, ei);
+}
+
+SCX_OPS_DEFINE(simple_ops,
+ .select_cpu = (void *)simple_select_cpu,
+ .enqueue = (void *)simple_enqueue,
+ .exit = (void *)simple_exit,
+ .name = "simple");
diff --git a/tools/sched_ext/scx_simple.c b/tools/sched_ext/scx_simple.c
new file mode 100644
index 000000000000..789ac62fea8e
--- /dev/null
+++ b/tools/sched_ext/scx_simple.c
@@ -0,0 +1,99 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2022 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2022 Tejun Heo <tj@kernel.org>
+ * Copyright (c) 2022 David Vernet <dvernet@meta.com>
+ */
+#include <stdio.h>
+#include <unistd.h>
+#include <signal.h>
+#include <libgen.h>
+#include <bpf/bpf.h>
+#include <scx/common.h>
+#include "scx_simple.bpf.skel.h"
+
+const char help_fmt[] =
+"A simple sched_ext scheduler.\n"
+"\n"
+"See the top-level comment in .bpf.c for more details.\n"
+"\n"
+"Usage: %s [-v]\n"
+"\n"
+" -v Print libbpf debug messages\n"
+" -h Display this help and exit\n";
+
+static bool verbose;
+static volatile int exit_req;
+
+static int libbpf_print_fn(enum libbpf_print_level level, const char *format, va_list args)
+{
+ if (level == LIBBPF_DEBUG && !verbose)
+ return 0;
+ return vfprintf(stderr, format, args);
+}
+
+static void sigint_handler(int simple)
+{
+ exit_req = 1;
+}
+
+static void read_stats(struct scx_simple *skel, __u64 *stats)
+{
+ int nr_cpus = libbpf_num_possible_cpus();
+ __u64 cnts[2][nr_cpus];
+ __u32 idx;
+
+ memset(stats, 0, sizeof(stats[0]) * 2);
+
+ for (idx = 0; idx < 2; idx++) {
+ int ret, cpu;
+
+ ret = bpf_map_lookup_elem(bpf_map__fd(skel->maps.stats),
+ &idx, cnts[idx]);
+ if (ret < 0)
+ continue;
+ for (cpu = 0; cpu < nr_cpus; cpu++)
+ stats[idx] += cnts[idx][cpu];
+ }
+}
+
+int main(int argc, char **argv)
+{
+ struct scx_simple *skel;
+ struct bpf_link *link;
+ __u32 opt;
+
+ libbpf_set_print(libbpf_print_fn);
+ signal(SIGINT, sigint_handler);
+ signal(SIGTERM, sigint_handler);
+
+ skel = SCX_OPS_OPEN(simple_ops, scx_simple);
+
+ while ((opt = getopt(argc, argv, "vh")) != -1) {
+ switch (opt) {
+ case 'v':
+ verbose = true;
+ break;
+ default:
+ fprintf(stderr, help_fmt, basename(argv[0]));
+ return opt != 'h';
+ }
+ }
+
+ SCX_OPS_LOAD(skel, simple_ops, scx_simple);
+ link = SCX_OPS_ATTACH(skel, simple_ops, scx_simple);
+
+ while (!exit_req && !UEI_EXITED(skel, uei)) {
+ __u64 stats[2];
+
+ read_stats(skel, stats);
+ printf("local=%llu global=%llu\n", stats[0], stats[1]);
+ fflush(stdout);
+ sleep(1);
+ }
+
+ bpf_link__destroy(link);
+ UEI_REPORT(skel, uei);
+ scx_simple__destroy(skel);
+ return 0;
+}
--
2.45.2
Implementation Analysis
Overview
This patch (PATCH 10/30) adds two working BPF schedulers to tools/sched_ext/, along with the shared header infrastructure they both depend on. These are not toy programs — they ship with the kernel tree and serve multiple purposes simultaneously: they demonstrate the sched_ext API to new BPF scheduler authors, they provide regression targets for the selftests added in PATCH 30/30, and scx_qmap is the testbed for the watchdog (PATCH 12/30) and other safety features.
The two schedulers are designed to illustrate a spectrum of complexity:
scx_simple: Minimum viable BPF scheduler. Global FIFO with per-CPU stat tracking. If you understand this scheduler, you understand the fundamental dispatch loop.scx_qmap: Five-level weighted priority scheduler usingBPF_MAP_TYPE_QUEUEmaps. Demonstrates BPF-side task queueing, per-task storage, custom DSQs, and theops.dispatch()callback.
Architecture Context
These schedulers demonstrate the two fundamental dispatch patterns that all BPF schedulers choose between:
Pattern 1 — Direct dispatch in ops.enqueue(): The scheduler calls scx_bpf_dispatch() inside ops.enqueue(), placing the task directly into a DSQ. The kernel's dispatch loop then consumes from that DSQ when the CPU needs work. scx_simple uses this pattern exclusively.
Pattern 2 — BPF-side queueing with ops.dispatch(): The scheduler holds tasks in BPF maps (the "BPF side") inside ops.enqueue() without calling scx_bpf_dispatch(). When the kernel needs work for a CPU, it calls ops.dispatch(), and the BPF scheduler pulls tasks from its maps and dispatches them then. scx_qmap uses this pattern for its priority queues.
The second pattern allows the BPF scheduler to maintain its own ordering and priority logic independently of when the kernel needs the next task. It is essential for implementing any policy more sophisticated than a single FIFO.
Code Walkthrough
scx_simple (the minimal BPF scheduler)
scx_simple.bpf.c — the BPF program
s32 BPF_STRUCT_OPS(simple_select_cpu, struct task_struct *p,
s32 prev_cpu, u64 wake_flags)
{
bool is_idle = false;
s32 cpu;
cpu = scx_bpf_select_cpu_dfl(p, prev_cpu, wake_flags, &is_idle);
if (is_idle) {
stat_inc(0); /* count local queueing */
scx_bpf_dispatch(p, SCX_DSQ_LOCAL, SCX_SLICE_DFL, 0);
}
return cpu;
}
void BPF_STRUCT_OPS(simple_enqueue, struct task_struct *p, u64 enq_flags)
{
stat_inc(1); /* count global queueing */
scx_bpf_dispatch(p, SCX_DSQ_GLOBAL, SCX_SLICE_DFL, enq_flags);
}
simple_select_cpu() calls scx_bpf_select_cpu_dfl(), the sched_ext default idle-CPU picker. If it finds an idle CPU (is_idle = true), the task is dispatched directly to that CPU's local DSQ (SCX_DSQ_LOCAL) from inside ops.select_cpu(). When this happens, ops.enqueue() is not called — the dispatch in select_cpu() short-circuits the enqueue path.
If no idle CPU is found, the task falls through to ops.enqueue(), which dispatches it to SCX_DSQ_GLOBAL — the global FIFO that any CPU can consume from.
This two-path design (local dispatch when idle CPU available, global fallback otherwise) is the canonical pattern for avoiding unnecessary cross-CPU wakeups. It is so common that understanding it is a prerequisite for reviewing any non-trivial BPF scheduler.
The stats map is BPF_MAP_TYPE_PERCPU_ARRAY — each CPU has its own copy of the counters, eliminating contention. The userspace binary aggregates them across CPUs using bpf_map_lookup_elem() with all-CPU semantics.
scx_simple.c — the userspace control plane
skel = SCX_OPS_OPEN(simple_ops, scx_simple);
/* ... option parsing ... */
SCX_OPS_LOAD(skel, simple_ops, scx_simple);
link = SCX_OPS_ATTACH(skel, simple_ops, scx_simple);
while (!exit_req && !UEI_EXITED(skel, uei)) {
read_stats(skel, stats);
printf("local=%llu global=%llu\n", stats[0], stats[1]);
sleep(1);
}
bpf_link__destroy(link);
UEI_REPORT(skel, uei);
The lifecycle is: open the BPF skeleton → load (verify and JIT the BPF program) → attach (register simple_ops as the active sched_ext_ops) → poll until exit → destroy the link (which unregisters the BPF scheduler and returns all tasks to CFS). UEI_EXITED() and UEI_REPORT() are macros from user_exit_info.h that read the scx_exit_info structure the kernel fills in when the BPF scheduler exits for any reason (watchdog, sysrq-S, etc.).
The critical detail: destroying the bpf_link is what triggers the orderly shutdown. If the process exits without calling bpf_link__destroy(), the kernel detects the orphaned link and initiates the same shutdown path. This is the "run binary to start, terminate binary to stop" model described in the cover letter.
scx_qmap (the priority-queue BPF scheduler)
Map structure
struct qmap {
__uint(type, BPF_MAP_TYPE_QUEUE);
__uint(max_entries, 4096);
__type(value, u32); /* stores PIDs */
} queue0, queue1, queue2, queue3, queue4;
struct {
__uint(type, BPF_MAP_TYPE_ARRAY_OF_MAPS);
__uint(max_entries, 5);
__array(values, struct qmap);
} queue_arr = { .values = { &queue0, ... &queue4 } };
Five BPF_MAP_TYPE_QUEUE maps implement five priority FIFOs. They are accessed through an ARRAY_OF_MAPS so that ops.dispatch() can select a queue by index without a chain of if statements. The queue values store PIDs (u32), not task pointers — task pointers cannot be stored in BPF maps because their lifetime is not managed by the map. The PID is looked up in ops.dispatch() using bpf_task_from_pid().
struct {
__uint(type, BPF_MAP_TYPE_TASK_STORAGE);
__uint(map_flags, BPF_F_NO_PREALLOC);
__type(value, struct task_ctx);
} task_ctx_stor;
BPF_MAP_TYPE_TASK_STORAGE provides per-task storage that is automatically allocated and freed with the task. BPF_F_NO_PREALLOC means entries are allocated on demand (sleepable context, in ops.init_task()) rather than at map creation time, which is correct for per-task storage that may cover all tasks on the system.
weight_to_idx() — priority mapping
static int weight_to_idx(u32 weight)
{
if (weight <= 25) return 0; /* lowest priority */
else if (weight <= 50) return 1;
else if (weight < 200) return 2; /* default nice=0 ~ weight 100 */
else if (weight < 400) return 3;
else return 4; /* highest priority */
}
p->scx.weight is the CFS-compatible compound weight (1–10000). The default nice=0 weight is 100, which maps to queue index 2. Nice-19 tasks (minimum priority) have weight 1 and map to queue 0. Nice -20 tasks (maximum priority) have weight 88761 and map to queue 4.
qmap_enqueue() — the BPF-side queueing path
void BPF_STRUCT_OPS(qmap_enqueue, struct task_struct *p, u64 enq_flags)
{
if (tctx->force_local) {
tctx->force_local = false;
scx_bpf_dispatch(p, SCX_DSQ_LOCAL, slice_ns, enq_flags);
return;
}
ring = bpf_map_lookup_elem(&queue_arr, &idx);
if (bpf_map_push_elem(ring, &pid, 0)) {
/* Queue full — fall back to the shared custom DSQ */
scx_bpf_dispatch(p, SHARED_DSQ, slice_ns, enq_flags);
return;
}
__sync_fetch_and_add(&nr_enqueued, 1);
/* NOTE: no scx_bpf_dispatch() call — task is now in BPF queue */
}
When the queue is not full, qmap_enqueue() pushes the PID into the queue and returns without calling scx_bpf_dispatch(). The task is now held in BPF-side storage. The kernel will call ops.dispatch() when a CPU needs work. The queue overflow fallback to SHARED_DSQ (a custom DSQ created in ops.init()) is important: BPF_MAP_TYPE_QUEUE has a fixed size of 4096 entries per queue, and a buggy scheduler that never dispatches would eventually fill them. The fallback prevents ops.enqueue() from silently dropping tasks — which would be detected by the watchdog but only after timeout_ms seconds.
qmap_dispatch() — the weighted round-robin dispatch
void BPF_STRUCT_OPS(qmap_dispatch, s32 cpu, struct task_struct *prev)
{
if (scx_bpf_consume(SHARED_DSQ))
return;
/* Round-robin through queues, dispatching 2^idx tasks per turn */
for (i = 0; i < 5; i++) {
if (!cpuc->dsp_cnt) {
cpuc->dsp_idx = (cpuc->dsp_idx + 1) % 5;
cpuc->dsp_cnt = 1 << cpuc->dsp_idx;
}
/* ... pop PID from queue, look up task, dispatch to SHARED_DSQ ... */
}
}
The dispatch policy: queue 4 (highest priority) gets 2^4 = 16 tasks dispatched per round-robin step; queue 0 (lowest priority) gets 2^0 = 1. This gives higher-priority tasks exponentially more CPU time relative to lower-priority tasks. The dispatch state (dsp_idx, dsp_cnt) is stored in a BPF_MAP_TYPE_PERCPU_ARRAY keyed by zero, giving each CPU its own independent dispatch cursor.
The bpf_task_from_pid() lookup is necessary because PIDs can be reused: if a task exits after being enqueued in the queue but before being dispatched, bpf_task_from_pid() returns NULL and the PID is silently skipped. This is why the commit comment notes: "The BPF queue map doesn't support removal and sched_ext can handle spurious dispatches."
qmap_init_task() — sleepable per-task initialization
s32 BPF_STRUCT_OPS_SLEEPABLE(qmap_init_task, struct task_struct *p, ...)
{
if (bpf_task_storage_get(&task_ctx_stor, p, 0,
BPF_LOCAL_STORAGE_GET_F_CREATE))
return 0;
return -ENOMEM;
}
BPF_STRUCT_OPS_SLEEPABLE marks ops.init_task() as a sleepable BPF callback. This is necessary because bpf_task_storage_get() with BPF_LOCAL_STORAGE_GET_F_CREATE allocates memory, which may sleep. The sched_ext infrastructure calls ops.init_task() from a context where sleeping is safe (it is not on the scheduling hot path). The allocation failure return of -ENOMEM will cause the task's initialization to fail and sched_ext to abort.
Key Concepts Introduced
The BPF skeleton (*.skel.h): Generated by bpftool gen skeleton from the compiled BPF object file (.o). The skeleton provides a typed C API for opening, loading, attaching, and destroying the BPF program. scx_simple.bpf.skel.h contains struct scx_simple with typed access to all maps and global variables. The SCX_OPS_OPEN/LOAD/ATTACH macros in common.h wrap the standard libbpf skeleton lifecycle with sched_ext-specific validation.
UEI_DEFINE / UEI_EXITED / UEI_REPORT (User Exit Info): Macros from user_exit_info.h that implement the exit notification protocol. The kernel writes scx_exit_info (exit kind, message, backtrace) into a BPF map when the scheduler exits. UEI_EXITED() polls that map. UEI_REPORT() prints the exit reason. This pattern enables BPF schedulers to provide rich exit diagnostics without requiring a special kernel interface.
const volatile globals as userspace-tunable parameters: scx_qmap uses const volatile u64 slice_ns = SCX_SLICE_DFL. The const volatile combination is a BPF idiom: const tells the verifier the value will not change during BPF execution (enabling optimizations), while volatile prevents the compiler from optimizing away reads (since the value can be modified by userspace before the program is loaded). The skeleton exposes these as skel->rodata->slice_ns, writable before the BPF program is loaded via SCX_OPS_LOAD().
BPF_MAP_TYPE_QUEUE as a FIFO: bpf_map_push_elem() enqueues; bpf_map_pop_elem() dequeues in FIFO order. Unlike BPF_MAP_TYPE_HASH or BPF_MAP_TYPE_ARRAY, queues do not have user-controllable keys — they are pure FIFOs. They have bounded capacity (max_entries = 4096), which requires the overflow handling seen in qmap_enqueue().
Custom DSQ creation: qmap_init() calls scx_bpf_create_dsq(SHARED_DSQ, -1). The second argument (-1) means the DSQ is not associated with any NUMA node. SHARED_DSQ = 0 is a user-defined ID that must not collide with the built-in IDs (SCX_DSQ_GLOBAL = 0 is actually a well-known ID — in the real code this would need to be a value that does not conflict; in this example SHARED_DSQ = 0 works only because the scheduler does not use SCX_DSQ_GLOBAL directly).
Why This Matters for Maintainers
scx_simple is the canonical reference implementation: When a new sched_ext_ops callback is added to the kernel, scx_simple (or its successor) is typically the first scheduler updated to demonstrate correct usage. If scx_simple cannot be compiled and run against the current kernel, something is broken. Maintainers should treat a broken scx_simple build as a P0 regression.
The PID-in-queue pattern requires bpf_task_from_pid() null-checking: Any BPF scheduler that queues task identifiers (PIDs, task pointers cast to integers) rather than dispatching immediately must handle the case where the task exits before dispatch. The scx_qmap pattern of if (!p) continue; after bpf_task_from_pid() is the correct idiom. A scheduler that fails to handle this null case will call scx_bpf_dispatch() with a NULL task pointer, which will trigger scx_ops_error().
Queue overflow fallback is mandatory for BPF_MAP_TYPE_QUEUE: Any BPF scheduler using fixed-size BPF maps for task queueing must have a fallback path for when the map is full. The scx_qmap pattern of falling back to a custom DSQ on bpf_map_push_elem() failure is the correct approach. Failing to handle the overflow silently drops tasks from the BPF scheduler's perspective — they are still in the kernel's runnable_list and will trigger the watchdog.
ops.dequeue() semantics: qmap_dequeue() only collects statistics; it does not remove the PID from the BPF queue because BPF_MAP_TYPE_QUEUE does not support random removal. This is acknowledged in the comment: "sched_ext can handle spurious dispatches." When a dequeued task is later found by bpf_task_from_pid() to have exited, the dispatch is silently skipped. Maintainers reviewing BPF schedulers that use non-random-access data structures should check that this pattern is consistently applied.
The sleepable ops.init_task() is the only place to do per-task allocation: Any per-task BPF state that requires allocation (BPF_LOCAL_STORAGE_GET_F_CREATE) must be initialized in ops.init_task(). Attempting to allocate per-task storage in ops.enqueue() or ops.dispatch() is not safe because those callbacks run in contexts that cannot sleep.
Connection to Other Patches
scx_qmap is the primary test harness for the watchdog added in PATCH 12/30 (patch-11.md). The stall_user_nth and stall_kernel_nth variables added to scx_qmap in that patch enable deterministic triggering of watchdog conditions for testing.
The infrastructure added here (tools/sched_ext/include/scx/, user_exit_info.h, common.bpf.h) is the foundation for all subsequent example schedulers in the series: scx_central (PATCH 18/30) uses the same skeleton pattern and UEI macros. The SCX_OPS_OPEN/LOAD/ATTACH macro pattern from common.h becomes the standard lifecycle template used by all downstream schedulers in the sched-ext/scx repository.
PATCH 29/30 (documentation) references scx_simple as the starting point for newcomers learning to write BPF schedulers, making this patch the practical entry point to the entire sched_ext ecosystem.
[PATCH 11/30] sched_ext: Add sysrq-S which disables the BPF scheduler
View on Lore: https://lore.kernel.org/all/20240618212056.2833381-12-tj@kernel.org
Commit Message
This enables the admin to abort the BPF scheduler and revert to CFS anytime.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: David Vernet <dvernet@meta.com>
Acked-by: Josh Don <joshdon@google.com>
Acked-by: Hao Luo <haoluo@google.com>
Acked-by: Barret Rhoden <brho@google.com>
---
drivers/tty/sysrq.c | 1 +
kernel/sched/build_policy.c | 1 +
kernel/sched/ext.c | 20 ++++++++++++++++++++
3 files changed, 22 insertions(+)
diff --git a/drivers/tty/sysrq.c b/drivers/tty/sysrq.c
index e5974b8239c9..167e877b8bef 100644
--- a/drivers/tty/sysrq.c
+++ b/drivers/tty/sysrq.c
@@ -531,6 +531,7 @@ static const struct sysrq_key_op *sysrq_key_table[62] = {
NULL, /* P */
NULL, /* Q */
&sysrq_replay_logs_op, /* R */
+ /* S: May be registered by sched_ext for resetting */
NULL, /* S */
NULL, /* T */
NULL, /* U */
diff --git a/kernel/sched/build_policy.c b/kernel/sched/build_policy.c
index f0c148fcd2df..9223c49ddcf3 100644
--- a/kernel/sched/build_policy.c
+++ b/kernel/sched/build_policy.c
@@ -32,6 +32,7 @@
#include <linux/suspend.h>
#include <linux/tsacct_kern.h>
#include <linux/vtime.h>
+#include <linux/sysrq.h>
#include <linux/percpu-rwsem.h>
#include <uapi/linux/sched/types.h>
diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c
index 49b115f5b052..1f5d80df263a 100644
--- a/kernel/sched/ext.c
+++ b/kernel/sched/ext.c
@@ -20,6 +20,7 @@ enum scx_exit_kind {
SCX_EXIT_UNREG = 64, /* user-space initiated unregistration */
SCX_EXIT_UNREG_BPF, /* BPF-initiated unregistration */
SCX_EXIT_UNREG_KERN, /* kernel-initiated unregistration */
+ SCX_EXIT_SYSRQ, /* requested by 'S' sysrq */
SCX_EXIT_ERROR = 1024, /* runtime error, error msg contains details */
SCX_EXIT_ERROR_BPF, /* ERROR but triggered through scx_bpf_error() */
@@ -2776,6 +2777,8 @@ static const char *scx_exit_reason(enum scx_exit_kind kind)
return "Scheduler unregistered from BPF";
case SCX_EXIT_UNREG_KERN:
return "Scheduler unregistered from the main kernel";
+ case SCX_EXIT_SYSRQ:
+ return "disabled by sysrq-S";
case SCX_EXIT_ERROR:
return "runtime error";
case SCX_EXIT_ERROR_BPF:
@@ -3526,6 +3529,21 @@ static struct bpf_struct_ops bpf_sched_ext_ops = {
* System integration and init.
*/
+static void sysrq_handle_sched_ext_reset(u8 key)
+{
+ if (scx_ops_helper)
+ scx_ops_disable(SCX_EXIT_SYSRQ);
+ else
+ pr_info("sched_ext: BPF scheduler not yet used\n");
+}
+
+static const struct sysrq_key_op sysrq_sched_ext_reset_op = {
+ .handler = sysrq_handle_sched_ext_reset,
+ .help_msg = "reset-sched-ext(S)",
+ .action_msg = "Disable sched_ext and revert all tasks to CFS",
+ .enable_mask = SYSRQ_ENABLE_RTNICE,
+};
+
void __init init_sched_ext_class(void)
{
s32 cpu, v;
@@ -3549,6 +3567,8 @@ void __init init_sched_ext_class(void)
init_dsq(&rq->scx.local_dsq, SCX_DSQ_LOCAL);
INIT_LIST_HEAD(&rq->scx.runnable_list);
}
+
+ register_sysrq_key('S', &sysrq_sched_ext_reset_op);
}
--
2.45.2
Diff
---
drivers/tty/sysrq.c | 1 +
kernel/sched/build_policy.c | 1 +
kernel/sched/ext.c | 20 ++++++++++++++++++++
3 files changed, 22 insertions(+)
diff --git a/drivers/tty/sysrq.c b/drivers/tty/sysrq.c
index e5974b8239c9..167e877b8bef 100644
--- a/drivers/tty/sysrq.c
+++ b/drivers/tty/sysrq.c
@@ -531,6 +531,7 @@ static const struct sysrq_key_op *sysrq_key_table[62] = {
NULL, /* P */
NULL, /* Q */
&sysrq_replay_logs_op, /* R */
+ /* S: May be registered by sched_ext for resetting */
NULL, /* S */
NULL, /* T */
NULL, /* U */
diff --git a/kernel/sched/build_policy.c b/kernel/sched/build_policy.c
index f0c148fcd2df..9223c49ddcf3 100644
--- a/kernel/sched/build_policy.c
+++ b/kernel/sched/build_policy.c
@@ -32,6 +32,7 @@
#include <linux/suspend.h>
#include <linux/tsacct_kern.h>
#include <linux/vtime.h>
+#include <linux/sysrq.h>
#include <linux/percpu-rwsem.h>
#include <uapi/linux/sched/types.h>
diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c
index 49b115f5b052..1f5d80df263a 100644
--- a/kernel/sched/ext.c
+++ b/kernel/sched/ext.c
@@ -20,6 +20,7 @@ enum scx_exit_kind {
SCX_EXIT_UNREG = 64, /* user-space initiated unregistration */
SCX_EXIT_UNREG_BPF, /* BPF-initiated unregistration */
SCX_EXIT_UNREG_KERN, /* kernel-initiated unregistration */
+ SCX_EXIT_SYSRQ, /* requested by 'S' sysrq */
SCX_EXIT_ERROR = 1024, /* runtime error, error msg contains details */
SCX_EXIT_ERROR_BPF, /* ERROR but triggered through scx_bpf_error() */
@@ -2776,6 +2777,8 @@ static const char *scx_exit_reason(enum scx_exit_kind kind)
return "Scheduler unregistered from BPF";
case SCX_EXIT_UNREG_KERN:
return "Scheduler unregistered from the main kernel";
+ case SCX_EXIT_SYSRQ:
+ return "disabled by sysrq-S";
case SCX_EXIT_ERROR:
return "runtime error";
case SCX_EXIT_ERROR_BPF:
@@ -3526,6 +3529,21 @@ static struct bpf_struct_ops bpf_sched_ext_ops = {
* System integration and init.
*/
+static void sysrq_handle_sched_ext_reset(u8 key)
+{
+ if (scx_ops_helper)
+ scx_ops_disable(SCX_EXIT_SYSRQ);
+ else
+ pr_info("sched_ext: BPF scheduler not yet used\n");
+}
+
+static const struct sysrq_key_op sysrq_sched_ext_reset_op = {
+ .handler = sysrq_handle_sched_ext_reset,
+ .help_msg = "reset-sched-ext(S)",
+ .action_msg = "Disable sched_ext and revert all tasks to CFS",
+ .enable_mask = SYSRQ_ENABLE_RTNICE,
+};
+
void __init init_sched_ext_class(void)
{
s32 cpu, v;
@@ -3549,6 +3567,8 @@ void __init init_sched_ext_class(void)
init_dsq(&rq->scx.local_dsq, SCX_DSQ_LOCAL);
INIT_LIST_HEAD(&rq->scx.runnable_list);
}
+
+ register_sysrq_key('S', &sysrq_sched_ext_reset_op);
}
--
2.45.2
Implementation Analysis
Overview
This patch (PATCH 11/30) adds the sysrq-S escape hatch: pressing Alt+SysRq+S on a system running a BPF scheduler calls scx_ops_disable(SCX_EXIT_SYSRQ), which performs an orderly shutdown of the BPF scheduler and migrates all tasks back to CFS. The patch is small (22 lines of meaningful code across three files) but provides a critical operator safety guarantee: no matter how badly a BPF scheduler misbehaves, a human at a console can recover the system without a reboot.
Architecture Context
The sysrq subsystem is the kernel's last-resort operator interface. It is designed to function even when the system is severely degraded — it bypasses normal kernel locking and work queues and is triggered directly from the interrupt handler for the SysRq key. This makes it suitable as the manual complement to the watchdog (PATCH 12/30), which handles the automated "scheduler appears stuck" case.
The full set of sched_ext safety mechanisms forms a layered defense:
- BPF verifier (static): Prevents obviously unsafe BPF programs from loading.
- Callback return value checking (dynamic): sched_ext validates every value returned from a BPF callback and calls
scx_ops_error()if invalid. - Watchdog (automatic runtime): Detects runnable tasks that have not been scheduled for longer than
timeout_msand triggers shutdown. - sysrq-S (manual runtime): Operator-initiated shutdown when the above automatics have not fired but the system is clearly misbehaving.
This patch implements layer 4.
Code Walkthrough
drivers/tty/sysrq.c — slot reservation
/* S: May be registered by sched_ext for resetting */
NULL, /* S */
The sysrq key table uses static registration for most keys. sched_ext uses register_sysrq_key() instead because ext.c is in kernel/sched/ and cannot directly initialize the drivers/tty/sysrq.c table. The comment is added to the existing NULL entry so that developers looking at the sysrq table know slot 'S' is intentionally reserved — preventing another subsystem from claiming it.
This is a deliberate documentation-by-comment approach: the slot is not locked in any runtime sense, but the comment establishes ownership so that future maintainers do not inadvertently steal 'S' for an unrelated feature.
kernel/sched/build_policy.c — build system include
#include <linux/sysrq.h>
build_policy.c is the translation unit that includes ext.c (along with the other policy files like fair.c, rt.c, etc.) via #include. Adding sysrq.h here rather than directly in ext.c is consistent with how other kernel-wide headers are managed in the scheduler build system — build_policy.c owns the external includes so that ext.c can focus on scheduler logic.
kernel/sched/ext.c — the handler and registration
static void sysrq_handle_sched_ext_reset(u8 key)
{
if (scx_ops_helper)
scx_ops_disable(SCX_EXIT_SYSRQ);
else
pr_info("sched_ext: BPF scheduler not yet used\n");
}
static const struct sysrq_key_op sysrq_sched_ext_reset_op = {
.handler = sysrq_handle_sched_ext_reset,
.help_msg = "reset-sched-ext(S)",
.action_msg = "Disable sched_ext and revert all tasks to CFS",
.enable_mask = SYSRQ_ENABLE_RTNICE,
};
The scx_ops_helper check is the correct way to determine whether a BPF scheduler has ever been loaded. scx_ops_helper is a kthread created during scx_ops_enable() and set to NULL when no scheduler is active. Checking it avoids calling scx_ops_disable() in an inconsistent state if the key is pressed before any BPF scheduler is loaded, instead printing a diagnostic message.
The .enable_mask = SYSRQ_ENABLE_RTNICE field means this sysrq action is restricted to kernels with CONFIG_MAGIC_SYSRQ_DEFAULT_ENABLE that includes the RTNICE bit, or to systems where sysrq has been explicitly unlocked via /proc/sys/kernel/sysrq. This is a deliberate access control: the ability to forcibly terminate a production BPF scheduler should require explicit administrative intent, not be available to any user who knows the sysrq sequence.
void __init init_sched_ext_class(void)
{
/* ... per-CPU DSQ init ... */
register_sysrq_key('S', &sysrq_sched_ext_reset_op);
}
Registration happens in init_sched_ext_class(), which is called from sched_init() early in boot. The handler is therefore always registered from the moment the scheduler subsystem initializes, regardless of whether a BPF scheduler is ever loaded.
enum scx_exit_kind — new exit code
SCX_EXIT_SYSRQ, /* requested by 'S' sysrq */
Every BPF scheduler shutdown carries an scx_exit_kind value that is passed to ops.exit(). The BPF scheduler userspace binary can inspect this to distinguish a normal unload from a watchdog-triggered abort from an operator-triggered sysrq reset. This allows the userspace control plane (e.g., scx_simple.c) to log or restart appropriately.
The exit kind value SCX_EXIT_SYSRQ sits in the SCX_EXIT_UNREG range (64–1023), not the SCX_EXIT_ERROR range (1024+). This is semantically important: sysrq-S is an orderly administrative action, not an error. The BPF scheduler did not do anything wrong; the operator decided to terminate it.
Key Concepts Introduced
scx_ops_disable(kind): The central function for shutting down the BPF scheduler. It is asynchronous — it schedules a kthread work item (scx_ops_disable_workfn) rather than performing the shutdown inline. This is necessary because sysrq_handle_sched_ext_reset() is called in interrupt context where many of the operations required for an orderly shutdown (sleeping, acquiring mutexes, iterating over all tasks) are not safe. The work item runs in a kthread and can safely perform all cleanup.
SCX_EXIT_SYSRQ vs SCX_EXIT_ERROR_STALL: The distinction between these two exit codes is fundamental to how the userspace scheduler binary reacts. SCX_EXIT_SYSRQ means "operator asked us to stop" — a clean exit. SCX_EXIT_ERROR_STALL means "scheduler was detected as broken" — a fault requiring investigation. BPF schedulers should handle these differently in their ops.exit() implementation.
Why This Matters for Maintainers
The scx_ops_helper NULL check is load-bearing: The handler must check scx_ops_helper before calling scx_ops_disable(). Calling scx_ops_disable() with no active BPF scheduler would operate on uninitialized state. If this check is ever removed or changed, the exit path in scx_ops_disable_workfn() must be audited for use-after-free and null dereference.
Registration before first use: The sysrq key is registered in init_sched_ext_class() which runs at late_initcall time during boot, long before any BPF scheduler could be loaded. This means sysrq_handle_sched_ext_reset() is always registered and safe to call. If this registration is moved later (e.g., into scx_ops_enable()), there would be a window between boot and the first scheduler load during which pressing sysrq-S would silently do nothing — a regression in the safety guarantee.
The enable_mask constraint: SYSRQ_ENABLE_RTNICE is not the most permissive mask. On a locked-down system (e.g., kernel.sysrq = 0), sysrq-S will not work. This is intentional for production security posture, but it means that in environments where sysrq is disabled for security reasons, the watchdog (PATCH 12/30) is the only automated recovery mechanism. Maintainers should document this interaction.
Interaction with the disable path: scx_ops_disable_workfn() calls cancel_delayed_work_sync(&scx_watchdog_work) (added in PATCH 12/30). The ordering matters: the watchdog work must be cancelled before the BPF scheduler state is torn down, otherwise the watchdog could fire on a partially-disabled scheduler and trigger a second, spurious scx_ops_error().
Connection to Other Patches
This patch is directly paired with PATCH 12/30 (patch-11.md, the watchdog). Together they implement the two recovery modes:
- sysrq-S (this patch): Manual, operator-triggered, orderly shutdown with exit kind
SCX_EXIT_SYSRQ. - Watchdog (PATCH 12/30): Automatic, timer-triggered, error shutdown with exit kind
SCX_EXIT_ERROR_STALL.
Both ultimately call scx_ops_disable() / scx_ops_error(), which feeds into scx_ops_disable_workfn(). The disable workfn is the single chokepoint for all BPF scheduler shutdown paths, and its correctness is critical to the safety guarantee advertised in the cover letter (patch-08.md).
[PATCH 12/30] sched_ext: Implement runnable task stall watchdog
View on Lore: https://lore.kernel.org/all/20240618212056.2833381-13-tj@kernel.org
Commit Message
From: David Vernet <dvernet@meta.com>
The most common and critical way that a BPF scheduler can misbehave is by
failing to run runnable tasks for too long. This patch implements a
watchdog.
* All tasks record when they become runnable.
* A watchdog work periodically scans all runnable tasks. If any task has
stayed runnable for too long, the BPF scheduler is aborted.
* scheduler_tick() monitors whether the watchdog itself is stuck. If so, the
BPF scheduler is aborted.
Because the watchdog only scans the tasks which are currently runnable and
usually very infrequently, the overhead should be negligible.
scx_qmap is updated so that it can be told to stall user and/or
kernel tasks.
A detected task stall looks like the following:
sched_ext: BPF scheduler "qmap" errored, disabling
sched_ext: runnable task stall (dbus-daemon[953] failed to run for 6.478s)
scx_check_timeout_workfn+0x10e/0x1b0
process_one_work+0x287/0x560
worker_thread+0x234/0x420
kthread+0xe9/0x100
ret_from_fork+0x1f/0x30
A detected watchdog stall:
sched_ext: BPF scheduler "qmap" errored, disabling
sched_ext: runnable task stall (watchdog failed to check in for 5.001s)
scheduler_tick+0x2eb/0x340
update_process_times+0x7a/0x90
tick_sched_timer+0xd8/0x130
__hrtimer_run_queues+0x178/0x3b0
hrtimer_interrupt+0xfc/0x390
__sysvec_apic_timer_interrupt+0xb7/0x2b0
sysvec_apic_timer_interrupt+0x90/0xb0
asm_sysvec_apic_timer_interrupt+0x1b/0x20
default_idle+0x14/0x20
arch_cpu_idle+0xf/0x20
default_idle_call+0x50/0x90
do_idle+0xe8/0x240
cpu_startup_entry+0x1d/0x20
kernel_init+0x0/0x190
start_kernel+0x0/0x392
start_kernel+0x324/0x392
x86_64_start_reservations+0x2a/0x2c
x86_64_start_kernel+0x104/0x109
secondary_startup_64_no_verify+0xce/0xdb
Note that this patch exposes scx_ops_error[_type]() in kernel/sched/ext.h to
inline scx_notify_sched_tick().
v4: - While disabling, cancel_delayed_work_sync(&scx_watchdog_work) was
being called before forward progress was guaranteed and thus could
lead to system lockup. Relocated.
- While enabling, it was comparing msecs against jiffies without
conversion leading to spurious load failures on lower HZ kernels.
Fixed.
- runnable list management is now used by core bypass logic and moved to
the patch implementing sched_ext core.
v3: - bpf_scx_init_member() was incorrectly comparing ops->timeout_ms
against SCX_WATCHDOG_MAX_TIMEOUT which is in jiffies without
conversion leading to spurious load failures in lower HZ kernels.
Fixed.
v2: - Julia Lawall noticed that the watchdog code was mixing msecs and
jiffies. Fix by using jiffies for everything.
Signed-off-by: David Vernet <dvernet@meta.com>
Reviewed-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
Acked-by: Josh Don <joshdon@google.com>
Acked-by: Hao Luo <haoluo@google.com>
Acked-by: Barret Rhoden <brho@google.com>
Cc: Julia Lawall <julia.lawall@inria.fr>
---
include/linux/sched/ext.h | 1 +
init/init_task.c | 1 +
kernel/sched/core.c | 1 +
kernel/sched/ext.c | 130 ++++++++++++++++++++++++++++++++-
kernel/sched/ext.h | 2 +
tools/sched_ext/scx_qmap.bpf.c | 12 +++
tools/sched_ext/scx_qmap.c | 12 ++-
7 files changed, 153 insertions(+), 6 deletions(-)
diff --git a/include/linux/sched/ext.h b/include/linux/sched/ext.h
index c1530a7992cc..96031252436f 100644
--- a/include/linux/sched/ext.h
+++ b/include/linux/sched/ext.h
@@ -122,6 +122,7 @@ struct sched_ext_entity {
atomic_long_t ops_state;
struct list_head runnable_node; /* rq->scx.runnable_list */
+ unsigned long runnable_at;
u64 ddsp_dsq_id;
u64 ddsp_enq_flags;
diff --git a/init/init_task.c b/init/init_task.c
index c6804396fe12..8a44c932d10f 100644
--- a/init/init_task.c
+++ b/init/init_task.c
@@ -106,6 +106,7 @@ struct task_struct init_task __aligned(L1_CACHE_BYTES) = {
.sticky_cpu = -1,
.holding_cpu = -1,
.runnable_node = LIST_HEAD_INIT(init_task.scx.runnable_node),
+ .runnable_at = INITIAL_JIFFIES,
.ddsp_dsq_id = SCX_DSQ_INVALID,
.slice = SCX_SLICE_DFL,
},
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 6042ce3bfee0..f4365becdc13 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -5516,6 +5516,7 @@ void sched_tick(void)
calc_global_load_tick(rq);
sched_core_tick(rq);
task_tick_mm_cid(rq, curr);
+ scx_tick(rq);
rq_unlock(rq, &rf);
diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c
index 1f5d80df263a..3dc515b3351f 100644
--- a/kernel/sched/ext.c
+++ b/kernel/sched/ext.c
@@ -8,6 +8,7 @@
enum scx_consts {
SCX_DSP_DFL_MAX_BATCH = 32,
+ SCX_WATCHDOG_MAX_TIMEOUT = 30 * HZ,
SCX_EXIT_BT_LEN = 64,
SCX_EXIT_MSG_LEN = 1024,
@@ -24,6 +25,7 @@ enum scx_exit_kind {
SCX_EXIT_ERROR = 1024, /* runtime error, error msg contains details */
SCX_EXIT_ERROR_BPF, /* ERROR but triggered through scx_bpf_error() */
+ SCX_EXIT_ERROR_STALL, /* watchdog detected stalled runnable tasks */
};
/*
@@ -319,6 +321,15 @@ struct sched_ext_ops {
*/
u64 flags;
+ /**
+ * timeout_ms - The maximum amount of time, in milliseconds, that a
+ * runnable task should be able to wait before being scheduled. The
+ * maximum timeout may not exceed the default timeout of 30 seconds.
+ *
+ * Defaults to the maximum allowed timeout value of 30 seconds.
+ */
+ u32 timeout_ms;
+
/**
* name - BPF scheduler's name
*
@@ -472,6 +483,23 @@ struct static_key_false scx_has_op[SCX_OPI_END] =
static atomic_t scx_exit_kind = ATOMIC_INIT(SCX_EXIT_DONE);
static struct scx_exit_info *scx_exit_info;
+/*
+ * The maximum amount of time in jiffies that a task may be runnable without
+ * being scheduled on a CPU. If this timeout is exceeded, it will trigger
+ * scx_ops_error().
+ */
+static unsigned long scx_watchdog_timeout;
+
+/*
+ * The last time the delayed work was run. This delayed work relies on
+ * ksoftirqd being able to run to service timer interrupts, so it's possible
+ * that this work itself could get wedged. To account for this, we check that
+ * it's not stalled in the timer tick, and trigger an error if it is.
+ */
+static unsigned long scx_watchdog_timestamp = INITIAL_JIFFIES;
+
+static struct delayed_work scx_watchdog_work;
+
/* idle tracking */
#ifdef CONFIG_SMP
#ifdef CONFIG_CPUMASK_OFFSTACK
@@ -1170,6 +1198,11 @@ static void set_task_runnable(struct rq *rq, struct task_struct *p)
{
lockdep_assert_rq_held(rq);
+ if (p->scx.flags & SCX_TASK_RESET_RUNNABLE_AT) {
+ p->scx.runnable_at = jiffies;
+ p->scx.flags &= ~SCX_TASK_RESET_RUNNABLE_AT;
+ }
+
/*
* list_add_tail() must be used. scx_ops_bypass() depends on tasks being
* appened to the runnable_list.
@@ -1177,9 +1210,11 @@ static void set_task_runnable(struct rq *rq, struct task_struct *p)
list_add_tail(&p->scx.runnable_node, &rq->scx.runnable_list);
}
-static void clr_task_runnable(struct task_struct *p)
+static void clr_task_runnable(struct task_struct *p, bool reset_runnable_at)
{
list_del_init(&p->scx.runnable_node);
+ if (reset_runnable_at)
+ p->scx.flags |= SCX_TASK_RESET_RUNNABLE_AT;
}
static void enqueue_task_scx(struct rq *rq, struct task_struct *p, int enq_flags)
@@ -1217,7 +1252,8 @@ static void ops_dequeue(struct task_struct *p, u64 deq_flags)
{
unsigned long opss;
- clr_task_runnable(p);
+ /* dequeue is always temporary, don't reset runnable_at */
+ clr_task_runnable(p, false);
/* acquire ensures that we see the preceding updates on QUEUED */
opss = atomic_long_read_acquire(&p->scx.ops_state);
@@ -1826,7 +1862,7 @@ static void set_next_task_scx(struct rq *rq, struct task_struct *p, bool first)
p->se.exec_start = rq_clock_task(rq);
- clr_task_runnable(p);
+ clr_task_runnable(p, true);
}
static void put_prev_task_scx(struct rq *rq, struct task_struct *p)
@@ -2176,9 +2212,71 @@ static void reset_idle_masks(void) {}
#endif /* CONFIG_SMP */
-static void task_tick_scx(struct rq *rq, struct task_struct *curr, int queued)
+static bool check_rq_for_timeouts(struct rq *rq)
+{
+ struct task_struct *p;
+ struct rq_flags rf;
+ bool timed_out = false;
+
+ rq_lock_irqsave(rq, &rf);
+ list_for_each_entry(p, &rq->scx.runnable_list, scx.runnable_node) {
+ unsigned long last_runnable = p->scx.runnable_at;
+
+ if (unlikely(time_after(jiffies,
+ last_runnable + scx_watchdog_timeout))) {
+ u32 dur_ms = jiffies_to_msecs(jiffies - last_runnable);
+
+ scx_ops_error_kind(SCX_EXIT_ERROR_STALL,
+ "%s[%d] failed to run for %u.%03us",
+ p->comm, p->pid,
+ dur_ms / 1000, dur_ms % 1000);
+ timed_out = true;
+ break;
+ }
+ }
+ rq_unlock_irqrestore(rq, &rf);
+
+ return timed_out;
+}
+
+static void scx_watchdog_workfn(struct work_struct *work)
+{
+ int cpu;
+
+ WRITE_ONCE(scx_watchdog_timestamp, jiffies);
+
+ for_each_online_cpu(cpu) {
+ if (unlikely(check_rq_for_timeouts(cpu_rq(cpu))))
+ break;
+
+ cond_resched();
+ }
+ queue_delayed_work(system_unbound_wq, to_delayed_work(work),
+ scx_watchdog_timeout / 2);
+}
+
+void scx_tick(struct rq *rq)
{
+ unsigned long last_check;
+
+ if (!scx_enabled())
+ return;
+
+ last_check = READ_ONCE(scx_watchdog_timestamp);
+ if (unlikely(time_after(jiffies,
+ last_check + READ_ONCE(scx_watchdog_timeout)))) {
+ u32 dur_ms = jiffies_to_msecs(jiffies - last_check);
+
+ scx_ops_error_kind(SCX_EXIT_ERROR_STALL,
+ "watchdog failed to check in for %u.%03us",
+ dur_ms / 1000, dur_ms % 1000);
+ }
+
update_other_load_avgs(rq);
+}
+
+static void task_tick_scx(struct rq *rq, struct task_struct *curr, int queued)
+{
update_curr_scx(rq);
/*
@@ -2248,6 +2346,7 @@ static int scx_ops_init_task(struct task_struct *p, struct task_group *tg, bool
scx_set_task_state(p, SCX_TASK_INIT);
+ p->scx.flags |= SCX_TASK_RESET_RUNNABLE_AT;
return 0;
}
@@ -2326,6 +2425,7 @@ void init_scx_entity(struct sched_ext_entity *scx)
scx->sticky_cpu = -1;
scx->holding_cpu = -1;
INIT_LIST_HEAD(&scx->runnable_node);
+ scx->runnable_at = jiffies;
scx->ddsp_dsq_id = SCX_DSQ_INVALID;
scx->slice = SCX_SLICE_DFL;
}
@@ -2783,6 +2883,8 @@ static const char *scx_exit_reason(enum scx_exit_kind kind)
return "runtime error";
case SCX_EXIT_ERROR_BPF:
return "scx_bpf_error";
+ case SCX_EXIT_ERROR_STALL:
+ return "runnable task stall";
default:
return "<UNKNOWN>";
}
@@ -2904,6 +3006,8 @@ static void scx_ops_disable_workfn(struct kthread_work *work)
if (scx_ops.exit)
SCX_CALL_OP(SCX_KF_UNLOCKED, exit, ei);
+ cancel_delayed_work_sync(&scx_watchdog_work);
+
/*
* Delete the kobject from the hierarchy eagerly in addition to just
* dropping a reference. Otherwise, if the object is deleted
@@ -3026,6 +3130,7 @@ static int scx_ops_enable(struct sched_ext_ops *ops, struct bpf_link *link)
{
struct scx_task_iter sti;
struct task_struct *p;
+ unsigned long timeout;
int i, ret;
mutex_lock(&scx_ops_enable_mutex);
@@ -3103,6 +3208,16 @@ static int scx_ops_enable(struct sched_ext_ops *ops, struct bpf_link *link)
goto err_disable;
}
+ if (ops->timeout_ms)
+ timeout = msecs_to_jiffies(ops->timeout_ms);
+ else
+ timeout = SCX_WATCHDOG_MAX_TIMEOUT;
+
+ WRITE_ONCE(scx_watchdog_timeout, timeout);
+ WRITE_ONCE(scx_watchdog_timestamp, jiffies);
+ queue_delayed_work(system_unbound_wq, &scx_watchdog_work,
+ scx_watchdog_timeout / 2);
+
/*
* Lock out forks before opening the floodgate so that they don't wander
* into the operations prematurely.
@@ -3413,6 +3528,12 @@ static int bpf_scx_init_member(const struct btf_type *t,
if (ret == 0)
return -EINVAL;
return 1;
+ case offsetof(struct sched_ext_ops, timeout_ms):
+ if (msecs_to_jiffies(*(u32 *)(udata + moff)) >
+ SCX_WATCHDOG_MAX_TIMEOUT)
+ return -E2BIG;
+ ops->timeout_ms = *(u32 *)(udata + moff);
+ return 1;
}
return 0;
@@ -3569,6 +3690,7 @@ void __init init_sched_ext_class(void)
}
register_sysrq_key('S', &sysrq_sched_ext_reset_op);
+ INIT_DELAYED_WORK(&scx_watchdog_work, scx_watchdog_workfn);
}
diff --git a/kernel/sched/ext.h b/kernel/sched/ext.h
index 9c5a2d928281..56fcdb0b2c05 100644
--- a/kernel/sched/ext.h
+++ b/kernel/sched/ext.h
@@ -29,6 +29,7 @@ static inline bool task_on_scx(const struct task_struct *p)
return scx_enabled() && p->sched_class == &ext_sched_class;
}
+void scx_tick(struct rq *rq);
void init_scx_entity(struct sched_ext_entity *scx);
void scx_pre_fork(struct task_struct *p);
int scx_fork(struct task_struct *p);
@@ -66,6 +67,7 @@ static inline const struct sched_class *next_active_class(const struct sched_cla
#define scx_enabled() false
#define scx_switched_all() false
+static inline void scx_tick(struct rq *rq) {}
static inline void scx_pre_fork(struct task_struct *p) {}
static inline int scx_fork(struct task_struct *p) { return 0; }
static inline void scx_post_fork(struct task_struct *p) {}
diff --git a/tools/sched_ext/scx_qmap.bpf.c b/tools/sched_ext/scx_qmap.bpf.c
index 976a2693da71..8beae08dfdc7 100644
--- a/tools/sched_ext/scx_qmap.bpf.c
+++ b/tools/sched_ext/scx_qmap.bpf.c
@@ -29,6 +29,8 @@ enum consts {
char _license[] SEC("license") = "GPL";
const volatile u64 slice_ns = SCX_SLICE_DFL;
+const volatile u32 stall_user_nth;
+const volatile u32 stall_kernel_nth;
const volatile u32 dsp_batch;
u32 test_error_cnt;
@@ -129,11 +131,20 @@ static int weight_to_idx(u32 weight)
void BPF_STRUCT_OPS(qmap_enqueue, struct task_struct *p, u64 enq_flags)
{
+ static u32 user_cnt, kernel_cnt;
struct task_ctx *tctx;
u32 pid = p->pid;
int idx = weight_to_idx(p->scx.weight);
void *ring;
+ if (p->flags & PF_KTHREAD) {
+ if (stall_kernel_nth && !(++kernel_cnt % stall_kernel_nth))
+ return;
+ } else {
+ if (stall_user_nth && !(++user_cnt % stall_user_nth))
+ return;
+ }
+
if (test_error_cnt && !--test_error_cnt)
scx_bpf_error("test triggering error");
@@ -261,4 +272,5 @@ SCX_OPS_DEFINE(qmap_ops,
.init_task = (void *)qmap_init_task,
.init = (void *)qmap_init,
.exit = (void *)qmap_exit,
+ .timeout_ms = 5000U,
.name = "qmap");
diff --git a/tools/sched_ext/scx_qmap.c b/tools/sched_ext/scx_qmap.c
index 7c84ade7ecfb..6e9e9726cd62 100644
--- a/tools/sched_ext/scx_qmap.c
+++ b/tools/sched_ext/scx_qmap.c
@@ -19,10 +19,12 @@ const char help_fmt[] =
"\n"
"See the top-level comment in .bpf.c for more details.\n"
"\n"
-"Usage: %s [-s SLICE_US] [-e COUNT] [-b COUNT] [-p] [-v]\n"
+"Usage: %s [-s SLICE_US] [-e COUNT] [-t COUNT] [-T COUNT] [-b COUNT] [-p] [-v]\n"
"\n"
" -s SLICE_US Override slice duration\n"
" -e COUNT Trigger scx_bpf_error() after COUNT enqueues\n"
+" -t COUNT Stall every COUNT'th user thread\n"
+" -T COUNT Stall every COUNT'th kernel thread\n"
" -b COUNT Dispatch upto COUNT tasks together\n"
" -p Switch only tasks on SCHED_EXT policy intead of all\n"
" -v Print libbpf debug messages\n"
@@ -55,7 +57,7 @@ int main(int argc, char **argv)
skel = SCX_OPS_OPEN(qmap_ops, scx_qmap);
- while ((opt = getopt(argc, argv, "s:e:b:pvh")) != -1) {
+ while ((opt = getopt(argc, argv, "s:e:t:T:b:pvh")) != -1) {
switch (opt) {
case 's':
skel->rodata->slice_ns = strtoull(optarg, NULL, 0) * 1000;
@@ -63,6 +65,12 @@ int main(int argc, char **argv)
case 'e':
skel->bss->test_error_cnt = strtoul(optarg, NULL, 0);
break;
+ case 't':
+ skel->rodata->stall_user_nth = strtoul(optarg, NULL, 0);
+ break;
+ case 'T':
+ skel->rodata->stall_kernel_nth = strtoul(optarg, NULL, 0);
+ break;
case 'b':
skel->rodata->dsp_batch = strtoul(optarg, NULL, 0);
break;
--
2.45.2
Diff
---
include/linux/sched/ext.h | 1 +
init/init_task.c | 1 +
kernel/sched/core.c | 1 +
kernel/sched/ext.c | 130 ++++++++++++++++++++++++++++++++-
kernel/sched/ext.h | 2 +
tools/sched_ext/scx_qmap.bpf.c | 12 +++
tools/sched_ext/scx_qmap.c | 12 ++-
7 files changed, 153 insertions(+), 6 deletions(-)
diff --git a/include/linux/sched/ext.h b/include/linux/sched/ext.h
index c1530a7992cc..96031252436f 100644
--- a/include/linux/sched/ext.h
+++ b/include/linux/sched/ext.h
@@ -122,6 +122,7 @@ struct sched_ext_entity {
atomic_long_t ops_state;
struct list_head runnable_node; /* rq->scx.runnable_list */
+ unsigned long runnable_at;
u64 ddsp_dsq_id;
u64 ddsp_enq_flags;
diff --git a/init/init_task.c b/init/init_task.c
index c6804396fe12..8a44c932d10f 100644
--- a/init/init_task.c
+++ b/init/init_task.c
@@ -106,6 +106,7 @@ struct task_struct init_task __aligned(L1_CACHE_BYTES) = {
.sticky_cpu = -1,
.holding_cpu = -1,
.runnable_node = LIST_HEAD_INIT(init_task.scx.runnable_node),
+ .runnable_at = INITIAL_JIFFIES,
.ddsp_dsq_id = SCX_DSQ_INVALID,
.slice = SCX_SLICE_DFL,
},
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 6042ce3bfee0..f4365becdc13 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -5516,6 +5516,7 @@ void sched_tick(void)
calc_global_load_tick(rq);
sched_core_tick(rq);
task_tick_mm_cid(rq, curr);
+ scx_tick(rq);
rq_unlock(rq, &rf);
diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c
index 1f5d80df263a..3dc515b3351f 100644
--- a/kernel/sched/ext.c
+++ b/kernel/sched/ext.c
@@ -8,6 +8,7 @@
enum scx_consts {
SCX_DSP_DFL_MAX_BATCH = 32,
+ SCX_WATCHDOG_MAX_TIMEOUT = 30 * HZ,
SCX_EXIT_BT_LEN = 64,
SCX_EXIT_MSG_LEN = 1024,
@@ -24,6 +25,7 @@ enum scx_exit_kind {
SCX_EXIT_ERROR = 1024, /* runtime error, error msg contains details */
SCX_EXIT_ERROR_BPF, /* ERROR but triggered through scx_bpf_error() */
+ SCX_EXIT_ERROR_STALL, /* watchdog detected stalled runnable tasks */
};
/*
@@ -319,6 +321,15 @@ struct sched_ext_ops {
*/
u64 flags;
+ /**
+ * timeout_ms - The maximum amount of time, in milliseconds, that a
+ * runnable task should be able to wait before being scheduled. The
+ * maximum timeout may not exceed the default timeout of 30 seconds.
+ *
+ * Defaults to the maximum allowed timeout value of 30 seconds.
+ */
+ u32 timeout_ms;
+
/**
* name - BPF scheduler's name
*
@@ -472,6 +483,23 @@ struct static_key_false scx_has_op[SCX_OPI_END] =
static atomic_t scx_exit_kind = ATOMIC_INIT(SCX_EXIT_DONE);
static struct scx_exit_info *scx_exit_info;
+/*
+ * The maximum amount of time in jiffies that a task may be runnable without
+ * being scheduled on a CPU. If this timeout is exceeded, it will trigger
+ * scx_ops_error().
+ */
+static unsigned long scx_watchdog_timeout;
+
+/*
+ * The last time the delayed work was run. This delayed work relies on
+ * ksoftirqd being able to run to service timer interrupts, so it's possible
+ * that this work itself could get wedged. To account for this, we check that
+ * it's not stalled in the timer tick, and trigger an error if it is.
+ */
+static unsigned long scx_watchdog_timestamp = INITIAL_JIFFIES;
+
+static struct delayed_work scx_watchdog_work;
+
/* idle tracking */
#ifdef CONFIG_SMP
#ifdef CONFIG_CPUMASK_OFFSTACK
@@ -1170,6 +1198,11 @@ static void set_task_runnable(struct rq *rq, struct task_struct *p)
{
lockdep_assert_rq_held(rq);
+ if (p->scx.flags & SCX_TASK_RESET_RUNNABLE_AT) {
+ p->scx.runnable_at = jiffies;
+ p->scx.flags &= ~SCX_TASK_RESET_RUNNABLE_AT;
+ }
+
/*
* list_add_tail() must be used. scx_ops_bypass() depends on tasks being
* appened to the runnable_list.
@@ -1177,9 +1210,11 @@ static void set_task_runnable(struct rq *rq, struct task_struct *p)
list_add_tail(&p->scx.runnable_node, &rq->scx.runnable_list);
}
-static void clr_task_runnable(struct task_struct *p)
+static void clr_task_runnable(struct task_struct *p, bool reset_runnable_at)
{
list_del_init(&p->scx.runnable_node);
+ if (reset_runnable_at)
+ p->scx.flags |= SCX_TASK_RESET_RUNNABLE_AT;
}
static void enqueue_task_scx(struct rq *rq, struct task_struct *p, int enq_flags)
@@ -1217,7 +1252,8 @@ static void ops_dequeue(struct task_struct *p, u64 deq_flags)
{
unsigned long opss;
- clr_task_runnable(p);
+ /* dequeue is always temporary, don't reset runnable_at */
+ clr_task_runnable(p, false);
/* acquire ensures that we see the preceding updates on QUEUED */
opss = atomic_long_read_acquire(&p->scx.ops_state);
@@ -1826,7 +1862,7 @@ static void set_next_task_scx(struct rq *rq, struct task_struct *p, bool first)
p->se.exec_start = rq_clock_task(rq);
- clr_task_runnable(p);
+ clr_task_runnable(p, true);
}
static void put_prev_task_scx(struct rq *rq, struct task_struct *p)
@@ -2176,9 +2212,71 @@ static void reset_idle_masks(void) {}
#endif /* CONFIG_SMP */
-static void task_tick_scx(struct rq *rq, struct task_struct *curr, int queued)
+static bool check_rq_for_timeouts(struct rq *rq)
+{
+ struct task_struct *p;
+ struct rq_flags rf;
+ bool timed_out = false;
+
+ rq_lock_irqsave(rq, &rf);
+ list_for_each_entry(p, &rq->scx.runnable_list, scx.runnable_node) {
+ unsigned long last_runnable = p->scx.runnable_at;
+
+ if (unlikely(time_after(jiffies,
+ last_runnable + scx_watchdog_timeout))) {
+ u32 dur_ms = jiffies_to_msecs(jiffies - last_runnable);
+
+ scx_ops_error_kind(SCX_EXIT_ERROR_STALL,
+ "%s[%d] failed to run for %u.%03us",
+ p->comm, p->pid,
+ dur_ms / 1000, dur_ms % 1000);
+ timed_out = true;
+ break;
+ }
+ }
+ rq_unlock_irqrestore(rq, &rf);
+
+ return timed_out;
+}
+
+static void scx_watchdog_workfn(struct work_struct *work)
+{
+ int cpu;
+
+ WRITE_ONCE(scx_watchdog_timestamp, jiffies);
+
+ for_each_online_cpu(cpu) {
+ if (unlikely(check_rq_for_timeouts(cpu_rq(cpu))))
+ break;
+
+ cond_resched();
+ }
+ queue_delayed_work(system_unbound_wq, to_delayed_work(work),
+ scx_watchdog_timeout / 2);
+}
+
+void scx_tick(struct rq *rq)
{
+ unsigned long last_check;
+
+ if (!scx_enabled())
+ return;
+
+ last_check = READ_ONCE(scx_watchdog_timestamp);
+ if (unlikely(time_after(jiffies,
+ last_check + READ_ONCE(scx_watchdog_timeout)))) {
+ u32 dur_ms = jiffies_to_msecs(jiffies - last_check);
+
+ scx_ops_error_kind(SCX_EXIT_ERROR_STALL,
+ "watchdog failed to check in for %u.%03us",
+ dur_ms / 1000, dur_ms % 1000);
+ }
+
update_other_load_avgs(rq);
+}
+
+static void task_tick_scx(struct rq *rq, struct task_struct *curr, int queued)
+{
update_curr_scx(rq);
/*
@@ -2248,6 +2346,7 @@ static int scx_ops_init_task(struct task_struct *p, struct task_group *tg, bool
scx_set_task_state(p, SCX_TASK_INIT);
+ p->scx.flags |= SCX_TASK_RESET_RUNNABLE_AT;
return 0;
}
@@ -2326,6 +2425,7 @@ void init_scx_entity(struct sched_ext_entity *scx)
scx->sticky_cpu = -1;
scx->holding_cpu = -1;
INIT_LIST_HEAD(&scx->runnable_node);
+ scx->runnable_at = jiffies;
scx->ddsp_dsq_id = SCX_DSQ_INVALID;
scx->slice = SCX_SLICE_DFL;
}
@@ -2783,6 +2883,8 @@ static const char *scx_exit_reason(enum scx_exit_kind kind)
return "runtime error";
case SCX_EXIT_ERROR_BPF:
return "scx_bpf_error";
+ case SCX_EXIT_ERROR_STALL:
+ return "runnable task stall";
default:
return "<UNKNOWN>";
}
@@ -2904,6 +3006,8 @@ static void scx_ops_disable_workfn(struct kthread_work *work)
if (scx_ops.exit)
SCX_CALL_OP(SCX_KF_UNLOCKED, exit, ei);
+ cancel_delayed_work_sync(&scx_watchdog_work);
+
/*
* Delete the kobject from the hierarchy eagerly in addition to just
* dropping a reference. Otherwise, if the object is deleted
@@ -3026,6 +3130,7 @@ static int scx_ops_enable(struct sched_ext_ops *ops, struct bpf_link *link)
{
struct scx_task_iter sti;
struct task_struct *p;
+ unsigned long timeout;
int i, ret;
mutex_lock(&scx_ops_enable_mutex);
@@ -3103,6 +3208,16 @@ static int scx_ops_enable(struct sched_ext_ops *ops, struct bpf_link *link)
goto err_disable;
}
+ if (ops->timeout_ms)
+ timeout = msecs_to_jiffies(ops->timeout_ms);
+ else
+ timeout = SCX_WATCHDOG_MAX_TIMEOUT;
+
+ WRITE_ONCE(scx_watchdog_timeout, timeout);
+ WRITE_ONCE(scx_watchdog_timestamp, jiffies);
+ queue_delayed_work(system_unbound_wq, &scx_watchdog_work,
+ scx_watchdog_timeout / 2);
+
/*
* Lock out forks before opening the floodgate so that they don't wander
* into the operations prematurely.
@@ -3413,6 +3528,12 @@ static int bpf_scx_init_member(const struct btf_type *t,
if (ret == 0)
return -EINVAL;
return 1;
+ case offsetof(struct sched_ext_ops, timeout_ms):
+ if (msecs_to_jiffies(*(u32 *)(udata + moff)) >
+ SCX_WATCHDOG_MAX_TIMEOUT)
+ return -E2BIG;
+ ops->timeout_ms = *(u32 *)(udata + moff);
+ return 1;
}
return 0;
@@ -3569,6 +3690,7 @@ void __init init_sched_ext_class(void)
}
register_sysrq_key('S', &sysrq_sched_ext_reset_op);
+ INIT_DELAYED_WORK(&scx_watchdog_work, scx_watchdog_workfn);
}
diff --git a/kernel/sched/ext.h b/kernel/sched/ext.h
index 9c5a2d928281..56fcdb0b2c05 100644
--- a/kernel/sched/ext.h
+++ b/kernel/sched/ext.h
@@ -29,6 +29,7 @@ static inline bool task_on_scx(const struct task_struct *p)
return scx_enabled() && p->sched_class == &ext_sched_class;
}
+void scx_tick(struct rq *rq);
void init_scx_entity(struct sched_ext_entity *scx);
void scx_pre_fork(struct task_struct *p);
int scx_fork(struct task_struct *p);
@@ -66,6 +67,7 @@ static inline const struct sched_class *next_active_class(const struct sched_cla
#define scx_enabled() false
#define scx_switched_all() false
+static inline void scx_tick(struct rq *rq) {}
static inline void scx_pre_fork(struct task_struct *p) {}
static inline int scx_fork(struct task_struct *p) { return 0; }
static inline void scx_post_fork(struct task_struct *p) {}
diff --git a/tools/sched_ext/scx_qmap.bpf.c b/tools/sched_ext/scx_qmap.bpf.c
index 976a2693da71..8beae08dfdc7 100644
--- a/tools/sched_ext/scx_qmap.bpf.c
+++ b/tools/sched_ext/scx_qmap.bpf.c
@@ -29,6 +29,8 @@ enum consts {
char _license[] SEC("license") = "GPL";
const volatile u64 slice_ns = SCX_SLICE_DFL;
+const volatile u32 stall_user_nth;
+const volatile u32 stall_kernel_nth;
const volatile u32 dsp_batch;
u32 test_error_cnt;
@@ -129,11 +131,20 @@ static int weight_to_idx(u32 weight)
void BPF_STRUCT_OPS(qmap_enqueue, struct task_struct *p, u64 enq_flags)
{
+ static u32 user_cnt, kernel_cnt;
struct task_ctx *tctx;
u32 pid = p->pid;
int idx = weight_to_idx(p->scx.weight);
void *ring;
+ if (p->flags & PF_KTHREAD) {
+ if (stall_kernel_nth && !(++kernel_cnt % stall_kernel_nth))
+ return;
+ } else {
+ if (stall_user_nth && !(++user_cnt % stall_user_nth))
+ return;
+ }
+
if (test_error_cnt && !--test_error_cnt)
scx_bpf_error("test triggering error");
@@ -261,4 +272,5 @@ SCX_OPS_DEFINE(qmap_ops,
.init_task = (void *)qmap_init_task,
.init = (void *)qmap_init,
.exit = (void *)qmap_exit,
+ .timeout_ms = 5000U,
.name = "qmap");
diff --git a/tools/sched_ext/scx_qmap.c b/tools/sched_ext/scx_qmap.c
index 7c84ade7ecfb..6e9e9726cd62 100644
--- a/tools/sched_ext/scx_qmap.c
+++ b/tools/sched_ext/scx_qmap.c
@@ -19,10 +19,12 @@ const char help_fmt[] =
"\n"
"See the top-level comment in .bpf.c for more details.\n"
"\n"
-"Usage: %s [-s SLICE_US] [-e COUNT] [-b COUNT] [-p] [-v]\n"
+"Usage: %s [-s SLICE_US] [-e COUNT] [-t COUNT] [-T COUNT] [-b COUNT] [-p] [-v]\n"
"\n"
" -s SLICE_US Override slice duration\n"
" -e COUNT Trigger scx_bpf_error() after COUNT enqueues\n"
+" -t COUNT Stall every COUNT'th user thread\n"
+" -T COUNT Stall every COUNT'th kernel thread\n"
" -b COUNT Dispatch upto COUNT tasks together\n"
" -p Switch only tasks on SCHED_EXT policy intead of all\n"
" -v Print libbpf debug messages\n"
@@ -55,7 +57,7 @@ int main(int argc, char **argv)
skel = SCX_OPS_OPEN(qmap_ops, scx_qmap);
- while ((opt = getopt(argc, argv, "s:e:b:pvh")) != -1) {
+ while ((opt = getopt(argc, argv, "s:e:t:T:b:pvh")) != -1) {
switch (opt) {
case 's':
skel->rodata->slice_ns = strtoull(optarg, NULL, 0) * 1000;
@@ -63,6 +65,12 @@ int main(int argc, char **argv)
case 'e':
skel->bss->test_error_cnt = strtoul(optarg, NULL, 0);
break;
+ case 't':
+ skel->rodata->stall_user_nth = strtoul(optarg, NULL, 0);
+ break;
+ case 'T':
+ skel->rodata->stall_kernel_nth = strtoul(optarg, NULL, 0);
+ break;
case 'b':
skel->rodata->dsp_batch = strtoul(optarg, NULL, 0);
break;
--
2.45.2
Implementation Analysis
Overview
This patch (PATCH 12/30, authored by David Vernet) implements the runnable-task stall watchdog. It addresses the most common failure mode of a buggy BPF scheduler: forgetting to dispatch a task. If any SCX task remains runnable (enqueued but not running) for longer than scx_watchdog_timeout jiffies, the watchdog fires scx_ops_error(), which triggers an orderly BPF scheduler shutdown and falls all tasks back to CFS.
The implementation has two detection paths: a periodic delayed-work scan (the primary path) and a timer-tick secondary check that detects if the delayed work itself has become stuck (the meta-watchdog).
Architecture Context
The watchdog is a critical safety layer between the BPF verifier (which catches statically unsafe programs) and sysrq-S (which requires operator intervention). Without it, a BPF scheduler that passes verification but has a logic bug — for example, an ops.enqueue() that silently drops tasks of a certain priority — could permanently starve those tasks with no automatic recovery.
The commit message shows two canonical detection scenarios with real kernel stack traces:
- Task stall:
dbus-daemon[953] failed to run for 6.478s— detected by the delayed-work scan. - Watchdog stall:
watchdog failed to check in for 5.001s— detected by the scheduler tick, indicating that ksoftirqd itself is stuck and the delayed work cannot run.
The second scenario is particularly subtle: if the BPF scheduler is so broken that it starves ksoftirqd, the delayed work can never fire. The timer-tick meta-watchdog catches exactly this case.
Code Walkthrough
sched_ext_entity.runnable_at — the per-task timestamp
/* include/linux/sched/ext.h */
unsigned long runnable_at;
/* init/init_task.c */
.runnable_at = INITIAL_JIFFIES,
Every sched_ext_entity gains an unsigned long runnable_at field. This records the jiffies value at which the task most recently became runnable. It is initialized to INITIAL_JIFFIES (not zero) to avoid a false positive immediately after boot when jiffies is small.
SCX_TASK_RESET_RUNNABLE_AT — the lazy timestamp update
The timestamp is not updated immediately when a task is re-enqueued after running. Instead, a flag SCX_TASK_RESET_RUNNABLE_AT is set in clr_task_runnable(p, true) when a task is removed from the runnable list because it started executing:
/* in set_next_task_scx() — task is about to run */
clr_task_runnable(p, true); /* sets SCX_TASK_RESET_RUNNABLE_AT */
When the task later returns to the runnable list (via set_task_runnable()), the flag is checked:
if (p->scx.flags & SCX_TASK_RESET_RUNNABLE_AT) {
p->scx.runnable_at = jiffies;
p->scx.flags &= ~SCX_TASK_RESET_RUNNABLE_AT;
}
This lazy update design means runnable_at is stamped at the moment the task actually becomes runnable again, not at the moment it finished running. This is the correct semantics: we want to know "how long has this task been waiting to run" not "when did it last run."
By contrast, ops_dequeue() calls clr_task_runnable(p, false) — dequeue without resetting the flag. This is because dequeue is a temporary removal (e.g., for migration), not a "task ran" event. The timestamp should not be refreshed.
The global watchdog state
static unsigned long scx_watchdog_timeout;
static unsigned long scx_watchdog_timestamp = INITIAL_JIFFIES;
static struct delayed_work scx_watchdog_work;
scx_watchdog_timeout is set at BPF scheduler load time from ops->timeout_ms (or defaults to SCX_WATCHDOG_MAX_TIMEOUT = 30 * HZ if timeout_ms is zero). scx_watchdog_timestamp records when the delayed work last ran — this is what the meta-watchdog checks.
check_rq_for_timeouts() — per-CPU scan
rq_lock_irqsave(rq, &rf);
list_for_each_entry(p, &rq->scx.runnable_list, scx.runnable_node) {
unsigned long last_runnable = p->scx.runnable_at;
if (unlikely(time_after(jiffies, last_runnable + scx_watchdog_timeout))) {
scx_ops_error_kind(SCX_EXIT_ERROR_STALL,
"%s[%d] failed to run for %u.%03us", ...);
break;
}
}
rq_unlock_irqrestore(rq, &rf);
The function acquires the runqueue lock and walks rq->scx.runnable_list — the list of SCX tasks that are currently runnable on this CPU. It uses time_after() rather than a direct comparison to correctly handle jiffies wraparound. The break after the first timed-out task is correct: scx_ops_error_kind() will trigger scheduler shutdown asynchronously, so there is no point scanning further.
Note that this function holds the runqueue lock while scanning, which is safe because the list can only be modified under the runqueue lock. However, this also means the function must complete quickly — it cannot sleep or do I/O.
scx_watchdog_workfn() — the delayed work function
static void scx_watchdog_workfn(struct work_struct *work)
{
WRITE_ONCE(scx_watchdog_timestamp, jiffies);
for_each_online_cpu(cpu) {
if (unlikely(check_rq_for_timeouts(cpu_rq(cpu))))
break;
cond_resched();
}
queue_delayed_work(system_unbound_wq, to_delayed_work(work),
scx_watchdog_timeout / 2);
}
The work function updates scx_watchdog_timestamp first (before scanning), so that if the scan takes a long time, the timestamp accurately reflects when the watchdog was last active. The scan calls cond_resched() between CPUs to avoid monopolizing the kworker thread and to allow the BPF scheduler to make progress if it can.
The work is re-queued every scx_watchdog_timeout / 2 jiffies. Using half the timeout ensures the watchdog can detect a stall within the timeout period even if one iteration runs just before the stall begins.
The work runs on system_unbound_wq, which is not bound to a specific CPU. This is important: a bound work queue could end up waiting for a CPU that is itself starved by the buggy BPF scheduler.
scx_tick() — the meta-watchdog
void scx_tick(struct rq *rq)
{
unsigned long last_check;
if (!scx_enabled())
return;
last_check = READ_ONCE(scx_watchdog_timestamp);
if (unlikely(time_after(jiffies,
last_check + READ_ONCE(scx_watchdog_timeout)))) {
scx_ops_error_kind(SCX_EXIT_ERROR_STALL,
"watchdog failed to check in for %u.%03us", ...);
}
update_other_load_avgs(rq);
}
scx_tick() is called from sched_tick() on every scheduler tick on every CPU. It checks whether scx_watchdog_timestamp was updated within scx_watchdog_timeout jiffies. If not, the watchdog work itself is stuck, and the meta-watchdog triggers scx_ops_error().
Note that update_other_load_avgs(rq) was previously in task_tick_scx(). This refactoring moves load average updates into scx_tick() so they happen regardless of whether the current task is an SCX task — load averages should be updated on every tick, not just when an SCX task is running.
BPF scheduler API: ops.timeout_ms
/* struct sched_ext_ops */
u32 timeout_ms;
BPF schedulers can set timeout_ms to request a shorter watchdog timeout than the 30-second default. This is useful for testing (scx_qmap sets it to 5000ms) and for latency-sensitive schedulers that should detect stalls quickly. The maximum is SCX_WATCHDOG_MAX_TIMEOUT = 30 * HZ; attempts to set a larger value return -E2BIG from bpf_scx_init_member().
scx_ops_enable() — watchdog startup
WRITE_ONCE(scx_watchdog_timeout, timeout);
WRITE_ONCE(scx_watchdog_timestamp, jiffies);
queue_delayed_work(system_unbound_wq, &scx_watchdog_work,
scx_watchdog_timeout / 2);
The watchdog is armed as part of scx_ops_enable(), after the BPF scheduler has been verified and initialized but before tasks are migrated to SCX. The timestamp is pre-set to jiffies so that the very first tick does not see a stale value.
scx_ops_disable_workfn() — watchdog shutdown
cancel_delayed_work_sync(&scx_watchdog_work);
When the BPF scheduler is disabled, the watchdog work is cancelled synchronously before the scheduler state is torn down. cancel_delayed_work_sync() blocks until any currently-executing iteration of the work function completes. This is necessary to avoid scx_watchdog_workfn() accessing freed scheduler state after teardown.
scx_qmap test harness additions
The scx_qmap example scheduler gains stall_user_nth and stall_kernel_nth variables, settable from userspace via -t and -T flags. When set, qmap_enqueue() deliberately drops every Nth user or kernel task by returning without calling scx_bpf_dispatch():
if (p->flags & PF_KTHREAD) {
if (stall_kernel_nth && !(++kernel_cnt % stall_kernel_nth))
return; /* intentional stall — watchdog should fire */
}
This is a deliberate test vector for the watchdog. scx_qmap also sets .timeout_ms = 5000U so that tests run with a 5-second timeout rather than the 30-second default, making CI runs faster.
Key Concepts Introduced
runnable_list as the watchdog's data source: Every SCX task that is runnable (enqueued in a DSQ or in BPF-side data structures awaiting dispatch) is also on its runqueue's scx.runnable_list. The watchdog iterates this list, not the DSQs themselves, because DSQs are scheduler-private and can span multiple CPUs. The runnable_list provides a CPU-local view of all tasks that should be making progress.
Dual-path detection (work + tick): The design deliberately redundancy: the delayed work catches the common case of a task being dropped, and the ticker catches the pathological case of the work queue itself being blocked. Without the ticker, a BPF scheduler that starves ksoftirqd would defeat the primary watchdog.
SCX_EXIT_ERROR_STALL vs SCX_EXIT_SYSRQ: Stall detection triggers scx_ops_error_kind(SCX_EXIT_ERROR_STALL, ...), not scx_ops_disable(SCX_EXIT_SYSRQ). The ERROR prefix matters: it signals to the BPF scheduler's userspace binary that a fault occurred, enabling it to log diagnostic information and potentially restart with a safer configuration.
Why This Matters for Maintainers
The clr_task_runnable(p, false) vs clr_task_runnable(p, true) distinction is an invariant: true (reset the timestamp on next enqueue) must be used only when the task actually ran — set_next_task_scx(). false must be used for temporary removals — ops_dequeue(). Breaking this distinction causes either false watchdog fires (reset too eagerly) or missed stalls (reset too late).
cancel_delayed_work_sync ordering: The cancel must happen in scx_ops_disable_workfn() before any sched_ext per-task state is freed. If moved later, there is a race where the watchdog work fires after scx_watchdog_timeout has been zeroed but before the tasks' scx.runnable_node has been removed from the runnable list, causing a list walk on partially-freed state.
system_unbound_wq is not optional: If the watchdog work were queued on a bound work queue or the default system_wq, a BPF scheduler that starves all work queues on a particular CPU could prevent the watchdog from running on that CPU. system_unbound_wq workers can migrate.
timeout_ms = 0 means "use the default": The bpf_scx_init_member() validation path allows timeout_ms = 0 (zero is valid and means "use SCX_WATCHDOG_MAX_TIMEOUT"). This is checked explicitly in scx_ops_enable():
if (ops->timeout_ms)
timeout = msecs_to_jiffies(ops->timeout_ms);
else
timeout = SCX_WATCHDOG_MAX_TIMEOUT;
Reviewers should be careful that this zero-means-default semantic is not accidentally changed.
Connection to Other Patches
This patch and PATCH 11/30 (patch-10.md, sysrq-S) complete the safety layer described in the cover letter (patch-08.md). Together they ensure:
- No BPF scheduler can starve tasks indefinitely without automatic detection (this patch).
- Any operator can manually terminate a misbehaving BPF scheduler without a reboot (patch-10.md).
PATCH 19/30 later extends the watchdog to handle the case where ops.dispatch() loops without making progress — a different kind of liveness failure where tasks are being dispatched but the dispatch loop itself is spinning.
The scx_qmap test harness additions here (stall flags, 5-second timeout) are used by the selftests added in PATCH 30/30 to provide regression coverage for the watchdog functionality.
Debugging and Monitoring (Patches 13–16, 18)
Overview
A scheduler that can be loaded as a BPF program is only as useful as the operator's ability to observe it. When a BPF scheduler misbehaves — starving tasks, consuming too much CPU, entering an error state — the operator needs tools to diagnose the problem and recover. This group of five patches builds the observability and diagnostic infrastructure around the sched_ext core.
Patches 13–16 are directly about visibility: who can use the ext class, what state is printed
when the scheduler is examined, how debug output is captured at error time, and how to inspect
the live system from userspace. Patch 18 is the scx_central example scheduler, which belongs
here rather than with the core examples (patch 10) because its primary contribution is
demonstrating a multi-CPU coordination pattern that has important implications for how operators
think about centralized scheduling and its failure modes.
Why These Patches Are Needed
The core sched_ext implementation (patch 09) is designed for correctness and performance. It correctly handles the BPF scheduler lifecycle, dispatches tasks, and exits gracefully on error. But it is essentially a black box from an operator's perspective:
- How do you know which tasks are using the ext class and which are not?
- When the system appears to slow down, how do you know whether the BPF scheduler is responsible?
- When the BPF scheduler exits with an error, what was it doing at the time?
- How do you verify the scheduler is actually registered and processing tasks?
Without answers to these questions, sched_ext would be difficult to operate safely in production. These patches provide those answers at four different levels of granularity: per-task policy control, system-level state dumps, error-time debug capture, and live monitoring.
Key Concepts
PATCH 13 — Per-Task Disallow Flag (SCX_TASK_DISALLOW)
Not all tasks should use a BPF scheduler. Real-time tasks (SCHED_FIFO, SCHED_RR) and
deadline tasks (SCHED_DEADLINE) never use sched_ext — they use their own higher-priority
classes. But among SCHED_EXT tasks, there may be specific tasks that the operator or BPF
program wants to keep on CFS even when the ext class is loaded.
Patch 13 introduces SCX_TASK_DISALLOW, a flag in scx_entity.flags. When set:
enqueue_task_scx()does not callops.enqueue().- The task is forwarded to
fair_sched_class.enqueue_task()instead. - The task runs on CFS as if sched_ext were not loaded.
The flag can be set by the BPF program from within ops.enable() — this is how a BPF
scheduler can "opt out" specific tasks (e.g., watchdog threads, init, specific system daemons)
from BPF management.
This is architecturally significant: it means a BPF scheduler does not need to handle
every SCHED_EXT task. A BPF scheduler can focus on a specific workload class (e.g., only
latency-sensitive application threads) while leaving infrastructure threads on CFS.
From a maintainer perspective, SCX_TASK_DISALLOW establishes the principle that the ext class
is not an all-or-nothing switch. This has implications for every subsequent patch that adds
per-task state or transitions — each must correctly handle the disallowed case. In particular,
when the BPF scheduler is disabled and all tasks must return to CFS, disallowed tasks are already
on CFS and must not be double-migrated.
PATCH 14 — scx_dump_state() in sysrq-T / show_state()
Linux's show_state() function (triggered by Alt+SysRq+T or writing to /proc/sysrq-trigger)
dumps the state of all runnable tasks to the kernel log. This is the traditional first tool
for diagnosing scheduler problems or deadlocks.
Patch 14 adds scx_dump_state(), called from show_state() when CONFIG_SCHED_CLASS_EXT is
enabled. The output includes:
- The name of the currently loaded BPF scheduler (
ops.name). - The global DSQ depth (number of tasks waiting in
SCX_DSQ_GLOBAL). - Per-CPU local DSQ depths.
- Total number of runnable SCX tasks.
- Error state and reason if the scheduler has exited.
This information is prepended to the existing per-task state dump, giving operators an immediate summary of the ext class state before they read through individual task entries.
The implementation hooks into kernel/sched/core.c's show_state_filter(). The key challenge
is lock ordering: scx_dump_state() must not acquire any lock that show_state() might
already hold while iterating over tasks. The patch uses RCU and lock-free reads where possible,
falling back to trylock semantics for fields that require the runqueue lock. This lock-order
discipline is a recurring concern in scheduler observability code and is worth studying carefully.
PATCH 15 — ops.exit_dump_len and the Debug Ring Buffer
When a BPF scheduler exits with an error, the most valuable diagnostic information is often available only inside the BPF program — the state of BPF maps, the task that was being processed, the DSQ the program was trying to dispatch to. The kernel's own error message (e.g., "invalid DSQ ID") tells you what went wrong but not why the BPF program thought it was valid.
Patch 15 adds a debug dump mechanism:
sched_ext_ops.exit_dump_len— BPF programs set this to the number of bytes they want to allocate for debug output. Set to 0 to disable.- A ring buffer of that size is allocated when the BPF scheduler is enabled.
- The BPF program can write to this ring buffer at any time via
bpf_printk()-style helpers that target the SCX debug buffer. - When
scx_ops_error()is called, the kernel prints the lastexit_dump_lenbytes of the ring buffer to the kernel log before completing the disable sequence.
The ring buffer is a fixed-size circular buffer: if the BPF program writes more than
exit_dump_len bytes, older entries are overwritten. This means the output always contains
the most recent diagnostic information — exactly what is needed for debugging the final
moments before an error.
From a design perspective, this patch demonstrates a general principle for BPF observability: the kernel provides a fixed-size storage mechanism and a commit point (the error exit), and the BPF program is responsible for writing meaningful content. The kernel never interprets the content — it just captures and prints it verbatim.
The ring buffer is allocated during scx_ops_enable() and freed during
scx_ops_disable_workfn(). Allocation is at enable time, not error time, because at error time
the system may be under memory pressure — possibly the very cause of the error. This is a
recurring pattern in kernel error reporting infrastructure.
PATCH 16 — scx_show_state.py
While patches 14 and 15 provide kernel-level output (triggered by specific events), patch 16
provides a userspace tool for live, on-demand monitoring: tools/sched_ext/scx_show_state.py.
The script reads from /sys/kernel/debug/sched/ext (the SCX debugfs directory created by the
core patch) and formats the output for human consumption:
- Whether an SCX scheduler is currently loaded.
- The scheduler's name and when it was loaded.
- Per-CPU statistics: dispatches per second, local DSQ depth, idle time.
- Global DSQ statistics.
- Error state and reason if the scheduler has exited.
The script is intentionally simple — it is a reference implementation, not a production monitoring tool. Its value is documenting which debugfs files expose which information, making it straightforward for operators to integrate SCX monitoring into their own tooling (Prometheus exporters, grafana dashboards, systemd watchdog scripts).
For a maintainer, this script is important because it documents the debugfs interface contract.
When reviewing changes to the debugfs output format, check whether scx_show_state.py would
need to be updated and whether the change preserves backward compatibility for existing scripts.
The kernel's official stance is that debugfs interfaces are not stable, but sched_ext's debugfs
interface is explicitly documented by the Python script, creating a de facto stability expectation.
PATCH 18 — scx_central: Centralized Dispatch Pattern
scx_central is the third example BPF scheduler. Its architecture is fundamentally different
from scx_simple (global FIFO) and scx_example_qmap (per-CPU priority queues):
- One designated "central" CPU is responsible for all scheduling decisions.
- When
ops.dispatch(cpu, prev)is called on any non-central CPU, it does nothing and returns. - The central CPU's
ops.dispatch()iterates over all other CPUs, examines their local DSQ depths, and dispatches tasks to fill them usingscx_bpf_dispatch()withSCX_DSQ_LOCAL_ON(target_cpu).
This pattern is motivated by scheduling algorithms that require global visibility to make decisions — work-stealing, NUMA-aware placement, gang scheduling. On such algorithms, having each CPU make independent decisions leads to suboptimal outcomes because no single CPU has the full picture.
scx_central demonstrates several important mechanisms:
Cross-CPU dispatch: A BPF program on the central CPU dispatches tasks to other CPUs' local
DSQs. This requires SCX_DSQ_LOCAL_ON(cpu) rather than SCX_DSQ_LOCAL, and the BPF program
must handle the case where a target CPU's local DSQ is already full (the dispatch call fails
and the task stays in the user DSQ for the next dispatch cycle).
CPU affinity in dispatch: When filling another CPU's local DSQ, scx_central must respect
the task's CPU affinity mask. scx_bpf_cpumask_test_cpu() checks whether the target CPU is
allowed for the task before dispatching.
Kick idle CPUs: After filling a CPU's local DSQ, the central scheduler uses
scx_bpf_kick_cpu(cpu, SCX_KICK_IDLE) (from patch 17, the cpu-coordination group) to wake the
idle CPU so it actually picks up the dispatched task. Without this, the idle CPU might remain
in the idle loop even though its local DSQ is now non-empty.
Central CPU overhead and failure mode: The central CPU is busier than others. If the central
CPU's ops.dispatch() does not complete in time, other CPUs' local DSQs drain empty, those
CPUs have no work to run, and the watchdog (patch 12) detects the stall. This is the primary
operational failure mode for the centralized scheduling pattern, and scx_central teaches
operators to monitor the central CPU's scheduling latency as a leading indicator.
Connections Between Patches
PATCH 13 (SCX_TASK_DISALLOW)
└─→ Affects what scx_dump_state() (PATCH 14) counts as an "SCX task"
└─→ Disallowed tasks don't call ops.enqueue() so they don't write to the debug buffer
PATCH 14 (scx_dump_state)
└─→ Reads DSQ depth state that PATCH 09 core maintains
└─→ Reads error state that scx_ops_error() sets
└─→ Is the kernel-side counterpart to PATCH 16's userspace script
PATCH 15 (exit_dump_len / debug ring buffer)
└─→ Extends the error exit path from PATCH 09 (scx_ops_error)
└─→ The ring buffer content is printed before the state PATCH 14 shows
PATCH 16 (scx_show_state.py)
└─→ Reads debugfs files created by PATCH 09
└─→ Formats information that PATCH 14 dumps to kernel log
└─→ References the debug buffer output from PATCH 15
PATCH 18 (scx_central)
└─→ Requires scx_bpf_kick_cpu() from PATCH 17 (cpu-coordination)
└─→ Demonstrates the watchdog failure mode (PATCH 12) for centralized scheduling
└─→ Uses SCX_TASK_DISALLOW (PATCH 13) to keep the central CPU thread on CFS
What to Focus On
For a maintainer, the critical lessons from this group:
-
The disallow flag interaction with class transitions.
SCX_TASK_DISALLOWcreates a case where a task hasSCHED_EXTpolicy but runs onfair_sched_class. This two-level dispatch is a source of subtle bugs in transitions. When reviewing future changes to the class transition logic or the disable path, verify explicitly that disallowed tasks are handled correctly — they must not be double-migrated back to CFS when the scheduler is disabled, andops.disable()must not be called for tasks that never hadops.enable()called. -
Lock ordering in dump functions.
scx_dump_state()operates in a constrained locking environment. The pattern — RCU for reading live state, avoid runqueue locks, use trylocks with graceful fallback — must be followed in any future dump function added toext.c. Violating this will cause deadlocks onAlt+SysRq+T, which is exactly the tool operators use when the system appears hung. -
Fixed-size debug capture vs. dynamic allocation. The ring buffer in patch 15 is fixed-size and allocated at enable time, not at error time. This is correct: at error time, the system may be under memory pressure (possibly the cause of the error). Any future addition to the error reporting path in
ext.cshould follow this pre-allocation pattern. -
debugfs interface as a stability contract. The
scx_show_state.pyscript documents the debugfs interface. Once a debugfs file is consumed by external tooling, changing its format is a user-visible regression. Future changes to debugfs output must update the Python script atomically and should note the format change in the commit message. -
Centralized scheduling and the watchdog interaction.
scx_centralteaches that the watchdog timeout must be calibrated against the dispatch latency of the centralized scheduler. If the central CPU takes longer thanscx_watchdog_timeout / 2to process one dispatch round, tasks on other CPUs will appear stalled to the watchdog. Future changes to watchdog thresholds or dispatch batching must consider this interaction.
[PATCH 13/30] sched_ext: Allow BPF schedulers to disallow specific tasks from joining SCHED_EXT
View on Lore: https://lore.kernel.org/all/20240618212056.2833381-14-tj@kernel.org
Commit Message
BPF schedulers might not want to schedule certain tasks - e.g. kernel
threads. This patch adds p->scx.disallow which can be set by BPF schedulers
in such cases. The field can be changed anytime and setting it in
ops.prep_enable() guarantees that the task can never be scheduled by
sched_ext.
scx_qmap is updated with the -d option to disallow a specific PID:
# echo $$
1092
# grep -E '(policy)|(ext\.enabled)' /proc/self/sched
policy : 0
ext.enabled : 0
# ./set-scx 1092
# grep -E '(policy)|(ext\.enabled)' /proc/self/sched
policy : 7
ext.enabled : 0
Run "scx_qmap -p -d 1092" in another terminal.
# cat /sys/kernel/sched_ext/nr_rejected
1
# grep -E '(policy)|(ext\.enabled)' /proc/self/sched
policy : 0
ext.enabled : 0
# ./set-scx 1092
setparam failed for 1092 (Permission denied)
- v4: Refreshed on top of tip:sched/core.
- v3: Update description to reflect /sys/kernel/sched_ext interface change.
- v2: Use atomic_long_t instead of atomic64_t for scx_kick_cpus_pnt_seqs to
accommodate 32bit archs.
Signed-off-by: Tejun Heo <tj@kernel.org>
Suggested-by: Barret Rhoden <brho@google.com>
Reviewed-by: David Vernet <dvernet@meta.com>
Acked-by: Josh Don <joshdon@google.com>
Acked-by: Hao Luo <haoluo@google.com>
Acked-by: Barret Rhoden <brho@google.com>
---
include/linux/sched/ext.h | 12 ++++++++
kernel/sched/ext.c | 50 ++++++++++++++++++++++++++++++++++
kernel/sched/ext.h | 2 ++
kernel/sched/syscalls.c | 4 +++
tools/sched_ext/scx_qmap.bpf.c | 4 +++
tools/sched_ext/scx_qmap.c | 11 ++++++--
6 files changed, 81 insertions(+), 2 deletions(-)
diff --git a/include/linux/sched/ext.h b/include/linux/sched/ext.h
index 96031252436f..ea7c501ac819 100644
--- a/include/linux/sched/ext.h
+++ b/include/linux/sched/ext.h
@@ -137,6 +137,18 @@ struct sched_ext_entity {
*/
u64 slice;
+ /*
+ * If set, reject future sched_setscheduler(2) calls updating the policy
+ * to %SCHED_EXT with -%EACCES.
+ *
+ * If set from ops.init_task() and the task's policy is already
+ * %SCHED_EXT, which can happen while the BPF scheduler is being loaded
+ * or by inhering the parent's policy during fork, the task's policy is
+ * rejected and forcefully reverted to %SCHED_NORMAL. The number of
+ * such events are reported through /sys/kernel/debug/sched_ext::nr_rejected.
+ */
+ bool disallow; /* reject switching into SCX */
+
/* cold fields */
/* must be the last field, see init_scx_entity() */
struct list_head tasks_node;
diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c
index 3dc515b3351f..8ff30b80e862 100644
--- a/kernel/sched/ext.c
+++ b/kernel/sched/ext.c
@@ -483,6 +483,8 @@ struct static_key_false scx_has_op[SCX_OPI_END] =
static atomic_t scx_exit_kind = ATOMIC_INIT(SCX_EXIT_DONE);
static struct scx_exit_info *scx_exit_info;
+static atomic_long_t scx_nr_rejected = ATOMIC_LONG_INIT(0);
+
/*
* The maximum amount of time in jiffies that a task may be runnable without
* being scheduled on a CPU. If this timeout is exceeded, it will trigger
@@ -2332,6 +2334,8 @@ static int scx_ops_init_task(struct task_struct *p, struct task_group *tg, bool
{
int ret;
+ p->scx.disallow = false;
+
if (SCX_HAS_OP(init_task)) {
struct scx_init_task_args args = {
.fork = fork,
@@ -2346,6 +2350,27 @@ static int scx_ops_init_task(struct task_struct *p, struct task_group *tg, bool
scx_set_task_state(p, SCX_TASK_INIT);
+ if (p->scx.disallow) {
+ struct rq *rq;
+ struct rq_flags rf;
+
+ rq = task_rq_lock(p, &rf);
+
+ /*
+ * We're either in fork or load path and @p->policy will be
+ * applied right after. Reverting @p->policy here and rejecting
+ * %SCHED_EXT transitions from scx_check_setscheduler()
+ * guarantees that if ops.init_task() sets @p->disallow, @p can
+ * never be in SCX.
+ */
+ if (p->policy == SCHED_EXT) {
+ p->policy = SCHED_NORMAL;
+ atomic_long_inc(&scx_nr_rejected);
+ }
+
+ task_rq_unlock(rq, p, &rf);
+ }
+
p->scx.flags |= SCX_TASK_RESET_RUNNABLE_AT;
return 0;
}
@@ -2549,6 +2574,18 @@ static void switched_from_scx(struct rq *rq, struct task_struct *p)
static void wakeup_preempt_scx(struct rq *rq, struct task_struct *p,int wake_flags) {}
static void switched_to_scx(struct rq *rq, struct task_struct *p) {}
+int scx_check_setscheduler(struct task_struct *p, int policy)
+{
+ lockdep_assert_rq_held(task_rq(p));
+
+ /* if disallow, reject transitioning into SCX */
+ if (scx_enabled() && READ_ONCE(p->scx.disallow) &&
+ p->policy != policy && policy == SCHED_EXT)
+ return -EACCES;
+
+ return 0;
+}
+
/*
* Omitted operations:
*
@@ -2703,9 +2740,17 @@ static ssize_t scx_attr_switch_all_show(struct kobject *kobj,
}
SCX_ATTR(switch_all);
+static ssize_t scx_attr_nr_rejected_show(struct kobject *kobj,
+ struct kobj_attribute *ka, char *buf)
+{
+ return sysfs_emit(buf, "%ld\n", atomic_long_read(&scx_nr_rejected));
+}
+SCX_ATTR(nr_rejected);
+
static struct attribute *scx_global_attrs[] = {
&scx_attr_state.attr,
&scx_attr_switch_all.attr,
+ &scx_attr_nr_rejected.attr,
NULL,
};
@@ -3178,6 +3223,8 @@ static int scx_ops_enable(struct sched_ext_ops *ops, struct bpf_link *link)
atomic_set(&scx_exit_kind, SCX_EXIT_NONE);
scx_warned_zero_slice = false;
+ atomic_long_set(&scx_nr_rejected, 0);
+
/*
* Keep CPUs stable during enable so that the BPF scheduler can track
* online CPUs by watching ->on/offline_cpu() after ->init().
@@ -3476,6 +3523,9 @@ static int bpf_scx_btf_struct_access(struct bpf_verifier_log *log,
if (off >= offsetof(struct task_struct, scx.slice) &&
off + size <= offsetofend(struct task_struct, scx.slice))
return SCALAR_VALUE;
+ if (off >= offsetof(struct task_struct, scx.disallow) &&
+ off + size <= offsetofend(struct task_struct, scx.disallow))
+ return SCALAR_VALUE;
}
return -EACCES;
diff --git a/kernel/sched/ext.h b/kernel/sched/ext.h
index 56fcdb0b2c05..33a9f7fe5832 100644
--- a/kernel/sched/ext.h
+++ b/kernel/sched/ext.h
@@ -35,6 +35,7 @@ void scx_pre_fork(struct task_struct *p);
int scx_fork(struct task_struct *p);
void scx_post_fork(struct task_struct *p);
void scx_cancel_fork(struct task_struct *p);
+int scx_check_setscheduler(struct task_struct *p, int policy);
bool task_should_scx(struct task_struct *p);
void init_sched_ext_class(void);
@@ -72,6 +73,7 @@ static inline void scx_pre_fork(struct task_struct *p) {}
static inline int scx_fork(struct task_struct *p) { return 0; }
static inline void scx_post_fork(struct task_struct *p) {}
static inline void scx_cancel_fork(struct task_struct *p) {}
+static inline int scx_check_setscheduler(struct task_struct *p, int policy) { return 0; }
static inline bool task_on_scx(const struct task_struct *p) { return false; }
static inline void init_sched_ext_class(void) {}
diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
index 18d44d180db1..4fa59c9f69ac 100644
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -714,6 +714,10 @@ int __sched_setscheduler(struct task_struct *p,
goto unlock;
}
+ retval = scx_check_setscheduler(p, policy);
+ if (retval)
+ goto unlock;
+
/*
* If not changing anything there's no need to proceed further,
* but store a possible modification of reset_on_fork.
diff --git a/tools/sched_ext/scx_qmap.bpf.c b/tools/sched_ext/scx_qmap.bpf.c
index 8beae08dfdc7..5ff217c4bfa0 100644
--- a/tools/sched_ext/scx_qmap.bpf.c
+++ b/tools/sched_ext/scx_qmap.bpf.c
@@ -32,6 +32,7 @@ const volatile u64 slice_ns = SCX_SLICE_DFL;
const volatile u32 stall_user_nth;
const volatile u32 stall_kernel_nth;
const volatile u32 dsp_batch;
+const volatile s32 disallow_tgid;
u32 test_error_cnt;
@@ -243,6 +244,9 @@ void BPF_STRUCT_OPS(qmap_dispatch, s32 cpu, struct task_struct *prev)
s32 BPF_STRUCT_OPS(qmap_init_task, struct task_struct *p,
struct scx_init_task_args *args)
{
+ if (p->tgid == disallow_tgid)
+ p->scx.disallow = true;
+
/*
* @p is new. Let's ensure that its task_ctx is available. We can sleep
* in this function and the following will automatically use GFP_KERNEL.
diff --git a/tools/sched_ext/scx_qmap.c b/tools/sched_ext/scx_qmap.c
index 6e9e9726cd62..a2614994cfaa 100644
--- a/tools/sched_ext/scx_qmap.c
+++ b/tools/sched_ext/scx_qmap.c
@@ -19,13 +19,15 @@ const char help_fmt[] =
"\n"
"See the top-level comment in .bpf.c for more details.\n"
"\n"
-"Usage: %s [-s SLICE_US] [-e COUNT] [-t COUNT] [-T COUNT] [-b COUNT] [-p] [-v]\n"
+"Usage: %s [-s SLICE_US] [-e COUNT] [-t COUNT] [-T COUNT] [-b COUNT]\n"
+" [-d PID] [-p] [-v]\n"
"\n"
" -s SLICE_US Override slice duration\n"
" -e COUNT Trigger scx_bpf_error() after COUNT enqueues\n"
" -t COUNT Stall every COUNT'th user thread\n"
" -T COUNT Stall every COUNT'th kernel thread\n"
" -b COUNT Dispatch upto COUNT tasks together\n"
+" -d PID Disallow a process from switching into SCHED_EXT (-1 for self)\n"
" -p Switch only tasks on SCHED_EXT policy intead of all\n"
" -v Print libbpf debug messages\n"
" -h Display this help and exit\n";
@@ -57,7 +59,7 @@ int main(int argc, char **argv)
skel = SCX_OPS_OPEN(qmap_ops, scx_qmap);
- while ((opt = getopt(argc, argv, "s:e:t:T:b:pvh")) != -1) {
+ while ((opt = getopt(argc, argv, "s:e:t:T:b:d:pvh")) != -1) {
switch (opt) {
case 's':
skel->rodata->slice_ns = strtoull(optarg, NULL, 0) * 1000;
@@ -74,6 +76,11 @@ int main(int argc, char **argv)
case 'b':
skel->rodata->dsp_batch = strtoul(optarg, NULL, 0);
break;
+ case 'd':
+ skel->rodata->disallow_tgid = strtol(optarg, NULL, 0);
+ if (skel->rodata->disallow_tgid < 0)
+ skel->rodata->disallow_tgid = getpid();
+ break;
case 'p':
skel->struct_ops.qmap_ops->flags |= SCX_OPS_SWITCH_PARTIAL;
break;
--
2.45.2
Diff
---
include/linux/sched/ext.h | 12 ++++++++
kernel/sched/ext.c | 50 ++++++++++++++++++++++++++++++++++
kernel/sched/ext.h | 2 ++
kernel/sched/syscalls.c | 4 +++
tools/sched_ext/scx_qmap.bpf.c | 4 +++
tools/sched_ext/scx_qmap.c | 11 ++++++--
6 files changed, 81 insertions(+), 2 deletions(-)
diff --git a/include/linux/sched/ext.h b/include/linux/sched/ext.h
index 96031252436f..ea7c501ac819 100644
--- a/include/linux/sched/ext.h
+++ b/include/linux/sched/ext.h
@@ -137,6 +137,18 @@ struct sched_ext_entity {
*/
u64 slice;
+ /*
+ * If set, reject future sched_setscheduler(2) calls updating the policy
+ * to %SCHED_EXT with -%EACCES.
+ *
+ * If set from ops.init_task() and the task's policy is already
+ * %SCHED_EXT, which can happen while the BPF scheduler is being loaded
+ * or by inhering the parent's policy during fork, the task's policy is
+ * rejected and forcefully reverted to %SCHED_NORMAL. The number of
+ * such events are reported through /sys/kernel/debug/sched_ext::nr_rejected.
+ */
+ bool disallow; /* reject switching into sched_ext */
+
/* cold fields */
/* must be the last field, see init_scx_entity() */
struct list_head tasks_node;
diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c
index 3dc515b3351f..8ff30b80e862 100644
--- a/kernel/sched/ext.c
+++ b/kernel/sched/ext.c
@@ -483,6 +483,8 @@ struct static_key_false scx_has_op[SCX_OPI_END] =
static atomic_t scx_exit_kind = ATOMIC_INIT(SCX_EXIT_DONE);
static struct scx_exit_info *scx_exit_info;
+static atomic_long_t scx_nr_rejected = ATOMIC_LONG_INIT(0);
+
/*
* The maximum amount of time in jiffies that a task may be runnable without
* being scheduled on a CPU. If this timeout is exceeded, it will trigger
@@ -2332,6 +2334,8 @@ static int scx_ops_init_task(struct task_struct *p, struct task_group *tg, bool
{
int ret;
+ p->scx.disallow = false;
+
if (SCX_HAS_OP(init_task)) {
struct scx_init_task_args args = {
.fork = fork,
@@ -2346,6 +2350,27 @@ static int scx_ops_init_task(struct task_struct *p, struct task_group *tg, bool
scx_set_task_state(p, SCX_TASK_INIT);
+ if (p->scx.disallow) {
+ struct rq *rq;
+ struct rq_flags rf;
+
+ rq = task_rq_lock(p, &rf);
+
+ /*
+ * We're either in fork or load path and @p->policy will be
+ * applied right after. Reverting @p->policy here and rejecting
+ * %SCHED_EXT transitions from scx_check_setscheduler()
+ * guarantees that if ops.init_task() sets @p->disallow, @p can
+ * never be in sched_ext.
+ */
+ if (p->policy == SCHED_EXT) {
+ p->policy = SCHED_NORMAL;
+ atomic_long_inc(&scx_nr_rejected);
+ }
+
+ task_rq_unlock(rq, p, &rf);
+ }
+
p->scx.flags |= SCX_TASK_RESET_RUNNABLE_AT;
return 0;
}
@@ -2549,6 +2574,18 @@ static void switched_from_scx(struct rq *rq, struct task_struct *p)
static void wakeup_preempt_scx(struct rq *rq, struct task_struct *p,int wake_flags) {}
static void switched_to_scx(struct rq *rq, struct task_struct *p) {}
+int scx_check_setscheduler(struct task_struct *p, int policy)
+{
+ lockdep_assert_rq_held(task_rq(p));
+
+ /* if disallow, reject transitioning into sched_ext */
+ if (scx_enabled() && READ_ONCE(p->scx.disallow) &&
+ p->policy != policy && policy == SCHED_EXT)
+ return -EACCES;
+
+ return 0;
+}
+
/*
* Omitted operations:
*
@@ -2703,9 +2740,17 @@ static ssize_t scx_attr_switch_all_show(struct kobject *kobj,
}
SCX_ATTR(switch_all);
+static ssize_t scx_attr_nr_rejected_show(struct kobject *kobj,
+ struct kobj_attribute *ka, char *buf)
+{
+ return sysfs_emit(buf, "%ld\n", atomic_long_read(&scx_nr_rejected));
+}
+SCX_ATTR(nr_rejected);
+
static struct attribute *scx_global_attrs[] = {
&scx_attr_state.attr,
&scx_attr_switch_all.attr,
+ &scx_attr_nr_rejected.attr,
NULL,
};
@@ -3178,6 +3223,8 @@ static int scx_ops_enable(struct sched_ext_ops *ops, struct bpf_link *link)
atomic_set(&scx_exit_kind, SCX_EXIT_NONE);
scx_warned_zero_slice = false;
+ atomic_long_set(&scx_nr_rejected, 0);
+
/*
* Keep CPUs stable during enable so that the BPF scheduler can track
* online CPUs by watching ->on/offline_cpu() after ->init().
@@ -3476,6 +3523,9 @@ static int bpf_scx_btf_struct_access(struct bpf_verifier_log *log,
if (off >= offsetof(struct task_struct, scx.slice) &&
off + size <= offsetofend(struct task_struct, scx.slice))
return SCALAR_VALUE;
+ if (off >= offsetof(struct task_struct, scx.disallow) &&
+ off + size <= offsetofend(struct task_struct, scx.disallow))
+ return SCALAR_VALUE;
}
return -EACCES;
diff --git a/kernel/sched/ext.h b/kernel/sched/ext.h
index 56fcdb0b2c05..33a9f7fe5832 100644
--- a/kernel/sched/ext.h
+++ b/kernel/sched/ext.h
@@ -35,6 +35,7 @@ void scx_pre_fork(struct task_struct *p);
int scx_fork(struct task_struct *p);
void scx_post_fork(struct task_struct *p);
void scx_cancel_fork(struct task_struct *p);
+int scx_check_setscheduler(struct task_struct *p, int policy);
bool task_should_scx(struct task_struct *p);
void init_sched_ext_class(void);
@@ -72,6 +73,7 @@ static inline void scx_pre_fork(struct task_struct *p) {}
static inline int scx_fork(struct task_struct *p) { return 0; }
static inline void scx_post_fork(struct task_struct *p) {}
static inline void scx_cancel_fork(struct task_struct *p) {}
+static inline int scx_check_setscheduler(struct task_struct *p, int policy) { return 0; }
static inline bool task_on_scx(const struct task_struct *p) { return false; }
static inline void init_sched_ext_class(void) {}
diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
index 18d44d180db1..4fa59c9f69ac 100644
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -714,6 +714,10 @@ int __sched_setscheduler(struct task_struct *p,
goto unlock;
}
+ retval = scx_check_setscheduler(p, policy);
+ if (retval)
+ goto unlock;
+
/*
* If not changing anything there's no need to proceed further,
* but store a possible modification of reset_on_fork.
diff --git a/tools/sched_ext/scx_qmap.bpf.c b/tools/sched_ext/scx_qmap.bpf.c
index 8beae08dfdc7..5ff217c4bfa0 100644
--- a/tools/sched_ext/scx_qmap.bpf.c
+++ b/tools/sched_ext/scx_qmap.bpf.c
@@ -32,6 +32,7 @@ const volatile u64 slice_ns = SCX_SLICE_DFL;
const volatile u32 stall_user_nth;
const volatile u32 stall_kernel_nth;
const volatile u32 dsp_batch;
+const volatile s32 disallow_tgid;
u32 test_error_cnt;
@@ -243,6 +244,9 @@ void BPF_STRUCT_OPS(qmap_dispatch, s32 cpu, struct task_struct *prev)
s32 BPF_STRUCT_OPS(qmap_init_task, struct task_struct *p,
struct scx_init_task_args *args)
{
+ if (p->tgid == disallow_tgid)
+ p->scx.disallow = true;
+
/*
* @p is new. Let's ensure that its task_ctx is available. We can sleep
* in this function and the following will automatically use GFP_KERNEL.
diff --git a/tools/sched_ext/scx_qmap.c b/tools/sched_ext/scx_qmap.c
index 6e9e9726cd62..a2614994cfaa 100644
--- a/tools/sched_ext/scx_qmap.c
+++ b/tools/sched_ext/scx_qmap.c
@@ -19,13 +19,15 @@ const char help_fmt[] =
"\n"
"See the top-level comment in .bpf.c for more details.\n"
"\n"
-"Usage: %s [-s SLICE_US] [-e COUNT] [-t COUNT] [-T COUNT] [-b COUNT] [-p] [-v]\n"
+"Usage: %s [-s SLICE_US] [-e COUNT] [-t COUNT] [-T COUNT] [-b COUNT]\n"
+" [-d PID] [-p] [-v]\n"
"\n"
" -s SLICE_US Override slice duration\n"
" -e COUNT Trigger scx_bpf_error() after COUNT enqueues\n"
" -t COUNT Stall every COUNT'th user thread\n"
" -T COUNT Stall every COUNT'th kernel thread\n"
" -b COUNT Dispatch upto COUNT tasks together\n"
+" -d PID Disallow a process from switching into SCHED_EXT (-1 for self)\n"
" -p Switch only tasks on SCHED_EXT policy intead of all\n"
" -v Print libbpf debug messages\n"
" -h Display this help and exit\n";
@@ -57,7 +59,7 @@ int main(int argc, char **argv)
skel = SCX_OPS_OPEN(qmap_ops, scx_qmap);
- while ((opt = getopt(argc, argv, "s:e:t:T:b:pvh")) != -1) {
+ while ((opt = getopt(argc, argv, "s:e:t:T:b:d:pvh")) != -1) {
switch (opt) {
case 's':
skel->rodata->slice_ns = strtoull(optarg, NULL, 0) * 1000;
@@ -74,6 +76,11 @@ int main(int argc, char **argv)
case 'b':
skel->rodata->dsp_batch = strtoul(optarg, NULL, 0);
break;
+ case 'd':
+ skel->rodata->disallow_tgid = strtol(optarg, NULL, 0);
+ if (skel->rodata->disallow_tgid < 0)
+ skel->rodata->disallow_tgid = getpid();
+ break;
case 'p':
skel->struct_ops.qmap_ops->flags |= SCX_OPS_SWITCH_PARTIAL;
break;
--
2.45.2
Implementation Analysis
Overview
This patch lets BPF schedulers permanently exclude specific tasks from SCHED_EXT. The motivating case is kernel threads that must not be managed by an arbitrary BPF scheduler (e.g., a kthread that is critical for boot or PM operations). The mechanism is a single boolean p->scx.disallow on sched_ext_entity. When a BPF scheduler sets this flag during ops.init_task(), the kernel reverts the task's policy to SCHED_NORMAL and blocks any future sched_setscheduler(2) calls that would move it to SCHED_EXT.
Code Walkthrough
include/linux/sched/ext.h — the disallow field
bool disallow; /* reject switching into sched_ext */
Added to sched_ext_entity after slice. The comment explains two distinct effects: (1) reject future sched_setscheduler calls with -EACCES, and (2) if set during ops.init_task() while the task already has SCHED_EXT policy (possible during BPF scheduler load or via fork inheriting the parent's policy), force-revert the policy to SCHED_NORMAL. The counter scx_nr_rejected tracks how many such forced reversions happen and is exposed at /sys/kernel/debug/sched_ext::nr_rejected.
kernel/sched/ext.c — enforcing disallow in scx_ops_init_task()
p->scx.disallow = false;
if (SCX_HAS_OP(init_task)) {
...
SCX_CALL_OP_RET(SCX_KF_SLEEPABLE, init_task, p, &args);
...
}
scx_set_task_state(p, SCX_TASK_INIT);
if (p->scx.disallow) {
struct rq *rq;
struct rq_flags rf;
rq = task_rq_lock(p, &rf);
if (p->policy == SCHED_EXT) {
p->policy = SCHED_NORMAL;
atomic_long_inc(&scx_nr_rejected);
}
task_rq_unlock(rq, p, &rf);
}
The flag is explicitly cleared before calling ops.init_task() so stale values cannot persist. After the BPF callback returns, the kernel checks the flag and, if set, acquires the task's rq lock to safely modify p->policy. This is important: the caller is either in fork or in the load path and p->policy is about to be applied, so reverting it here ensures the task never actually enters the ext class.
kernel/sched/ext.c — scx_check_setscheduler()
int scx_check_setscheduler(struct task_struct *p, int policy)
{
lockdep_assert_rq_held(task_rq(p));
if (scx_enabled() && READ_ONCE(p->scx.disallow) &&
p->policy != policy && policy == SCHED_EXT)
return -EACCES;
return 0;
}
This new function is called from __sched_setscheduler() in kernel/sched/syscalls.c (after the existing capability check, before the "nothing to do" early-return). The lockdep assertion documents that rq->lock must be held by the caller, which __sched_setscheduler() guarantees. The READ_ONCE is used because disallow can be set from BPF context on a different CPU without holding any lock.
kernel/sched/ext.c — BPF BTF accessor
if (off >= offsetof(struct task_struct, scx.disallow) &&
off + size <= offsetofend(struct task_struct, scx.disallow))
return SCALAR_VALUE;
Added to bpf_scx_btf_struct_access() so BPF programs can write p->scx.disallow directly from within ops.init_task(). Without this, the BPF verifier would reject the store.
tools/sched_ext/scx_qmap.bpf.c — example usage
s32 BPF_STRUCT_OPS(qmap_init_task, struct task_struct *p,
struct scx_init_task_args *args)
{
if (p->tgid == disallow_tgid)
p->scx.disallow = true;
...
}
The BPF scheduler checks if the task's TGID matches a configured value and sets disallow. This demonstrates the intended usage pattern: set the flag in ops.init_task() to guarantee the task can never run under SCHED_EXT.
Key Concepts
p->scx.disallow: A boolean insched_ext_entitythat acts as a per-task veto on SCHED_EXT membership. The flag is checked in two places:scx_ops_init_task()(for new tasks) andscx_check_setscheduler()(forsched_setscheduler(2)calls).scx_nr_rejected: Anatomic_long_tcounter incremented whenever a task is force-reverted from SCHED_EXT to SCHED_NORMAL due todisallowbeing set. It resets on each BPF scheduler load (scx_ops_enable()). Exposed via sysfs at/sys/kernel/sched_ext/nr_rejected.scx_check_setscheduler(): A new hook inserted into the policy-change syscall path. It is the runtime guard that preventsdisallowed tasks from being moved to SCHED_EXT after initial setup.- Two-path enforcement: The
disallowflag is checked both atinit_tasktime (for tasks that inherit policy or are already SCHED_EXT when the BPF scheduler loads) and atsched_setschedulertime (for future attempts). This two-path design is what makes the guarantee watertight.
Locking and Concurrency Notes
- When
disallowis checked inscx_ops_init_task(), the code acquirestask_rq_lock(p, &rf)before modifyingp->policy. This is the correct protocol for changing a task's scheduling policy outside the full setscheduler path. scx_check_setscheduler()useslockdep_assert_rq_held(task_rq(p))to document that it always runs under the task's rq lock. This is satisfied because__sched_setscheduler()holds the lock before calling it.READ_ONCE(p->scx.disallow)inscx_check_setscheduler()provides ordering against concurrent BPF writes fromops.init_task()which may run without any lock.
Why Maintainers Need to Know This
- The
disallowflag is racy by design: The comment in the header explicitly says the flag "can be changed anytime". The guarantee is weaker than a lock: it is possible for a window to exist betweenops.init_task()returning and the check inscx_ops_init_task(). The two-path check (init_task + check_setscheduler) is the mitigation. - Setting
disallowoutsideops.init_task()does not revoke SCHED_EXT: If a BPF scheduler setsp->scx.disallow = trueafter the task is already running under SCHED_EXT (e.g., fromops.enqueue()), the kernel does not immediately demote the task. It only blocks futuresched_setschedulertransitions. A maintainer reviewing BPF schedulers should flag any attempt to usedisallowas a "revoke" mechanism outsideops.init_task(). nr_rejectedresets on each BPF scheduler load: Monitoring tools should not treat this counter as a cumulative system-wide metric across scheduler restarts.- BTF write access is required: Any new
sched_ext_entityfield that BPF schedulers need to write fromops.init_task()must be explicitly added tobpf_scx_btf_struct_access(). This is a common oversight when extending the interface.
Connection to Other Patches
- Builds on the
ops.init_task()callback from earlier patches that first allowed BPF schedulers to reject tasks by returning an error code; this patch adds a complementary per-task flag that survives initial setup. scx_nr_rejectedfeeds into thescx_show_state.pydebugging tool added in PATCH 16/30, which reads the counter via drgn.- The BTF accessor pattern used here (
bpf_scx_btf_struct_access()) is the same mechanism used to exposep->scx.sliceto BPF; maintainers adding new writable fields tosched_ext_entitymust follow this pattern.
[PATCH 14/30] sched_ext: Print sched_ext info when dumping stack
View on Lore: https://lore.kernel.org/all/20240618212056.2833381-15-tj@kernel.org
Commit Message
From: David Vernet <void@manifault.com>
It would be useful to see what the sched_ext scheduler state is, and what
scheduler is running, when we're dumping a task's stack. This patch
therefore adds a new print_scx_info() function that's called in the same
context as print_worker_info() and print_stop_info(). An example dump
follows.
BUG: kernel NULL pointer dereference, address: 0000000000000999
#PF: supervisor write access in kernel mode
#PF: error_code(0x0002) - not-present page
PGD 0 P4D 0
Oops: 0002 [#1] PREEMPT SMP
CPU: 13 PID: 2047 Comm: insmod Tainted: G O 6.6.0-work-10323-gb58d4cae8e99-dirty #34
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS unknown 2/2/2022
Sched_ext: qmap (enabled+all), task: runnable_at=-17ms
RIP: 0010:init_module+0x9/0x1000 [test_module]
...
v3: - scx_ops_enable_state_str[] definition moved to an earlier patch as
it's now used by core implementation.
- Convert jiffy delta to msecs using jiffies_to_msecs() instead of
multiplying by (HZ / MSEC_PER_SEC). The conversion is implemented in
jiffies_delta_msecs().
v2: - We are now using scx_ops_enable_state_str[] outside
CONFIG_SCHED_DEBUG. Move it outside of CONFIG_SCHED_DEBUG and to the
top. This was reported by Changwoo and Andrea.
Signed-off-by: David Vernet <void@manifault.com>
Reported-by: Changwoo Min <changwoo@igalia.com>
Reported-by: Andrea Righi <andrea.righi@canonical.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
---
include/linux/sched/ext.h | 2 ++
kernel/sched/core.c | 1 +
kernel/sched/ext.c | 53 +++++++++++++++++++++++++++++++++++++++
lib/dump_stack.c | 1 +
4 files changed, 57 insertions(+)
diff --git a/include/linux/sched/ext.h b/include/linux/sched/ext.h
index ea7c501ac819..85fb5dc725ef 100644
--- a/include/linux/sched/ext.h
+++ b/include/linux/sched/ext.h
@@ -155,10 +155,12 @@ struct sched_ext_entity {
};
void sched_ext_free(struct task_struct *p);
+void print_scx_info(const char *log_lvl, struct task_struct *p);
#else /* !CONFIG_SCHED_CLASS_EXT */
static inline void sched_ext_free(struct task_struct *p) {}
+static inline void print_scx_info(const char *log_lvl, struct task_struct *p) {}
#endif /* CONFIG_SCHED_CLASS_EXT */
#endif /* _LINUX_SCHED_EXT_H */
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index f4365becdc13..1a3144c80af8 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -7486,6 +7486,7 @@ void sched_show_task(struct task_struct *p)
print_worker_info(KERN_INFO, p);
print_stop_info(KERN_INFO, p);
+ print_scx_info(KERN_INFO, p);
show_stack(p, NULL, KERN_INFO);
put_task_stack(p);
}
diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c
index 8ff30b80e862..6f4de29d7372 100644
--- a/kernel/sched/ext.c
+++ b/kernel/sched/ext.c
@@ -586,6 +586,14 @@ static __printf(3, 4) void scx_ops_exit_kind(enum scx_exit_kind kind,
#define SCX_HAS_OP(op) static_branch_likely(&scx_has_op[SCX_OP_IDX(op)])
+static long jiffies_delta_msecs(unsigned long at, unsigned long now)
+{
+ if (time_after(at, now))
+ return jiffies_to_msecs(at - now);
+ else
+ return -(long)jiffies_to_msecs(now - at);
+}
+
/* if the highest set bit is N, return a mask with bits [N+1, 31] set */
static u32 higher_bits(u32 flags)
{
@@ -3715,6 +3723,51 @@ static const struct sysrq_key_op sysrq_sched_ext_reset_op = {
.enable_mask = SYSRQ_ENABLE_RTNICE,
};
+/**
+ * print_scx_info - print out sched_ext scheduler state
+ * @log_lvl: the log level to use when printing
+ * @p: target task
+ *
+ * If a sched_ext scheduler is enabled, print the name and state of the
+ * scheduler. If @p is on sched_ext, print further information about the task.
+ *
+ * This function can be safely called on any task as long as the task_struct
+ * itself is accessible. While safe, this function isn't synchronized and may
+ * print out mixups or garbages of limited length.
+ */
+void print_scx_info(const char *log_lvl, struct task_struct *p)
+{
+ enum scx_ops_enable_state state = scx_ops_enable_state();
+ const char *all = READ_ONCE(scx_switching_all) ? "+all" : "";
+ char runnable_at_buf[22] = "?";
+ struct sched_class *class;
+ unsigned long runnable_at;
+
+ if (state == SCX_OPS_DISABLED)
+ return;
+
+ /*
+ * Carefully check if the task was running on sched_ext, and then
+ * carefully copy the time it's been runnable, and its state.
+ */
+ if (copy_from_kernel_nofault(&class, &p->sched_class, sizeof(class)) ||
+ class != &ext_sched_class) {
+ printk("%sSched_ext: %s (%s%s)", log_lvl, scx_ops.name,
+ scx_ops_enable_state_str[state], all);
+ return;
+ }
+
+ if (!copy_from_kernel_nofault(&runnable_at, &p->scx.runnable_at,
+ sizeof(runnable_at)))
+ scnprintf(runnable_at_buf, sizeof(runnable_at_buf), "%+ldms",
+ jiffies_delta_msecs(runnable_at, jiffies));
+
+ /* print everything onto one line to conserve console space */
+ printk("%sSched_ext: %s (%s%s), task: runnable_at=%s",
+ log_lvl, scx_ops.name, scx_ops_enable_state_str[state], all,
+ runnable_at_buf);
+}
+
void __init init_sched_ext_class(void)
{
s32 cpu, v;
diff --git a/lib/dump_stack.c b/lib/dump_stack.c
index 222c6d6c8281..9581ef4efec5 100644
--- a/lib/dump_stack.c
+++ b/lib/dump_stack.c
@@ -68,6 +68,7 @@ void dump_stack_print_info(const char *log_lvl)
print_worker_info(log_lvl, current);
print_stop_info(log_lvl, current);
+ print_scx_info(log_lvl, current);
}
/**
--
2.45.2
Diff
---
include/linux/sched/ext.h | 2 ++
kernel/sched/core.c | 1 +
kernel/sched/ext.c | 53 +++++++++++++++++++++++++++++++++++++++
lib/dump_stack.c | 1 +
4 files changed, 57 insertions(+)
diff --git a/include/linux/sched/ext.h b/include/linux/sched/ext.h
index ea7c501ac819..85fb5dc725ef 100644
--- a/include/linux/sched/ext.h
+++ b/include/linux/sched/ext.h
@@ -155,10 +155,12 @@ struct sched_ext_entity {
};
void sched_ext_free(struct task_struct *p);
+void print_scx_info(const char *log_lvl, struct task_struct *p);
#else /* !CONFIG_SCHED_CLASS_EXT */
static inline void sched_ext_free(struct task_struct *p) {}
+static inline void print_scx_info(const char *log_lvl, struct task_struct *p) {}
#endif /* CONFIG_SCHED_CLASS_EXT */
#endif /* _LINUX_SCHED_EXT_H */
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index f4365becdc13..1a3144c80af8 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -7486,6 +7486,7 @@ void sched_show_task(struct task_struct *p)
print_worker_info(KERN_INFO, p);
print_stop_info(KERN_INFO, p);
+ print_scx_info(KERN_INFO, p);
show_stack(p, NULL, KERN_INFO);
put_task_stack(p);
}
diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c
index 8ff30b80e862..6f4de29d7372 100644
--- a/kernel/sched/ext.c
+++ b/kernel/sched/ext.c
@@ -586,6 +586,14 @@ static __printf(3, 4) void scx_ops_exit_kind(enum scx_exit_kind kind,
#define SCX_HAS_OP(op) static_branch_likely(&scx_has_op[SCX_OP_IDX(op)])
+static long jiffies_delta_msecs(unsigned long at, unsigned long now)
+{
+ if (time_after(at, now))
+ return jiffies_to_msecs(at - now);
+ else
+ return -(long)jiffies_to_msecs(now - at);
+}
+
/* if the highest set bit is N, return a mask with bits [N+1, 31] set */
static u32 higher_bits(u32 flags)
{
@@ -3715,6 +3723,51 @@ static const struct sysrq_key_op sysrq_sched_ext_reset_op = {
.enable_mask = SYSRQ_ENABLE_RTNICE,
};
+/**
+ * print_scx_info - print out sched_ext scheduler state
+ * @log_lvl: the log level to use when printing
+ * @p: target task
+ *
+ * If a sched_ext scheduler is enabled, print the name and state of the
+ * scheduler. If @p is on sched_ext, print further information about the task.
+ *
+ * This function can be safely called on any task as long as the task_struct
+ * itself is accessible. While safe, this function isn't synchronized and may
+ * print out mixups or garbages of limited length.
+ */
+void print_scx_info(const char *log_lvl, struct task_struct *p)
+{
+ enum scx_ops_enable_state state = scx_ops_enable_state();
+ const char *all = READ_ONCE(scx_switching_all) ? "+all" : "";
+ char runnable_at_buf[22] = "?";
+ struct sched_class *class;
+ unsigned long runnable_at;
+
+ if (state == SCX_OPS_DISABLED)
+ return;
+
+ /*
+ * Carefully check if the task was running on sched_ext, and then
+ * carefully copy the time it's been runnable, and its state.
+ */
+ if (copy_from_kernel_nofault(&class, &p->sched_class, sizeof(class)) ||
+ class != &ext_sched_class) {
+ printk("%sSched_ext: %s (%s%s)", log_lvl, scx_ops.name,
+ scx_ops_enable_state_str[state], all);
+ return;
+ }
+
+ if (!copy_from_kernel_nofault(&runnable_at, &p->scx.runnable_at,
+ sizeof(runnable_at)))
+ scnprintf(runnable_at_buf, sizeof(runnable_at_buf), "%+ldms",
+ jiffies_delta_msecs(runnable_at, jiffies));
+
+ /* print everything onto one line to conserve console space */
+ printk("%sSched_ext: %s (%s%s), task: runnable_at=%s",
+ log_lvl, scx_ops.name, scx_ops_enable_state_str[state], all,
+ runnable_at_buf);
+}
+
void __init init_sched_ext_class(void)
{
s32 cpu, v;
diff --git a/lib/dump_stack.c b/lib/dump_stack.c
index 222c6d6c8281..9581ef4efec5 100644
--- a/lib/dump_stack.c
+++ b/lib/dump_stack.c
@@ -68,6 +68,7 @@ void dump_stack_print_info(const char *log_lvl)
print_worker_info(log_lvl, current);
print_stop_info(log_lvl, current);
+ print_scx_info(log_lvl, current);
}
/**
--
2.45.2
Implementation Analysis
Overview
When a kernel panic, oops, or task stack dump (sysrq-T) occurs, it is extremely useful to know which BPF scheduler is loaded and how that scheduler sees the task being dumped. This patch adds print_scx_info(), a function called from the same context as print_worker_info() and print_stop_info(), that emits sched_ext state as part of the standard stack dump line. The function is explicitly designed to be safe to call without any locks, at any time the task_struct pointer is valid, even in oops context.
Code Walkthrough
include/linux/sched/ext.h — declaration
Two declarations are added: the real print_scx_info() under CONFIG_SCHED_CLASS_EXT and a no-op stub when the config is absent. This is the standard pattern for optional scheduler features that get called from kernel/sched/core.c regardless of config.
kernel/sched/core.c and lib/dump_stack.c — call sites
// sched_show_task() in core.c — called by sysrq-T and /proc/<pid>/stack
print_worker_info(KERN_INFO, p);
print_stop_info(KERN_INFO, p);
print_scx_info(KERN_INFO, p); // NEW
show_stack(p, NULL, KERN_INFO);
// dump_stack_print_info() in lib/dump_stack.c — called on oops/panic
print_worker_info(log_lvl, current);
print_stop_info(log_lvl, current);
print_scx_info(log_lvl, current); // NEW
Two call sites are added so that SCX state appears in both task-specific dumps (sysrq-T iterates tasks and calls sched_show_task()) and the current-task dump on oops (via dump_stack_print_info()).
kernel/sched/ext.c — jiffies_delta_msecs() helper
static long jiffies_delta_msecs(unsigned long at, unsigned long now)
{
if (time_after(at, now))
return jiffies_to_msecs(at - now);
else
return -(long)jiffies_to_msecs(now - at);
}
A signed millisecond delta from a jiffies timestamp. A positive value means the event is in the future; negative means it is in the past. Used to show how long a task has been waiting to run (runnable_at).
kernel/sched/ext.c — print_scx_info()
void print_scx_info(const char *log_lvl, struct task_struct *p)
{
enum scx_ops_enable_state state = scx_ops_enable_state();
const char *all = READ_ONCE(scx_switching_all) ? "+all" : "";
char runnable_at_buf[22] = "?";
struct sched_class *class;
unsigned long runnable_at;
if (state == SCX_OPS_DISABLED)
return;
if (copy_from_kernel_nofault(&class, &p->sched_class, sizeof(class)) ||
class != &ext_sched_class) {
printk("%sSched_ext: %s (%s%s)", log_lvl, scx_ops.name,
scx_ops_enable_state_str[state], all);
return;
}
if (!copy_from_kernel_nofault(&runnable_at, &p->scx.runnable_at,
sizeof(runnable_at)))
scnprintf(runnable_at_buf, sizeof(runnable_at_buf), "%+ldms",
jiffies_delta_msecs(runnable_at, jiffies));
printk("%sSched_ext: %s (%s%s), task: runnable_at=%s",
log_lvl, scx_ops.name, scx_ops_enable_state_str[state], all,
runnable_at_buf);
}
Key design decisions:
copy_from_kernel_nofault()is used instead of direct dereference because this function may be called in contexts where thetask_structmemory is partially or fully inaccessible (e.g., after a use-after-free or during an oops where memory is in an unknown state). The nofault variant returns an error instead of faulting.- If the task is not an SCX task, only the scheduler name and state are printed (no per-task fields). This is still useful: you learn which BPF scheduler was loaded and in what state.
- If the task is an SCX task, the
runnable_atjiffies timestamp is read (again with nofault) and converted to a signed millisecond offset from "now". A value like-17msmeans the task has been waiting to run for 17 milliseconds. - Everything is printed on one line to conserve console space in oops output.
Key Concepts
scx_ops_enable_state_str[]: A string array mappingenum scx_ops_enable_statevalues (SCX_OPS_PREPPING,SCX_OPS_ENABLING,SCX_OPS_ENABLED,SCX_OPS_DISABLING,SCX_OPS_DISABLED) to human-readable names. This array was moved to an earlier patch because it is now needed outsideCONFIG_SCHED_DEBUG.scx_switching_all: A flag indicating whether all tasks are under the BPF scheduler (vs. only tasks withSCHED_EXTpolicy). The+allsuffix in the output reflects this.p->scx.runnable_at: A jiffies timestamp set when the task becomes runnable. The delta shown in the dump tells you how long the task has been stuck waiting — critical for diagnosing starvation or dispatch bugs.copy_from_kernel_nofault(): Safe dereference that handles faults gracefully. The comment in the implementation explicitly notes the function is "not synchronized and may print out mixups or garbages of limited length" — this is an acceptable trade-off for a debugging aid.
Locking and Concurrency Notes
print_scx_info() deliberately holds no locks and uses no synchronization. It reads scx_ops.name, scx_ops_enable_state_var, scx_switching_all, and p->scx.runnable_at all without protection. The function comment acknowledges this: it may print "mixups or garbages". This is intentional — the function must remain safe to call in any context, including NMI and oops handlers where acquiring locks is forbidden. The READ_ONCE() on scx_switching_all provides minimal compiler barrier protection without locking.
Why Maintainers Need to Know This
- This is the first line of debugging for hung systems: When a system is unresponsive with a BPF scheduler loaded, sysrq-T will now tell you which scheduler is loaded, its state, and which tasks are stuck waiting. Without this, you would have no kernel-side indication that a BPF scheduler is involved.
- The example output in the commit message is the expected format:
Sched_ext: qmap (enabled+all), task: runnable_at=-17ms. A maintainer reviewing a bug report should look for this line in the oops/sysrq output. copy_from_kernel_nofaultis the right tool here: Any future extensions toprint_scx_info()that read additional task fields should continue using nofault accessors. Direct dereferences in this path are a latent crash risk.- State transitions during dump: Because the function is unsynchronized, it is possible for the state to change (e.g., BPF scheduler being unloaded) between the
scx_ops_enable_state()read and theprintk. The output may be stale or inconsistent. This is acceptable.
Connection to Other Patches
- The
scx_ops_enable_state_str[]array used here was moved out ofCONFIG_SCHED_DEBUGgating by an earlier patch precisely because this function needs it in non-debug builds. p->scx.runnable_atis a field established by the core sched_ext task tracking patches; its purpose as a starvation detection timestamp is directly exploited here for human-readable output.- The watchdog (PATCH 12) detects starvation and triggers an error exit;
print_scx_info()provides the complementary operator-facing visibility into how long a task has been waiting, useful for diagnosing watchdog triggers before they escalate.
[PATCH 15/30] sched_ext: Print debug dump after an error exit
View on Lore: https://lore.kernel.org/all/20240618212056.2833381-16-tj@kernel.org
Commit Message
If a BPF scheduler triggers an error, the scheduler is aborted and the
system is reverted to the built-in scheduler. In the process, a lot of
information which may be useful for figuring out what happened can be lost.
This patch adds debug dump which captures information which may be useful
for debugging including runqueue and runnable thread states at the time of
failure. The following shows a debug dump after triggering the watchdog:
root@test ~# os/work/tools/sched_ext/build/bin/scx_qmap -t 100
stats : enq=1 dsp=0 delta=1 deq=0
stats : enq=90 dsp=90 delta=0 deq=0
stats : enq=156 dsp=156 delta=0 deq=0
stats : enq=218 dsp=218 delta=0 deq=0
stats : enq=255 dsp=255 delta=0 deq=0
stats : enq=271 dsp=271 delta=0 deq=0
stats : enq=284 dsp=284 delta=0 deq=0
stats : enq=293 dsp=293 delta=0 deq=0
DEBUG DUMP
================================================================================
kworker/u32:12[320] triggered exit kind 1026:
runnable task stall (stress[1530] failed to run for 6.841s)
Backtrace:
scx_watchdog_workfn+0x136/0x1c0
process_scheduled_works+0x2b5/0x600
worker_thread+0x269/0x360
kthread+0xeb/0x110
ret_from_fork+0x36/0x40
ret_from_fork_asm+0x1a/0x30
QMAP FIFO[0]:
QMAP FIFO[1]:
QMAP FIFO[2]: 1436
QMAP FIFO[3]:
QMAP FIFO[4]:
CPU states
----------
CPU 0 : nr_run=1 ops_qseq=244
curr=swapper/0[0] class=idle_sched_class
QMAP: dsp_idx=1 dsp_cnt=0
R stress[1530] -6841ms
scx_state/flags=3/0x1 ops_state/qseq=2/20
sticky/holding_cpu=-1/-1 dsq_id=(n/a)
cpus=ff
QMAP: force_local=0
asm_sysvec_apic_timer_interrupt+0x16/0x20
CPU 2 : nr_run=2 ops_qseq=142
curr=swapper/2[0] class=idle_sched_class
QMAP: dsp_idx=1 dsp_cnt=0
R sshd[1703] -5905ms
scx_state/flags=3/0x9 ops_state/qseq=2/88
sticky/holding_cpu=-1/-1 dsq_id=(n/a)
cpus=ff
QMAP: force_local=1
__x64_sys_ppoll+0xf6/0x120
do_syscall_64+0x7b/0x150
entry_SYSCALL_64_after_hwframe+0x76/0x7e
R fish[1539] -4141ms
scx_state/flags=3/0x9 ops_state/qseq=2/124
sticky/holding_cpu=-1/-1 dsq_id=(n/a)
cpus=ff
QMAP: force_local=1
futex_wait+0x60/0xe0
do_futex+0x109/0x180
__x64_sys_futex+0x117/0x190
do_syscall_64+0x7b/0x150
entry_SYSCALL_64_after_hwframe+0x76/0x7e
CPU 3 : nr_run=2 ops_qseq=162
curr=kworker/u32:12[320] class=ext_sched_class
QMAP: dsp_idx=1 dsp_cnt=0
*R kworker/u32:12[320] +0ms
scx_state/flags=3/0xd ops_state/qseq=0/0
sticky/holding_cpu=-1/-1 dsq_id=(n/a)
cpus=ff
QMAP: force_local=0
scx_dump_state+0x613/0x6f0
scx_ops_error_irq_workfn+0x1f/0x40
irq_work_run_list+0x82/0xd0
irq_work_run+0x14/0x30
__sysvec_irq_work+0x40/0x140
sysvec_irq_work+0x60/0x70
asm_sysvec_irq_work+0x16/0x20
scx_watchdog_workfn+0x15f/0x1c0
process_scheduled_works+0x2b5/0x600
worker_thread+0x269/0x360
kthread+0xeb/0x110
ret_from_fork+0x36/0x40
ret_from_fork_asm+0x1a/0x30
R kworker/3:2[1436] +0ms
scx_state/flags=3/0x9 ops_state/qseq=2/160
sticky/holding_cpu=-1/-1 dsq_id=(n/a)
cpus=08
QMAP: force_local=0
kthread+0xeb/0x110
ret_from_fork+0x36/0x40
ret_from_fork_asm+0x1a/0x30
CPU 7 : nr_run=0 ops_qseq=76
curr=swapper/7[0] class=idle_sched_class
================================================================================
EXIT: runnable task stall (stress[1530] failed to run for 6.841s)
It shows that CPU 3 was running the watchdog when it triggered the error
condition and the scx_qmap thread has been queued on CPU 0 for over 5
seconds but failed to run. It also prints out scx_qmap specific information
- e.g. which tasks are queued on each FIFO and so on using the dump_*() ops.
This dump has proved pretty useful for developing and debugging BPF
schedulers.
Debug dump is generated automatically when the BPF scheduler exits due to an
error. The debug buffer used in such cases is determined by
sched_ext_ops.exit_dump_len and defaults to 32k. If the debug dump overruns
the available buffer, the output is truncated and marked accordingly.
Debug dump output can also be read through the sched_ext_dump tracepoint.
When read through the tracepoint, there is no length limit.
SysRq-D can be used to trigger debug dump at any time while a BPF scheduler
is loaded. This is non-destructive - the scheduler keeps running afterwards.
The output can be read through the sched_ext_dump tracepoint.
v2: - The size of exit debug dump buffer can now be customized using
sched_ext_ops.exit_dump_len.
- sched_ext_ops.dump*() added to enable dumping of BPF scheduler
specific information.
- Tracpoint output and SysRq-D triggering added.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: David Vernet <dvernet@meta.com>
---
include/trace/events/sched_ext.h | 32 ++
kernel/sched/ext.c | 421 ++++++++++++++++++-
tools/sched_ext/include/scx/common.bpf.h | 12 +
tools/sched_ext/include/scx/compat.h | 9 +-
tools/sched_ext/include/scx/user_exit_info.h | 19 +
tools/sched_ext/scx_qmap.bpf.c | 54 +++
tools/sched_ext/scx_qmap.c | 14 +-
tools/sched_ext/scx_simple.c | 2 +-
8 files changed, 555 insertions(+), 8 deletions(-)
create mode 100644 include/trace/events/sched_ext.h
diff --git a/include/trace/events/sched_ext.h b/include/trace/events/sched_ext.h
new file mode 100644
index 000000000000..fe19da7315a9
--- /dev/null
+++ b/include/trace/events/sched_ext.h
@@ -0,0 +1,32 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#undef TRACE_SYSTEM
+#define TRACE_SYSTEM sched_ext
+
+#if !defined(_TRACE_SCHED_EXT_H) || defined(TRACE_HEADER_MULTI_READ)
+#define _TRACE_SCHED_EXT_H
+
+#include <linux/tracepoint.h>
+
+TRACE_EVENT(sched_ext_dump,
+
+ TP_PROTO(const char *line),
+
+ TP_ARGS(line),
+
+ TP_STRUCT__entry(
+ __string(line, line)
+ ),
+
+ TP_fast_assign(
+ __assign_str(line);
+ ),
+
+ TP_printk("%s",
+ __get_str(line)
+ )
+);
+
+#endif /* _TRACE_SCHED_EXT_H */
+
+/* This part must be outside protection */
+#include <trace/define_trace.h>
diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c
index 6f4de29d7372..66bb9cf075f0 100644
--- a/kernel/sched/ext.c
+++ b/kernel/sched/ext.c
@@ -12,6 +12,7 @@ enum scx_consts {
SCX_EXIT_BT_LEN = 64,
SCX_EXIT_MSG_LEN = 1024,
+ SCX_EXIT_DUMP_DFL_LEN = 32768,
};
enum scx_exit_kind {
@@ -48,6 +49,9 @@ struct scx_exit_info {
/* informational message */
char *msg;
+
+ /* debug dump */
+ char *dump;
};
/* sched_ext_ops.flags */
@@ -105,6 +109,17 @@ struct scx_exit_task_args {
bool cancelled;
};
+/*
+ * Informational context provided to dump operations.
+ */
+struct scx_dump_ctx {
+ enum scx_exit_kind kind;
+ s64 exit_code;
+ const char *reason;
+ u64 at_ns;
+ u64 at_jiffies;
+};
+
/**
* struct sched_ext_ops - Operation table for BPF scheduler implementation
*
@@ -296,6 +311,36 @@ struct sched_ext_ops {
*/
void (*disable)(struct task_struct *p);
+ /**
+ * dump - Dump BPF scheduler state on error
+ * @ctx: debug dump context
+ *
+ * Use scx_bpf_dump() to generate BPF scheduler specific debug dump.
+ */
+ void (*dump)(struct scx_dump_ctx *ctx);
+
+ /**
+ * dump_cpu - Dump BPF scheduler state for a CPU on error
+ * @ctx: debug dump context
+ * @cpu: CPU to generate debug dump for
+ * @idle: @cpu is currently idle without any runnable tasks
+ *
+ * Use scx_bpf_dump() to generate BPF scheduler specific debug dump for
+ * @cpu. If @idle is %true and this operation doesn't produce any
+ * output, @cpu is skipped for dump.
+ */
+ void (*dump_cpu)(struct scx_dump_ctx *ctx, s32 cpu, bool idle);
+
+ /**
+ * dump_task - Dump BPF scheduler state for a runnable task on error
+ * @ctx: debug dump context
+ * @p: runnable task to generate debug dump for
+ *
+ * Use scx_bpf_dump() to generate BPF scheduler specific debug dump for
+ * @p.
+ */
+ void (*dump_task)(struct scx_dump_ctx *ctx, struct task_struct *p);
+
/*
* All online ops must come before ops.init().
*/
@@ -330,6 +375,12 @@ struct sched_ext_ops {
*/
u32 timeout_ms;
+ /**
+ * exit_dump_len - scx_exit_info.dump buffer length. If 0, the default
+ * value of 32768 is used.
+ */
+ u32 exit_dump_len;
+
/**
* name - BPF scheduler's name
*
@@ -567,10 +618,27 @@ struct scx_bstr_buf {
static DEFINE_RAW_SPINLOCK(scx_exit_bstr_buf_lock);
static struct scx_bstr_buf scx_exit_bstr_buf;
+/* ops debug dump */
+struct scx_dump_data {
+ s32 cpu;
+ bool first;
+ s32 cursor;
+ struct seq_buf *s;
+ const char *prefix;
+ struct scx_bstr_buf buf;
+};
+
+struct scx_dump_data scx_dump_data = {
+ .cpu = -1,
+};
+
/* /sys/kernel/sched_ext interface */
static struct kset *scx_kset;
static struct kobject *scx_root_kobj;
+#define CREATE_TRACE_POINTS
+#include <trace/events/sched_ext.h>
+
static __printf(3, 4) void scx_ops_exit_kind(enum scx_exit_kind kind,
s64 exit_code,
const char *fmt, ...);
@@ -2897,12 +2965,13 @@ static void scx_ops_bypass(bool bypass)
static void free_exit_info(struct scx_exit_info *ei)
{
+ kfree(ei->dump);
kfree(ei->msg);
kfree(ei->bt);
kfree(ei);
}
-static struct scx_exit_info *alloc_exit_info(void)
+static struct scx_exit_info *alloc_exit_info(size_t exit_dump_len)
{
struct scx_exit_info *ei;
@@ -2912,8 +2981,9 @@ static struct scx_exit_info *alloc_exit_info(void)
ei->bt = kcalloc(sizeof(ei->bt[0]), SCX_EXIT_BT_LEN, GFP_KERNEL);
ei->msg = kzalloc(SCX_EXIT_MSG_LEN, GFP_KERNEL);
+ ei->dump = kzalloc(exit_dump_len, GFP_KERNEL);
- if (!ei->bt || !ei->msg) {
+ if (!ei->bt || !ei->msg || !ei->dump) {
free_exit_info(ei);
return NULL;
}
@@ -3125,8 +3195,274 @@ static void scx_ops_disable(enum scx_exit_kind kind)
schedule_scx_ops_disable_work();
}
+static void dump_newline(struct seq_buf *s)
+{
+ trace_sched_ext_dump("");
+
+ /* @s may be zero sized and seq_buf triggers WARN if so */
+ if (s->size)
+ seq_buf_putc(s, '\n');
+}
+
+static __printf(2, 3) void dump_line(struct seq_buf *s, const char *fmt, ...)
+{
+ va_list args;
+
+#ifdef CONFIG_TRACEPOINTS
+ if (trace_sched_ext_dump_enabled()) {
+ /* protected by scx_dump_state()::dump_lock */
+ static char line_buf[SCX_EXIT_MSG_LEN];
+
+ va_start(args, fmt);
+ vscnprintf(line_buf, sizeof(line_buf), fmt, args);
+ va_end(args);
+
+ trace_sched_ext_dump(line_buf);
+ }
+#endif
+ /* @s may be zero sized and seq_buf triggers WARN if so */
+ if (s->size) {
+ va_start(args, fmt);
+ seq_buf_vprintf(s, fmt, args);
+ va_end(args);
+
+ seq_buf_putc(s, '\n');
+ }
+}
+
+static void dump_stack_trace(struct seq_buf *s, const char *prefix,
+ const unsigned long *bt, unsigned int len)
+{
+ unsigned int i;
+
+ for (i = 0; i < len; i++)
+ dump_line(s, "%s%pS", prefix, (void *)bt[i]);
+}
+
+static void ops_dump_init(struct seq_buf *s, const char *prefix)
+{
+ struct scx_dump_data *dd = &scx_dump_data;
+
+ lockdep_assert_irqs_disabled();
+
+ dd->cpu = smp_processor_id(); /* allow scx_bpf_dump() */
+ dd->first = true;
+ dd->cursor = 0;
+ dd->s = s;
+ dd->prefix = prefix;
+}
+
+static void ops_dump_flush(void)
+{
+ struct scx_dump_data *dd = &scx_dump_data;
+ char *line = dd->buf.line;
+
+ if (!dd->cursor)
+ return;
+
+ /*
+ * There's something to flush and this is the first line. Insert a blank
+ * line to distinguish ops dump.
+ */
+ if (dd->first) {
+ dump_newline(dd->s);
+ dd->first = false;
+ }
+
+ /*
+ * There may be multiple lines in $line. Scan and emit each line
+ * separately.
+ */
+ while (true) {
+ char *end = line;
+ char c;
+
+ while (*end != '\n' && *end != '\0')
+ end++;
+
+ /*
+ * If $line overflowed, it may not have newline at the end.
+ * Always emit with a newline.
+ */
+ c = *end;
+ *end = '\0';
+ dump_line(dd->s, "%s%s", dd->prefix, line);
+ if (c == '\0')
+ break;
+
+ /* move to the next line */
+ end++;
+ if (*end == '\0')
+ break;
+ line = end;
+ }
+
+ dd->cursor = 0;
+}
+
+static void ops_dump_exit(void)
+{
+ ops_dump_flush();
+ scx_dump_data.cpu = -1;
+}
+
+static void scx_dump_task(struct seq_buf *s, struct scx_dump_ctx *dctx,
+ struct task_struct *p, char marker)
+{
+ static unsigned long bt[SCX_EXIT_BT_LEN];
+ char dsq_id_buf[19] = "(n/a)";
+ unsigned long ops_state = atomic_long_read(&p->scx.ops_state);
+ unsigned int bt_len;
+
+ if (p->scx.dsq)
+ scnprintf(dsq_id_buf, sizeof(dsq_id_buf), "0x%llx",
+ (unsigned long long)p->scx.dsq->id);
+
+ dump_newline(s);
+ dump_line(s, " %c%c %s[%d] %+ldms",
+ marker, task_state_to_char(p), p->comm, p->pid,
+ jiffies_delta_msecs(p->scx.runnable_at, dctx->at_jiffies));
+ dump_line(s, " scx_state/flags=%u/0x%x ops_state/qseq=%lu/%lu",
+ scx_get_task_state(p), p->scx.flags & ~SCX_TASK_STATE_MASK,
+ ops_state & SCX_OPSS_STATE_MASK,
+ ops_state >> SCX_OPSS_QSEQ_SHIFT);
+ dump_line(s, " sticky/holding_cpu=%d/%d dsq_id=%s",
+ p->scx.sticky_cpu, p->scx.holding_cpu, dsq_id_buf);
+ dump_line(s, " cpus=%*pb", cpumask_pr_args(p->cpus_ptr));
+
+ if (SCX_HAS_OP(dump_task)) {
+ ops_dump_init(s, " ");
+ SCX_CALL_OP(SCX_KF_REST, dump_task, dctx, p);
+ ops_dump_exit();
+ }
+
+ bt_len = stack_trace_save_tsk(p, bt, SCX_EXIT_BT_LEN, 1);
+ if (bt_len) {
+ dump_newline(s);
+ dump_stack_trace(s, " ", bt, bt_len);
+ }
+}
+
+static void scx_dump_state(struct scx_exit_info *ei, size_t dump_len)
+{
+ static DEFINE_SPINLOCK(dump_lock);
+ static const char trunc_marker[] = "\n\n~~~~ TRUNCATED ~~~~\n";
+ struct scx_dump_ctx dctx = {
+ .kind = ei->kind,
+ .exit_code = ei->exit_code,
+ .reason = ei->reason,
+ .at_ns = ktime_get_ns(),
+ .at_jiffies = jiffies,
+ };
+ struct seq_buf s;
+ unsigned long flags;
+ char *buf;
+ int cpu;
+
+ spin_lock_irqsave(&dump_lock, flags);
+
+ seq_buf_init(&s, ei->dump, dump_len);
+
+ if (ei->kind == SCX_EXIT_NONE) {
+ dump_line(&s, "Debug dump triggered by %s", ei->reason);
+ } else {
+ dump_line(&s, "%s[%d] triggered exit kind %d:",
+ current->comm, current->pid, ei->kind);
+ dump_line(&s, " %s (%s)", ei->reason, ei->msg);
+ dump_newline(&s);
+ dump_line(&s, "Backtrace:");
+ dump_stack_trace(&s, " ", ei->bt, ei->bt_len);
+ }
+
+ if (SCX_HAS_OP(dump)) {
+ ops_dump_init(&s, "");
+ SCX_CALL_OP(SCX_KF_UNLOCKED, dump, &dctx);
+ ops_dump_exit();
+ }
+
+ dump_newline(&s);
+ dump_line(&s, "CPU states");
+ dump_line(&s, "----------");
+
+ for_each_possible_cpu(cpu) {
+ struct rq *rq = cpu_rq(cpu);
+ struct rq_flags rf;
+ struct task_struct *p;
+ struct seq_buf ns;
+ size_t avail, used;
+ bool idle;
+
+ rq_lock(rq, &rf);
+
+ idle = list_empty(&rq->scx.runnable_list) &&
+ rq->curr->sched_class == &idle_sched_class;
+
+ if (idle && !SCX_HAS_OP(dump_cpu))
+ goto next;
+
+ /*
+ * We don't yet know whether ops.dump_cpu() will produce output
+ * and we may want to skip the default CPU dump if it doesn't.
+ * Use a nested seq_buf to generate the standard dump so that we
+ * can decide whether to commit later.
+ */
+ avail = seq_buf_get_buf(&s, &buf);
+ seq_buf_init(&ns, buf, avail);
+
+ dump_newline(&ns);
+ dump_line(&ns, "CPU %-4d: nr_run=%u ops_qseq=%lu",
+ cpu, rq->scx.nr_running, rq->scx.ops_qseq);
+ dump_line(&ns, " curr=%s[%d] class=%ps",
+ rq->curr->comm, rq->curr->pid,
+ rq->curr->sched_class);
+
+ used = seq_buf_used(&ns);
+ if (SCX_HAS_OP(dump_cpu)) {
+ ops_dump_init(&ns, " ");
+ SCX_CALL_OP(SCX_KF_REST, dump_cpu, &dctx, cpu, idle);
+ ops_dump_exit();
+ }
+
+ /*
+ * If idle && nothing generated by ops.dump_cpu(), there's
+ * nothing interesting. Skip.
+ */
+ if (idle && used == seq_buf_used(&ns))
+ goto next;
+
+ /*
+ * $s may already have overflowed when $ns was created. If so,
+ * calling commit on it will trigger BUG.
+ */
+ if (avail) {
+ seq_buf_commit(&s, seq_buf_used(&ns));
+ if (seq_buf_has_overflowed(&ns))
+ seq_buf_set_overflow(&s);
+ }
+
+ if (rq->curr->sched_class == &ext_sched_class)
+ scx_dump_task(&s, &dctx, rq->curr, '*');
+
+ list_for_each_entry(p, &rq->scx.runnable_list, scx.runnable_node)
+ scx_dump_task(&s, &dctx, p, ' ');
+ next:
+ rq_unlock(rq, &rf);
+ }
+
+ if (seq_buf_has_overflowed(&s) && dump_len >= sizeof(trunc_marker))
+ memcpy(ei->dump + dump_len - sizeof(trunc_marker),
+ trunc_marker, sizeof(trunc_marker));
+
+ spin_unlock_irqrestore(&dump_lock, flags);
+}
+
static void scx_ops_error_irq_workfn(struct irq_work *irq_work)
{
+ struct scx_exit_info *ei = scx_exit_info;
+
+ if (ei->kind >= SCX_EXIT_ERROR)
+ scx_dump_state(ei, scx_ops.exit_dump_len);
+
schedule_scx_ops_disable_work();
}
@@ -3152,6 +3488,13 @@ static __printf(3, 4) void scx_ops_exit_kind(enum scx_exit_kind kind,
vscnprintf(ei->msg, SCX_EXIT_MSG_LEN, fmt, args);
va_end(args);
+ /*
+ * Set ei->kind and ->reason for scx_dump_state(). They'll be set again
+ * in scx_ops_disable_workfn().
+ */
+ ei->kind = kind;
+ ei->reason = scx_exit_reason(ei->kind);
+
irq_work_queue(&scx_ops_error_irq_work);
}
@@ -3213,7 +3556,7 @@ static int scx_ops_enable(struct sched_ext_ops *ops, struct bpf_link *link)
if (ret < 0)
goto err;
- scx_exit_info = alloc_exit_info();
+ scx_exit_info = alloc_exit_info(ops->exit_dump_len);
if (!scx_exit_info) {
ret = -ENOMEM;
goto err_del;
@@ -3592,6 +3935,10 @@ static int bpf_scx_init_member(const struct btf_type *t,
return -E2BIG;
ops->timeout_ms = *(u32 *)(udata + moff);
return 1;
+ case offsetof(struct sched_ext_ops, exit_dump_len):
+ ops->exit_dump_len =
+ *(u32 *)(udata + moff) ?: SCX_EXIT_DUMP_DFL_LEN;
+ return 1;
}
return 0;
@@ -3723,6 +4070,21 @@ static const struct sysrq_key_op sysrq_sched_ext_reset_op = {
.enable_mask = SYSRQ_ENABLE_RTNICE,
};
+static void sysrq_handle_sched_ext_dump(u8 key)
+{
+ struct scx_exit_info ei = { .kind = SCX_EXIT_NONE, .reason = "SysRq-D" };
+
+ if (scx_enabled())
+ scx_dump_state(&ei, 0);
+}
+
+static const struct sysrq_key_op sysrq_sched_ext_dump_op = {
+ .handler = sysrq_handle_sched_ext_dump,
+ .help_msg = "dump-sched-ext(D)",
+ .action_msg = "Trigger sched_ext debug dump",
+ .enable_mask = SYSRQ_ENABLE_RTNICE,
+};
+
/**
* print_scx_info - print out sched_ext scheduler state
* @log_lvl: the log level to use when printing
@@ -3793,6 +4155,7 @@ void __init init_sched_ext_class(void)
}
register_sysrq_key('S', &sysrq_sched_ext_reset_op);
+ register_sysrq_key('D', &sysrq_sched_ext_dump_op);
INIT_DELAYED_WORK(&scx_watchdog_work, scx_watchdog_workfn);
}
@@ -4218,6 +4581,57 @@ __bpf_kfunc void scx_bpf_error_bstr(char *fmt, unsigned long long *data,
raw_spin_unlock_irqrestore(&scx_exit_bstr_buf_lock, flags);
}
+/**
+ * scx_bpf_dump - Generate extra debug dump specific to the BPF scheduler
+ * @fmt: format string
+ * @data: format string parameters packaged using ___bpf_fill() macro
+ * @data__sz: @data len, must end in '__sz' for the verifier
+ *
+ * To be called through scx_bpf_dump() helper from ops.dump(), dump_cpu() and
+ * dump_task() to generate extra debug dump specific to the BPF scheduler.
+ *
+ * The extra dump may be multiple lines. A single line may be split over
+ * multiple calls. The last line is automatically terminated.
+ */
+__bpf_kfunc void scx_bpf_dump_bstr(char *fmt, unsigned long long *data,
+ u32 data__sz)
+{
+ struct scx_dump_data *dd = &scx_dump_data;
+ struct scx_bstr_buf *buf = &dd->buf;
+ s32 ret;
+
+ if (raw_smp_processor_id() != dd->cpu) {
+ scx_ops_error("scx_bpf_dump() must only be called from ops.dump() and friends");
+ return;
+ }
+
+ /* append the formatted string to the line buf */
+ ret = __bstr_format(buf->data, buf->line + dd->cursor,
+ sizeof(buf->line) - dd->cursor, fmt, data, data__sz);
+ if (ret < 0) {
+ dump_line(dd->s, "%s[!] (\"%s\", %p, %u) failed to format (%d)",
+ dd->prefix, fmt, data, data__sz, ret);
+ return;
+ }
+
+ dd->cursor += ret;
+ dd->cursor = min_t(s32, dd->cursor, sizeof(buf->line));
+
+ if (!dd->cursor)
+ return;
+
+ /*
+ * If the line buf overflowed or ends in a newline, flush it into the
+ * dump. This is to allow the caller to generate a single line over
+ * multiple calls. As ops_dump_flush() can also handle multiple lines in
+ * the line buf, the only case which can lead to an unexpected
+ * truncation is when the caller keeps generating newlines in the middle
+ * instead of the end consecutively. Don't do that.
+ */
+ if (dd->cursor >= sizeof(buf->line) || buf->line[dd->cursor - 1] == '\n')
+ ops_dump_flush();
+}
+
/**
* scx_bpf_nr_cpu_ids - Return the number of possible CPU IDs
*
@@ -4426,6 +4840,7 @@ BTF_ID_FLAGS(func, scx_bpf_dsq_nr_queued)
BTF_ID_FLAGS(func, scx_bpf_destroy_dsq)
BTF_ID_FLAGS(func, scx_bpf_exit_bstr, KF_TRUSTED_ARGS)
BTF_ID_FLAGS(func, scx_bpf_error_bstr, KF_TRUSTED_ARGS)
+BTF_ID_FLAGS(func, scx_bpf_dump_bstr, KF_TRUSTED_ARGS)
BTF_ID_FLAGS(func, scx_bpf_nr_cpu_ids)
BTF_ID_FLAGS(func, scx_bpf_get_possible_cpumask, KF_ACQUIRE)
BTF_ID_FLAGS(func, scx_bpf_get_online_cpumask, KF_ACQUIRE)
diff --git a/tools/sched_ext/include/scx/common.bpf.h b/tools/sched_ext/include/scx/common.bpf.h
index 833fe1bdccf9..3ea5cdf58bc7 100644
--- a/tools/sched_ext/include/scx/common.bpf.h
+++ b/tools/sched_ext/include/scx/common.bpf.h
@@ -38,6 +38,7 @@ s32 scx_bpf_dsq_nr_queued(u64 dsq_id) __ksym;
void scx_bpf_destroy_dsq(u64 dsq_id) __ksym;
void scx_bpf_exit_bstr(s64 exit_code, char *fmt, unsigned long long *data, u32 data__sz) __ksym __weak;
void scx_bpf_error_bstr(char *fmt, unsigned long long *data, u32 data_len) __ksym;
+void scx_bpf_dump_bstr(char *fmt, unsigned long long *data, u32 data_len) __ksym __weak;
u32 scx_bpf_nr_cpu_ids(void) __ksym __weak;
const struct cpumask *scx_bpf_get_possible_cpumask(void) __ksym __weak;
const struct cpumask *scx_bpf_get_online_cpumask(void) __ksym __weak;
@@ -97,6 +98,17 @@ void ___scx_bpf_bstr_format_checker(const char *fmt, ...) {}
___scx_bpf_bstr_format_checker(fmt, ##args); \
})
+/*
+ * scx_bpf_dump() wraps the scx_bpf_dump_bstr() kfunc with variadic arguments
+ * instead of an array of u64. To be used from ops.dump() and friends.
+ */
+#define scx_bpf_dump(fmt, args...) \
+({ \
+ scx_bpf_bstr_preamble(fmt, args) \
+ scx_bpf_dump_bstr(___fmt, ___param, sizeof(___param)); \
+ ___scx_bpf_bstr_format_checker(fmt, ##args); \
+})
+
#define BPF_STRUCT_OPS(name, args...) \
SEC("struct_ops/"#name) \
BPF_PROG(name, ##args)
diff --git a/tools/sched_ext/include/scx/compat.h b/tools/sched_ext/include/scx/compat.h
index a7fdaf8a858e..c58024c980c8 100644
--- a/tools/sched_ext/include/scx/compat.h
+++ b/tools/sched_ext/include/scx/compat.h
@@ -111,16 +111,23 @@ static inline bool __COMPAT_struct_has_field(const char *type, const char *field
* is used to define ops and compat.h::SCX_OPS_LOAD/ATTACH() are used to load
* and attach it, backward compatibility is automatically maintained where
* reasonable.
+ *
+ * ec7e3b0463e1 ("implement-ops") in https://github.com/sched-ext/sched_ext is
+ * the current minimum required kernel version.
*/
#define SCX_OPS_OPEN(__ops_name, __scx_name) ({ \
struct __scx_name *__skel; \
\
+ SCX_BUG_ON(!__COMPAT_struct_has_field("sched_ext_ops", "dump"), \
+ "sched_ext_ops.dump() missing, kernel too old?"); \
+ \
__skel = __scx_name##__open(); \
SCX_BUG_ON(!__skel, "Could not open " #__scx_name); \
__skel; \
})
-#define SCX_OPS_LOAD(__skel, __ops_name, __scx_name) ({ \
+#define SCX_OPS_LOAD(__skel, __ops_name, __scx_name, __uei_name) ({ \
+ UEI_SET_SIZE(__skel, __ops_name, __uei_name); \
SCX_BUG_ON(__scx_name##__load((__skel)), "Failed to load skel"); \
})
diff --git a/tools/sched_ext/include/scx/user_exit_info.h b/tools/sched_ext/include/scx/user_exit_info.h
index 8c3b7fac4d05..c2ef85c645e1 100644
--- a/tools/sched_ext/include/scx/user_exit_info.h
+++ b/tools/sched_ext/include/scx/user_exit_info.h
@@ -13,6 +13,7 @@
enum uei_sizes {
UEI_REASON_LEN = 128,
UEI_MSG_LEN = 1024,
+ UEI_DUMP_DFL_LEN = 32768,
};
struct user_exit_info {
@@ -28,6 +29,8 @@ struct user_exit_info {
#include <bpf/bpf_core_read.h>
#define UEI_DEFINE(__name) \
+ char RESIZABLE_ARRAY(data, __name##_dump); \
+ const volatile u32 __name##_dump_len; \
struct user_exit_info __name SEC(".data")
#define UEI_RECORD(__uei_name, __ei) ({ \
@@ -35,6 +38,8 @@ struct user_exit_info {
sizeof(__uei_name.reason), (__ei)->reason); \
bpf_probe_read_kernel_str(__uei_name.msg, \
sizeof(__uei_name.msg), (__ei)->msg); \
+ bpf_probe_read_kernel_str(__uei_name##_dump, \
+ __uei_name##_dump_len, (__ei)->dump); \
if (bpf_core_field_exists((__ei)->exit_code)) \
__uei_name.exit_code = (__ei)->exit_code; \
/* use __sync to force memory barrier */ \
@@ -47,6 +52,13 @@ struct user_exit_info {
#include <stdio.h>
#include <stdbool.h>
+/* no need to call the following explicitly if SCX_OPS_LOAD() is used */
+#define UEI_SET_SIZE(__skel, __ops_name, __uei_name) ({ \
+ u32 __len = (__skel)->struct_ops.__ops_name->exit_dump_len ?: UEI_DUMP_DFL_LEN; \
+ (__skel)->rodata->__uei_name##_dump_len = __len; \
+ RESIZE_ARRAY((__skel), data, __uei_name##_dump, __len); \
+})
+
#define UEI_EXITED(__skel, __uei_name) ({ \
/* use __sync to force memory barrier */ \
__sync_val_compare_and_swap(&(__skel)->data->__uei_name.kind, -1, -1); \
@@ -54,6 +66,13 @@ struct user_exit_info {
#define UEI_REPORT(__skel, __uei_name) ({ \
struct user_exit_info *__uei = &(__skel)->data->__uei_name; \
+ char *__uei_dump = (__skel)->data_##__uei_name##_dump->__uei_name##_dump; \
+ if (__uei_dump[0] != '\0') { \
+ fputs("\nDEBUG DUMP\n", stderr); \
+ fputs("================================================================================\n\n", stderr); \
+ fputs(__uei_dump, stderr); \
+ fputs("\n================================================================================\n\n", stderr); \
+ } \
fprintf(stderr, "EXIT: %s", __uei->reason); \
if (__uei->msg[0] != '\0') \
fprintf(stderr, " (%s)", __uei->msg); \
diff --git a/tools/sched_ext/scx_qmap.bpf.c b/tools/sched_ext/scx_qmap.bpf.c
index 5ff217c4bfa0..5b3da28bf042 100644
--- a/tools/sched_ext/scx_qmap.bpf.c
+++ b/tools/sched_ext/scx_qmap.bpf.c
@@ -33,6 +33,7 @@ const volatile u32 stall_user_nth;
const volatile u32 stall_kernel_nth;
const volatile u32 dsp_batch;
const volatile s32 disallow_tgid;
+const volatile bool suppress_dump;
u32 test_error_cnt;
@@ -258,6 +259,56 @@ s32 BPF_STRUCT_OPS(qmap_init_task, struct task_struct *p,
return -ENOMEM;
}
+void BPF_STRUCT_OPS(qmap_dump, struct scx_dump_ctx *dctx)
+{
+ s32 i, pid;
+
+ if (suppress_dump)
+ return;
+
+ bpf_for(i, 0, 5) {
+ void *fifo;
+
+ if (!(fifo = bpf_map_lookup_elem(&queue_arr, &i)))
+ return;
+
+ scx_bpf_dump("QMAP FIFO[%d]:", i);
+ bpf_repeat(4096) {
+ if (bpf_map_pop_elem(fifo, &pid))
+ break;
+ scx_bpf_dump(" %d", pid);
+ }
+ scx_bpf_dump("\n");
+ }
+}
+
+void BPF_STRUCT_OPS(qmap_dump_cpu, struct scx_dump_ctx *dctx, s32 cpu, bool idle)
+{
+ u32 zero = 0;
+ struct cpu_ctx *cpuc;
+
+ if (suppress_dump || idle)
+ return;
+ if (!(cpuc = bpf_map_lookup_percpu_elem(&cpu_ctx_stor, &zero, cpu)))
+ return;
+
+ scx_bpf_dump("QMAP: dsp_idx=%llu dsp_cnt=%llu",
+ cpuc->dsp_idx, cpuc->dsp_cnt);
+}
+
+void BPF_STRUCT_OPS(qmap_dump_task, struct scx_dump_ctx *dctx, struct task_struct *p)
+{
+ struct task_ctx *taskc;
+
+ if (suppress_dump)
+ return;
+ if (!(taskc = bpf_task_storage_get(&task_ctx_stor, p, 0, 0)))
+ return;
+
+ scx_bpf_dump("QMAP: force_local=%d",
+ taskc->force_local);
+}
+
s32 BPF_STRUCT_OPS_SLEEPABLE(qmap_init)
{
return scx_bpf_create_dsq(SHARED_DSQ, -1);
@@ -274,6 +325,9 @@ SCX_OPS_DEFINE(qmap_ops,
.dequeue = (void *)qmap_dequeue,
.dispatch = (void *)qmap_dispatch,
.init_task = (void *)qmap_init_task,
+ .dump = (void *)qmap_dump,
+ .dump_cpu = (void *)qmap_dump_cpu,
+ .dump_task = (void *)qmap_dump_task,
.init = (void *)qmap_init,
.exit = (void *)qmap_exit,
.timeout_ms = 5000U,
diff --git a/tools/sched_ext/scx_qmap.c b/tools/sched_ext/scx_qmap.c
index a2614994cfaa..a1123a17581b 100644
--- a/tools/sched_ext/scx_qmap.c
+++ b/tools/sched_ext/scx_qmap.c
@@ -20,7 +20,7 @@ const char help_fmt[] =
"See the top-level comment in .bpf.c for more details.\n"
"\n"
"Usage: %s [-s SLICE_US] [-e COUNT] [-t COUNT] [-T COUNT] [-b COUNT]\n"
-" [-d PID] [-p] [-v]\n"
+" [-d PID] [-D LEN] [-p] [-v]\n"
"\n"
" -s SLICE_US Override slice duration\n"
" -e COUNT Trigger scx_bpf_error() after COUNT enqueues\n"
@@ -28,6 +28,8 @@ const char help_fmt[] =
" -T COUNT Stall every COUNT'th kernel thread\n"
" -b COUNT Dispatch upto COUNT tasks together\n"
" -d PID Disallow a process from switching into SCHED_EXT (-1 for self)\n"
+" -D LEN Set scx_exit_info.dump buffer length\n"
+" -S Suppress qmap-specific debug dump\n"
" -p Switch only tasks on SCHED_EXT policy intead of all\n"
" -v Print libbpf debug messages\n"
" -h Display this help and exit\n";
@@ -59,7 +61,7 @@ int main(int argc, char **argv)
skel = SCX_OPS_OPEN(qmap_ops, scx_qmap);
- while ((opt = getopt(argc, argv, "s:e:t:T:b:d:pvh")) != -1) {
+ while ((opt = getopt(argc, argv, "s:e:t:T:b:d:D:Spvh")) != -1) {
switch (opt) {
case 's':
skel->rodata->slice_ns = strtoull(optarg, NULL, 0) * 1000;
@@ -81,6 +83,12 @@ int main(int argc, char **argv)
if (skel->rodata->disallow_tgid < 0)
skel->rodata->disallow_tgid = getpid();
break;
+ case 'D':
+ skel->struct_ops.qmap_ops->exit_dump_len = strtoul(optarg, NULL, 0);
+ break;
+ case 'S':
+ skel->rodata->suppress_dump = true;
+ break;
case 'p':
skel->struct_ops.qmap_ops->flags |= SCX_OPS_SWITCH_PARTIAL;
break;
@@ -93,7 +101,7 @@ int main(int argc, char **argv)
}
}
- SCX_OPS_LOAD(skel, qmap_ops, scx_qmap);
+ SCX_OPS_LOAD(skel, qmap_ops, scx_qmap, uei);
link = SCX_OPS_ATTACH(skel, qmap_ops, scx_qmap);
while (!exit_req && !UEI_EXITED(skel, uei)) {
diff --git a/tools/sched_ext/scx_simple.c b/tools/sched_ext/scx_simple.c
index 789ac62fea8e..7f500d1d56ac 100644
--- a/tools/sched_ext/scx_simple.c
+++ b/tools/sched_ext/scx_simple.c
@@ -80,7 +80,7 @@ int main(int argc, char **argv)
}
}
- SCX_OPS_LOAD(skel, simple_ops, scx_simple);
+ SCX_OPS_LOAD(skel, simple_ops, scx_simple, uei);
link = SCX_OPS_ATTACH(skel, simple_ops, scx_simple);
while (!exit_req && !UEI_EXITED(skel, uei)) {
--
2.45.2
Diff
---
include/trace/events/sched_ext.h | 32 ++
kernel/sched/ext.c | 421 ++++++++++++++++++-
tools/sched_ext/include/scx/common.bpf.h | 12 +
tools/sched_ext/include/scx/compat.h | 9 +-
tools/sched_ext/include/scx/user_exit_info.h | 19 +
tools/sched_ext/scx_qmap.bpf.c | 54 +++
tools/sched_ext/scx_qmap.c | 14 +-
tools/sched_ext/scx_simple.c | 2 +-
8 files changed, 555 insertions(+), 8 deletions(-)
create mode 100644 include/trace/events/sched_ext.h
diff --git a/include/trace/events/sched_ext.h b/include/trace/events/sched_ext.h
new file mode 100644
index 000000000000..fe19da7315a9
--- /dev/null
+++ b/include/trace/events/sched_ext.h
@@ -0,0 +1,32 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#undef TRACE_SYSTEM
+#define TRACE_SYSTEM sched_ext
+
+#if !defined(_TRACE_SCHED_EXT_H) || defined(TRACE_HEADER_MULTI_READ)
+#define _TRACE_SCHED_EXT_H
+
+#include <linux/tracepoint.h>
+
+TRACE_EVENT(sched_ext_dump,
+
+ TP_PROTO(const char *line),
+
+ TP_ARGS(line),
+
+ TP_STRUCT__entry(
+ __string(line, line)
+ ),
+
+ TP_fast_assign(
+ __assign_str(line);
+ ),
+
+ TP_printk("%s",
+ __get_str(line)
+ )
+);
+
+#endif /* _TRACE_SCHED_EXT_H */
+
+/* This part must be outside protection */
+#include <trace/define_trace.h>
diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c
index 6f4de29d7372..66bb9cf075f0 100644
--- a/kernel/sched/ext.c
+++ b/kernel/sched/ext.c
@@ -12,6 +12,7 @@ enum scx_consts {
SCX_EXIT_BT_LEN = 64,
SCX_EXIT_MSG_LEN = 1024,
+ SCX_EXIT_DUMP_DFL_LEN = 32768,
};
enum scx_exit_kind {
@@ -48,6 +49,9 @@ struct scx_exit_info {
/* informational message */
char *msg;
+
+ /* debug dump */
+ char *dump;
};
/* sched_ext_ops.flags */
@@ -105,6 +109,17 @@ struct scx_exit_task_args {
bool cancelled;
};
+/*
+ * Informational context provided to dump operations.
+ */
+struct scx_dump_ctx {
+ enum scx_exit_kind kind;
+ s64 exit_code;
+ const char *reason;
+ u64 at_ns;
+ u64 at_jiffies;
+};
+
/**
* struct sched_ext_ops - Operation table for BPF scheduler implementation
*
@@ -296,6 +311,36 @@ struct sched_ext_ops {
*/
void (*disable)(struct task_struct *p);
+ /**
+ * dump - Dump BPF scheduler state on error
+ * @ctx: debug dump context
+ *
+ * Use scx_bpf_dump() to generate BPF scheduler specific debug dump.
+ */
+ void (*dump)(struct scx_dump_ctx *ctx);
+
+ /**
+ * dump_cpu - Dump BPF scheduler state for a CPU on error
+ * @ctx: debug dump context
+ * @cpu: CPU to generate debug dump for
+ * @idle: @cpu is currently idle without any runnable tasks
+ *
+ * Use scx_bpf_dump() to generate BPF scheduler specific debug dump for
+ * @cpu. If @idle is %true and this operation doesn't produce any
+ * output, @cpu is skipped for dump.
+ */
+ void (*dump_cpu)(struct scx_dump_ctx *ctx, s32 cpu, bool idle);
+
+ /**
+ * dump_task - Dump BPF scheduler state for a runnable task on error
+ * @ctx: debug dump context
+ * @p: runnable task to generate debug dump for
+ *
+ * Use scx_bpf_dump() to generate BPF scheduler specific debug dump for
+ * @p.
+ */
+ void (*dump_task)(struct scx_dump_ctx *ctx, struct task_struct *p);
+
/*
* All online ops must come before ops.init().
*/
@@ -330,6 +375,12 @@ struct sched_ext_ops {
*/
u32 timeout_ms;
+ /**
+ * exit_dump_len - scx_exit_info.dump buffer length. If 0, the default
+ * value of 32768 is used.
+ */
+ u32 exit_dump_len;
+
/**
* name - BPF scheduler's name
*
@@ -567,10 +618,27 @@ struct scx_bstr_buf {
static DEFINE_RAW_SPINLOCK(scx_exit_bstr_buf_lock);
static struct scx_bstr_buf scx_exit_bstr_buf;
+/* ops debug dump */
+struct scx_dump_data {
+ s32 cpu;
+ bool first;
+ s32 cursor;
+ struct seq_buf *s;
+ const char *prefix;
+ struct scx_bstr_buf buf;
+};
+
+struct scx_dump_data scx_dump_data = {
+ .cpu = -1,
+};
+
/* /sys/kernel/sched_ext interface */
static struct kset *scx_kset;
static struct kobject *scx_root_kobj;
+#define CREATE_TRACE_POINTS
+#include <trace/events/sched_ext.h>
+
static __printf(3, 4) void scx_ops_exit_kind(enum scx_exit_kind kind,
s64 exit_code,
const char *fmt, ...);
@@ -2897,12 +2965,13 @@ static void scx_ops_bypass(bool bypass)
static void free_exit_info(struct scx_exit_info *ei)
{
+ kfree(ei->dump);
kfree(ei->msg);
kfree(ei->bt);
kfree(ei);
}
-static struct scx_exit_info *alloc_exit_info(void)
+static struct scx_exit_info *alloc_exit_info(size_t exit_dump_len)
{
struct scx_exit_info *ei;
@@ -2912,8 +2981,9 @@ static struct scx_exit_info *alloc_exit_info(void)
ei->bt = kcalloc(sizeof(ei->bt[0]), SCX_EXIT_BT_LEN, GFP_KERNEL);
ei->msg = kzalloc(SCX_EXIT_MSG_LEN, GFP_KERNEL);
+ ei->dump = kzalloc(exit_dump_len, GFP_KERNEL);
- if (!ei->bt || !ei->msg) {
+ if (!ei->bt || !ei->msg || !ei->dump) {
free_exit_info(ei);
return NULL;
}
@@ -3125,8 +3195,274 @@ static void scx_ops_disable(enum scx_exit_kind kind)
schedule_scx_ops_disable_work();
}
+static void dump_newline(struct seq_buf *s)
+{
+ trace_sched_ext_dump("");
+
+ /* @s may be zero sized and seq_buf triggers WARN if so */
+ if (s->size)
+ seq_buf_putc(s, '\n');
+}
+
+static __printf(2, 3) void dump_line(struct seq_buf *s, const char *fmt, ...)
+{
+ va_list args;
+
+#ifdef CONFIG_TRACEPOINTS
+ if (trace_sched_ext_dump_enabled()) {
+ /* protected by scx_dump_state()::dump_lock */
+ static char line_buf[SCX_EXIT_MSG_LEN];
+
+ va_start(args, fmt);
+ vscnprintf(line_buf, sizeof(line_buf), fmt, args);
+ va_end(args);
+
+ trace_sched_ext_dump(line_buf);
+ }
+#endif
+ /* @s may be zero sized and seq_buf triggers WARN if so */
+ if (s->size) {
+ va_start(args, fmt);
+ seq_buf_vprintf(s, fmt, args);
+ va_end(args);
+
+ seq_buf_putc(s, '\n');
+ }
+}
+
+static void dump_stack_trace(struct seq_buf *s, const char *prefix,
+ const unsigned long *bt, unsigned int len)
+{
+ unsigned int i;
+
+ for (i = 0; i < len; i++)
+ dump_line(s, "%s%pS", prefix, (void *)bt[i]);
+}
+
+static void ops_dump_init(struct seq_buf *s, const char *prefix)
+{
+ struct scx_dump_data *dd = &scx_dump_data;
+
+ lockdep_assert_irqs_disabled();
+
+ dd->cpu = smp_processor_id(); /* allow scx_bpf_dump() */
+ dd->first = true;
+ dd->cursor = 0;
+ dd->s = s;
+ dd->prefix = prefix;
+}
+
+static void ops_dump_flush(void)
+{
+ struct scx_dump_data *dd = &scx_dump_data;
+ char *line = dd->buf.line;
+
+ if (!dd->cursor)
+ return;
+
+ /*
+ * There's something to flush and this is the first line. Insert a blank
+ * line to distinguish ops dump.
+ */
+ if (dd->first) {
+ dump_newline(dd->s);
+ dd->first = false;
+ }
+
+ /*
+ * There may be multiple lines in $line. Scan and emit each line
+ * separately.
+ */
+ while (true) {
+ char *end = line;
+ char c;
+
+ while (*end != '\n' && *end != '\0')
+ end++;
+
+ /*
+ * If $line overflowed, it may not have newline at the end.
+ * Always emit with a newline.
+ */
+ c = *end;
+ *end = '\0';
+ dump_line(dd->s, "%s%s", dd->prefix, line);
+ if (c == '\0')
+ break;
+
+ /* move to the next line */
+ end++;
+ if (*end == '\0')
+ break;
+ line = end;
+ }
+
+ dd->cursor = 0;
+}
+
+static void ops_dump_exit(void)
+{
+ ops_dump_flush();
+ scx_dump_data.cpu = -1;
+}
+
+static void scx_dump_task(struct seq_buf *s, struct scx_dump_ctx *dctx,
+ struct task_struct *p, char marker)
+{
+ static unsigned long bt[SCX_EXIT_BT_LEN];
+ char dsq_id_buf[19] = "(n/a)";
+ unsigned long ops_state = atomic_long_read(&p->scx.ops_state);
+ unsigned int bt_len;
+
+ if (p->scx.dsq)
+ scnprintf(dsq_id_buf, sizeof(dsq_id_buf), "0x%llx",
+ (unsigned long long)p->scx.dsq->id);
+
+ dump_newline(s);
+ dump_line(s, " %c%c %s[%d] %+ldms",
+ marker, task_state_to_char(p), p->comm, p->pid,
+ jiffies_delta_msecs(p->scx.runnable_at, dctx->at_jiffies));
+ dump_line(s, " scx_state/flags=%u/0x%x ops_state/qseq=%lu/%lu",
+ scx_get_task_state(p), p->scx.flags & ~SCX_TASK_STATE_MASK,
+ ops_state & SCX_OPSS_STATE_MASK,
+ ops_state >> SCX_OPSS_QSEQ_SHIFT);
+ dump_line(s, " sticky/holding_cpu=%d/%d dsq_id=%s",
+ p->scx.sticky_cpu, p->scx.holding_cpu, dsq_id_buf);
+ dump_line(s, " cpus=%*pb", cpumask_pr_args(p->cpus_ptr));
+
+ if (SCX_HAS_OP(dump_task)) {
+ ops_dump_init(s, " ");
+ SCX_CALL_OP(SCX_KF_REST, dump_task, dctx, p);
+ ops_dump_exit();
+ }
+
+ bt_len = stack_trace_save_tsk(p, bt, SCX_EXIT_BT_LEN, 1);
+ if (bt_len) {
+ dump_newline(s);
+ dump_stack_trace(s, " ", bt, bt_len);
+ }
+}
+
+static void scx_dump_state(struct scx_exit_info *ei, size_t dump_len)
+{
+ static DEFINE_SPINLOCK(dump_lock);
+ static const char trunc_marker[] = "\n\n~~~~ TRUNCATED ~~~~\n";
+ struct scx_dump_ctx dctx = {
+ .kind = ei->kind,
+ .exit_code = ei->exit_code,
+ .reason = ei->reason,
+ .at_ns = ktime_get_ns(),
+ .at_jiffies = jiffies,
+ };
+ struct seq_buf s;
+ unsigned long flags;
+ char *buf;
+ int cpu;
+
+ spin_lock_irqsave(&dump_lock, flags);
+
+ seq_buf_init(&s, ei->dump, dump_len);
+
+ if (ei->kind == SCX_EXIT_NONE) {
+ dump_line(&s, "Debug dump triggered by %s", ei->reason);
+ } else {
+ dump_line(&s, "%s[%d] triggered exit kind %d:",
+ current->comm, current->pid, ei->kind);
+ dump_line(&s, " %s (%s)", ei->reason, ei->msg);
+ dump_newline(&s);
+ dump_line(&s, "Backtrace:");
+ dump_stack_trace(&s, " ", ei->bt, ei->bt_len);
+ }
+
+ if (SCX_HAS_OP(dump)) {
+ ops_dump_init(&s, "");
+ SCX_CALL_OP(SCX_KF_UNLOCKED, dump, &dctx);
+ ops_dump_exit();
+ }
+
+ dump_newline(&s);
+ dump_line(&s, "CPU states");
+ dump_line(&s, "----------");
+
+ for_each_possible_cpu(cpu) {
+ struct rq *rq = cpu_rq(cpu);
+ struct rq_flags rf;
+ struct task_struct *p;
+ struct seq_buf ns;
+ size_t avail, used;
+ bool idle;
+
+ rq_lock(rq, &rf);
+
+ idle = list_empty(&rq->scx.runnable_list) &&
+ rq->curr->sched_class == &idle_sched_class;
+
+ if (idle && !SCX_HAS_OP(dump_cpu))
+ goto next;
+
+ /*
+ * We don't yet know whether ops.dump_cpu() will produce output
+ * and we may want to skip the default CPU dump if it doesn't.
+ * Use a nested seq_buf to generate the standard dump so that we
+ * can decide whether to commit later.
+ */
+ avail = seq_buf_get_buf(&s, &buf);
+ seq_buf_init(&ns, buf, avail);
+
+ dump_newline(&ns);
+ dump_line(&ns, "CPU %-4d: nr_run=%u ops_qseq=%lu",
+ cpu, rq->scx.nr_running, rq->scx.ops_qseq);
+ dump_line(&ns, " curr=%s[%d] class=%ps",
+ rq->curr->comm, rq->curr->pid,
+ rq->curr->sched_class);
+
+ used = seq_buf_used(&ns);
+ if (SCX_HAS_OP(dump_cpu)) {
+ ops_dump_init(&ns, " ");
+ SCX_CALL_OP(SCX_KF_REST, dump_cpu, &dctx, cpu, idle);
+ ops_dump_exit();
+ }
+
+ /*
+ * If idle && nothing generated by ops.dump_cpu(), there's
+ * nothing interesting. Skip.
+ */
+ if (idle && used == seq_buf_used(&ns))
+ goto next;
+
+ /*
+ * $s may already have overflowed when $ns was created. If so,
+ * calling commit on it will trigger BUG.
+ */
+ if (avail) {
+ seq_buf_commit(&s, seq_buf_used(&ns));
+ if (seq_buf_has_overflowed(&ns))
+ seq_buf_set_overflow(&s);
+ }
+
+ if (rq->curr->sched_class == &ext_sched_class)
+ scx_dump_task(&s, &dctx, rq->curr, '*');
+
+ list_for_each_entry(p, &rq->scx.runnable_list, scx.runnable_node)
+ scx_dump_task(&s, &dctx, p, ' ');
+ next:
+ rq_unlock(rq, &rf);
+ }
+
+ if (seq_buf_has_overflowed(&s) && dump_len >= sizeof(trunc_marker))
+ memcpy(ei->dump + dump_len - sizeof(trunc_marker),
+ trunc_marker, sizeof(trunc_marker));
+
+ spin_unlock_irqrestore(&dump_lock, flags);
+}
+
static void scx_ops_error_irq_workfn(struct irq_work *irq_work)
{
+ struct scx_exit_info *ei = scx_exit_info;
+
+ if (ei->kind >= SCX_EXIT_ERROR)
+ scx_dump_state(ei, scx_ops.exit_dump_len);
+
schedule_scx_ops_disable_work();
}
@@ -3152,6 +3488,13 @@ static __printf(3, 4) void scx_ops_exit_kind(enum scx_exit_kind kind,
vscnprintf(ei->msg, SCX_EXIT_MSG_LEN, fmt, args);
va_end(args);
+ /*
+ * Set ei->kind and ->reason for scx_dump_state(). They'll be set again
+ * in scx_ops_disable_workfn().
+ */
+ ei->kind = kind;
+ ei->reason = scx_exit_reason(ei->kind);
+
irq_work_queue(&scx_ops_error_irq_work);
}
@@ -3213,7 +3556,7 @@ static int scx_ops_enable(struct sched_ext_ops *ops, struct bpf_link *link)
if (ret < 0)
goto err;
- scx_exit_info = alloc_exit_info();
+ scx_exit_info = alloc_exit_info(ops->exit_dump_len);
if (!scx_exit_info) {
ret = -ENOMEM;
goto err_del;
@@ -3592,6 +3935,10 @@ static int bpf_scx_init_member(const struct btf_type *t,
return -E2BIG;
ops->timeout_ms = *(u32 *)(udata + moff);
return 1;
+ case offsetof(struct sched_ext_ops, exit_dump_len):
+ ops->exit_dump_len =
+ *(u32 *)(udata + moff) ?: SCX_EXIT_DUMP_DFL_LEN;
+ return 1;
}
return 0;
@@ -3723,6 +4070,21 @@ static const struct sysrq_key_op sysrq_sched_ext_reset_op = {
.enable_mask = SYSRQ_ENABLE_RTNICE,
};
+static void sysrq_handle_sched_ext_dump(u8 key)
+{
+ struct scx_exit_info ei = { .kind = SCX_EXIT_NONE, .reason = "SysRq-D" };
+
+ if (scx_enabled())
+ scx_dump_state(&ei, 0);
+}
+
+static const struct sysrq_key_op sysrq_sched_ext_dump_op = {
+ .handler = sysrq_handle_sched_ext_dump,
+ .help_msg = "dump-sched-ext(D)",
+ .action_msg = "Trigger sched_ext debug dump",
+ .enable_mask = SYSRQ_ENABLE_RTNICE,
+};
+
/**
* print_scx_info - print out sched_ext scheduler state
* @log_lvl: the log level to use when printing
@@ -3793,6 +4155,7 @@ void __init init_sched_ext_class(void)
}
register_sysrq_key('S', &sysrq_sched_ext_reset_op);
+ register_sysrq_key('D', &sysrq_sched_ext_dump_op);
INIT_DELAYED_WORK(&scx_watchdog_work, scx_watchdog_workfn);
}
@@ -4218,6 +4581,57 @@ __bpf_kfunc void scx_bpf_error_bstr(char *fmt, unsigned long long *data,
raw_spin_unlock_irqrestore(&scx_exit_bstr_buf_lock, flags);
}
+/**
+ * scx_bpf_dump - Generate extra debug dump specific to the BPF scheduler
+ * @fmt: format string
+ * @data: format string parameters packaged using ___bpf_fill() macro
+ * @data__sz: @data len, must end in '__sz' for the verifier
+ *
+ * To be called through scx_bpf_dump() helper from ops.dump(), dump_cpu() and
+ * dump_task() to generate extra debug dump specific to the BPF scheduler.
+ *
+ * The extra dump may be multiple lines. A single line may be split over
+ * multiple calls. The last line is automatically terminated.
+ */
+__bpf_kfunc void scx_bpf_dump_bstr(char *fmt, unsigned long long *data,
+ u32 data__sz)
+{
+ struct scx_dump_data *dd = &scx_dump_data;
+ struct scx_bstr_buf *buf = &dd->buf;
+ s32 ret;
+
+ if (raw_smp_processor_id() != dd->cpu) {
+ scx_ops_error("scx_bpf_dump() must only be called from ops.dump() and friends");
+ return;
+ }
+
+ /* append the formatted string to the line buf */
+ ret = __bstr_format(buf->data, buf->line + dd->cursor,
+ sizeof(buf->line) - dd->cursor, fmt, data, data__sz);
+ if (ret < 0) {
+ dump_line(dd->s, "%s[!] (\"%s\", %p, %u) failed to format (%d)",
+ dd->prefix, fmt, data, data__sz, ret);
+ return;
+ }
+
+ dd->cursor += ret;
+ dd->cursor = min_t(s32, dd->cursor, sizeof(buf->line));
+
+ if (!dd->cursor)
+ return;
+
+ /*
+ * If the line buf overflowed or ends in a newline, flush it into the
+ * dump. This is to allow the caller to generate a single line over
+ * multiple calls. As ops_dump_flush() can also handle multiple lines in
+ * the line buf, the only case which can lead to an unexpected
+ * truncation is when the caller keeps generating newlines in the middle
+ * instead of the end consecutively. Don't do that.
+ */
+ if (dd->cursor >= sizeof(buf->line) || buf->line[dd->cursor - 1] == '\n')
+ ops_dump_flush();
+}
+
/**
* scx_bpf_nr_cpu_ids - Return the number of possible CPU IDs
*
@@ -4426,6 +4840,7 @@ BTF_ID_FLAGS(func, scx_bpf_dsq_nr_queued)
BTF_ID_FLAGS(func, scx_bpf_destroy_dsq)
BTF_ID_FLAGS(func, scx_bpf_exit_bstr, KF_TRUSTED_ARGS)
BTF_ID_FLAGS(func, scx_bpf_error_bstr, KF_TRUSTED_ARGS)
+BTF_ID_FLAGS(func, scx_bpf_dump_bstr, KF_TRUSTED_ARGS)
BTF_ID_FLAGS(func, scx_bpf_nr_cpu_ids)
BTF_ID_FLAGS(func, scx_bpf_get_possible_cpumask, KF_ACQUIRE)
BTF_ID_FLAGS(func, scx_bpf_get_online_cpumask, KF_ACQUIRE)
diff --git a/tools/sched_ext/include/scx/common.bpf.h b/tools/sched_ext/include/scx/common.bpf.h
index 833fe1bdccf9..3ea5cdf58bc7 100644
--- a/tools/sched_ext/include/scx/common.bpf.h
+++ b/tools/sched_ext/include/scx/common.bpf.h
@@ -38,6 +38,7 @@ s32 scx_bpf_dsq_nr_queued(u64 dsq_id) __ksym;
void scx_bpf_destroy_dsq(u64 dsq_id) __ksym;
void scx_bpf_exit_bstr(s64 exit_code, char *fmt, unsigned long long *data, u32 data__sz) __ksym __weak;
void scx_bpf_error_bstr(char *fmt, unsigned long long *data, u32 data_len) __ksym;
+void scx_bpf_dump_bstr(char *fmt, unsigned long long *data, u32 data_len) __ksym __weak;
u32 scx_bpf_nr_cpu_ids(void) __ksym __weak;
const struct cpumask *scx_bpf_get_possible_cpumask(void) __ksym __weak;
const struct cpumask *scx_bpf_get_online_cpumask(void) __ksym __weak;
@@ -97,6 +98,17 @@ void ___scx_bpf_bstr_format_checker(const char *fmt, ...) {}
___scx_bpf_bstr_format_checker(fmt, ##args); \
})
+/*
+ * scx_bpf_dump() wraps the scx_bpf_dump_bstr() kfunc with variadic arguments
+ * instead of an array of u64. To be used from ops.dump() and friends.
+ */
+#define scx_bpf_dump(fmt, args...) \
+({ \
+ scx_bpf_bstr_preamble(fmt, args) \
+ scx_bpf_dump_bstr(___fmt, ___param, sizeof(___param)); \
+ ___scx_bpf_bstr_format_checker(fmt, ##args); \
+})
+
#define BPF_STRUCT_OPS(name, args...) \
SEC("struct_ops/"#name) \
BPF_PROG(name, ##args)
diff --git a/tools/sched_ext/include/scx/compat.h b/tools/sched_ext/include/scx/compat.h
index a7fdaf8a858e..c58024c980c8 100644
--- a/tools/sched_ext/include/scx/compat.h
+++ b/tools/sched_ext/include/scx/compat.h
@@ -111,16 +111,23 @@ static inline bool __COMPAT_struct_has_field(const char *type, const char *field
* is used to define ops and compat.h::SCX_OPS_LOAD/ATTACH() are used to load
* and attach it, backward compatibility is automatically maintained where
* reasonable.
+ *
+ * ec7e3b0463e1 ("implement-ops") in https://github.com/sched-ext/sched_ext is
+ * the current minimum required kernel version.
*/
#define SCX_OPS_OPEN(__ops_name, __scx_name) ({ \
struct __scx_name *__skel; \
\
+ SCX_BUG_ON(!__COMPAT_struct_has_field("sched_ext_ops", "dump"), \
+ "sched_ext_ops.dump() missing, kernel too old?"); \
+ \
__skel = __scx_name##__open(); \
SCX_BUG_ON(!__skel, "Could not open " #__scx_name); \
__skel; \
})
-#define SCX_OPS_LOAD(__skel, __ops_name, __scx_name) ({ \
+#define SCX_OPS_LOAD(__skel, __ops_name, __scx_name, __uei_name) ({ \
+ UEI_SET_SIZE(__skel, __ops_name, __uei_name); \
SCX_BUG_ON(__scx_name##__load((__skel)), "Failed to load skel"); \
})
diff --git a/tools/sched_ext/include/scx/user_exit_info.h b/tools/sched_ext/include/scx/user_exit_info.h
index 8c3b7fac4d05..c2ef85c645e1 100644
--- a/tools/sched_ext/include/scx/user_exit_info.h
+++ b/tools/sched_ext/include/scx/user_exit_info.h
@@ -13,6 +13,7 @@
enum uei_sizes {
UEI_REASON_LEN = 128,
UEI_MSG_LEN = 1024,
+ UEI_DUMP_DFL_LEN = 32768,
};
struct user_exit_info {
@@ -28,6 +29,8 @@ struct user_exit_info {
#include <bpf/bpf_core_read.h>
#define UEI_DEFINE(__name) \
+ char RESIZABLE_ARRAY(data, __name##_dump); \
+ const volatile u32 __name##_dump_len; \
struct user_exit_info __name SEC(".data")
#define UEI_RECORD(__uei_name, __ei) ({ \
@@ -35,6 +38,8 @@ struct user_exit_info {
sizeof(__uei_name.reason), (__ei)->reason); \
bpf_probe_read_kernel_str(__uei_name.msg, \
sizeof(__uei_name.msg), (__ei)->msg); \
+ bpf_probe_read_kernel_str(__uei_name##_dump, \
+ __uei_name##_dump_len, (__ei)->dump); \
if (bpf_core_field_exists((__ei)->exit_code)) \
__uei_name.exit_code = (__ei)->exit_code; \
/* use __sync to force memory barrier */ \
@@ -47,6 +52,13 @@ struct user_exit_info {
#include <stdio.h>
#include <stdbool.h>
+/* no need to call the following explicitly if SCX_OPS_LOAD() is used */
+#define UEI_SET_SIZE(__skel, __ops_name, __uei_name) ({ \
+ u32 __len = (__skel)->struct_ops.__ops_name->exit_dump_len ?: UEI_DUMP_DFL_LEN; \
+ (__skel)->rodata->__uei_name##_dump_len = __len; \
+ RESIZE_ARRAY((__skel), data, __uei_name##_dump, __len); \
+})
+
#define UEI_EXITED(__skel, __uei_name) ({ \
/* use __sync to force memory barrier */ \
__sync_val_compare_and_swap(&(__skel)->data->__uei_name.kind, -1, -1); \
@@ -54,6 +66,13 @@ struct user_exit_info {
#define UEI_REPORT(__skel, __uei_name) ({ \
struct user_exit_info *__uei = &(__skel)->data->__uei_name; \
+ char *__uei_dump = (__skel)->data_##__uei_name##_dump->__uei_name##_dump; \
+ if (__uei_dump[0] != '\0') { \
+ fputs("\nDEBUG DUMP\n", stderr); \
+ fputs("================================================================================\n\n", stderr); \
+ fputs(__uei_dump, stderr); \
+ fputs("\n================================================================================\n\n", stderr); \
+ } \
fprintf(stderr, "EXIT: %s", __uei->reason); \
if (__uei->msg[0] != '\0') \
fprintf(stderr, " (%s)", __uei->msg); \
diff --git a/tools/sched_ext/scx_qmap.bpf.c b/tools/sched_ext/scx_qmap.bpf.c
index 5ff217c4bfa0..5b3da28bf042 100644
--- a/tools/sched_ext/scx_qmap.bpf.c
+++ b/tools/sched_ext/scx_qmap.bpf.c
@@ -33,6 +33,7 @@ const volatile u32 stall_user_nth;
const volatile u32 stall_kernel_nth;
const volatile u32 dsp_batch;
const volatile s32 disallow_tgid;
+const volatile bool suppress_dump;
u32 test_error_cnt;
@@ -258,6 +259,56 @@ s32 BPF_STRUCT_OPS(qmap_init_task, struct task_struct *p,
return -ENOMEM;
}
+void BPF_STRUCT_OPS(qmap_dump, struct scx_dump_ctx *dctx)
+{
+ s32 i, pid;
+
+ if (suppress_dump)
+ return;
+
+ bpf_for(i, 0, 5) {
+ void *fifo;
+
+ if (!(fifo = bpf_map_lookup_elem(&queue_arr, &i)))
+ return;
+
+ scx_bpf_dump("QMAP FIFO[%d]:", i);
+ bpf_repeat(4096) {
+ if (bpf_map_pop_elem(fifo, &pid))
+ break;
+ scx_bpf_dump(" %d", pid);
+ }
+ scx_bpf_dump("\n");
+ }
+}
+
+void BPF_STRUCT_OPS(qmap_dump_cpu, struct scx_dump_ctx *dctx, s32 cpu, bool idle)
+{
+ u32 zero = 0;
+ struct cpu_ctx *cpuc;
+
+ if (suppress_dump || idle)
+ return;
+ if (!(cpuc = bpf_map_lookup_percpu_elem(&cpu_ctx_stor, &zero, cpu)))
+ return;
+
+ scx_bpf_dump("QMAP: dsp_idx=%llu dsp_cnt=%llu",
+ cpuc->dsp_idx, cpuc->dsp_cnt);
+}
+
+void BPF_STRUCT_OPS(qmap_dump_task, struct scx_dump_ctx *dctx, struct task_struct *p)
+{
+ struct task_ctx *taskc;
+
+ if (suppress_dump)
+ return;
+ if (!(taskc = bpf_task_storage_get(&task_ctx_stor, p, 0, 0)))
+ return;
+
+ scx_bpf_dump("QMAP: force_local=%d",
+ taskc->force_local);
+}
+
s32 BPF_STRUCT_OPS_SLEEPABLE(qmap_init)
{
return scx_bpf_create_dsq(SHARED_DSQ, -1);
@@ -274,6 +325,9 @@ SCX_OPS_DEFINE(qmap_ops,
.dequeue = (void *)qmap_dequeue,
.dispatch = (void *)qmap_dispatch,
.init_task = (void *)qmap_init_task,
+ .dump = (void *)qmap_dump,
+ .dump_cpu = (void *)qmap_dump_cpu,
+ .dump_task = (void *)qmap_dump_task,
.init = (void *)qmap_init,
.exit = (void *)qmap_exit,
.timeout_ms = 5000U,
diff --git a/tools/sched_ext/scx_qmap.c b/tools/sched_ext/scx_qmap.c
index a2614994cfaa..a1123a17581b 100644
--- a/tools/sched_ext/scx_qmap.c
+++ b/tools/sched_ext/scx_qmap.c
@@ -20,7 +20,7 @@ const char help_fmt[] =
"See the top-level comment in .bpf.c for more details.\n"
"\n"
"Usage: %s [-s SLICE_US] [-e COUNT] [-t COUNT] [-T COUNT] [-b COUNT]\n"
-" [-d PID] [-p] [-v]\n"
+" [-d PID] [-D LEN] [-p] [-v]\n"
"\n"
" -s SLICE_US Override slice duration\n"
" -e COUNT Trigger scx_bpf_error() after COUNT enqueues\n"
@@ -28,6 +28,8 @@ const char help_fmt[] =
" -T COUNT Stall every COUNT'th kernel thread\n"
" -b COUNT Dispatch upto COUNT tasks together\n"
" -d PID Disallow a process from switching into SCHED_EXT (-1 for self)\n"
+" -D LEN Set scx_exit_info.dump buffer length\n"
+" -S Suppress qmap-specific debug dump\n"
" -p Switch only tasks on SCHED_EXT policy intead of all\n"
" -v Print libbpf debug messages\n"
" -h Display this help and exit\n";
@@ -59,7 +61,7 @@ int main(int argc, char **argv)
skel = SCX_OPS_OPEN(qmap_ops, scx_qmap);
- while ((opt = getopt(argc, argv, "s:e:t:T:b:d:pvh")) != -1) {
+ while ((opt = getopt(argc, argv, "s:e:t:T:b:d:D:Spvh")) != -1) {
switch (opt) {
case 's':
skel->rodata->slice_ns = strtoull(optarg, NULL, 0) * 1000;
@@ -81,6 +83,12 @@ int main(int argc, char **argv)
if (skel->rodata->disallow_tgid < 0)
skel->rodata->disallow_tgid = getpid();
break;
+ case 'D':
+ skel->struct_ops.qmap_ops->exit_dump_len = strtoul(optarg, NULL, 0);
+ break;
+ case 'S':
+ skel->rodata->suppress_dump = true;
+ break;
case 'p':
skel->struct_ops.qmap_ops->flags |= SCX_OPS_SWITCH_PARTIAL;
break;
@@ -93,7 +101,7 @@ int main(int argc, char **argv)
}
}
- SCX_OPS_LOAD(skel, qmap_ops, scx_qmap);
+ SCX_OPS_LOAD(skel, qmap_ops, scx_qmap, uei);
link = SCX_OPS_ATTACH(skel, qmap_ops, scx_qmap);
while (!exit_req && !UEI_EXITED(skel, uei)) {
diff --git a/tools/sched_ext/scx_simple.c b/tools/sched_ext/scx_simple.c
index 789ac62fea8e..7f500d1d56ac 100644
--- a/tools/sched_ext/scx_simple.c
+++ b/tools/sched_ext/scx_simple.c
@@ -80,7 +80,7 @@ int main(int argc, char **argv)
}
}
- SCX_OPS_LOAD(skel, simple_ops, scx_simple);
+ SCX_OPS_LOAD(skel, simple_ops, scx_simple, uei);
link = SCX_OPS_ATTACH(skel, simple_ops, scx_simple);
while (!exit_req && !UEI_EXITED(skel, uei)) {
--
2.45.2
Implementation Analysis
Overview
When a BPF scheduler crashes, violates an invariant, or is killed by the watchdog, the kernel unloads it and falls back to CFS. At that moment, a great deal of state that would explain what went wrong — which tasks were queued, what the scheduler's internal state was — becomes inaccessible. This patch adds a post-mortem debug dump mechanism: at the time of error, the kernel captures CPU and task state, and optionally lets the BPF scheduler write its own diagnostic data. All of this is printed after the BPF program is unloaded, giving a clear picture of the failure.
Code Walkthrough
kernel/sched/ext.c — scx_exit_info gains a dump buffer
SCX_EXIT_DUMP_DFL_LEN = 32768, // new constant
struct scx_exit_info {
...
char *dump; // NEW: BPF scheduler's debug dump
};
alloc_exit_info() now takes a exit_dump_len parameter (defaulting to SCX_EXIT_DUMP_DFL_LEN = 32 KB if the BPF scheduler specifies 0 via ops.exit_dump_len). This buffer is where both the kernel-generated CPU/task dump and optional BPF-written data land.
sched_ext_ops — new fields and callbacks
u32 exit_dump_len; // how many bytes of dump buffer to allocate
void (*dump)(struct scx_dump_ctx *ctx); // global dump
void (*dump_cpu)(struct scx_dump_ctx *ctx, s32 cpu, bool idle); // per-CPU dump
void (*dump_task)(struct scx_dump_ctx *ctx, struct task_struct *p); // per-task dump
These optional callbacks are invoked during the dump phase. ops.dump() is called once for global scheduler state. ops.dump_cpu() is called for each CPU. ops.dump_task() is called for each runnable task. BPF schedulers call scx_bpf_dump() (a kfunc backed by scx_bpf_dump_bstr()) within these callbacks to write formatted text into the dump buffer.
kernel/sched/ext.c — scx_dump_state() orchestration
The existing scx_dump_state() function is significantly extended. At error-exit time it:
- Allocates a
seq_bufbacked byei->dump. - Calls
ops.dump()if provided (global scheduler state). - Iterates each CPU: prints
nr_run,ops_qseq, andcurr. Callsops.dump_cpu()if provided. - For each runnable task on each CPU: prints
scx_state,flags,ops_state, timing info (runnable_at), stack trace, and callsops.dump_task()if provided.
The dump is written to ei->dump and then printed via pr_info() after the BPF program is fully unloaded. This ordering is deliberate: the BPF program must be gone before printing so that it cannot interfere with the output.
include/trace/events/sched_ext.h — tracepoints
A new sched_ext.h tracepoint header is added. The CREATE_TRACE_POINTS define is moved inside ext.c (from outside in earlier patches) to accommodate this. Tracepoints are added for the dump path so that tools like perf or BPF programs can observe dump events without parsing kernel log output.
tools/sched_ext/include/scx/user_exit_info.h — userspace dump reading
UEI_DUMP_DFL_LEN = 32768,
#define UEI_REPORT(__skel, __uei_name)
The UEI_REPORT macro (User Exit Info) is updated to also print the dump buffer that the BPF program has written. This is the userspace counterpart to the kernel's ei->dump: after the BPF scheduler exits, the controlling userspace process calls UEI_REPORT() which prints both the exit reason and the dump contents.
tools/sched_ext/scx_qmap.bpf.c — demonstration
const volatile bool suppress_dump;
void BPF_STRUCT_OPS(qmap_dump, struct scx_dump_ctx *ctx) { ... }
void BPF_STRUCT_OPS(qmap_dump_cpu, struct scx_dump_ctx *ctx, s32 cpu, bool idle) { ... }
void BPF_STRUCT_OPS(qmap_dump_task, struct scx_dump_ctx *ctx, struct task_struct *p) { ... }
scx_qmap is updated with all three dump callbacks as a working example. The -D LEN flag sets exit_dump_len. The suppress_dump flag disables the BPF dump callbacks for testing.
Key Concepts
scx_exit_info.dump: A pre-allocated ring buffer (size configured byops.exit_dump_len) that holds both kernel-generated and BPF-generated debug text. Allocated at BPF scheduler load time so it is always available even during OOM conditions.ops.exit_dump_len: The BPF scheduler declares how much dump buffer it needs. A scheduler that writes verbose per-task state should request more. 0 defaults to 32 KB.ops.dump/ops.dump_cpu/ops.dump_task: Three optional hooks called during the dump phase. They receive ascx_dump_ctxand usescx_bpf_dump()to write formatted output.scx_bpf_dump(): A__bpf_kfuncthat appends a formatted string to the current dump buffer. It is only callable from within the dump callbacks. Declared__weakin the BPF header so schedulers built against older kernels that lack it can still compile.- Post-mortem timing: The dump is captured at the moment of error (CPU/task state snapshot) but printed after BPF program unload. This means the printed state is from the moment of failure, not from the moment the printk runs.
Locking and Concurrency Notes
scx_dump_state()is called from the BPF scheduler disable path which holds no rq locks globally. Per-CPU state is read under each CPU's rq lock (taken and released per-CPU during the dump iteration).- The
scx_dump_dataglobal is protected by being accessed only from the error exit path, which is serialized by the global SCX state machine (only one BPF scheduler can be active at a time). - BPF dump callbacks (
ops.dump,ops.dump_cpu,ops.dump_task) are called withSCX_KF_UNLOCKEDcontext — no locks are held when they run, so the BPF program can safely call sleep-able helpers.
Why Maintainers Need to Know This
- The dump is your primary post-mortem tool: When a user reports a BPF scheduler crash, ask for the kernel log. The dump will show CPU run queues, runnable task states with
runnable_atdeltas, the exit kind and message, and any BPF-provided diagnostics. ops.exit_dump_lenmust be set large enough: If the BPF scheduler writes more dump data than the buffer holds, output is truncated. A common mistake is leavingexit_dump_lenat 0 and then writing extensive per-task debug data fromops.dump_task().suppress_dumpin scx_qmap is a test fixture: It exists so that automated tests can trigger error exits without generating verbose kernel log output. This pattern is useful in test schedulers.- The dump buffer is allocated at load time: If the system is memory-constrained when the BPF scheduler is loaded,
alloc_exit_info()may fail, and the dump will be unavailable. The core handles this gracefully (no dump printed), but a maintainer should be aware that absence of a dump does not mean the error was clean.
Connection to Other Patches
- The
scx_exit_infostruct (withexit_kind,exit_code,msg, and nowdump) is consumed by theUEI_REPORTmacro in userspace tool headers — this patch completes the exit information flow from kernel to userspace. p->scx.runnable_atand thejiffies_delta_msecs()helper from PATCH 14/30 are used here indump_taskto show how long each runnable task has been waiting — the same value thatprint_scx_info()uses in stack dumps.- The
ops.dump_cpuandops.dump_taskcallbacks build on the same callback dispatch infrastructure (SCX_CALL_OP) used by all other sched_ext ops.
[PATCH 16/30] tools/sched_ext: Add scx_show_state.py
View on Lore: https://lore.kernel.org/all/20240618212056.2833381-17-tj@kernel.org
Commit Message
There are states which are interesting but don't quite fit the interface
exposed under /sys/kernel/sched_ext. Add tools/scx_show_state.py to show
them.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: David Vernet <dvernet@meta.com>
---
tools/sched_ext/scx_show_state.py | 39 +++++++++++++++++++++++++++++++
1 file changed, 39 insertions(+)
create mode 100644 tools/sched_ext/scx_show_state.py
diff --git a/tools/sched_ext/scx_show_state.py b/tools/sched_ext/scx_show_state.py
new file mode 100644
index 000000000000..d457d2a74e1e
--- /dev/null
+++ b/tools/sched_ext/scx_show_state.py
@@ -0,0 +1,39 @@
+#!/usr/bin/env drgn
+#
+# Copyright (C) 2024 Tejun Heo <tj@kernel.org>
+# Copyright (C) 2024 Meta Platforms, Inc. and affiliates.
+
+desc = """
+This is a drgn script to show the current sched_ext state.
+For more info on drgn, visit https://github.com/osandov/drgn.
+"""
+
+import drgn
+import sys
+
+def err(s):
+ print(s, file=sys.stderr, flush=True)
+ sys.exit(1)
+
+def read_int(name):
+ return int(prog[name].value_())
+
+def read_atomic(name):
+ return prog[name].counter.value_()
+
+def read_static_key(name):
+ return prog[name].key.enabled.counter.value_()
+
+def ops_state_str(state):
+ return prog['scx_ops_enable_state_str'][state].string_().decode()
+
+ops = prog['scx_ops']
+enable_state = read_atomic("scx_ops_enable_state_var")
+
+print(f'ops : {ops.name.string_().decode()}')
+print(f'enabled : {read_static_key("__scx_ops_enabled")}')
+print(f'switching_all : {read_int("scx_switching_all")}')
+print(f'switched_all : {read_static_key("__scx_switched_all")}')
+print(f'enable_state : {ops_state_str(enable_state)} ({enable_state})')
+print(f'bypass_depth : {read_atomic("scx_ops_bypass_depth")}')
+print(f'nr_rejected : {read_atomic("scx_nr_rejected")}')
--
2.45.2
Diff
---
tools/sched_ext/scx_show_state.py | 39 +++++++++++++++++++++++++++++++
1 file changed, 39 insertions(+)
create mode 100644 tools/sched_ext/scx_show_state.py
diff --git a/tools/sched_ext/scx_show_state.py b/tools/sched_ext/scx_show_state.py
new file mode 100644
index 000000000000..d457d2a74e1e
--- /dev/null
+++ b/tools/sched_ext/scx_show_state.py
@@ -0,0 +1,39 @@
+#!/usr/bin/env drgn
+#
+# Copyright (C) 2024 Tejun Heo <tj@kernel.org>
+# Copyright (C) 2024 Meta Platforms, Inc. and affiliates.
+
+desc = """
+This is a drgn script to show the current sched_ext state.
+For more info on drgn, visit https://github.com/osandov/drgn.
+"""
+
+import drgn
+import sys
+
+def err(s):
+ print(s, file=sys.stderr, flush=True)
+ sys.exit(1)
+
+def read_int(name):
+ return int(prog[name].value_())
+
+def read_atomic(name):
+ return prog[name].counter.value_()
+
+def read_static_key(name):
+ return prog[name].key.enabled.counter.value_()
+
+def ops_state_str(state):
+ return prog['scx_ops_enable_state_str'][state].string_().decode()
+
+ops = prog['scx_ops']
+enable_state = read_atomic("scx_ops_enable_state_var")
+
+print(f'ops : {ops.name.string_().decode()}')
+print(f'enabled : {read_static_key("__scx_ops_enabled")}')
+print(f'switching_all : {read_int("scx_switching_all")}')
+print(f'switched_all : {read_static_key("__scx_switched_all")}')
+print(f'enable_state : {ops_state_str(enable_state)} ({enable_state})')
+print(f'bypass_depth : {read_atomic("scx_ops_bypass_depth")}')
+print(f'nr_rejected : {read_atomic("scx_nr_rejected")}')
--
2.45.2
Implementation Analysis
Overview
Some critical sched_ext runtime state is not exposed through the sysfs interface at /sys/kernel/sched_ext and must be read directly from kernel memory. This patch adds tools/sched_ext/scx_show_state.py, a drgn script that reads live kernel state and displays a snapshot of the sched_ext subsystem: which BPF scheduler is loaded, its enable state, whether it is running in full-switch mode, the bypass depth, and how many tasks have been rejected due to the disallow flag.
Code Walkthrough
tools/sched_ext/scx_show_state.py — full file
The script uses drgn, a programmable Linux kernel debugger that reads live kernel memory from a running system. It requires no kernel modules or BPF programs — drgn reads directly from /proc/kcore (or a kernel core dump).
#!/usr/bin/env drgn
def read_int(name):
return int(prog[name].value_())
def read_atomic(name):
return prog[name].counter.value_()
def read_static_key(name):
return prog[name].key.enabled.counter.value_()
def ops_state_str(state):
return prog['scx_ops_enable_state_str'][state].string_().decode()
ops = prog['scx_ops']
enable_state = read_atomic("scx_ops_enable_state_var")
print(f'ops : {ops.name.string_().decode()}')
print(f'enabled : {read_static_key("__scx_ops_enabled")}')
print(f'switching_all : {read_int("scx_switching_all")}')
print(f'switched_all : {read_static_key("__scx_switched_all")}')
print(f'enable_state : {ops_state_str(enable_state)} ({enable_state})')
print(f'bypass_depth : {read_atomic("scx_ops_bypass_depth")}')
print(f'nr_rejected : {read_atomic("scx_nr_rejected")}')
Each field maps directly to a kernel variable:
ops.name: Thenamefield of the currently registeredsched_ext_opsstruct — tells you which BPF scheduler is loaded.__scx_ops_enabled: A static key (jump label) that is1when any BPF scheduler is active. This is the fast-path check used in the hot scheduling path.scx_switching_all: Whether the BPF scheduler is running in "switch all" mode (all tasks use SCHED_EXT, not just those with explicit SCHED_EXT policy).__scx_switched_all: A static key that is1when switch-all mode is fully active (distinct fromscx_switching_allwhich is the intent;__scx_switched_allreflects the actual active state).scx_ops_enable_state_var: The currentenum scx_ops_enable_statevalue (PREPPING, ENABLING, ENABLED, DISABLING, DISABLED). Reading this as an atomic counter and mapping throughscx_ops_enable_state_str[]gives the human-readable state.scx_ops_bypass_depth: How deeply the bypass mode is nested. Non-zero means the BPF scheduler is bypassed (e.g., during CPU hotplug or PM operations) and the system is running with built-in fallback behavior.scx_nr_rejected: Count of tasks rejected from SCHED_EXT due top->scx.disallowsince the last BPF scheduler load. Added by PATCH 13/30.
Key Concepts
- drgn vs. debugfs: This tool reads kernel variables that are not exposed via
/sys/kernel/sched_ext. The sysfs interface only exposesstate,switch_all, andnr_rejected. The drgn script can access any kernel symbol, making it more flexible for debugging scenarios where intermediate state matters. - Static keys (
read_static_key()): sched_ext uses static branch/jump labels for performance-critical checks likescx_enabled(). A static key's runtime value is stored inkey.enabled.counter— this is the internal implementation detail that drgn must access since these are not simple variables. - Atomic variables (
read_atomic()):scx_ops_enable_state_var,scx_ops_bypass_depth, andscx_nr_rejectedareatomic_t/atomic_long_t— drgn reads.counter.value_()for these. bypass_depthas a debugging signal: A non-zerobypass_depthwhen a BPF scheduler should be active indicates the system is in a transitional or suspended state. Ifbypass_depthstays non-zero indefinitely, it suggests a bypass entry/exit imbalance bug.
Locking and Concurrency Notes
This is a read-only userspace tool that accesses kernel memory without any synchronization. All values read may be transiently inconsistent with each other (e.g., enable_state might be ENABLED while enabled is 0 during a transition). This is acceptable for a diagnostic snapshot tool. The script should be used to get a general picture of system state, not as a definitive single-point-in-time snapshot.
Why Maintainers Need to Know This
- Use this tool to verify BPF scheduler load: After loading a BPF scheduler, run
sudo python scx_show_state.pyto confirmops,enabled, andenable_stateall match expectations. A mismatch betweenscx_switching_allandswitched_allindicates the mode transition is not yet complete. bypass_depth > 0indicates suppressed scheduling: If users report that a BPF scheduler is loaded but not making scheduling decisions, checkbypass_depth. A stuck bypass is a known failure mode during PM suspend/resume sequences.nr_rejectedmonitors disallow policy: If you have a BPF scheduler that usesp->scx.disallow, watchnr_rejectedto confirm the policy is working. A value of 0 when you expect rejections means thedisallowflag is not being set correctly.- Tool depends on kernel symbol names: If kernel variables are renamed or restructured, this script will break. It is tied to the internal variable names of a specific kernel version. Users should ensure the drgn script matches their kernel.
Connection to Other Patches
- PATCH 13/30 introduced
scx_nr_rejected— this tool is the first way to read that counter without grep-ing/sys/kernel/sched_ext/nr_rejected. - PATCH 15/30 introduced the debug dump mechanism for error exits; this tool complements it by showing the live state before an error occurs.
- The
scx_ops_enable_state_str[]array read by this tool was made available outsideCONFIG_SCHED_DEBUGin PATCH 14/30, which is a prerequisite for this script to work on production kernels without debug config.
[PATCH 18/30] sched_ext: Add a central scheduler which makes all scheduling decisions on one CPU
View on Lore: https://lore.kernel.org/all/20240618212056.2833381-19-tj@kernel.org
Commit Message
This patch adds a new example scheduler, scx_central, which demonstrates
central scheduling where one CPU is responsible for making all scheduling
decisions in the system using scx_bpf_kick_cpu(). The central CPU makes
scheduling decisions for all CPUs in the system, queues tasks on the
appropriate local dsq's and preempts the worker CPUs. The worker CPUs in
turn preempt the central CPU when it needs tasks to run.
Currently, every CPU depends on its own tick to expire the current task. A
follow-up patch implementing tickless support for sched_ext will allow the
worker CPUs to go full tickless so that they can run completely undisturbed.
v3: - Kumar fixed a bug where the dispatch path could overflow the dispatch
buffer if too many are dispatched to the fallback DSQ.
- Use the new SCX_KICK_IDLE to wake up non-central CPUs.
- Dropped '-p' option.
v2: - Use RESIZABLE_ARRAY() instead of fixed MAX_CPUS and use SCX_BUG[_ON]()
to simplify error handling.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: David Vernet <dvernet@meta.com>
Acked-by: Josh Don <joshdon@google.com>
Acked-by: Hao Luo <haoluo@google.com>
Acked-by: Barret Rhoden <brho@google.com>
Cc: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Cc: Julia Lawall <julia.lawall@inria.fr>
---
tools/sched_ext/Makefile | 2 +-
tools/sched_ext/scx_central.bpf.c | 214 ++++++++++++++++++++++++++++++
tools/sched_ext/scx_central.c | 105 +++++++++++++++
3 files changed, 320 insertions(+), 1 deletion(-)
create mode 100644 tools/sched_ext/scx_central.bpf.c
create mode 100644 tools/sched_ext/scx_central.c
diff --git a/tools/sched_ext/Makefile b/tools/sched_ext/Makefile
index 626782a21375..bf7e108f5ae1 100644
--- a/tools/sched_ext/Makefile
+++ b/tools/sched_ext/Makefile
@@ -176,7 +176,7 @@ $(INCLUDE_DIR)/%.bpf.skel.h: $(SCXOBJ_DIR)/%.bpf.o $(INCLUDE_DIR)/vmlinux.h $(BP
SCX_COMMON_DEPS := include/scx/common.h include/scx/user_exit_info.h | $(BINDIR)
-c-sched-targets = scx_simple scx_qmap
+c-sched-targets = scx_simple scx_qmap scx_central
$(addprefix $(BINDIR)/,$(c-sched-targets)): \
$(BINDIR)/%: \
diff --git a/tools/sched_ext/scx_central.bpf.c b/tools/sched_ext/scx_central.bpf.c
new file mode 100644
index 000000000000..428b2262faa3
--- /dev/null
+++ b/tools/sched_ext/scx_central.bpf.c
@@ -0,0 +1,214 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * A central FIFO sched_ext scheduler which demonstrates the followings:
+ *
+ * a. Making all scheduling decisions from one CPU:
+ *
+ * The central CPU is the only one making scheduling decisions. All other
+ * CPUs kick the central CPU when they run out of tasks to run.
+ *
+ * There is one global BPF queue and the central CPU schedules all CPUs by
+ * dispatching from the global queue to each CPU's local dsq from dispatch().
+ * This isn't the most straightforward. e.g. It'd be easier to bounce
+ * through per-CPU BPF queues. The current design is chosen to maximally
+ * utilize and verify various SCX mechanisms such as LOCAL_ON dispatching.
+ *
+ * b. Preemption
+ *
+ * SCX_KICK_PREEMPT is used to trigger scheduling and CPUs to move to the
+ * next tasks.
+ *
+ * This scheduler is designed to maximize usage of various SCX mechanisms. A
+ * more practical implementation would likely put the scheduling loop outside
+ * the central CPU's dispatch() path and add some form of priority mechanism.
+ *
+ * Copyright (c) 2022 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2022 Tejun Heo <tj@kernel.org>
+ * Copyright (c) 2022 David Vernet <dvernet@meta.com>
+ */
+#include <scx/common.bpf.h>
+
+char _license[] SEC("license") = "GPL";
+
+enum {
+ FALLBACK_DSQ_ID = 0,
+};
+
+const volatile s32 central_cpu;
+const volatile u32 nr_cpu_ids = 1; /* !0 for veristat, set during init */
+const volatile u64 slice_ns = SCX_SLICE_DFL;
+
+u64 nr_total, nr_locals, nr_queued, nr_lost_pids;
+u64 nr_dispatches, nr_mismatches, nr_retries;
+u64 nr_overflows;
+
+UEI_DEFINE(uei);
+
+struct {
+ __uint(type, BPF_MAP_TYPE_QUEUE);
+ __uint(max_entries, 4096);
+ __type(value, s32);
+} central_q SEC(".maps");
+
+/* can't use percpu map due to bad lookups */
+bool RESIZABLE_ARRAY(data, cpu_gimme_task);
+
+s32 BPF_STRUCT_OPS(central_select_cpu, struct task_struct *p,
+ s32 prev_cpu, u64 wake_flags)
+{
+ /*
+ * Steer wakeups to the central CPU as much as possible to avoid
+ * disturbing other CPUs. It's safe to blindly return the central cpu as
+ * select_cpu() is a hint and if @p can't be on it, the kernel will
+ * automatically pick a fallback CPU.
+ */
+ return central_cpu;
+}
+
+void BPF_STRUCT_OPS(central_enqueue, struct task_struct *p, u64 enq_flags)
+{
+ s32 pid = p->pid;
+
+ __sync_fetch_and_add(&nr_total, 1);
+
+ if (bpf_map_push_elem(¢ral_q, &pid, 0)) {
+ __sync_fetch_and_add(&nr_overflows, 1);
+ scx_bpf_dispatch(p, FALLBACK_DSQ_ID, SCX_SLICE_DFL, enq_flags);
+ return;
+ }
+
+ __sync_fetch_and_add(&nr_queued, 1);
+
+ if (!scx_bpf_task_running(p))
+ scx_bpf_kick_cpu(central_cpu, SCX_KICK_PREEMPT);
+}
+
+static bool dispatch_to_cpu(s32 cpu)
+{
+ struct task_struct *p;
+ s32 pid;
+
+ bpf_repeat(BPF_MAX_LOOPS) {
+ if (bpf_map_pop_elem(¢ral_q, &pid))
+ break;
+
+ __sync_fetch_and_sub(&nr_queued, 1);
+
+ p = bpf_task_from_pid(pid);
+ if (!p) {
+ __sync_fetch_and_add(&nr_lost_pids, 1);
+ continue;
+ }
+
+ /*
+ * If we can't run the task at the top, do the dumb thing and
+ * bounce it to the fallback dsq.
+ */
+ if (!bpf_cpumask_test_cpu(cpu, p->cpus_ptr)) {
+ __sync_fetch_and_add(&nr_mismatches, 1);
+ scx_bpf_dispatch(p, FALLBACK_DSQ_ID, SCX_SLICE_DFL, 0);
+ bpf_task_release(p);
+ /*
+ * We might run out of dispatch buffer slots if we continue dispatching
+ * to the fallback DSQ, without dispatching to the local DSQ of the
+ * target CPU. In such a case, break the loop now as will fail the
+ * next dispatch operation.
+ */
+ if (!scx_bpf_dispatch_nr_slots())
+ break;
+ continue;
+ }
+
+ /* dispatch to local and mark that @cpu doesn't need more */
+ scx_bpf_dispatch(p, SCX_DSQ_LOCAL_ON | cpu, SCX_SLICE_DFL, 0);
+
+ if (cpu != central_cpu)
+ scx_bpf_kick_cpu(cpu, SCX_KICK_IDLE);
+
+ bpf_task_release(p);
+ return true;
+ }
+
+ return false;
+}
+
+void BPF_STRUCT_OPS(central_dispatch, s32 cpu, struct task_struct *prev)
+{
+ if (cpu == central_cpu) {
+ /* dispatch for all other CPUs first */
+ __sync_fetch_and_add(&nr_dispatches, 1);
+
+ bpf_for(cpu, 0, nr_cpu_ids) {
+ bool *gimme;
+
+ if (!scx_bpf_dispatch_nr_slots())
+ break;
+
+ /* central's gimme is never set */
+ gimme = ARRAY_ELEM_PTR(cpu_gimme_task, cpu, nr_cpu_ids);
+ if (gimme && !*gimme)
+ continue;
+
+ if (dispatch_to_cpu(cpu))
+ *gimme = false;
+ }
+
+ /*
+ * Retry if we ran out of dispatch buffer slots as we might have
+ * skipped some CPUs and also need to dispatch for self. The ext
+ * core automatically retries if the local dsq is empty but we
+ * can't rely on that as we're dispatching for other CPUs too.
+ * Kick self explicitly to retry.
+ */
+ if (!scx_bpf_dispatch_nr_slots()) {
+ __sync_fetch_and_add(&nr_retries, 1);
+ scx_bpf_kick_cpu(central_cpu, SCX_KICK_PREEMPT);
+ return;
+ }
+
+ /* look for a task to run on the central CPU */
+ if (scx_bpf_consume(FALLBACK_DSQ_ID))
+ return;
+ dispatch_to_cpu(central_cpu);
+ } else {
+ bool *gimme;
+
+ if (scx_bpf_consume(FALLBACK_DSQ_ID))
+ return;
+
+ gimme = ARRAY_ELEM_PTR(cpu_gimme_task, cpu, nr_cpu_ids);
+ if (gimme)
+ *gimme = true;
+
+ /*
+ * Force dispatch on the scheduling CPU so that it finds a task
+ * to run for us.
+ */
+ scx_bpf_kick_cpu(central_cpu, SCX_KICK_PREEMPT);
+ }
+}
+
+int BPF_STRUCT_OPS_SLEEPABLE(central_init)
+{
+ return scx_bpf_create_dsq(FALLBACK_DSQ_ID, -1);
+}
+
+void BPF_STRUCT_OPS(central_exit, struct scx_exit_info *ei)
+{
+ UEI_RECORD(uei, ei);
+}
+
+SCX_OPS_DEFINE(central_ops,
+ /*
+ * We are offloading all scheduling decisions to the central CPU
+ * and thus being the last task on a given CPU doesn't mean
+ * anything special. Enqueue the last tasks like any other tasks.
+ */
+ .flags = SCX_OPS_ENQ_LAST,
+
+ .select_cpu = (void *)central_select_cpu,
+ .enqueue = (void *)central_enqueue,
+ .dispatch = (void *)central_dispatch,
+ .init = (void *)central_init,
+ .exit = (void *)central_exit,
+ .name = "central");
diff --git a/tools/sched_ext/scx_central.c b/tools/sched_ext/scx_central.c
new file mode 100644
index 000000000000..5f09fc666a63
--- /dev/null
+++ b/tools/sched_ext/scx_central.c
@@ -0,0 +1,105 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2022 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2022 Tejun Heo <tj@kernel.org>
+ * Copyright (c) 2022 David Vernet <dvernet@meta.com>
+ */
+#define _GNU_SOURCE
+#include <sched.h>
+#include <stdio.h>
+#include <unistd.h>
+#include <inttypes.h>
+#include <signal.h>
+#include <libgen.h>
+#include <bpf/bpf.h>
+#include <scx/common.h>
+#include "scx_central.bpf.skel.h"
+
+const char help_fmt[] =
+"A central FIFO sched_ext scheduler.\n"
+"\n"
+"See the top-level comment in .bpf.c for more details.\n"
+"\n"
+"Usage: %s [-s SLICE_US] [-c CPU]\n"
+"\n"
+" -s SLICE_US Override slice duration\n"
+" -c CPU Override the central CPU (default: 0)\n"
+" -v Print libbpf debug messages\n"
+" -h Display this help and exit\n";
+
+static bool verbose;
+static volatile int exit_req;
+
+static int libbpf_print_fn(enum libbpf_print_level level, const char *format, va_list args)
+{
+ if (level == LIBBPF_DEBUG && !verbose)
+ return 0;
+ return vfprintf(stderr, format, args);
+}
+
+static void sigint_handler(int dummy)
+{
+ exit_req = 1;
+}
+
+int main(int argc, char **argv)
+{
+ struct scx_central *skel;
+ struct bpf_link *link;
+ __u64 seq = 0;
+ __s32 opt;
+
+ libbpf_set_print(libbpf_print_fn);
+ signal(SIGINT, sigint_handler);
+ signal(SIGTERM, sigint_handler);
+
+ skel = SCX_OPS_OPEN(central_ops, scx_central);
+
+ skel->rodata->central_cpu = 0;
+ skel->rodata->nr_cpu_ids = libbpf_num_possible_cpus();
+
+ while ((opt = getopt(argc, argv, "s:c:pvh")) != -1) {
+ switch (opt) {
+ case 's':
+ skel->rodata->slice_ns = strtoull(optarg, NULL, 0) * 1000;
+ break;
+ case 'c':
+ skel->rodata->central_cpu = strtoul(optarg, NULL, 0);
+ break;
+ case 'v':
+ verbose = true;
+ break;
+ default:
+ fprintf(stderr, help_fmt, basename(argv[0]));
+ return opt != 'h';
+ }
+ }
+
+ /* Resize arrays so their element count is equal to cpu count. */
+ RESIZE_ARRAY(skel, data, cpu_gimme_task, skel->rodata->nr_cpu_ids);
+
+ SCX_OPS_LOAD(skel, central_ops, scx_central, uei);
+ link = SCX_OPS_ATTACH(skel, central_ops, scx_central);
+
+ while (!exit_req && !UEI_EXITED(skel, uei)) {
+ printf("[SEQ %llu]\n", seq++);
+ printf("total :%10" PRIu64 " local:%10" PRIu64 " queued:%10" PRIu64 " lost:%10" PRIu64 "\n",
+ skel->bss->nr_total,
+ skel->bss->nr_locals,
+ skel->bss->nr_queued,
+ skel->bss->nr_lost_pids);
+ printf(" dispatch:%10" PRIu64 " mismatch:%10" PRIu64 " retry:%10" PRIu64 "\n",
+ skel->bss->nr_dispatches,
+ skel->bss->nr_mismatches,
+ skel->bss->nr_retries);
+ printf("overflow:%10" PRIu64 "\n",
+ skel->bss->nr_overflows);
+ fflush(stdout);
+ sleep(1);
+ }
+
+ bpf_link__destroy(link);
+ UEI_REPORT(skel, uei);
+ scx_central__destroy(skel);
+ return 0;
+}
--
2.45.2
Diff
---
tools/sched_ext/Makefile | 2 +-
tools/sched_ext/scx_central.bpf.c | 214 ++++++++++++++++++++++++++++++
tools/sched_ext/scx_central.c | 105 +++++++++++++++
3 files changed, 320 insertions(+), 1 deletion(-)
create mode 100644 tools/sched_ext/scx_central.bpf.c
create mode 100644 tools/sched_ext/scx_central.c
diff --git a/tools/sched_ext/Makefile b/tools/sched_ext/Makefile
index 626782a21375..bf7e108f5ae1 100644
--- a/tools/sched_ext/Makefile
+++ b/tools/sched_ext/Makefile
@@ -176,7 +176,7 @@ $(INCLUDE_DIR)/%.bpf.skel.h: $(SCXOBJ_DIR)/%.bpf.o $(INCLUDE_DIR)/vmlinux.h $(BP
SCX_COMMON_DEPS := include/scx/common.h include/scx/user_exit_info.h | $(BINDIR)
-c-sched-targets = scx_simple scx_qmap
+c-sched-targets = scx_simple scx_qmap scx_central
$(addprefix $(BINDIR)/,$(c-sched-targets)): \
$(BINDIR)/%: \
diff --git a/tools/sched_ext/scx_central.bpf.c b/tools/sched_ext/scx_central.bpf.c
new file mode 100644
index 000000000000..428b2262faa3
--- /dev/null
+++ b/tools/sched_ext/scx_central.bpf.c
@@ -0,0 +1,214 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * A central FIFO sched_ext scheduler which demonstrates the followings:
+ *
+ * a. Making all scheduling decisions from one CPU:
+ *
+ * The central CPU is the only one making scheduling decisions. All other
+ * CPUs kick the central CPU when they run out of tasks to run.
+ *
+ * There is one global BPF queue and the central CPU schedules all CPUs by
+ * dispatching from the global queue to each CPU's local dsq from dispatch().
+ * This isn't the most straightforward. e.g. It'd be easier to bounce
+ * through per-CPU BPF queues. The current design is chosen to maximally
+ * utilize and verify various sched_ext mechanisms such as LOCAL_ON dispatching.
+ *
+ * b. Preemption
+ *
+ * SCX_KICK_PREEMPT is used to trigger scheduling and CPUs to move to the
+ * next tasks.
+ *
+ * This scheduler is designed to maximize usage of various sched_ext mechanisms. A
+ * more practical implementation would likely put the scheduling loop outside
+ * the central CPU's dispatch() path and add some form of priority mechanism.
+ *
+ * Copyright (c) 2022 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2022 Tejun Heo <tj@kernel.org>
+ * Copyright (c) 2022 David Vernet <dvernet@meta.com>
+ */
+#include <scx/common.bpf.h>
+
+char _license[] SEC("license") = "GPL";
+
+enum {
+ FALLBACK_DSQ_ID = 0,
+};
+
+const volatile s32 central_cpu;
+const volatile u32 nr_cpu_ids = 1; /* !0 for veristat, set during init */
+const volatile u64 slice_ns = SCX_SLICE_DFL;
+
+u64 nr_total, nr_locals, nr_queued, nr_lost_pids;
+u64 nr_dispatches, nr_mismatches, nr_retries;
+u64 nr_overflows;
+
+UEI_DEFINE(uei);
+
+struct {
+ __uint(type, BPF_MAP_TYPE_QUEUE);
+ __uint(max_entries, 4096);
+ __type(value, s32);
+} central_q SEC(".maps");
+
+/* can't use percpu map due to bad lookups */
+bool RESIZABLE_ARRAY(data, cpu_gimme_task);
+
+s32 BPF_STRUCT_OPS(central_select_cpu, struct task_struct *p,
+ s32 prev_cpu, u64 wake_flags)
+{
+ /*
+ * Steer wakeups to the central CPU as much as possible to avoid
+ * disturbing other CPUs. It's safe to blindly return the central cpu as
+ * select_cpu() is a hint and if @p can't be on it, the kernel will
+ * automatically pick a fallback CPU.
+ */
+ return central_cpu;
+}
+
+void BPF_STRUCT_OPS(central_enqueue, struct task_struct *p, u64 enq_flags)
+{
+ s32 pid = p->pid;
+
+ __sync_fetch_and_add(&nr_total, 1);
+
+ if (bpf_map_push_elem(¢ral_q, &pid, 0)) {
+ __sync_fetch_and_add(&nr_overflows, 1);
+ scx_bpf_dispatch(p, FALLBACK_DSQ_ID, SCX_SLICE_DFL, enq_flags);
+ return;
+ }
+
+ __sync_fetch_and_add(&nr_queued, 1);
+
+ if (!scx_bpf_task_running(p))
+ scx_bpf_kick_cpu(central_cpu, SCX_KICK_PREEMPT);
+}
+
+static bool dispatch_to_cpu(s32 cpu)
+{
+ struct task_struct *p;
+ s32 pid;
+
+ bpf_repeat(BPF_MAX_LOOPS) {
+ if (bpf_map_pop_elem(¢ral_q, &pid))
+ break;
+
+ __sync_fetch_and_sub(&nr_queued, 1);
+
+ p = bpf_task_from_pid(pid);
+ if (!p) {
+ __sync_fetch_and_add(&nr_lost_pids, 1);
+ continue;
+ }
+
+ /*
+ * If we can't run the task at the top, do the dumb thing and
+ * bounce it to the fallback dsq.
+ */
+ if (!bpf_cpumask_test_cpu(cpu, p->cpus_ptr)) {
+ __sync_fetch_and_add(&nr_mismatches, 1);
+ scx_bpf_dispatch(p, FALLBACK_DSQ_ID, SCX_SLICE_DFL, 0);
+ bpf_task_release(p);
+ /*
+ * We might run out of dispatch buffer slots if we continue dispatching
+ * to the fallback DSQ, without dispatching to the local DSQ of the
+ * target CPU. In such a case, break the loop now as will fail the
+ * next dispatch operation.
+ */
+ if (!scx_bpf_dispatch_nr_slots())
+ break;
+ continue;
+ }
+
+ /* dispatch to local and mark that @cpu doesn't need more */
+ scx_bpf_dispatch(p, SCX_DSQ_LOCAL_ON | cpu, SCX_SLICE_DFL, 0);
+
+ if (cpu != central_cpu)
+ scx_bpf_kick_cpu(cpu, SCX_KICK_IDLE);
+
+ bpf_task_release(p);
+ return true;
+ }
+
+ return false;
+}
+
+void BPF_STRUCT_OPS(central_dispatch, s32 cpu, struct task_struct *prev)
+{
+ if (cpu == central_cpu) {
+ /* dispatch for all other CPUs first */
+ __sync_fetch_and_add(&nr_dispatches, 1);
+
+ bpf_for(cpu, 0, nr_cpu_ids) {
+ bool *gimme;
+
+ if (!scx_bpf_dispatch_nr_slots())
+ break;
+
+ /* central's gimme is never set */
+ gimme = ARRAY_ELEM_PTR(cpu_gimme_task, cpu, nr_cpu_ids);
+ if (gimme && !*gimme)
+ continue;
+
+ if (dispatch_to_cpu(cpu))
+ *gimme = false;
+ }
+
+ /*
+ * Retry if we ran out of dispatch buffer slots as we might have
+ * skipped some CPUs and also need to dispatch for self. The ext
+ * core automatically retries if the local dsq is empty but we
+ * can't rely on that as we're dispatching for other CPUs too.
+ * Kick self explicitly to retry.
+ */
+ if (!scx_bpf_dispatch_nr_slots()) {
+ __sync_fetch_and_add(&nr_retries, 1);
+ scx_bpf_kick_cpu(central_cpu, SCX_KICK_PREEMPT);
+ return;
+ }
+
+ /* look for a task to run on the central CPU */
+ if (scx_bpf_consume(FALLBACK_DSQ_ID))
+ return;
+ dispatch_to_cpu(central_cpu);
+ } else {
+ bool *gimme;
+
+ if (scx_bpf_consume(FALLBACK_DSQ_ID))
+ return;
+
+ gimme = ARRAY_ELEM_PTR(cpu_gimme_task, cpu, nr_cpu_ids);
+ if (gimme)
+ *gimme = true;
+
+ /*
+ * Force dispatch on the scheduling CPU so that it finds a task
+ * to run for us.
+ */
+ scx_bpf_kick_cpu(central_cpu, SCX_KICK_PREEMPT);
+ }
+}
+
+int BPF_STRUCT_OPS_SLEEPABLE(central_init)
+{
+ return scx_bpf_create_dsq(FALLBACK_DSQ_ID, -1);
+}
+
+void BPF_STRUCT_OPS(central_exit, struct scx_exit_info *ei)
+{
+ UEI_RECORD(uei, ei);
+}
+
+SCX_OPS_DEFINE(central_ops,
+ /*
+ * We are offloading all scheduling decisions to the central CPU
+ * and thus being the last task on a given CPU doesn't mean
+ * anything special. Enqueue the last tasks like any other tasks.
+ */
+ .flags = SCX_OPS_ENQ_LAST,
+
+ .select_cpu = (void *)central_select_cpu,
+ .enqueue = (void *)central_enqueue,
+ .dispatch = (void *)central_dispatch,
+ .init = (void *)central_init,
+ .exit = (void *)central_exit,
+ .name = "central");
diff --git a/tools/sched_ext/scx_central.c b/tools/sched_ext/scx_central.c
new file mode 100644
index 000000000000..5f09fc666a63
--- /dev/null
+++ b/tools/sched_ext/scx_central.c
@@ -0,0 +1,105 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2022 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2022 Tejun Heo <tj@kernel.org>
+ * Copyright (c) 2022 David Vernet <dvernet@meta.com>
+ */
+#define _GNU_SOURCE
+#include <sched.h>
+#include <stdio.h>
+#include <unistd.h>
+#include <inttypes.h>
+#include <signal.h>
+#include <libgen.h>
+#include <bpf/bpf.h>
+#include <scx/common.h>
+#include "scx_central.bpf.skel.h"
+
+const char help_fmt[] =
+"A central FIFO sched_ext scheduler.\n"
+"\n"
+"See the top-level comment in .bpf.c for more details.\n"
+"\n"
+"Usage: %s [-s SLICE_US] [-c CPU]\n"
+"\n"
+" -s SLICE_US Override slice duration\n"
+" -c CPU Override the central CPU (default: 0)\n"
+" -v Print libbpf debug messages\n"
+" -h Display this help and exit\n";
+
+static bool verbose;
+static volatile int exit_req;
+
+static int libbpf_print_fn(enum libbpf_print_level level, const char *format, va_list args)
+{
+ if (level == LIBBPF_DEBUG && !verbose)
+ return 0;
+ return vfprintf(stderr, format, args);
+}
+
+static void sigint_handler(int dummy)
+{
+ exit_req = 1;
+}
+
+int main(int argc, char **argv)
+{
+ struct scx_central *skel;
+ struct bpf_link *link;
+ __u64 seq = 0;
+ __s32 opt;
+
+ libbpf_set_print(libbpf_print_fn);
+ signal(SIGINT, sigint_handler);
+ signal(SIGTERM, sigint_handler);
+
+ skel = SCX_OPS_OPEN(central_ops, scx_central);
+
+ skel->rodata->central_cpu = 0;
+ skel->rodata->nr_cpu_ids = libbpf_num_possible_cpus();
+
+ while ((opt = getopt(argc, argv, "s:c:pvh")) != -1) {
+ switch (opt) {
+ case 's':
+ skel->rodata->slice_ns = strtoull(optarg, NULL, 0) * 1000;
+ break;
+ case 'c':
+ skel->rodata->central_cpu = strtoul(optarg, NULL, 0);
+ break;
+ case 'v':
+ verbose = true;
+ break;
+ default:
+ fprintf(stderr, help_fmt, basename(argv[0]));
+ return opt != 'h';
+ }
+ }
+
+ /* Resize arrays so their element count is equal to cpu count. */
+ RESIZE_ARRAY(skel, data, cpu_gimme_task, skel->rodata->nr_cpu_ids);
+
+ SCX_OPS_LOAD(skel, central_ops, scx_central, uei);
+ link = SCX_OPS_ATTACH(skel, central_ops, scx_central);
+
+ while (!exit_req && !UEI_EXITED(skel, uei)) {
+ printf("[SEQ %llu]\n", seq++);
+ printf("total :%10" PRIu64 " local:%10" PRIu64 " queued:%10" PRIu64 " lost:%10" PRIu64 "\n",
+ skel->bss->nr_total,
+ skel->bss->nr_locals,
+ skel->bss->nr_queued,
+ skel->bss->nr_lost_pids);
+ printf(" dispatch:%10" PRIu64 " mismatch:%10" PRIu64 " retry:%10" PRIu64 "\n",
+ skel->bss->nr_dispatches,
+ skel->bss->nr_mismatches,
+ skel->bss->nr_retries);
+ printf("overflow:%10" PRIu64 "\n",
+ skel->bss->nr_overflows);
+ fflush(stdout);
+ sleep(1);
+ }
+
+ bpf_link__destroy(link);
+ UEI_REPORT(skel, uei);
+ scx_central__destroy(skel);
+ return 0;
+}
--
2.45.2
Implementation Analysis
Overview
scx_central is an example BPF scheduler demonstrating a "central scheduling" architecture: a single designated CPU (central_cpu, default CPU 0) makes all scheduling decisions for the entire system. All other CPUs ("worker CPUs") execute tasks but do not dispatch — they signal the central CPU when they need work. The central CPU uses scx_bpf_dispatch() with SCX_DSQ_LOCAL_ON | cpu to push tasks directly into remote CPUs' local DSQs, then kicks those CPUs with scx_bpf_kick_cpu(cpu, SCX_KICK_IDLE) to wake them.
This design is deliberately chosen to maximally exercise sched_ext mechanisms (LOCAL_ON dispatching, preemption via kick, the SCX_OPS_ENQ_LAST flag) rather than to be a practical production scheduler.
Code Walkthrough
scx_central.bpf.c — data structures
struct {
__uint(type, BPF_MAP_TYPE_QUEUE);
__uint(max_entries, 4096);
__type(value, s32);
} central_q SEC(".maps"); // global FIFO queue of PIDs
bool RESIZABLE_ARRAY(data, cpu_gimme_task); // per-CPU signal: "I need a task"
The global queue holds PIDs (not task pointers — BPF maps cannot hold pointers to kernel objects). Worker CPUs set their cpu_gimme_task[cpu] = true when they have no work; the central CPU checks this array to know which CPUs need dispatching.
central_select_cpu() — steer to central CPU
s32 BPF_STRUCT_OPS(central_select_cpu, struct task_struct *p,
s32 prev_cpu, u64 wake_flags)
{
return central_cpu;
}
All wakeups are steered toward the central CPU. select_cpu() is a hint — if the task cannot run on central_cpu (e.g., CPU affinity), the kernel picks a fallback. This minimizes cache disturbance on worker CPUs.
central_enqueue() — push PID to global queue
void BPF_STRUCT_OPS(central_enqueue, struct task_struct *p, u64 enq_flags)
{
s32 pid = p->pid;
if (bpf_map_push_elem(¢ral_q, &pid, 0)) {
// overflow: dispatch to fallback DSQ
scx_bpf_dispatch(p, FALLBACK_DSQ_ID, SCX_SLICE_DFL, enq_flags);
return;
}
if (!scx_bpf_task_running(p))
scx_bpf_kick_cpu(central_cpu, SCX_KICK_PREEMPT);
}
Tasks are enqueued as PIDs into a BPF QUEUE map (FIFO). If the queue is full, the task falls back to FALLBACK_DSQ_ID (a regular shared DSQ). If the task is not currently running anywhere, the central CPU is preempted (SCX_KICK_PREEMPT) to process the new arrival immediately.
dispatch_to_cpu() — central CPU dispatches to a specific worker
static bool dispatch_to_cpu(s32 cpu)
{
bpf_repeat(BPF_MAX_LOOPS) {
if (bpf_map_pop_elem(¢ral_q, &pid)) break;
p = bpf_task_from_pid(pid);
if (!p) { nr_lost_pids++; continue; }
if (!bpf_cpumask_test_cpu(cpu, p->cpus_ptr)) {
// CPU affinity mismatch: bounce to fallback DSQ
scx_bpf_dispatch(p, FALLBACK_DSQ_ID, SCX_SLICE_DFL, 0);
if (!scx_bpf_dispatch_nr_slots()) break;
continue;
}
// Dispatch directly to the target CPU's local DSQ
scx_bpf_dispatch(p, SCX_DSQ_LOCAL_ON | cpu, SCX_SLICE_DFL, 0);
if (cpu != central_cpu)
scx_bpf_kick_cpu(cpu, SCX_KICK_IDLE);
return true;
}
return false;
}
This is the heart of the central scheduler. It pops PIDs from the global queue, resolves them to tasks via bpf_task_from_pid(), checks CPU affinity, and dispatches to SCX_DSQ_LOCAL_ON | cpu (the specific CPU's local DSQ). Tasks with mismatched affinity are bounced to the fallback DSQ where they will be picked up by a CPU that can run them.
central_dispatch() — per-CPU dispatch callback
void BPF_STRUCT_OPS(central_dispatch, s32 cpu, struct task_struct *prev)
{
if (cpu == central_cpu) {
// Central CPU: dispatch for ALL other CPUs
bpf_for(cpu, 0, nr_cpu_ids) {
if (!scx_bpf_dispatch_nr_slots()) break;
gimme = ARRAY_ELEM_PTR(cpu_gimme_task, cpu, nr_cpu_ids);
if (gimme && !*gimme) continue;
if (dispatch_to_cpu(cpu)) *gimme = false;
}
// Self-kick if dispatch buffer exhausted
if (!scx_bpf_dispatch_nr_slots()) {
scx_bpf_kick_cpu(central_cpu, SCX_KICK_PREEMPT);
return;
}
// Try to find work for central CPU itself
if (scx_bpf_consume(FALLBACK_DSQ_ID)) return;
dispatch_to_cpu(central_cpu);
} else {
// Worker CPU: signal central CPU
if (scx_bpf_consume(FALLBACK_DSQ_ID)) return;
*gimme = true;
scx_bpf_kick_cpu(central_cpu, SCX_KICK_PREEMPT);
}
}
The central CPU iterates all CPUs, checks cpu_gimme_task[cpu], and dispatches work. The dispatch buffer has a finite number of slots (scx_bpf_dispatch_nr_slots()); if it fills up, the central CPU kicks itself to retry in the next scheduling cycle. Worker CPUs that need work set their gimme flag and kick the central CPU.
SCX_OPS_ENQ_LAST flag
SCX_OPS_DEFINE(central_ops,
.flags = SCX_OPS_ENQ_LAST,
...);
This flag tells sched_ext to call ops.enqueue() even for the last task on a CPU (the task that would otherwise just keep running). Without this, the "last task" would not go through ops.enqueue() and would bypass the central queue entirely. For a central scheduler that needs to control all task placement, this is mandatory.
Key Concepts
SCX_DSQ_LOCAL_ON | cpu: Dispatching to a specific CPU's local DSQ from a different CPU. This is the core primitive that makes remote dispatch possible. The kernel handles the necessary cross-CPU bookkeeping.RESIZABLE_ARRAY(data, cpu_gimme_task): A global boolean array indexed by CPU number, used as a signaling mechanism between worker CPUs and the central CPU. It must be resized at load time to match the actual number of CPUs viaRESIZE_ARRAY().- PID-based queueing hazard: The global queue stores PIDs, not pointers. By the time the central CPU dequeues a PID, the task may have exited (leading to
nr_lost_pids++) or changed its CPU affinity. The scheduler handles both cases. - Dispatch buffer overflow:
scx_bpf_dispatch_nr_slots()returns the number of available dispatch buffer slots. Exhausting these slots (when dispatching for many CPUs in oneops.dispatch()call) is handled by self-kicking the central CPU for a retry. - Fallback DSQ: A regular shared DSQ created in
central_init(). Tasks with CPU affinity constraints that cannot be dispatched to a specific worker CPU land here. Both the central CPU and worker CPUs try to consume from it viascx_bpf_consume(FALLBACK_DSQ_ID).
Locking and Concurrency Notes
ops.enqueue()andops.dispatch()are called with the CPU'srq->lockheld. Operations inside them must not sleep or acquire locks that could deadlock withrq->lock.bpf_task_from_pid()is safe to call fromops.dispatch()but requires callingbpf_task_release()on the returned pointer. The code correctly releases the task pointer in all code paths including the fallback dispatch case.- The
cpu_gimme_task[]array is accessed concurrently (worker CPUs write, central CPU reads) without explicit locking. The__sync_fetch_and_addatomics on the counter variables (nr_total,nr_queued, etc.) use atomic operations, butcpu_gimme_taskis accessed with plain loads/stores. This is a deliberate choice for performance — the worst case is a spurious dispatch attempt, not a correctness issue.
Why Maintainers Need to Know This
- This is the canonical example of
LOCAL_ONdispatch andscx_bpf_kick_cpu(): When reviewing or writing BPF schedulers that do remote dispatch,scx_centralis the reference implementation. The pattern of checkingcpu_gimme_task, dispatching withSCX_DSQ_LOCAL_ON | cpu, and kicking withSCX_KICK_IDLEis the established idiom. - The dispatch buffer limit is a real constraint:
scx_bpf_dispatch_nr_slots()must be checked when dispatching for multiple CPUs in a singleops.dispatch()call. Ignoring this leads to silent dispatch failures (tasks dropped into fallback DSQ unexpectedly). - CPU affinity mismatches create fallback DSQ pressure: In systems with tasks pinned to specific CPUs, the central scheduler will frequently encounter affinity mismatches and bounce tasks to the fallback DSQ. High
nr_mismatchesin the output indicates this. Production central schedulers should either pre-sort tasks by CPU affinity or maintain per-CPU queues. SCX_OPS_ENQ_LASTis required for true centralization: Any BPF scheduler that wants to control placement of every task (not just runnable-but-not-running tasks) must set this flag. Without it, "last task" scenarios bypass the central queue.
Connection to Other Patches
- PATCH 17/30 introduced
scx_bpf_kick_cpu()withSCX_KICK_IDLEandSCX_KICK_PREEMPT—scx_centralis explicitly the motivating example for both flags. - PATCH 21/30 (tickless support) extends
scx_centralfurther: worker CPUs dispatch tasks withSCX_SLICE_INFand a BPF timer on the central CPU replaces the tick for preemption. That patch demonstrates the full realization of the central scheduling concept. - PATCH 19/30 (dispatch loop watchdog) is partly motivated by the fact that a buggy central scheduler dispatching ineligible tasks could trap a CPU in the dispatch loop indefinitely.
CPU Coordination (Patches 17 and 19)
Overview
Scheduling a task requires not just deciding which task to run, but also coordinating the CPUs
that will run it. A BPF scheduler that can only enqueue tasks passively — waiting for each CPU
to call ops.dispatch() on its own schedule — cannot implement latency-sensitive policies or
ensure timely preemption of lower-priority work. Patches 17 and 19 add the active control
primitives that allow a BPF scheduler to reach out and influence CPU behavior directly.
Patch 17 introduces scx_bpf_kick_cpu(), the primary inter-CPU signaling mechanism in
sched_ext. Patch 19 extends the watchdog to detect a specific failure mode introduced by the
dispatch mechanism: an infinite loop inside ops.dispatch() that never yields control back
to the kernel.
Together these two patches complete the control loop between the BPF scheduler and the CPUs it manages: kick gives the BPF program the ability to push CPUs, and the extended watchdog ensures that push mechanism cannot be abused to lock up the system.
Why These Patches Are Needed
The Problem with Purely Reactive Dispatch
In the base sched_ext design (patch 09), the flow is:
- A CPU needs a task to run.
- The CPU calls
pick_next_task_scx(). - If the local DSQ is empty, the kernel calls
ops.dispatch(cpu, prev). - The BPF program fills the local DSQ.
- The CPU runs the task.
This is purely reactive: the CPU asks, the BPF program answers. For many scheduling policies this is sufficient, but consider these scenarios:
Latency-sensitive wakeup: A high-priority task wakes up on a system where all CPUs are
running lower-priority tasks. In the reactive model, the high-priority task must wait until
one of those CPUs naturally calls ops.dispatch() — which happens only after its current
task yields or is preempted by a timer. Until then, the high-priority task sits idle in a DSQ.
Idle CPU with work available: A task is dispatched to the global DSQ. Several CPUs are idle and could pick it up, but they are in the deep idle state (halted). In the reactive model, they will not wake up until the next interrupt. The task experiences unnecessary latency.
Cross-CPU work stealing: A CPU has an empty local DSQ and needs to steal work from another CPU's queue. The stealing CPU can query other CPUs' DSQs and dispatch tasks to itself, but it cannot tell an overloaded CPU to immediately re-evaluate its queues.
scx_bpf_kick_cpu() solves all three by letting any BPF context signal any CPU to take a
scheduling action immediately.
The Dispatch Loop Problem
ops.dispatch() is called with the runqueue lock held. If a BPF program enters an infinite
loop inside ops.dispatch() — for example, calling scx_bpf_consume() in a tight loop that
never runs out of tasks — the CPU will never release the runqueue lock, other CPUs that need
to migrate tasks to or from this CPU will spin waiting for the lock, and the system will
gradually deadlock.
The base watchdog (patch 12) detects task stalls: a runnable task that hasn't been scheduled.
But a spinning ops.dispatch() doesn't produce a stalled task — it produces a CPU that is
nominally "busy scheduling" but actually spinning. The base watchdog cannot catch this.
Patch 19 extends the watchdog specifically for this dispatch loop failure mode.
Key Concepts
PATCH 17 — scx_bpf_kick_cpu()
scx_bpf_kick_cpu(cpu, flags) is a BPF helper that sends an inter-processor interrupt (IPI)
to the target CPU, causing it to re-evaluate its scheduling state. The flags argument controls
what kind of action the target CPU will take:
SCX_KICK_IDLE: If the target CPU is idle (in the idle loop or halted), wake it up. If it
is not idle, this is a no-op — the CPU is already running a task and will naturally call
ops.dispatch() when that task finishes or yields.
Use case: A task was just dispatched to a DSQ, and the BPF program wants to ensure an idle CPU picks it up promptly rather than waiting for the next timer interrupt to fire.
SCX_KICK_PREEMPT: If the target CPU is running a SCX task, preempt it immediately.
This causes the CPU to finish the current scheduling quantum early and call pick_next_task_scx()
sooner than it would have naturally.
Use case: A high-priority task wakes up and needs a CPU. The BPF program identifies which CPU
is running the lowest-priority current task, kicks it with SCX_KICK_PREEMPT, and dispatches
the high-priority task to that CPU's local DSQ. The preempted CPU immediately picks up the
high-priority task.
SCX_KICK_WAIT (added in patch 23, cpu-coordination group): Block the calling CPU until
the target CPU has completed one full scheduling round. This is used when the kicking CPU needs
a guarantee that the target CPU has actually processed the kick before proceeding.
Implementation details
scx_bpf_kick_cpu() sends an IPI using smp_send_reschedule(cpu), the same mechanism the
kernel uses for normal task migrations. The target CPU handles the IPI by setting
TIF_NEED_RESCHED and, if idle, exiting the idle loop.
For SCX_KICK_PREEMPT, the target CPU's scx_rq.flags has SCX_RQ_PREEMPT set before the
IPI is sent. When the target CPU processes the reschedule, it checks this flag and calls
resched_curr() on itself, which causes the current task to be preempted at the next
scheduler tick or preemption point.
The BPF helper is accessible only from BPF programs attached to sched_ext — it is not a
general-purpose IPI mechanism. It is registered in the BPF verifier's allowed helper set for
the BPF_PROG_TYPE_STRUCT_OPS program type that implements sched_ext_ops.
Interaction with scx_central
Patch 18 (scx_central) is the primary consumer of scx_bpf_kick_cpu(). The central
scheduler, after dispatching tasks to a CPU's local DSQ, uses SCX_KICK_IDLE to wake that
CPU. Without the kick, the idle CPU might remain halted for up to several milliseconds (until
the next timer interrupt), adding avoidable latency to task wakeups.
PATCH 19 — Watchdog Extension for Dispatch Loops
The dispatch loop watchdog works as follows:
Detection mechanism: Each CPU tracks how long it has been in ops.dispatch(). A timestamp
scx_rq.dispatch_start is set when ops.dispatch() is entered and cleared when it returns.
The watchdog timer (which fires every scx_watchdog_timeout / 2) checks whether any CPU has
been in ops.dispatch() for longer than scx_watchdog_timeout.
Stall condition: If ops.dispatch() has been executing for more than scx_watchdog_timeout,
the watchdog calls scx_ops_error() with reason "dispatch stall detected on CPU N". This
triggers the full disable sequence: the BPF scheduler is killed and all tasks return to CFS.
Why this is safe: The dispatch loop stall check is done by the watchdog timer, which runs
on a different CPU via the hrtimer infrastructure. The stalling CPU cannot prevent the watchdog
from firing because the watchdog runs in interrupt context on other CPUs. The watchdog CPU can
call scx_ops_error() even while the stalling CPU holds the runqueue lock, because
scx_ops_error() is designed to be called from any context and only sets a flag atomically
before deferring actual work to a workqueue.
Relationship to the base watchdog
The base watchdog (patch 12) tracks per-task runnable_at timestamps. This dispatch watchdog
tracks per-CPU dispatch_start timestamps. They are complementary: the task watchdog catches
"task never scheduled" bugs, the dispatch watchdog catches "CPU never returns from dispatch"
bugs. Both call the same scx_ops_error() function, producing the same disable sequence.
BPF program implications
Any BPF program that implements ops.dispatch() must ensure the callback returns within
scx_watchdog_timeout. Long-running dispatch logic (e.g., iterating over thousands of tasks)
must be broken into multiple dispatch calls. The scx_bpf_consume() helper returns true while
tasks are available and false when the DSQ is empty — a well-written dispatch loop checks this
return value and exits when it returns false, rather than consuming indefinitely.
A BPF program that does not implement ops.dispatch() is unaffected by this watchdog extension,
since the kernel will use its default (empty) dispatch implementation.
Connections Between Patches
PATCH 17 (scx_bpf_kick_cpu)
└─→ Required by PATCH 18 (scx_central): central CPU kicks idle CPUs after dispatch
└─→ Used by PATCH 23 (SCX_KICK_WAIT): adds a blocking variant of the kick
└─→ Enables preemption-based priority enforcement for BPF schedulers
PATCH 19 (dispatch watchdog)
└─→ Extends PATCH 12 (base watchdog): same error path, new detection condition
└─→ Makes scx_bpf_consume() loops in ops.dispatch() safe to write
└─→ Protects against a failure mode that scx_central (PATCH 18) could trigger
if its central dispatch loop ran without bound
Connections to Core Infrastructure
Both patches build directly on the core implementation:
-
scx_bpf_kick_cpu()usessmp_send_reschedule()which is part of the kernel's IPI infrastructure, not sched_ext-specific. The sched_ext addition is the BPF-accessible wrapper and theSCX_KICK_PREEMPTlogic that hooks into the SCX-specific reschedule path. -
The dispatch watchdog uses
scx_rq, the per-CPU sched_ext runqueue state introduced in patch 09. Thedispatch_starttimestamp is a new field inscx_rqadded by this patch. -
Both patches interact with
scx_ops_error(): kick enables controlled preemption that is safe to use even during error recovery, and the dispatch watchdog triggers the error exit.
What to Focus On
For a maintainer, the critical lessons from this group:
-
IPI overhead and batching.
scx_bpf_kick_cpu()sends a real IPI. IPIs are cheap but not free — each one is an interrupt that interrupts the target CPU's execution. A BPF scheduler that sends an IPI for every task wakeup (one task per kick) will generate significant IPI overhead on a large system. When reviewing BPF schedulers or changes to kick semantics, watch for unbounded kick rates. TheSCX_KICK_IDLEflag is specifically designed to be no-op when the CPU is already running, which reduces overhead in the common case. -
Preemption semantics and fairness.
SCX_KICK_PREEMPTcan be used to implement strict priority preemption. However, if a BPF scheduler aggressively preempts whenever a higher-priority task appears, it can cause starvation of lower-priority tasks if the system is always generating high-priority work. When reviewing schedulers that useSCX_KICK_PREEMPT, verify they have a mechanism to ensure lower-priority tasks eventually run. -
The dispatch watchdog timeout calibration. The watchdog timeout (
scx_watchdog_timeout, default 30 seconds) is a system-wide parameter. A BPF scheduler that does legitimate long-running dispatch work (e.g., sorting thousands of tasks) will be killed by the watchdog if its dispatch time exceeds this threshold. When reviewing BPF schedulers or changes to the watchdog timeout, verify that the timeout is appropriate for the intended workload. The timeout is configurable via a module parameter, but changing it affects all SCX schedulers on the system. -
Runqueue lock semantics in ops.dispatch().
ops.dispatch()is called with the runqueue lock held. This means the BPF program cannot call any function that would acquire the runqueue lock recursively. The BPF verifier enforces some of this, but maintainers reviewing new BPF helpers for use inops.dispatch()must verify that they do not acquire the runqueue lock or any lock that nests inside the runqueue lock. -
The dispatch watchdog and legitimate blocking. The dispatch watchdog fires if
ops.dispatch()doesn't return within the timeout. But what if a BPF program legitimately needs to wait for an external event during dispatch (e.g., a BPF spinlock protecting a complex data structure)? This is forbidden:ops.dispatch()must never block. Any synchronization inops.dispatch()must use non-blocking mechanisms (BPF spin_lock withbpf_spin_lock()). When reviewing changes to dispatch semantics, maintain this invariant.
[PATCH 17/30] sched_ext: Implement scx_bpf_kick_cpu() and task preemption support
View on Lore: https://lore.kernel.org/all/20240618212056.2833381-18-tj@kernel.org
Commit Message
It's often useful to wake up and/or trigger reschedule on other CPUs. This
patch adds scx_bpf_kick_cpu() kfunc helper that BPF scheduler can call to
kick the target CPU into the scheduling path.
As a sched_ext task relinquishes its CPU only after its slice is depleted,
this patch also adds SCX_KICK_PREEMPT and SCX_ENQ_PREEMPT which clears the
slice of the target CPU's current task to guarantee that sched_ext's
scheduling path runs on the CPU.
If SCX_KICK_IDLE is specified, the target CPU is kicked iff the CPU is idle
to guarantee that the target CPU will go through at least one full sched_ext
scheduling cycle after the kicking. This can be used to wake up idle CPUs
without incurring unnecessary overhead if it isn't currently idle.
As a demonstration of how backward compatibility can be supported using BPF
CO-RE, tools/sched_ext/include/scx/compat.bpf.h is added. It provides
__COMPAT_scx_bpf_kick_cpu_IDLE() which uses SCX_KICK_IDLE if available or
becomes a regular kicking otherwise. This allows schedulers to use the new
SCX_KICK_IDLE while maintaining support for older kernels. The plan is to
temporarily use compat helpers to ease API updates and drop them after a few
kernel releases.
v5: - SCX_KICK_IDLE added. Note that this also adds a compat mechanism for
schedulers so that they can support kernels without SCX_KICK_IDLE.
This is useful as a demonstration of how new feature flags can be
added in a backward compatible way.
- kick_cpus_irq_workfn() reimplemented so that it touches the pending
cpumasks only as necessary to reduce kicking overhead on machines with
a lot of CPUs.
- tools/sched_ext/include/scx/compat.bpf.h added.
v4: - Move example scheduler to its own patch.
v3: - Make scx_example_central switch all tasks by default.
- Convert to BPF inline iterators.
v2: - Julia Lawall reported that scx_example_central can overflow the
dispatch buffer and malfunction. As scheduling for other CPUs can't be
handled by the automatic retry mechanism, fix by implementing an
explicit overflow and retry handling.
- Updated to use generic BPF cpumask helpers.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: David Vernet <dvernet@meta.com>
Acked-by: Josh Don <joshdon@google.com>
Acked-by: Hao Luo <haoluo@google.com>
Acked-by: Barret Rhoden <brho@google.com>
---
include/linux/sched/ext.h | 4 +
kernel/sched/ext.c | 225 +++++++++++++++++++++--
kernel/sched/sched.h | 10 +
tools/sched_ext/include/scx/common.bpf.h | 1 +
4 files changed, 227 insertions(+), 13 deletions(-)
diff --git a/include/linux/sched/ext.h b/include/linux/sched/ext.h
index 85fb5dc725ef..3b2809b980ac 100644
--- a/include/linux/sched/ext.h
+++ b/include/linux/sched/ext.h
@@ -134,6 +134,10 @@ struct sched_ext_entity {
* scx_bpf_dispatch() but can also be modified directly by the BPF
* scheduler. Automatically decreased by SCX as the task executes. On
* depletion, a scheduling event is triggered.
+ *
+ * This value is cleared to zero if the task is preempted by
+ * %SCX_KICK_PREEMPT and shouldn't be used to determine how long the
+ * task ran. Use p->se.sum_exec_runtime instead.
*/
u64 slice;
diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c
index 66bb9cf075f0..213793d086d7 100644
--- a/kernel/sched/ext.c
+++ b/kernel/sched/ext.c
@@ -412,6 +412,14 @@ enum scx_enq_flags {
/* high 32bits are SCX specific */
+ /*
+ * Set the following to trigger preemption when calling
+ * scx_bpf_dispatch() with a local dsq as the target. The slice of the
+ * current task is cleared to zero and the CPU is kicked into the
+ * scheduling path. Implies %SCX_ENQ_HEAD.
+ */
+ SCX_ENQ_PREEMPT = 1LLU << 32,
+
/*
* The task being enqueued is the only task available for the cpu. By
* default, ext core keeps executing such tasks but when
@@ -441,6 +449,24 @@ enum scx_pick_idle_cpu_flags {
SCX_PICK_IDLE_CORE = 1LLU << 0, /* pick a CPU whose SMT siblings are also idle */
};
+enum scx_kick_flags {
+ /*
+ * Kick the target CPU if idle. Guarantees that the target CPU goes
+ * through at least one full scheduling cycle before going idle. If the
+ * target CPU can be determined to be currently not idle and going to go
+ * through a scheduling cycle before going idle, noop.
+ */
+ SCX_KICK_IDLE = 1LLU << 0,
+
+ /*
+ * Preempt the current task and execute the dispatch path. If the
+ * current task of the target CPU is an SCX task, its ->scx.slice is
+ * cleared to zero before the scheduling path is invoked so that the
+ * task expires and the dispatch path is invoked.
+ */
+ SCX_KICK_PREEMPT = 1LLU << 1,
+};
+
enum scx_ops_enable_state {
SCX_OPS_PREPPING,
SCX_OPS_ENABLING,
@@ -1019,7 +1045,7 @@ static void dispatch_enqueue(struct scx_dispatch_q *dsq, struct task_struct *p,
}
}
- if (enq_flags & SCX_ENQ_HEAD)
+ if (enq_flags & (SCX_ENQ_HEAD | SCX_ENQ_PREEMPT))
list_add(&p->scx.dsq_node, &dsq->list);
else
list_add_tail(&p->scx.dsq_node, &dsq->list);
@@ -1045,8 +1071,16 @@ static void dispatch_enqueue(struct scx_dispatch_q *dsq, struct task_struct *p,
if (is_local) {
struct rq *rq = container_of(dsq, struct rq, scx.local_dsq);
+ bool preempt = false;
+
+ if ((enq_flags & SCX_ENQ_PREEMPT) && p != rq->curr &&
+ rq->curr->sched_class == &ext_sched_class) {
+ rq->curr->scx.slice = 0;
+ preempt = true;
+ }
- if (sched_class_above(&ext_sched_class, rq->curr->sched_class))
+ if (preempt || sched_class_above(&ext_sched_class,
+ rq->curr->sched_class))
resched_curr(rq);
} else {
raw_spin_unlock(&dsq->lock);
@@ -1872,8 +1906,10 @@ static int balance_scx(struct rq *rq, struct task_struct *prev,
{
struct scx_dsp_ctx *dspc = this_cpu_ptr(scx_dsp_ctx);
bool prev_on_scx = prev->sched_class == &ext_sched_class;
+ bool has_tasks = false;
lockdep_assert_rq_held(rq);
+ rq->scx.flags |= SCX_RQ_BALANCING;
if (prev_on_scx) {
WARN_ON_ONCE(prev->scx.flags & SCX_TASK_BAL_KEEP);
@@ -1890,19 +1926,19 @@ static int balance_scx(struct rq *rq, struct task_struct *prev,
if ((prev->scx.flags & SCX_TASK_QUEUED) &&
prev->scx.slice && !scx_ops_bypassing()) {
prev->scx.flags |= SCX_TASK_BAL_KEEP;
- return 1;
+ goto has_tasks;
}
}
/* if there already are tasks to run, nothing to do */
if (rq->scx.local_dsq.nr)
- return 1;
+ goto has_tasks;
if (consume_dispatch_q(rq, rf, &scx_dsq_global))
- return 1;
+ goto has_tasks;
if (!SCX_HAS_OP(dispatch) || scx_ops_bypassing() || !scx_rq_online(rq))
- return 0;
+ goto out;
dspc->rq = rq;
dspc->rf = rf;
@@ -1923,12 +1959,18 @@ static int balance_scx(struct rq *rq, struct task_struct *prev,
flush_dispatch_buf(rq, rf);
if (rq->scx.local_dsq.nr)
- return 1;
+ goto has_tasks;
if (consume_dispatch_q(rq, rf, &scx_dsq_global))
- return 1;
+ goto has_tasks;
} while (dspc->nr_tasks);
- return 0;
+ goto out;
+
+has_tasks:
+ has_tasks = true;
+out:
+ rq->scx.flags &= ~SCX_RQ_BALANCING;
+ return has_tasks;
}
static void set_next_task_scx(struct rq *rq, struct task_struct *p, bool first)
@@ -2666,7 +2708,8 @@ int scx_check_setscheduler(struct task_struct *p, int policy)
* Omitted operations:
*
* - wakeup_preempt: NOOP as it isn't useful in the wakeup path because the task
- * isn't tied to the CPU at that point.
+ * isn't tied to the CPU at that point. Preemption is implemented by resetting
+ * the victim task's slice to 0 and triggering reschedule on the target CPU.
*
* - migrate_task_rq: Unnecessary as task to cpu mapping is transient.
*
@@ -2902,6 +2945,9 @@ bool task_should_scx(struct task_struct *p)
* of the queue.
*
* d. pick_next_task() suppresses zero slice warning.
+ *
+ * e. scx_bpf_kick_cpu() is disabled to avoid irq_work malfunction during PM
+ * operations.
*/
static void scx_ops_bypass(bool bypass)
{
@@ -3410,11 +3456,21 @@ static void scx_dump_state(struct scx_exit_info *ei, size_t dump_len)
seq_buf_init(&ns, buf, avail);
dump_newline(&ns);
- dump_line(&ns, "CPU %-4d: nr_run=%u ops_qseq=%lu",
- cpu, rq->scx.nr_running, rq->scx.ops_qseq);
+ dump_line(&ns, "CPU %-4d: nr_run=%u flags=0x%x ops_qseq=%lu",
+ cpu, rq->scx.nr_running, rq->scx.flags,
+ rq->scx.ops_qseq);
dump_line(&ns, " curr=%s[%d] class=%ps",
rq->curr->comm, rq->curr->pid,
rq->curr->sched_class);
+ if (!cpumask_empty(rq->scx.cpus_to_kick))
+ dump_line(&ns, " cpus_to_kick : %*pb",
+ cpumask_pr_args(rq->scx.cpus_to_kick));
+ if (!cpumask_empty(rq->scx.cpus_to_kick_if_idle))
+ dump_line(&ns, " idle_to_kick : %*pb",
+ cpumask_pr_args(rq->scx.cpus_to_kick_if_idle));
+ if (!cpumask_empty(rq->scx.cpus_to_preempt))
+ dump_line(&ns, " cpus_to_preempt: %*pb",
+ cpumask_pr_args(rq->scx.cpus_to_preempt));
used = seq_buf_used(&ns);
if (SCX_HAS_OP(dump_cpu)) {
@@ -4085,6 +4141,82 @@ static const struct sysrq_key_op sysrq_sched_ext_dump_op = {
.enable_mask = SYSRQ_ENABLE_RTNICE,
};
+static bool can_skip_idle_kick(struct rq *rq)
+{
+ lockdep_assert_rq_held(rq);
+
+ /*
+ * We can skip idle kicking if @rq is going to go through at least one
+ * full SCX scheduling cycle before going idle. Just checking whether
+ * curr is not idle is insufficient because we could be racing
+ * balance_one() trying to pull the next task from a remote rq, which
+ * may fail, and @rq may become idle afterwards.
+ *
+ * The race window is small and we don't and can't guarantee that @rq is
+ * only kicked while idle anyway. Skip only when sure.
+ */
+ return !is_idle_task(rq->curr) && !(rq->scx.flags & SCX_RQ_BALANCING);
+}
+
+static void kick_one_cpu(s32 cpu, struct rq *this_rq)
+{
+ struct rq *rq = cpu_rq(cpu);
+ struct scx_rq *this_scx = &this_rq->scx;
+ unsigned long flags;
+
+ raw_spin_rq_lock_irqsave(rq, flags);
+
+ /*
+ * During CPU hotplug, a CPU may depend on kicking itself to make
+ * forward progress. Allow kicking self regardless of online state.
+ */
+ if (cpu_online(cpu) || cpu == cpu_of(this_rq)) {
+ if (cpumask_test_cpu(cpu, this_scx->cpus_to_preempt)) {
+ if (rq->curr->sched_class == &ext_sched_class)
+ rq->curr->scx.slice = 0;
+ cpumask_clear_cpu(cpu, this_scx->cpus_to_preempt);
+ }
+
+ resched_curr(rq);
+ } else {
+ cpumask_clear_cpu(cpu, this_scx->cpus_to_preempt);
+ }
+
+ raw_spin_rq_unlock_irqrestore(rq, flags);
+}
+
+static void kick_one_cpu_if_idle(s32 cpu, struct rq *this_rq)
+{
+ struct rq *rq = cpu_rq(cpu);
+ unsigned long flags;
+
+ raw_spin_rq_lock_irqsave(rq, flags);
+
+ if (!can_skip_idle_kick(rq) &&
+ (cpu_online(cpu) || cpu == cpu_of(this_rq)))
+ resched_curr(rq);
+
+ raw_spin_rq_unlock_irqrestore(rq, flags);
+}
+
+static void kick_cpus_irq_workfn(struct irq_work *irq_work)
+{
+ struct rq *this_rq = this_rq();
+ struct scx_rq *this_scx = &this_rq->scx;
+ s32 cpu;
+
+ for_each_cpu(cpu, this_scx->cpus_to_kick) {
+ kick_one_cpu(cpu, this_rq);
+ cpumask_clear_cpu(cpu, this_scx->cpus_to_kick);
+ cpumask_clear_cpu(cpu, this_scx->cpus_to_kick_if_idle);
+ }
+
+ for_each_cpu(cpu, this_scx->cpus_to_kick_if_idle) {
+ kick_one_cpu_if_idle(cpu, this_rq);
+ cpumask_clear_cpu(cpu, this_scx->cpus_to_kick_if_idle);
+ }
+}
+
/**
* print_scx_info - print out sched_ext scheduler state
* @log_lvl: the log level to use when printing
@@ -4139,7 +4271,7 @@ void __init init_sched_ext_class(void)
* definitions so that BPF scheduler implementations can use them
* through the generated vmlinux.h.
*/
- WRITE_ONCE(v, SCX_ENQ_WAKEUP | SCX_DEQ_SLEEP);
+ WRITE_ONCE(v, SCX_ENQ_WAKEUP | SCX_DEQ_SLEEP | SCX_KICK_PREEMPT);
BUG_ON(rhashtable_init(&dsq_hash, &dsq_hash_params));
init_dsq(&scx_dsq_global, SCX_DSQ_GLOBAL);
@@ -4152,6 +4284,11 @@ void __init init_sched_ext_class(void)
init_dsq(&rq->scx.local_dsq, SCX_DSQ_LOCAL);
INIT_LIST_HEAD(&rq->scx.runnable_list);
+
+ BUG_ON(!zalloc_cpumask_var(&rq->scx.cpus_to_kick, GFP_KERNEL));
+ BUG_ON(!zalloc_cpumask_var(&rq->scx.cpus_to_kick_if_idle, GFP_KERNEL));
+ BUG_ON(!zalloc_cpumask_var(&rq->scx.cpus_to_preempt, GFP_KERNEL));
+ init_irq_work(&rq->scx.kick_cpus_irq_work, kick_cpus_irq_workfn);
}
register_sysrq_key('S', &sysrq_sched_ext_reset_op);
@@ -4438,6 +4575,67 @@ static const struct btf_kfunc_id_set scx_kfunc_set_dispatch = {
__bpf_kfunc_start_defs();
+/**
+ * scx_bpf_kick_cpu - Trigger reschedule on a CPU
+ * @cpu: cpu to kick
+ * @flags: %SCX_KICK_* flags
+ *
+ * Kick @cpu into rescheduling. This can be used to wake up an idle CPU or
+ * trigger rescheduling on a busy CPU. This can be called from any online
+ * scx_ops operation and the actual kicking is performed asynchronously through
+ * an irq work.
+ */
+__bpf_kfunc void scx_bpf_kick_cpu(s32 cpu, u64 flags)
+{
+ struct rq *this_rq;
+ unsigned long irq_flags;
+
+ if (!ops_cpu_valid(cpu, NULL))
+ return;
+
+ /*
+ * While bypassing for PM ops, IRQ handling may not be online which can
+ * lead to irq_work_queue() malfunction such as infinite busy wait for
+ * IRQ status update. Suppress kicking.
+ */
+ if (scx_ops_bypassing())
+ return;
+
+ local_irq_save(irq_flags);
+
+ this_rq = this_rq();
+
+ /*
+ * Actual kicking is bounced to kick_cpus_irq_workfn() to avoid nesting
+ * rq locks. We can probably be smarter and avoid bouncing if called
+ * from ops which don't hold a rq lock.
+ */
+ if (flags & SCX_KICK_IDLE) {
+ struct rq *target_rq = cpu_rq(cpu);
+
+ if (unlikely(flags & SCX_KICK_PREEMPT))
+ scx_ops_error("PREEMPT cannot be used with SCX_KICK_IDLE");
+
+ if (raw_spin_rq_trylock(target_rq)) {
+ if (can_skip_idle_kick(target_rq)) {
+ raw_spin_rq_unlock(target_rq);
+ goto out;
+ }
+ raw_spin_rq_unlock(target_rq);
+ }
+ cpumask_set_cpu(cpu, this_rq->scx.cpus_to_kick_if_idle);
+ } else {
+ cpumask_set_cpu(cpu, this_rq->scx.cpus_to_kick);
+
+ if (flags & SCX_KICK_PREEMPT)
+ cpumask_set_cpu(cpu, this_rq->scx.cpus_to_preempt);
+ }
+
+ irq_work_queue(&this_rq->scx.kick_cpus_irq_work);
+out:
+ local_irq_restore(irq_flags);
+}
+
/**
* scx_bpf_dsq_nr_queued - Return the number of queued tasks
* @dsq_id: id of the DSQ
@@ -4836,6 +5034,7 @@ __bpf_kfunc s32 scx_bpf_task_cpu(const struct task_struct *p)
__bpf_kfunc_end_defs();
BTF_KFUNCS_START(scx_kfunc_ids_any)
+BTF_ID_FLAGS(func, scx_bpf_kick_cpu)
BTF_ID_FLAGS(func, scx_bpf_dsq_nr_queued)
BTF_ID_FLAGS(func, scx_bpf_destroy_dsq)
BTF_ID_FLAGS(func, scx_bpf_exit_bstr, KF_TRUSTED_ARGS)
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 2960e153c3a7..d9054eb4ba82 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -724,12 +724,22 @@ struct cfs_rq {
};
#ifdef CONFIG_SCHED_CLASS_EXT
+/* scx_rq->flags, protected by the rq lock */
+enum scx_rq_flags {
+ SCX_RQ_BALANCING = 1 << 1,
+};
+
struct scx_rq {
struct scx_dispatch_q local_dsq;
struct list_head runnable_list; /* runnable tasks on this rq */
unsigned long ops_qseq;
u64 extra_enq_flags; /* see move_task_to_local_dsq() */
u32 nr_running;
+ u32 flags;
+ cpumask_var_t cpus_to_kick;
+ cpumask_var_t cpus_to_kick_if_idle;
+ cpumask_var_t cpus_to_preempt;
+ struct irq_work kick_cpus_irq_work;
};
#endif /* CONFIG_SCHED_CLASS_EXT */
diff --git a/tools/sched_ext/include/scx/common.bpf.h b/tools/sched_ext/include/scx/common.bpf.h
index 3ea5cdf58bc7..421118bc56ff 100644
--- a/tools/sched_ext/include/scx/common.bpf.h
+++ b/tools/sched_ext/include/scx/common.bpf.h
@@ -34,6 +34,7 @@ void scx_bpf_dispatch(struct task_struct *p, u64 dsq_id, u64 slice, u64 enq_flag
u32 scx_bpf_dispatch_nr_slots(void) __ksym;
void scx_bpf_dispatch_cancel(void) __ksym;
bool scx_bpf_consume(u64 dsq_id) __ksym;
+void scx_bpf_kick_cpu(s32 cpu, u64 flags) __ksym;
s32 scx_bpf_dsq_nr_queued(u64 dsq_id) __ksym;
void scx_bpf_destroy_dsq(u64 dsq_id) __ksym;
void scx_bpf_exit_bstr(s64 exit_code, char *fmt, unsigned long long *data, u32 data__sz) __ksym __weak;
--
2.45.2
Diff
---
include/linux/sched/ext.h | 4 +
kernel/sched/ext.c | 225 +++++++++++++++++++++--
kernel/sched/sched.h | 10 +
tools/sched_ext/include/scx/common.bpf.h | 1 +
4 files changed, 227 insertions(+), 13 deletions(-)
diff --git a/include/linux/sched/ext.h b/include/linux/sched/ext.h
index 85fb5dc725ef..3b2809b980ac 100644
--- a/include/linux/sched/ext.h
+++ b/include/linux/sched/ext.h
@@ -134,6 +134,10 @@ struct sched_ext_entity {
* scx_bpf_dispatch() but can also be modified directly by the BPF
* scheduler. Automatically decreased by sched_ext as the task executes. On
* depletion, a scheduling event is triggered.
+ *
+ * This value is cleared to zero if the task is preempted by
+ * %SCX_KICK_PREEMPT and shouldn't be used to determine how long the
+ * task ran. Use p->se.sum_exec_runtime instead.
*/
u64 slice;
diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c
index 66bb9cf075f0..213793d086d7 100644
--- a/kernel/sched/ext.c
+++ b/kernel/sched/ext.c
@@ -412,6 +412,14 @@ enum scx_enq_flags {
/* high 32bits are sched_ext specific */
+ /*
+ * Set the following to trigger preemption when calling
+ * scx_bpf_dispatch() with a local dsq as the target. The slice of the
+ * current task is cleared to zero and the CPU is kicked into the
+ * scheduling path. Implies %SCX_ENQ_HEAD.
+ */
+ SCX_ENQ_PREEMPT = 1LLU << 32,
+
/*
* The task being enqueued is the only task available for the cpu. By
* default, ext core keeps executing such tasks but when
@@ -441,6 +449,24 @@ enum scx_pick_idle_cpu_flags {
SCX_PICK_IDLE_CORE = 1LLU << 0, /* pick a CPU whose SMT siblings are also idle */
};
+enum scx_kick_flags {
+ /*
+ * Kick the target CPU if idle. Guarantees that the target CPU goes
+ * through at least one full scheduling cycle before going idle. If the
+ * target CPU can be determined to be currently not idle and going to go
+ * through a scheduling cycle before going idle, noop.
+ */
+ SCX_KICK_IDLE = 1LLU << 0,
+
+ /*
+ * Preempt the current task and execute the dispatch path. If the
+ * current task of the target CPU is an sched_ext task, its ->scx.slice is
+ * cleared to zero before the scheduling path is invoked so that the
+ * task expires and the dispatch path is invoked.
+ */
+ SCX_KICK_PREEMPT = 1LLU << 1,
+};
+
enum scx_ops_enable_state {
SCX_OPS_PREPPING,
SCX_OPS_ENABLING,
@@ -1019,7 +1045,7 @@ static void dispatch_enqueue(struct scx_dispatch_q *dsq, struct task_struct *p,
}
}
- if (enq_flags & SCX_ENQ_HEAD)
+ if (enq_flags & (SCX_ENQ_HEAD | SCX_ENQ_PREEMPT))
list_add(&p->scx.dsq_node, &dsq->list);
else
list_add_tail(&p->scx.dsq_node, &dsq->list);
@@ -1045,8 +1071,16 @@ static void dispatch_enqueue(struct scx_dispatch_q *dsq, struct task_struct *p,
if (is_local) {
struct rq *rq = container_of(dsq, struct rq, scx.local_dsq);
+ bool preempt = false;
+
+ if ((enq_flags & SCX_ENQ_PREEMPT) && p != rq->curr &&
+ rq->curr->sched_class == &ext_sched_class) {
+ rq->curr->scx.slice = 0;
+ preempt = true;
+ }
- if (sched_class_above(&ext_sched_class, rq->curr->sched_class))
+ if (preempt || sched_class_above(&ext_sched_class,
+ rq->curr->sched_class))
resched_curr(rq);
} else {
raw_spin_unlock(&dsq->lock);
@@ -1872,8 +1906,10 @@ static int balance_scx(struct rq *rq, struct task_struct *prev,
{
struct scx_dsp_ctx *dspc = this_cpu_ptr(scx_dsp_ctx);
bool prev_on_scx = prev->sched_class == &ext_sched_class;
+ bool has_tasks = false;
lockdep_assert_rq_held(rq);
+ rq->scx.flags |= SCX_RQ_BALANCING;
if (prev_on_scx) {
WARN_ON_ONCE(prev->scx.flags & SCX_TASK_BAL_KEEP);
@@ -1890,19 +1926,19 @@ static int balance_scx(struct rq *rq, struct task_struct *prev,
if ((prev->scx.flags & SCX_TASK_QUEUED) &&
prev->scx.slice && !scx_ops_bypassing()) {
prev->scx.flags |= SCX_TASK_BAL_KEEP;
- return 1;
+ goto has_tasks;
}
}
/* if there already are tasks to run, nothing to do */
if (rq->scx.local_dsq.nr)
- return 1;
+ goto has_tasks;
if (consume_dispatch_q(rq, rf, &scx_dsq_global))
- return 1;
+ goto has_tasks;
if (!SCX_HAS_OP(dispatch) || scx_ops_bypassing() || !scx_rq_online(rq))
- return 0;
+ goto out;
dspc->rq = rq;
dspc->rf = rf;
@@ -1923,12 +1959,18 @@ static int balance_scx(struct rq *rq, struct task_struct *prev,
flush_dispatch_buf(rq, rf);
if (rq->scx.local_dsq.nr)
- return 1;
+ goto has_tasks;
if (consume_dispatch_q(rq, rf, &scx_dsq_global))
- return 1;
+ goto has_tasks;
} while (dspc->nr_tasks);
- return 0;
+ goto out;
+
+has_tasks:
+ has_tasks = true;
+out:
+ rq->scx.flags &= ~SCX_RQ_BALANCING;
+ return has_tasks;
}
static void set_next_task_scx(struct rq *rq, struct task_struct *p, bool first)
@@ -2666,7 +2708,8 @@ int scx_check_setscheduler(struct task_struct *p, int policy)
* Omitted operations:
*
* - wakeup_preempt: NOOP as it isn't useful in the wakeup path because the task
- * isn't tied to the CPU at that point.
+ * isn't tied to the CPU at that point. Preemption is implemented by resetting
+ * the victim task's slice to 0 and triggering reschedule on the target CPU.
*
* - migrate_task_rq: Unnecessary as task to cpu mapping is transient.
*
@@ -2902,6 +2945,9 @@ bool task_should_scx(struct task_struct *p)
* of the queue.
*
* d. pick_next_task() suppresses zero slice warning.
+ *
+ * e. scx_bpf_kick_cpu() is disabled to avoid irq_work malfunction during PM
+ * operations.
*/
static void scx_ops_bypass(bool bypass)
{
@@ -3410,11 +3456,21 @@ static void scx_dump_state(struct scx_exit_info *ei, size_t dump_len)
seq_buf_init(&ns, buf, avail);
dump_newline(&ns);
- dump_line(&ns, "CPU %-4d: nr_run=%u ops_qseq=%lu",
- cpu, rq->scx.nr_running, rq->scx.ops_qseq);
+ dump_line(&ns, "CPU %-4d: nr_run=%u flags=0x%x ops_qseq=%lu",
+ cpu, rq->scx.nr_running, rq->scx.flags,
+ rq->scx.ops_qseq);
dump_line(&ns, " curr=%s[%d] class=%ps",
rq->curr->comm, rq->curr->pid,
rq->curr->sched_class);
+ if (!cpumask_empty(rq->scx.cpus_to_kick))
+ dump_line(&ns, " cpus_to_kick : %*pb",
+ cpumask_pr_args(rq->scx.cpus_to_kick));
+ if (!cpumask_empty(rq->scx.cpus_to_kick_if_idle))
+ dump_line(&ns, " idle_to_kick : %*pb",
+ cpumask_pr_args(rq->scx.cpus_to_kick_if_idle));
+ if (!cpumask_empty(rq->scx.cpus_to_preempt))
+ dump_line(&ns, " cpus_to_preempt: %*pb",
+ cpumask_pr_args(rq->scx.cpus_to_preempt));
used = seq_buf_used(&ns);
if (SCX_HAS_OP(dump_cpu)) {
@@ -4085,6 +4141,82 @@ static const struct sysrq_key_op sysrq_sched_ext_dump_op = {
.enable_mask = SYSRQ_ENABLE_RTNICE,
};
+static bool can_skip_idle_kick(struct rq *rq)
+{
+ lockdep_assert_rq_held(rq);
+
+ /*
+ * We can skip idle kicking if @rq is going to go through at least one
+ * full sched_ext scheduling cycle before going idle. Just checking whether
+ * curr is not idle is insufficient because we could be racing
+ * balance_one() trying to pull the next task from a remote rq, which
+ * may fail, and @rq may become idle afterwards.
+ *
+ * The race window is small and we don't and can't guarantee that @rq is
+ * only kicked while idle anyway. Skip only when sure.
+ */
+ return !is_idle_task(rq->curr) && !(rq->scx.flags & SCX_RQ_BALANCING);
+}
+
+static void kick_one_cpu(s32 cpu, struct rq *this_rq)
+{
+ struct rq *rq = cpu_rq(cpu);
+ struct scx_rq *this_scx = &this_rq->scx;
+ unsigned long flags;
+
+ raw_spin_rq_lock_irqsave(rq, flags);
+
+ /*
+ * During CPU hotplug, a CPU may depend on kicking itself to make
+ * forward progress. Allow kicking self regardless of online state.
+ */
+ if (cpu_online(cpu) || cpu == cpu_of(this_rq)) {
+ if (cpumask_test_cpu(cpu, this_scx->cpus_to_preempt)) {
+ if (rq->curr->sched_class == &ext_sched_class)
+ rq->curr->scx.slice = 0;
+ cpumask_clear_cpu(cpu, this_scx->cpus_to_preempt);
+ }
+
+ resched_curr(rq);
+ } else {
+ cpumask_clear_cpu(cpu, this_scx->cpus_to_preempt);
+ }
+
+ raw_spin_rq_unlock_irqrestore(rq, flags);
+}
+
+static void kick_one_cpu_if_idle(s32 cpu, struct rq *this_rq)
+{
+ struct rq *rq = cpu_rq(cpu);
+ unsigned long flags;
+
+ raw_spin_rq_lock_irqsave(rq, flags);
+
+ if (!can_skip_idle_kick(rq) &&
+ (cpu_online(cpu) || cpu == cpu_of(this_rq)))
+ resched_curr(rq);
+
+ raw_spin_rq_unlock_irqrestore(rq, flags);
+}
+
+static void kick_cpus_irq_workfn(struct irq_work *irq_work)
+{
+ struct rq *this_rq = this_rq();
+ struct scx_rq *this_scx = &this_rq->scx;
+ s32 cpu;
+
+ for_each_cpu(cpu, this_scx->cpus_to_kick) {
+ kick_one_cpu(cpu, this_rq);
+ cpumask_clear_cpu(cpu, this_scx->cpus_to_kick);
+ cpumask_clear_cpu(cpu, this_scx->cpus_to_kick_if_idle);
+ }
+
+ for_each_cpu(cpu, this_scx->cpus_to_kick_if_idle) {
+ kick_one_cpu_if_idle(cpu, this_rq);
+ cpumask_clear_cpu(cpu, this_scx->cpus_to_kick_if_idle);
+ }
+}
+
/**
* print_scx_info - print out sched_ext scheduler state
* @log_lvl: the log level to use when printing
@@ -4139,7 +4271,7 @@ void __init init_sched_ext_class(void)
* definitions so that BPF scheduler implementations can use them
* through the generated vmlinux.h.
*/
- WRITE_ONCE(v, SCX_ENQ_WAKEUP | SCX_DEQ_SLEEP);
+ WRITE_ONCE(v, SCX_ENQ_WAKEUP | SCX_DEQ_SLEEP | SCX_KICK_PREEMPT);
BUG_ON(rhashtable_init(&dsq_hash, &dsq_hash_params));
init_dsq(&scx_dsq_global, SCX_DSQ_GLOBAL);
@@ -4152,6 +4284,11 @@ void __init init_sched_ext_class(void)
init_dsq(&rq->scx.local_dsq, SCX_DSQ_LOCAL);
INIT_LIST_HEAD(&rq->scx.runnable_list);
+
+ BUG_ON(!zalloc_cpumask_var(&rq->scx.cpus_to_kick, GFP_KERNEL));
+ BUG_ON(!zalloc_cpumask_var(&rq->scx.cpus_to_kick_if_idle, GFP_KERNEL));
+ BUG_ON(!zalloc_cpumask_var(&rq->scx.cpus_to_preempt, GFP_KERNEL));
+ init_irq_work(&rq->scx.kick_cpus_irq_work, kick_cpus_irq_workfn);
}
register_sysrq_key('S', &sysrq_sched_ext_reset_op);
@@ -4438,6 +4575,67 @@ static const struct btf_kfunc_id_set scx_kfunc_set_dispatch = {
__bpf_kfunc_start_defs();
+/**
+ * scx_bpf_kick_cpu - Trigger reschedule on a CPU
+ * @cpu: cpu to kick
+ * @flags: %SCX_KICK_* flags
+ *
+ * Kick @cpu into rescheduling. This can be used to wake up an idle CPU or
+ * trigger rescheduling on a busy CPU. This can be called from any online
+ * scx_ops operation and the actual kicking is performed asynchronously through
+ * an irq work.
+ */
+__bpf_kfunc void scx_bpf_kick_cpu(s32 cpu, u64 flags)
+{
+ struct rq *this_rq;
+ unsigned long irq_flags;
+
+ if (!ops_cpu_valid(cpu, NULL))
+ return;
+
+ /*
+ * While bypassing for PM ops, IRQ handling may not be online which can
+ * lead to irq_work_queue() malfunction such as infinite busy wait for
+ * IRQ status update. Suppress kicking.
+ */
+ if (scx_ops_bypassing())
+ return;
+
+ local_irq_save(irq_flags);
+
+ this_rq = this_rq();
+
+ /*
+ * Actual kicking is bounced to kick_cpus_irq_workfn() to avoid nesting
+ * rq locks. We can probably be smarter and avoid bouncing if called
+ * from ops which don't hold a rq lock.
+ */
+ if (flags & SCX_KICK_IDLE) {
+ struct rq *target_rq = cpu_rq(cpu);
+
+ if (unlikely(flags & SCX_KICK_PREEMPT))
+ scx_ops_error("PREEMPT cannot be used with SCX_KICK_IDLE");
+
+ if (raw_spin_rq_trylock(target_rq)) {
+ if (can_skip_idle_kick(target_rq)) {
+ raw_spin_rq_unlock(target_rq);
+ goto out;
+ }
+ raw_spin_rq_unlock(target_rq);
+ }
+ cpumask_set_cpu(cpu, this_rq->scx.cpus_to_kick_if_idle);
+ } else {
+ cpumask_set_cpu(cpu, this_rq->scx.cpus_to_kick);
+
+ if (flags & SCX_KICK_PREEMPT)
+ cpumask_set_cpu(cpu, this_rq->scx.cpus_to_preempt);
+ }
+
+ irq_work_queue(&this_rq->scx.kick_cpus_irq_work);
+out:
+ local_irq_restore(irq_flags);
+}
+
/**
* scx_bpf_dsq_nr_queued - Return the number of queued tasks
* @dsq_id: id of the DSQ
@@ -4836,6 +5034,7 @@ __bpf_kfunc s32 scx_bpf_task_cpu(const struct task_struct *p)
__bpf_kfunc_end_defs();
BTF_KFUNCS_START(scx_kfunc_ids_any)
+BTF_ID_FLAGS(func, scx_bpf_kick_cpu)
BTF_ID_FLAGS(func, scx_bpf_dsq_nr_queued)
BTF_ID_FLAGS(func, scx_bpf_destroy_dsq)
BTF_ID_FLAGS(func, scx_bpf_exit_bstr, KF_TRUSTED_ARGS)
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 2960e153c3a7..d9054eb4ba82 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -724,12 +724,22 @@ struct cfs_rq {
};
#ifdef CONFIG_SCHED_CLASS_EXT
+/* scx_rq->flags, protected by the rq lock */
+enum scx_rq_flags {
+ SCX_RQ_BALANCING = 1 << 1,
+};
+
struct scx_rq {
struct scx_dispatch_q local_dsq;
struct list_head runnable_list; /* runnable tasks on this rq */
unsigned long ops_qseq;
u64 extra_enq_flags; /* see move_task_to_local_dsq() */
u32 nr_running;
+ u32 flags;
+ cpumask_var_t cpus_to_kick;
+ cpumask_var_t cpus_to_kick_if_idle;
+ cpumask_var_t cpus_to_preempt;
+ struct irq_work kick_cpus_irq_work;
};
#endif /* CONFIG_SCHED_CLASS_EXT */
diff --git a/tools/sched_ext/include/scx/common.bpf.h b/tools/sched_ext/include/scx/common.bpf.h
index 3ea5cdf58bc7..421118bc56ff 100644
--- a/tools/sched_ext/include/scx/common.bpf.h
+++ b/tools/sched_ext/include/scx/common.bpf.h
@@ -34,6 +34,7 @@ void scx_bpf_dispatch(struct task_struct *p, u64 dsq_id, u64 slice, u64 enq_flag
u32 scx_bpf_dispatch_nr_slots(void) __ksym;
void scx_bpf_dispatch_cancel(void) __ksym;
bool scx_bpf_consume(u64 dsq_id) __ksym;
+void scx_bpf_kick_cpu(s32 cpu, u64 flags) __ksym;
s32 scx_bpf_dsq_nr_queued(u64 dsq_id) __ksym;
void scx_bpf_destroy_dsq(u64 dsq_id) __ksym;
void scx_bpf_exit_bstr(s64 exit_code, char *fmt, unsigned long long *data, u32 data__sz) __ksym __weak;
--
2.45.2
Implementation Analysis
Overview
BPF schedulers need a way to trigger rescheduling on arbitrary CPUs — to wake idle CPUs when work arrives, or to preempt a running task when a higher-priority task should run. This patch adds scx_bpf_kick_cpu(cpu, flags), the primary inter-CPU signaling primitive for sched_ext. It also adds SCX_ENQ_PREEMPT, a flag for scx_bpf_dispatch() that preempts the current task on a local DSQ. Together these form the preemption and wakeup infrastructure that enables sophisticated multi-CPU scheduling policies.
Code Walkthrough
New enqueue flag: SCX_ENQ_PREEMPT
SCX_ENQ_PREEMPT = 1LLU << 32,
When scx_bpf_dispatch(p, SCX_DSQ_LOCAL, slice, SCX_ENQ_PREEMPT) is called with a local DSQ as target:
- The task is added to the HEAD of the local DSQ (implies
SCX_ENQ_HEAD). - The currently running task's
->scx.sliceis set to zero (forcing it off the CPU). resched_curr(rq)is called to trigger a reschedule.
This is implemented in dispatch_enqueue():
if ((enq_flags & SCX_ENQ_PREEMPT) && p != rq->curr &&
rq->curr->sched_class == &ext_sched_class) {
rq->curr->scx.slice = 0;
preempt = true;
}
if (preempt || sched_class_above(&ext_sched_class, rq->curr->sched_class))
resched_curr(rq);
Note: only SCX tasks are preempted this way; if curr is a higher-priority RT or DL task, sched_class_above() still handles normal preemption.
New enum: scx_kick_flags
enum scx_kick_flags {
SCX_KICK_IDLE = 1LLU << 0, // kick only if idle
SCX_KICK_PREEMPT = 1LLU << 1, // preempt running SCX task
};
SCX_KICK_IDLE: Kick the target CPU only if it is idle (going through a scheduling cycle before it idles). The implementation usescan_skip_idle_kick()which checks both whethercurris the idle task AND whether the CPU is currently inbalance_scx()(theSCX_RQ_BALANCINGflag). Skipping the kick when the CPU is already inbalance_scx()avoids unnecessary IPIs.SCX_KICK_PREEMPT: Clear the running SCX task's->scx.sliceto zero and callresched_curr(). Applied inkick_one_cpu()under the target CPU'srq->lock.- These flags are mutually exclusive:
SCX_KICK_IDLE | SCX_KICK_PREEMPTis an error caught byscx_ops_error().
scx_rq additions in kernel/sched/sched.h
enum scx_rq_flags {
SCX_RQ_BALANCING = 1 << 1, // CPU is inside balance_scx()
};
struct scx_rq {
...
u32 flags;
cpumask_var_t cpus_to_kick;
cpumask_var_t cpus_to_kick_if_idle;
cpumask_var_t cpus_to_preempt;
struct irq_work kick_cpus_irq_work;
};
Each CPU maintains three cpumask bitmaps tracking pending kick actions, plus an irq_work for deferred execution. The SCX_RQ_BALANCING flag is set at the start of balance_scx() and cleared at the end — it allows can_skip_idle_kick() to know the CPU is about to pick a task and does not need an additional kick.
scx_bpf_kick_cpu() — the BPF kfunc
__bpf_kfunc void scx_bpf_kick_cpu(s32 cpu, u64 flags)
{
if (!ops_cpu_valid(cpu, NULL)) return;
if (scx_ops_bypassing()) return; // disabled during PM ops
local_irq_save(irq_flags);
this_rq = this_rq();
if (flags & SCX_KICK_IDLE) {
// Optimistic early-out: if we can lock the target rq and it's
// already going through a scheduling cycle, skip the kick
if (raw_spin_rq_trylock(target_rq)) {
if (can_skip_idle_kick(target_rq)) { goto out; }
raw_spin_rq_unlock(target_rq);
}
cpumask_set_cpu(cpu, this_rq->scx.cpus_to_kick_if_idle);
} else {
cpumask_set_cpu(cpu, this_rq->scx.cpus_to_kick);
if (flags & SCX_KICK_PREEMPT)
cpumask_set_cpu(cpu, this_rq->scx.cpus_to_preempt);
}
irq_work_queue(&this_rq->scx.kick_cpus_irq_work);
out:
local_irq_restore(irq_flags);
}
The actual kicking is deferred through irq_work to avoid nesting rq locks (the caller may already hold a lock). The function sets bits in the calling CPU's cpumask bitmaps and queues the IRQ work; kick_cpus_irq_workfn() then processes the bitmaps on the next IRQ work invocation, acquiring each target CPU's rq->lock in turn.
kick_cpus_irq_workfn() — actual IPI delivery
static void kick_cpus_irq_workfn(struct irq_work *irq_work)
{
for_each_cpu(cpu, this_scx->cpus_to_kick) {
kick_one_cpu(cpu, this_rq);
cpumask_clear_cpu(cpu, this_scx->cpus_to_kick);
cpumask_clear_cpu(cpu, this_scx->cpus_to_kick_if_idle);
}
for_each_cpu(cpu, this_scx->cpus_to_kick_if_idle) {
kick_one_cpu_if_idle(cpu, this_rq);
cpumask_clear_cpu(cpu, this_scx->cpus_to_kick_if_idle);
}
}
kick_one_cpu() acquires the target's rq->lock, optionally zeros the SCX task's slice (for PREEMPT), and calls resched_curr(). The cpus_to_kick_if_idle bitmap is also cleared when a CPU is in the cpus_to_kick set (a regular kick subsumes an idle kick for the same CPU).
balance_scx() refactoring
Multiple return 1/return 0 statements are replaced with goto has_tasks/goto out to allow the SCX_RQ_BALANCING flag to be cleared at a single exit point. This is necessary because can_skip_idle_kick() must be able to observe the flag from another CPU.
init_sched_ext_class() — initialization
Three cpumask variables and one irq_work are allocated per CPU:
BUG_ON(!zalloc_cpumask_var(&rq->scx.cpus_to_kick, GFP_KERNEL));
BUG_ON(!zalloc_cpumask_var(&rq->scx.cpus_to_kick_if_idle, GFP_KERNEL));
BUG_ON(!zalloc_cpumask_var(&rq->scx.cpus_to_preempt, GFP_KERNEL));
init_irq_work(&rq->scx.kick_cpus_irq_work, kick_cpus_irq_workfn);
The SCX_KICK_PREEMPT constant is also added to the WRITE_ONCE(v, ...) dummy write in init_sched_ext_class() so that the BTF type information for scx_kick_flags is included in the kernel's vmlinux BTF and available to BPF programs.
SCX_KICK_IDLE disables bypass
scx_bpf_kick_cpu() returns immediately if scx_ops_bypassing() is true (the scheduler is in bypass mode during PM operations). Allowing kick during bypass could cause irq_work_queue() malfunction because IRQ handling may not be fully online.
Key Concepts
- Deferred kicking via
irq_work: Directresched_curr()from BPF context is impossible because it requires holding the target CPU'srq->lock. Usingirq_workdefers the actual lock acquisition and IPI to a safe context. The BPF program only sets bits and queues the work. cpus_to_kickvs.cpus_to_kick_if_idle: Two separate bitmaps per CPU. A kick incpus_to_kickalways reschedules; a kick incpus_to_kick_if_idleonly reschedules ifcan_skip_idle_kick()returns false. If the same CPU appears in both, the unconditional kick wins.SCX_RQ_BALANCINGflag: Set whilebalance_scx()executes, enablingcan_skip_idle_kick()to correctly identify CPUs that are about to pick a task (and thus do not need an idle kick). Without this flag,can_skip_idle_kick()would use onlyis_idle_task(rq->curr), which creates a race withbalance_one()pulling tasks from remote queues.- Preemption model: sched_ext does not implement
wakeup_preempt(the standard scheduler hook for preempting on wakeup). Instead, preemption is achieved by: (1) setting the victim task's->scx.slice = 0and (2) callingresched_curr(). A zero slice causes the dispatch path to not keep the task, effectively evicting it.
Locking and Concurrency Notes
scx_bpf_kick_cpu()is callable fromops.enqueue(),ops.dispatch(), and anySCX_KF_*context. It useslocal_irq_save/restore(notrq->lock) to protect its cpumask modifications. This means the cpumasks are protected by IRQ disabling, not rq locking.kick_cpus_irq_workfn()runs in IRQ work context (softirq-like), where it acquires each target CPU'srq->lockindividually. It must not hold the calling CPU'srq->lock.kick_one_cpu()holds the target CPU'srq->lockwhile zeroingscx.sliceand callingresched_curr(). Therq->lockis required forresched_curr().- During CPU hotplug, a CPU may need to kick itself. The code handles this:
if (cpu_online(cpu) || cpu == cpu_of(this_rq))— self-kicks are always allowed even if the CPU is offline. p->scx.slicecleared by preemption is documented ininclude/linux/sched/ext.h: it should not be used to determine how long the task ran; usep->se.sum_exec_runtimeinstead.
Why Maintainers Need to Know This
- The two-step preemption protocol is critical: Preempting an SCX task requires both zeroing
->scx.sliceAND callingresched_curr(). Missing either step leaves the task running past its intended preemption point. TheSCX_ENQ_PREEMPTflag andkick_one_cpu()both implement both steps atomically underrq->lock. scx_bpf_kick_cpu()is disabled during bypass: BPF schedulers that callscx_bpf_kick_cpu()must be aware that during PM suspend/resume (bypass mode), their kicks are silently dropped. This can cause issues if a scheduler relies on kicks for correctness rather than just performance.- Batch kicks are efficient: The cpumask + irq_work design means that if a BPF program calls
scx_bpf_kick_cpu()for many CPUs in oneops.dispatch()invocation, all kicks are batched into a single irq_work execution. This is more efficient than sending individual IPIs. can_skip_idle_kick()has a race window: The comment in the code says "The race window is small and we don't and can't guarantee that @rq is only kicked while idle anyway. Skip only when sure." Maintainers should understand this meansSCX_KICK_IDLEis a best-effort optimization, not a guarantee.
Connection to Other Patches
- PATCH 18/30 (
scx_central) is the primary user ofscx_bpf_kick_cpu(): it usesSCX_KICK_PREEMPTto interrupt the central CPU andSCX_KICK_IDLEto wake worker CPUs. - PATCH 23/30 adds
SCX_KICK_WAIT, extendingscx_bpf_kick_cpu()with a blocking variant that waits for the kicked CPU to complete one scheduling cycle. - PATCH 21/30 (tickless) adds the
SCX_RQ_CAN_STOP_TICKflag toscx_rq_flags, extending thescx_rq_flagsenum introduced here. - The
compat.bpf.hfile added in this patch introduces__COMPAT_scx_bpf_kick_cpu_IDLE()as a demonstration of the BPF CO-RE compatibility pattern: schedulers can useSCX_KICK_IDLEif available or fall back to a regular kick on older kernels.
[PATCH 19/30] sched_ext: Make watchdog handle ops.dispatch() looping stall
View on Lore: https://lore.kernel.org/all/20240618212056.2833381-20-tj@kernel.org
Commit Message
The dispatch path retries if the local DSQ is still empty after
ops.dispatch() either dispatched or consumed a task. This is both out of
necessity and for convenience. It has to retry because the dispatch path
might lose the tasks to dequeue while the rq lock is released while trying
to migrate tasks across CPUs, and the retry mechanism makes ops.dispatch()
implementation easier as it only needs to make some forward progress each
iteration.
However, this makes it possible for ops.dispatch() to stall CPUs by
repeatedly dispatching ineligible tasks. If all CPUs are stalled that way,
the watchdog or sysrq handler can't run and the system can't be saved. Let's
address the issue by breaking out of the dispatch loop after 32 iterations.
It is unlikely but not impossible for ops.dispatch() to legitimately go over
the iteration limit. We want to come back to the dispatch path in such cases
as not doing so risks stalling the CPU by idling with runnable tasks
pending. As the previous task is still current in balance_scx(),
resched_curr() doesn't do anything - it will just get cleared. Let's instead
use scx_kick_bpf() which will trigger reschedule after switching to the next
task which will likely be the idle task.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: David Vernet <dvernet@meta.com>
---
kernel/sched/ext.c | 17 +++++++++++++++++
tools/sched_ext/scx_qmap.bpf.c | 15 +++++++++++++++
tools/sched_ext/scx_qmap.c | 8 ++++++--
3 files changed, 38 insertions(+), 2 deletions(-)
diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c
index 213793d086d7..89bcca84d6b5 100644
--- a/kernel/sched/ext.c
+++ b/kernel/sched/ext.c
@@ -8,6 +8,7 @@
enum scx_consts {
SCX_DSP_DFL_MAX_BATCH = 32,
+ SCX_DSP_MAX_LOOPS = 32,
SCX_WATCHDOG_MAX_TIMEOUT = 30 * HZ,
SCX_EXIT_BT_LEN = 64,
@@ -665,6 +666,7 @@ static struct kobject *scx_root_kobj;
#define CREATE_TRACE_POINTS
#include <trace/events/sched_ext.h>
+static void scx_bpf_kick_cpu(s32 cpu, u64 flags);
static __printf(3, 4) void scx_ops_exit_kind(enum scx_exit_kind kind,
s64 exit_code,
const char *fmt, ...);
@@ -1906,6 +1908,7 @@ static int balance_scx(struct rq *rq, struct task_struct *prev,
{
struct scx_dsp_ctx *dspc = this_cpu_ptr(scx_dsp_ctx);
bool prev_on_scx = prev->sched_class == &ext_sched_class;
+ int nr_loops = SCX_DSP_MAX_LOOPS;
bool has_tasks = false;
lockdep_assert_rq_held(rq);
@@ -1962,6 +1965,20 @@ static int balance_scx(struct rq *rq, struct task_struct *prev,
goto has_tasks;
if (consume_dispatch_q(rq, rf, &scx_dsq_global))
goto has_tasks;
+
+ /*
+ * ops.dispatch() can trap us in this loop by repeatedly
+ * dispatching ineligible tasks. Break out once in a while to
+ * allow the watchdog to run. As IRQ can't be enabled in
+ * balance(), we want to complete this scheduling cycle and then
+ * start a new one. IOW, we want to call resched_curr() on the
+ * next, most likely idle, task, not the current one. Use
+ * scx_bpf_kick_cpu() for deferred kicking.
+ */
+ if (unlikely(!--nr_loops)) {
+ scx_bpf_kick_cpu(cpu_of(rq), 0);
+ break;
+ }
} while (dspc->nr_tasks);
goto out;
diff --git a/tools/sched_ext/scx_qmap.bpf.c b/tools/sched_ext/scx_qmap.bpf.c
index 5b3da28bf042..879fc9c788e5 100644
--- a/tools/sched_ext/scx_qmap.bpf.c
+++ b/tools/sched_ext/scx_qmap.bpf.c
@@ -31,6 +31,7 @@ char _license[] SEC("license") = "GPL";
const volatile u64 slice_ns = SCX_SLICE_DFL;
const volatile u32 stall_user_nth;
const volatile u32 stall_kernel_nth;
+const volatile u32 dsp_inf_loop_after;
const volatile u32 dsp_batch;
const volatile s32 disallow_tgid;
const volatile bool suppress_dump;
@@ -198,6 +199,20 @@ void BPF_STRUCT_OPS(qmap_dispatch, s32 cpu, struct task_struct *prev)
if (scx_bpf_consume(SHARED_DSQ))
return;
+ if (dsp_inf_loop_after && nr_dispatched > dsp_inf_loop_after) {
+ /*
+ * PID 2 should be kthreadd which should mostly be idle and off
+ * the scheduler. Let's keep dispatching it to force the kernel
+ * to call this function over and over again.
+ */
+ p = bpf_task_from_pid(2);
+ if (p) {
+ scx_bpf_dispatch(p, SCX_DSQ_LOCAL, slice_ns, 0);
+ bpf_task_release(p);
+ return;
+ }
+ }
+
if (!(cpuc = bpf_map_lookup_elem(&cpu_ctx_stor, &zero))) {
scx_bpf_error("failed to look up cpu_ctx");
return;
diff --git a/tools/sched_ext/scx_qmap.c b/tools/sched_ext/scx_qmap.c
index a1123a17581b..594147a710a8 100644
--- a/tools/sched_ext/scx_qmap.c
+++ b/tools/sched_ext/scx_qmap.c
@@ -19,13 +19,14 @@ const char help_fmt[] =
"\n"
"See the top-level comment in .bpf.c for more details.\n"
"\n"
-"Usage: %s [-s SLICE_US] [-e COUNT] [-t COUNT] [-T COUNT] [-b COUNT]\n"
+"Usage: %s [-s SLICE_US] [-e COUNT] [-t COUNT] [-T COUNT] [-l COUNT] [-b COUNT]\n"
" [-d PID] [-D LEN] [-p] [-v]\n"
"\n"
" -s SLICE_US Override slice duration\n"
" -e COUNT Trigger scx_bpf_error() after COUNT enqueues\n"
" -t COUNT Stall every COUNT'th user thread\n"
" -T COUNT Stall every COUNT'th kernel thread\n"
+" -l COUNT Trigger dispatch infinite looping after COUNT dispatches\n"
" -b COUNT Dispatch upto COUNT tasks together\n"
" -d PID Disallow a process from switching into SCHED_EXT (-1 for self)\n"
" -D LEN Set scx_exit_info.dump buffer length\n"
@@ -61,7 +62,7 @@ int main(int argc, char **argv)
skel = SCX_OPS_OPEN(qmap_ops, scx_qmap);
- while ((opt = getopt(argc, argv, "s:e:t:T:b:d:D:Spvh")) != -1) {
+ while ((opt = getopt(argc, argv, "s:e:t:T:l:b:d:D:Spvh")) != -1) {
switch (opt) {
case 's':
skel->rodata->slice_ns = strtoull(optarg, NULL, 0) * 1000;
@@ -75,6 +76,9 @@ int main(int argc, char **argv)
case 'T':
skel->rodata->stall_kernel_nth = strtoul(optarg, NULL, 0);
break;
+ case 'l':
+ skel->rodata->dsp_inf_loop_after = strtoul(optarg, NULL, 0);
+ break;
case 'b':
skel->rodata->dsp_batch = strtoul(optarg, NULL, 0);
break;
--
2.45.2
Diff
---
kernel/sched/ext.c | 17 +++++++++++++++++
tools/sched_ext/scx_qmap.bpf.c | 15 +++++++++++++++
tools/sched_ext/scx_qmap.c | 8 ++++++--
3 files changed, 38 insertions(+), 2 deletions(-)
diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c
index 213793d086d7..89bcca84d6b5 100644
--- a/kernel/sched/ext.c
+++ b/kernel/sched/ext.c
@@ -8,6 +8,7 @@
enum scx_consts {
SCX_DSP_DFL_MAX_BATCH = 32,
+ SCX_DSP_MAX_LOOPS = 32,
SCX_WATCHDOG_MAX_TIMEOUT = 30 * HZ,
SCX_EXIT_BT_LEN = 64,
@@ -665,6 +666,7 @@ static struct kobject *scx_root_kobj;
#define CREATE_TRACE_POINTS
#include <trace/events/sched_ext.h>
+static void scx_bpf_kick_cpu(s32 cpu, u64 flags);
static __printf(3, 4) void scx_ops_exit_kind(enum scx_exit_kind kind,
s64 exit_code,
const char *fmt, ...);
@@ -1906,6 +1908,7 @@ static int balance_scx(struct rq *rq, struct task_struct *prev,
{
struct scx_dsp_ctx *dspc = this_cpu_ptr(scx_dsp_ctx);
bool prev_on_scx = prev->sched_class == &ext_sched_class;
+ int nr_loops = SCX_DSP_MAX_LOOPS;
bool has_tasks = false;
lockdep_assert_rq_held(rq);
@@ -1962,6 +1965,20 @@ static int balance_scx(struct rq *rq, struct task_struct *prev,
goto has_tasks;
if (consume_dispatch_q(rq, rf, &scx_dsq_global))
goto has_tasks;
+
+ /*
+ * ops.dispatch() can trap us in this loop by repeatedly
+ * dispatching ineligible tasks. Break out once in a while to
+ * allow the watchdog to run. As IRQ can't be enabled in
+ * balance(), we want to complete this scheduling cycle and then
+ * start a new one. IOW, we want to call resched_curr() on the
+ * next, most likely idle, task, not the current one. Use
+ * scx_bpf_kick_cpu() for deferred kicking.
+ */
+ if (unlikely(!--nr_loops)) {
+ scx_bpf_kick_cpu(cpu_of(rq), 0);
+ break;
+ }
} while (dspc->nr_tasks);
goto out;
diff --git a/tools/sched_ext/scx_qmap.bpf.c b/tools/sched_ext/scx_qmap.bpf.c
index 5b3da28bf042..879fc9c788e5 100644
--- a/tools/sched_ext/scx_qmap.bpf.c
+++ b/tools/sched_ext/scx_qmap.bpf.c
@@ -31,6 +31,7 @@ char _license[] SEC("license") = "GPL";
const volatile u64 slice_ns = SCX_SLICE_DFL;
const volatile u32 stall_user_nth;
const volatile u32 stall_kernel_nth;
+const volatile u32 dsp_inf_loop_after;
const volatile u32 dsp_batch;
const volatile s32 disallow_tgid;
const volatile bool suppress_dump;
@@ -198,6 +199,20 @@ void BPF_STRUCT_OPS(qmap_dispatch, s32 cpu, struct task_struct *prev)
if (scx_bpf_consume(SHARED_DSQ))
return;
+ if (dsp_inf_loop_after && nr_dispatched > dsp_inf_loop_after) {
+ /*
+ * PID 2 should be kthreadd which should mostly be idle and off
+ * the scheduler. Let's keep dispatching it to force the kernel
+ * to call this function over and over again.
+ */
+ p = bpf_task_from_pid(2);
+ if (p) {
+ scx_bpf_dispatch(p, SCX_DSQ_LOCAL, slice_ns, 0);
+ bpf_task_release(p);
+ return;
+ }
+ }
+
if (!(cpuc = bpf_map_lookup_elem(&cpu_ctx_stor, &zero))) {
scx_bpf_error("failed to look up cpu_ctx");
return;
diff --git a/tools/sched_ext/scx_qmap.c b/tools/sched_ext/scx_qmap.c
index a1123a17581b..594147a710a8 100644
--- a/tools/sched_ext/scx_qmap.c
+++ b/tools/sched_ext/scx_qmap.c
@@ -19,13 +19,14 @@ const char help_fmt[] =
"\n"
"See the top-level comment in .bpf.c for more details.\n"
"\n"
-"Usage: %s [-s SLICE_US] [-e COUNT] [-t COUNT] [-T COUNT] [-b COUNT]\n"
+"Usage: %s [-s SLICE_US] [-e COUNT] [-t COUNT] [-T COUNT] [-l COUNT] [-b COUNT]\n"
" [-d PID] [-D LEN] [-p] [-v]\n"
"\n"
" -s SLICE_US Override slice duration\n"
" -e COUNT Trigger scx_bpf_error() after COUNT enqueues\n"
" -t COUNT Stall every COUNT'th user thread\n"
" -T COUNT Stall every COUNT'th kernel thread\n"
+" -l COUNT Trigger dispatch infinite looping after COUNT dispatches\n"
" -b COUNT Dispatch upto COUNT tasks together\n"
" -d PID Disallow a process from switching into SCHED_EXT (-1 for self)\n"
" -D LEN Set scx_exit_info.dump buffer length\n"
@@ -61,7 +62,7 @@ int main(int argc, char **argv)
skel = SCX_OPS_OPEN(qmap_ops, scx_qmap);
- while ((opt = getopt(argc, argv, "s:e:t:T:b:d:D:Spvh")) != -1) {
+ while ((opt = getopt(argc, argv, "s:e:t:T:l:b:d:D:Spvh")) != -1) {
switch (opt) {
case 's':
skel->rodata->slice_ns = strtoull(optarg, NULL, 0) * 1000;
@@ -75,6 +76,9 @@ int main(int argc, char **argv)
case 'T':
skel->rodata->stall_kernel_nth = strtoul(optarg, NULL, 0);
break;
+ case 'l':
+ skel->rodata->dsp_inf_loop_after = strtoul(optarg, NULL, 0);
+ break;
case 'b':
skel->rodata->dsp_batch = strtoul(optarg, NULL, 0);
break;
--
2.45.2
Implementation Analysis
Overview
The sched_ext dispatch loop in balance_scx() automatically retries when ops.dispatch() makes progress (dispatches or consumes a task) but the local DSQ is still empty afterward. This retry is both necessary (for correctness when rq lock drops cause lost tasks) and convenient (BPF schedulers don't need to dispatch exactly the right amount). However, it creates a DoS vector: a buggy ops.dispatch() that repeatedly dispatches tasks ineligible for the current CPU traps the CPU in an infinite loop. If all CPUs are trapped, the watchdog cannot run, and the system hangs. This patch breaks out of the dispatch loop after SCX_DSP_MAX_LOOPS = 32 iterations by using scx_bpf_kick_cpu() to reschedule after the current task is done.
Code Walkthrough
kernel/sched/ext.c — new constant
enum scx_consts {
SCX_DSP_DFL_MAX_BATCH = 32,
SCX_DSP_MAX_LOOPS = 32, // NEW
...
};
32 iterations is chosen as a balance: generous enough for legitimate dispatch bursts, strict enough to prevent indefinite stalls.
kernel/sched/ext.c — forward declaration
static void scx_bpf_kick_cpu(s32 cpu, u64 flags);
scx_bpf_kick_cpu() is defined later in the file (as a __bpf_kfunc). This forward declaration is needed because balance_scx() calls it but appears before the function definition. This is the first time the kernel-internal scheduler code calls a function that is also exposed as a BPF kfunc — maintainers should note this coupling.
kernel/sched/ext.c — loop counter in balance_scx()
static int balance_scx(struct rq *rq, struct task_struct *prev,
struct rq_flags *rf)
{
int nr_loops = SCX_DSP_MAX_LOOPS; // NEW
...
do {
...
if (rq->scx.local_dsq.nr) goto has_tasks;
if (consume_dispatch_q(rq, rf, &scx_dsq_global)) goto has_tasks;
if (unlikely(!--nr_loops)) { // NEW
scx_bpf_kick_cpu(cpu_of(rq), 0);
break;
}
} while (dspc->nr_tasks);
...
}
The countdown runs inside the do { ... } while (dspc->nr_tasks) loop. When nr_loops reaches zero, instead of calling resched_curr(rq) directly, scx_bpf_kick_cpu(cpu_of(rq), 0) is called. This distinction is critical.
Why scx_bpf_kick_cpu() and not resched_curr()?
balance_scx() runs during the scheduler's task-pick path. At this point, prev is still current — resched_curr(rq) would set TIF_NEED_RESCHED on the current task, but because the scheduler is in the middle of selecting the next task, this flag would just get cleared immediately. The system would continue scheduling as if nothing happened.
scx_bpf_kick_cpu(cpu_of(rq), 0) defers the reschedule via irq_work. The irq_work fires after balance_scx() returns and after the scheduler has picked the next task (likely the idle task, since the local DSQ is empty). At that point, TIF_NEED_RESCHED is set on the idle task and the CPU will immediately try to schedule again — this time entering balance_scx() fresh with nr_loops reset to 32.
This "deferred self-kick" pattern is specifically designed for the case where the balancer itself needs to trigger a retry but cannot do so synchronously.
tools/sched_ext/scx_qmap.bpf.c — test fixture
const volatile u32 dsp_inf_loop_after;
void BPF_STRUCT_OPS(qmap_dispatch, s32 cpu, struct task_struct *prev)
{
...
if (dsp_inf_loop_after && nr_dispatched > dsp_inf_loop_after) {
p = bpf_task_from_pid(2); // PID 2 = kthreadd (usually idle)
if (p) {
scx_bpf_dispatch(p, SCX_DSQ_LOCAL, slice_ns, 0);
bpf_task_release(p);
return;
}
}
...
}
This deliberately triggers the dispatch loop stall: after dsp_inf_loop_after dispatches, the scheduler repeatedly dispatches kthreadd (PID 2) to the local DSQ. kthreadd is typically not runnable or has a different CPU affinity, so it keeps getting dispatched but never satisfies the dispatch loop's termination condition. The -l COUNT flag in scx_qmap.c activates this behavior for testing.
Key Concepts
- The dispatch loop retry invariant:
balance_scx()retries whenops.dispatch()dispatched or consumed at least one task but the local DSQ is still empty. This retry is necessary for correctness because cross-CPU task migrations (which require dropping and re-acquiringrq->lock) can cause tasks to disappear from the local DSQ between dispatch rounds. SCX_DSP_MAX_LOOPS = 32: Hard limit on dispatch loop iterations. Legitimate schedulers should never need more than a handful of retries. If a scheduler hits this limit in production, it indicates either a bug inops.dispatch()or an unusual workload with very high affinity-mismatch rates.- Deferred self-kick pattern: When a CPU needs to retry scheduling after the current task has been selected, using
scx_bpf_kick_cpu(self, 0)via irq_work is the correct mechanism. Directresched_curr()inbalance_scx()context has no effect. nr_loopsis per-invocation: The counter resets every timebalance_scx()is called. 32 loops per scheduling invocation, not 32 loops globally. A legitimately slow dispatcher gets 32 attempts each time the CPU needs to pick a task.
Locking and Concurrency Notes
balance_scx()is called withrq->lockheld. The dispatch loop temporarily dropsrq->lockduringops.dispatch()(to allow cross-CPU migrations). Thenr_loopscounter is a local variable and is not affected by lock drops.scx_bpf_kick_cpu(cpu_of(rq), 0)called from withinbalance_scx()usesirq_work_queue()which is IRQ-safe but requires IRQs to be disabled.balance_scx()runs with IRQs disabled (rq->lock requirement), so this is safe.- The irq_work
kick_cpus_irq_workis per-CPU. Calling it frombalance_scx()for the same CPU (self-kick) will fire afterbalance_scx()completes and the lock is released.
Why Maintainers Need to Know This
- A looping stall is a BPF scheduler bug, not a kernel bug: If a system hangs with all CPUs stuck in dispatch loops, the root cause is
ops.dispatch()returning true (progress signal) without actually making the CPU runnable. The most common form is dispatching tasks to a local DSQ when those tasks have CPU affinity that prevents them from running on that CPU. - The 32-iteration limit gives the watchdog a chance to run: By breaking out every 32 iterations, the CPU eventually picks the idle task, which allows the watchdog kthread and sysrq handler to run. Without this limit, a single-CPU system with a looping
ops.dispatch()would hang permanently. - The forward declaration of
scx_bpf_kick_cpu()is a code smell: Having a__bpf_kfunccalled from the internal scheduler hot path creates a coupling between the BPF API surface and the scheduler internals. Future restructuring should consider whether this forward declaration should be replaced with a separate internal helper. - Test with
-lflag:scx_qmap -l 100triggers the infinite dispatch loop after 100 dispatches. This is the canonical test for this patch's fix. Running it should produce a watchdog-style exit rather than a system hang.
Connection to Other Patches
- PATCH 17/30 introduced
scx_bpf_kick_cpu(), which this patch repurposes as a deferred self-kick mechanism from withinbalance_scx(). The forward declaration added here is a direct consequence of that dependency. - The original watchdog (from an earlier patch in this series) detects task starvation — tasks not running for too long. This patch addresses a different failure mode: the dispatch path itself looping, which starves the watchdog rather than tasks.
- PATCH 18/30 (
scx_central) is susceptible to this exact failure mode: ifdispatch_to_cpu()keeps bouncing tasks toFALLBACK_DSQ_IDwithout successfully dispatching to any local DSQ, the dispatch retry loop could spin. Thescx_bpf_dispatch_nr_slots()check incentral_dispatchis the BPF-side guard against this; the 32-iteration limit is the kernel-side backstop.
Task and Operation Management (Patches 20–23)
Overview
This group of four patches refines how sched_ext manages the lifecycle of individual tasks and tracks in-flight operations. The core implementation (patch 09) established the basic enqueue and dispatch mechanism, but several important task state transitions were underspecified: What exactly is the BPF program notified of when a task's CPU run begins or ends? Can the kernel suppress scheduling ticks when the BPF scheduler doesn't need them? What prevents races when the BPF program and kernel code concurrently touch the same task? And how does a CPU coordinate with another to ensure a kicked CPU has actually rescheduled?
Patches 20–23 answer each of these questions, completing the task lifecycle interface and adding the concurrency controls needed for robust BPF scheduler implementation.
Why These Patches Are Needed
Fine-Grained Task State Visibility
The base sched_ext implementation notifies BPF programs of two task events: ops.enqueue()
(task becomes runnable) and dispatch (task is placed on a CPU's local DSQ). But between
"task is runnable" and "task is actually running on a CPU" there is a gap that matters for
scheduling algorithms:
- Work-conserving schedulers need to know when a CPU actually begins executing a task (not just when the task was dispatched) to update utilization estimates.
- Gang schedulers need to know when a group of related tasks all become runnable simultaneously (they are waiting for all members to be ready before dispatching any).
- Preemption-aware schedulers need to know when a task stops executing, distinguishing between "stopped voluntarily" (syscall, I/O wait) and "stopped involuntarily" (preempted).
The base implementation does not expose these transitions to BPF programs.
Tick Suppression
Linux's scheduler tick fires periodically (typically every 1–4ms) on each CPU to implement
time-slicing. The tick calls scheduler_tick(), which checks whether the current task has
exhausted its time slice and sets TIF_NEED_RESCHED if so.
Many latency-sensitive workloads benefit from nohz_full (tickless operation): when a CPU
runs a single task with no pending work, the tick can be suppressed, eliminating jitter from
interrupt-driven preemption. CFS supports this natively. sched_ext needs the same capability.
In-Flight Operation Tracking
When a BPF program calls scx_bpf_dispatch(), the kernel places the task in a DSQ. But
between the BPF program deciding to dispatch a task and the kernel actually executing the
dispatch, the task's state might change: the task could be migrated, preempted, or even exit.
The kernel needs a mechanism to track tasks that are subjects of an in-flight SCX operation
and ensure the operation completes safely or is cancelled.
Synchronous Kick Confirmation
scx_bpf_kick_cpu() (patch 17) sends an IPI and returns immediately — it does not wait for
the target CPU to actually reschedule. For most use cases this is fine (fire and forget), but
for some coordination patterns, the calling CPU needs a guarantee that the target CPU has
processed the kick. Without this, race conditions are possible.
Key Concepts
PATCH 20 — Task Lifecycle Callbacks
Patch 20 adds four new sched_ext_ops callbacks that give BPF programs fine-grained visibility
into task state:
ops.runnable(p, enq_flags): Called when a task transitions from sleeping/waiting to
runnable. This happens just before ops.enqueue() is called. The distinction from
ops.enqueue() is subtle but important: runnable() is called once per sleep-to-wake
transition, while enqueue() may be called multiple times (e.g., after preemption and
re-queueing). BPF schedulers use runnable() to track the number of runnable tasks (for
load estimation) without double-counting re-queued tasks.
ops.running(p): Called immediately before a task begins executing on a CPU
(just before context_switch()). This is the notification that the task is now consuming CPU
cycles. BPF schedulers use this to start per-task CPU time accounting, update utilization
estimates, and record task.start_time for scheduling analytics.
ops.stopping(p, runnable): Called when a task is about to stop running on a CPU, before
it is removed from rq->curr. The runnable argument indicates whether the task will remain
runnable (true: preempted or yielded) or become blocked (false: waiting for I/O or sleeping).
This is the symmetric counterpart to ops.running().
ops.quiescent(p, deq_flags): Called when a task transitions from runnable to not runnable
(the task has gone to sleep, waiting for I/O, or exited). This is the symmetric counterpart to
ops.runnable(). BPF schedulers use this to decrement their runnable task count.
The relationship between these four callbacks traces a complete task lifecycle:
ops.runnable() ← task wakes up
ops.enqueue() ← task placed in a DSQ
ops.running() ← task begins executing on CPU
ops.stopping() ← task stops executing (may still be runnable)
ops.enqueue() ← (if still runnable: re-queued after preemption)
ops.quiescent() ← task goes to sleep
Beyond these four, patch 20 also adds:
ops.enable(p): Called when a task first adopts SCHED_EXT scheduling policy (or on
system init for existing tasks). This is where BPF programs initialize per-task state
(e.g., allocating a BPF_MAP_TYPE_TASK_STORAGE entry). Called from switching_to() (patch 04),
before the task is enqueued.
ops.disable(p): Called when a task leaves SCHED_EXT policy (either by calling
sched_setscheduler() to switch to another policy, or when the BPF scheduler is being disabled
and all tasks are returned to CFS). This is where BPF programs free per-task state.
The enable()/disable() pair guarantees balanced allocation/deallocation. The kernel ensures
disable() is called exactly once for each enable() call, even during error exits.
PATCH 21 — Tickless Support (nohz_full Integration)
When a BPF scheduler runs a task on a CPU and that CPU has no other runnable tasks, the
scheduler tick can be suppressed. Patch 21 integrates sched_ext with nohz_full:
The relevant function is task_tick_scx(rq, curr, queued), the sched_ext implementation of
sched_class->task_tick(). In CFS, this function checks whether the current task's time slice
has expired. In sched_ext, the tick is suppressed if all of the following are true:
- The CPU's local DSQ is empty (no other SCX tasks waiting).
- The global DSQ is empty.
- The BPF scheduler has not set any "tick needed" flag.
- The BPF scheduler's
ops.dispatch()would return immediately (no pending work).
When these conditions hold, task_tick_scx() calls sched_can_stop_tick(rq) to signal that
the tick can be stopped. The nohz_full infrastructure then suppresses the timer interrupt
on this CPU until a new task becomes runnable or the CPU is kicked.
Interaction with time slices: sched_ext tasks have a scx_entity.slice field that holds
the remaining time slice. When the tick fires and slice reaches zero, the task is preempted.
With tickless operation, preemption is driven by timer expiry rather than periodic ticks,
which may result in longer time slices than configured — this is intentional for nohz_full
workloads where jitter reduction is more important than strict time-slice enforcement.
BPF program responsibility: A BPF scheduler that dispatches tasks with scx_bpf_dispatch()
using SCX_SLICE_INF (infinite slice) is declaring that it will handle preemption itself (via
scx_bpf_kick_cpu() with SCX_KICK_PREEMPT). For such schedulers, tick suppression is
always appropriate. A scheduler that uses finite slices should not enable tick suppression
unless it can tolerate some slice overshoot.
PATCH 22 — scx_entity::in_op_task
This patch adds in_op_task, a flag in scx_entity that marks tasks that are currently the
subject of an in-flight SCX operation. The primary use case is scx_bpf_dispatch(): when the
BPF program calls scx_bpf_dispatch(p, dsq_id, slice, enq_flags), the kernel must:
- Validate that
pis still in a valid state for dispatch. - Acquire the appropriate locks (the task's DSQ lock, possibly the runqueue lock).
- Move the task to the target DSQ.
Between steps 1 and 2, the task could be migrated, preempted, or exit. Without in_op_task,
the kernel would have to check every condition again after acquiring the locks, and if the
state changed, it would have to abort and potentially corrupt the DSQ.
in_op_task works as a serialization token:
- It is set when a BPF callback begins processing a task (e.g., at the start of
ops.enqueue()). - It is cleared when the callback returns.
- Any kernel code path that needs to modify a task's scheduling state checks
in_op_taskfirst. If set, it waits (viascx_task_iter_wait()) for the in-flight operation to complete.
This avoids TOCTOU (Time Of Check, Time Of Use) races where the BPF program checks a task's state, decides to dispatch it, and the kernel changes the state before the dispatch completes.
Relationship to BPF verifier: The BPF verifier cannot prevent all races — it can prevent
BPF programs from accessing invalid memory, but it cannot reason about kernel-side state
changes. in_op_task is the kernel-side mechanism that complements the verifier's static
analysis by providing runtime serialization.
PATCH 23 — SCX_KICK_WAIT
Patch 23 adds the SCX_KICK_WAIT flag to scx_bpf_kick_cpu(). When specified:
- The kick is sent as usual (IPI to target CPU).
- The calling CPU then waits (via
wait_event()on a per-CPU completion) until the target CPU has completed one full scheduling round (i.e.,schedule()has returned and a new task has been selected).
Use case: A BPF scheduler that dispatches a task to CPU A's local DSQ and then needs to know that CPU A has actually picked up the task (not just received the kick) before proceeding. For example, a gang scheduler waiting for all CPUs in a gang to have their tasks running.
Implementation: The target CPU sets a completion event at the end of __schedule() when
it detects a pending SCX_KICK_WAIT. The calling CPU's scx_bpf_kick_cpu() waits on this
completion with a timeout. If the timeout expires (e.g., the target CPU is stuck), the wait
returns an error but does not trigger a watchdog event — it is the calling CPU's BPF program's
responsibility to handle the timeout.
Cost: SCX_KICK_WAIT adds synchronization overhead: the calling CPU blocks until the
target CPU reschedules. For high-frequency scheduling decisions, this is too expensive. It is
intended for coarse-grained coordination (gang startup, barrier-style synchronization) rather
than per-task dispatch.
Connections Between Patches
PATCH 20 (lifecycle callbacks)
└─→ ops.enable/disable: required foundation for per-task BPF state
└─→ ops.running/stopping: consumed by PATCH 21 (tick suppression) to determine
whether tick can be stopped based on actual execution state
└─→ ops.runnable/quiescent: enable accurate task count tracking
PATCH 21 (tickless)
└─→ Depends on ops.stopping (PATCH 20): tick is only suppressed after stopping
└─→ Interacts with scx_entity.slice: inf-slice tasks are primary beneficiaries
PATCH 22 (in_op_task)
└─→ Serializes against PATCH 20 callbacks: in_op_task prevents ops.disable() from
racing with ops.enqueue() for the same task
└─→ Interacts with PATCH 23: in-flight dispatch (in_op_task) and synchronous kick
(KICK_WAIT) together allow BPF programs to implement deterministic scheduling rounds
PATCH 23 (SCX_KICK_WAIT)
└─→ Extends PATCH 17 (scx_bpf_kick_cpu) with a synchronous variant
└─→ Used alongside PATCH 22 to confirm task was dispatched and CPU has rescheduled
What to Focus On
For a maintainer, the critical lessons from this group:
-
The enable/disable pairing guarantee.
ops.enable()andops.disable()must be called in perfectly matched pairs. The kernel's guarantee:disable()is called exactly once for everyenable(), even during error exit (SCX_EXIT_ERROR). When reviewing any change to the class transition or disable path, verify this invariant is preserved. A misseddisable()call leaks BPF per-task resources; a spuriousdisable()call on a task that never hadenable()called is undefined behavior. -
runnable/quiescent vs enqueue/dequeue. The new lifecycle callbacks form two parallel pairs:
runnable/quiescenttrack the task's runnable state (independent of CPU), whilerunning/stoppingtrack CPU execution. These are distinct events and must not be confused. A task that is preempted firesstopping(runnable=true)(it is still runnable) and will fireenqueue()again when re-queued, but does not firequiescent(). When reviewing BPF schedulers that implement these callbacks, verify they correctly distinguish preemption from blocking. -
Tick suppression and time-slice accuracy.
nohz_fullintegration means tick-based preemption may fire late. A BPF scheduler that relies on precise time slices (e.g., for real-time guarantees) should not use tickless mode. When reviewing changes to the tick path intask_tick_scx(), verify that finite-slice tasks still receive timely preemption. -
in_op_task as a liveness concern.
in_op_taskserializes kernel code against in-flight BPF operations. If a BPF program holdsin_op_taskindefinitely (e.g., infinite loop inops.enqueue()), kernel code waiting for the serialization will also wait indefinitely. The dispatch watchdog (patch 19) catches the infinite loop case, butin_op_taskitself has no timeout. When reviewing future uses ofin_op_task, ensure the in-flight operation is always bounded in time. -
SCX_KICK_WAIT and priority inversion. If CPU A calls
scx_bpf_kick_cpu(B, SCX_KICK_WAIT)and CPU B is running a high-priority RT task (which cannot be preempted by sched_ext), CPU A will wait until that RT task voluntarily yields. This is a form of priority inversion: the BPF scheduler on CPU A (which may be handling a high-priority SCX task) is blocked waiting for an RT task on CPU B to yield. Future changes toSCX_KICK_WAITshould consider adding a timeout that the BPF program can configure to avoid unbounded waits.
[PATCH 20/30] sched_ext: Add task state tracking operations
View on Lore: https://lore.kernel.org/all/20240618212056.2833381-21-tj@kernel.org
Commit Message
Being able to track the task runnable and running state transitions are
useful for a variety of purposes including latency tracking and load factor
calculation.
Currently, BPF schedulers don't have a good way of tracking these
transitions. Becoming runnable can be determined from ops.enqueue() but
becoming quiescent can only be inferred from the lack of subsequent enqueue.
Also, as the local dsq can have multiple tasks and some events are handled
in the sched_ext core, it's difficult to determine when a given task starts
and stops executing.
This patch adds sched_ext_ops.runnable(), .running(), .stopping() and
.quiescent() operations to track the task runnable and running state
transitions. They're mostly self explanatory; however, we want to ensure
that running <-> stopping transitions are always contained within runnable
<-> quiescent transitions which is a bit different from how the scheduler
core behaves. This adds a bit of complication. See the comment in
dequeue_task_scx().
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: David Vernet <dvernet@meta.com>
Acked-by: Josh Don <joshdon@google.com>
Acked-by: Hao Luo <haoluo@google.com>
Acked-by: Barret Rhoden <brho@google.com>
---
kernel/sched/ext.c | 105 +++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 105 insertions(+)
diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c
index 89bcca84d6b5..2e652f7b8f54 100644
--- a/kernel/sched/ext.c
+++ b/kernel/sched/ext.c
@@ -214,6 +214,72 @@ struct sched_ext_ops {
*/
void (*tick)(struct task_struct *p);
+ /**
+ * runnable - A task is becoming runnable on its associated CPU
+ * @p: task becoming runnable
+ * @enq_flags: %SCX_ENQ_*
+ *
+ * This and the following three functions can be used to track a task's
+ * execution state transitions. A task becomes ->runnable() on a CPU,
+ * and then goes through one or more ->running() and ->stopping() pairs
+ * as it runs on the CPU, and eventually becomes ->quiescent() when it's
+ * done running on the CPU.
+ *
+ * @p is becoming runnable on the CPU because it's
+ *
+ * - waking up (%SCX_ENQ_WAKEUP)
+ * - being moved from another CPU
+ * - being restored after temporarily taken off the queue for an
+ * attribute change.
+ *
+ * This and ->enqueue() are related but not coupled. This operation
+ * notifies @p's state transition and may not be followed by ->enqueue()
+ * e.g. when @p is being dispatched to a remote CPU, or when @p is
+ * being enqueued on a CPU experiencing a hotplug event. Likewise, a
+ * task may be ->enqueue()'d without being preceded by this operation
+ * e.g. after exhausting its slice.
+ */
+ void (*runnable)(struct task_struct *p, u64 enq_flags);
+
+ /**
+ * running - A task is starting to run on its associated CPU
+ * @p: task starting to run
+ *
+ * See ->runnable() for explanation on the task state notifiers.
+ */
+ void (*running)(struct task_struct *p);
+
+ /**
+ * stopping - A task is stopping execution
+ * @p: task stopping to run
+ * @runnable: is task @p still runnable?
+ *
+ * See ->runnable() for explanation on the task state notifiers. If
+ * !@runnable, ->quiescent() will be invoked after this operation
+ * returns.
+ */
+ void (*stopping)(struct task_struct *p, bool runnable);
+
+ /**
+ * quiescent - A task is becoming not runnable on its associated CPU
+ * @p: task becoming not runnable
+ * @deq_flags: %SCX_DEQ_*
+ *
+ * See ->runnable() for explanation on the task state notifiers.
+ *
+ * @p is becoming quiescent on the CPU because it's
+ *
+ * - sleeping (%SCX_DEQ_SLEEP)
+ * - being moved to another CPU
+ * - being temporarily taken off the queue for an attribute change
+ * (%SCX_DEQ_SAVE)
+ *
+ * This and ->dequeue() are related but not coupled. This operation
+ * notifies @p's state transition and may not be preceded by ->dequeue()
+ * e.g. when @p is being dispatched to a remote CPU.
+ */
+ void (*quiescent)(struct task_struct *p, u64 deq_flags);
+
/**
* yield - Yield CPU
* @from: yielding task
@@ -1359,6 +1425,9 @@ static void enqueue_task_scx(struct rq *rq, struct task_struct *p, int enq_flags
rq->scx.nr_running++;
add_nr_running(rq, 1);
+ if (SCX_HAS_OP(runnable))
+ SCX_CALL_OP(SCX_KF_REST, runnable, p, enq_flags);
+
do_enqueue_task(rq, p, enq_flags, sticky_cpu);
}
@@ -1418,6 +1487,26 @@ static void dequeue_task_scx(struct rq *rq, struct task_struct *p, int deq_flags
ops_dequeue(p, deq_flags);
+ /*
+ * A currently running task which is going off @rq first gets dequeued
+ * and then stops running. As we want running <-> stopping transitions
+ * to be contained within runnable <-> quiescent transitions, trigger
+ * ->stopping() early here instead of in put_prev_task_scx().
+ *
+ * @p may go through multiple stopping <-> running transitions between
+ * here and put_prev_task_scx() if task attribute changes occur while
+ * balance_scx() leaves @rq unlocked. However, they don't contain any
+ * information meaningful to the BPF scheduler and can be suppressed by
+ * skipping the callbacks if the task is !QUEUED.
+ */
+ if (SCX_HAS_OP(stopping) && task_current(rq, p)) {
+ update_curr_scx(rq);
+ SCX_CALL_OP(SCX_KF_REST, stopping, p, false);
+ }
+
+ if (SCX_HAS_OP(quiescent))
+ SCX_CALL_OP(SCX_KF_REST, quiescent, p, deq_flags);
+
if (deq_flags & SCX_DEQ_SLEEP)
p->scx.flags |= SCX_TASK_DEQD_FOR_SLEEP;
else
@@ -1999,6 +2088,10 @@ static void set_next_task_scx(struct rq *rq, struct task_struct *p, bool first)
p->se.exec_start = rq_clock_task(rq);
+ /* see dequeue_task_scx() on why we skip when !QUEUED */
+ if (SCX_HAS_OP(running) && (p->scx.flags & SCX_TASK_QUEUED))
+ SCX_CALL_OP(SCX_KF_REST, running, p);
+
clr_task_runnable(p, true);
}
@@ -2037,6 +2130,10 @@ static void put_prev_task_scx(struct rq *rq, struct task_struct *p)
update_curr_scx(rq);
+ /* see dequeue_task_scx() on why we skip when !QUEUED */
+ if (SCX_HAS_OP(stopping) && (p->scx.flags & SCX_TASK_QUEUED))
+ SCX_CALL_OP(SCX_KF_REST, stopping, p, true);
+
/*
* If we're being called from put_prev_task_balance(), balance_scx() may
* have decided that @p should keep running.
@@ -4081,6 +4178,10 @@ static s32 select_cpu_stub(struct task_struct *p, s32 prev_cpu, u64 wake_flags)
static void enqueue_stub(struct task_struct *p, u64 enq_flags) {}
static void dequeue_stub(struct task_struct *p, u64 enq_flags) {}
static void dispatch_stub(s32 prev_cpu, struct task_struct *p) {}
+static void runnable_stub(struct task_struct *p, u64 enq_flags) {}
+static void running_stub(struct task_struct *p) {}
+static void stopping_stub(struct task_struct *p, bool runnable) {}
+static void quiescent_stub(struct task_struct *p, u64 deq_flags) {}
static bool yield_stub(struct task_struct *from, struct task_struct *to) { return false; }
static void set_weight_stub(struct task_struct *p, u32 weight) {}
static void set_cpumask_stub(struct task_struct *p, const struct cpumask *mask) {}
@@ -4097,6 +4198,10 @@ static struct sched_ext_ops __bpf_ops_sched_ext_ops = {
.enqueue = enqueue_stub,
.dequeue = dequeue_stub,
.dispatch = dispatch_stub,
+ .runnable = runnable_stub,
+ .running = running_stub,
+ .stopping = stopping_stub,
+ .quiescent = quiescent_stub,
.yield = yield_stub,
.set_weight = set_weight_stub,
.set_cpumask = set_cpumask_stub,
--
2.45.2
Diff
---
kernel/sched/ext.c | 105 +++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 105 insertions(+)
diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c
index 89bcca84d6b5..2e652f7b8f54 100644
--- a/kernel/sched/ext.c
+++ b/kernel/sched/ext.c
@@ -214,6 +214,72 @@ struct sched_ext_ops {
*/
void (*tick)(struct task_struct *p);
+ /**
+ * runnable - A task is becoming runnable on its associated CPU
+ * @p: task becoming runnable
+ * @enq_flags: %SCX_ENQ_*
+ *
+ * This and the following three functions can be used to track a task's
+ * execution state transitions. A task becomes ->runnable() on a CPU,
+ * and then goes through one or more ->running() and ->stopping() pairs
+ * as it runs on the CPU, and eventually becomes ->quiescent() when it's
+ * done running on the CPU.
+ *
+ * @p is becoming runnable on the CPU because it's
+ *
+ * - waking up (%SCX_ENQ_WAKEUP)
+ * - being moved from another CPU
+ * - being restored after temporarily taken off the queue for an
+ * attribute change.
+ *
+ * This and ->enqueue() are related but not coupled. This operation
+ * notifies @p's state transition and may not be followed by ->enqueue()
+ * e.g. when @p is being dispatched to a remote CPU, or when @p is
+ * being enqueued on a CPU experiencing a hotplug event. Likewise, a
+ * task may be ->enqueue()'d without being preceded by this operation
+ * e.g. after exhausting its slice.
+ */
+ void (*runnable)(struct task_struct *p, u64 enq_flags);
+
+ /**
+ * running - A task is starting to run on its associated CPU
+ * @p: task starting to run
+ *
+ * See ->runnable() for explanation on the task state notifiers.
+ */
+ void (*running)(struct task_struct *p);
+
+ /**
+ * stopping - A task is stopping execution
+ * @p: task stopping to run
+ * @runnable: is task @p still runnable?
+ *
+ * See ->runnable() for explanation on the task state notifiers. If
+ * !@runnable, ->quiescent() will be invoked after this operation
+ * returns.
+ */
+ void (*stopping)(struct task_struct *p, bool runnable);
+
+ /**
+ * quiescent - A task is becoming not runnable on its associated CPU
+ * @p: task becoming not runnable
+ * @deq_flags: %SCX_DEQ_*
+ *
+ * See ->runnable() for explanation on the task state notifiers.
+ *
+ * @p is becoming quiescent on the CPU because it's
+ *
+ * - sleeping (%SCX_DEQ_SLEEP)
+ * - being moved to another CPU
+ * - being temporarily taken off the queue for an attribute change
+ * (%SCX_DEQ_SAVE)
+ *
+ * This and ->dequeue() are related but not coupled. This operation
+ * notifies @p's state transition and may not be preceded by ->dequeue()
+ * e.g. when @p is being dispatched to a remote CPU.
+ */
+ void (*quiescent)(struct task_struct *p, u64 deq_flags);
+
/**
* yield - Yield CPU
* @from: yielding task
@@ -1359,6 +1425,9 @@ static void enqueue_task_scx(struct rq *rq, struct task_struct *p, int enq_flags
rq->scx.nr_running++;
add_nr_running(rq, 1);
+ if (SCX_HAS_OP(runnable))
+ SCX_CALL_OP(SCX_KF_REST, runnable, p, enq_flags);
+
do_enqueue_task(rq, p, enq_flags, sticky_cpu);
}
@@ -1418,6 +1487,26 @@ static void dequeue_task_scx(struct rq *rq, struct task_struct *p, int deq_flags
ops_dequeue(p, deq_flags);
+ /*
+ * A currently running task which is going off @rq first gets dequeued
+ * and then stops running. As we want running <-> stopping transitions
+ * to be contained within runnable <-> quiescent transitions, trigger
+ * ->stopping() early here instead of in put_prev_task_scx().
+ *
+ * @p may go through multiple stopping <-> running transitions between
+ * here and put_prev_task_scx() if task attribute changes occur while
+ * balance_scx() leaves @rq unlocked. However, they don't contain any
+ * information meaningful to the BPF scheduler and can be suppressed by
+ * skipping the callbacks if the task is !QUEUED.
+ */
+ if (SCX_HAS_OP(stopping) && task_current(rq, p)) {
+ update_curr_scx(rq);
+ SCX_CALL_OP(SCX_KF_REST, stopping, p, false);
+ }
+
+ if (SCX_HAS_OP(quiescent))
+ SCX_CALL_OP(SCX_KF_REST, quiescent, p, deq_flags);
+
if (deq_flags & SCX_DEQ_SLEEP)
p->scx.flags |= SCX_TASK_DEQD_FOR_SLEEP;
else
@@ -1999,6 +2088,10 @@ static void set_next_task_scx(struct rq *rq, struct task_struct *p, bool first)
p->se.exec_start = rq_clock_task(rq);
+ /* see dequeue_task_scx() on why we skip when !QUEUED */
+ if (SCX_HAS_OP(running) && (p->scx.flags & SCX_TASK_QUEUED))
+ SCX_CALL_OP(SCX_KF_REST, running, p);
+
clr_task_runnable(p, true);
}
@@ -2037,6 +2130,10 @@ static void put_prev_task_scx(struct rq *rq, struct task_struct *p)
update_curr_scx(rq);
+ /* see dequeue_task_scx() on why we skip when !QUEUED */
+ if (SCX_HAS_OP(stopping) && (p->scx.flags & SCX_TASK_QUEUED))
+ SCX_CALL_OP(SCX_KF_REST, stopping, p, true);
+
/*
* If we're being called from put_prev_task_balance(), balance_scx() may
* have decided that @p should keep running.
@@ -4081,6 +4178,10 @@ static s32 select_cpu_stub(struct task_struct *p, s32 prev_cpu, u64 wake_flags)
static void enqueue_stub(struct task_struct *p, u64 enq_flags) {}
static void dequeue_stub(struct task_struct *p, u64 enq_flags) {}
static void dispatch_stub(s32 prev_cpu, struct task_struct *p) {}
+static void runnable_stub(struct task_struct *p, u64 enq_flags) {}
+static void running_stub(struct task_struct *p) {}
+static void stopping_stub(struct task_struct *p, bool runnable) {}
+static void quiescent_stub(struct task_struct *p, u64 deq_flags) {}
static bool yield_stub(struct task_struct *from, struct task_struct *to) { return false; }
static void set_weight_stub(struct task_struct *p, u32 weight) {}
static void set_cpumask_stub(struct task_struct *p, const struct cpumask *mask) {}
@@ -4097,6 +4198,10 @@ static struct sched_ext_ops __bpf_ops_sched_ext_ops = {
.enqueue = enqueue_stub,
.dequeue = dequeue_stub,
.dispatch = dispatch_stub,
+ .runnable = runnable_stub,
+ .running = running_stub,
+ .stopping = stopping_stub,
+ .quiescent = quiescent_stub,
.yield = yield_stub,
.set_weight = set_weight_stub,
.set_cpumask = set_cpumask_stub,
--
2.45.2
Implementation Analysis
Overview
BPF schedulers need accurate per-task timing data to implement policies like load balancing, latency-aware scheduling, or utilization-based admission. Before this patch, a BPF scheduler could detect when a task became runnable (from ops.enqueue()) but had no direct way to know when it started executing, when it stopped, or when it went to sleep. This patch adds four state-transition callbacks — ops.runnable(), ops.running(), ops.stopping(), and ops.quiescent() — that together form a complete task lifecycle notification system.
Code Walkthrough
New callbacks in sched_ext_ops
void (*runnable)(struct task_struct *p, u64 enq_flags);
void (*running)(struct task_struct *p);
void (*stopping)(struct task_struct *p, bool runnable);
void (*quiescent)(struct task_struct *p, u64 deq_flags);
The four callbacks describe a two-level state machine:
- Outer level: runnable ↔ quiescent (is the task on a run queue?)
- Inner level: running ↔ stopping (is the task currently executing on a CPU?)
The invariant sched_ext enforces is: running ↔ stopping transitions are always contained within runnable ↔ quiescent transitions. This is a stronger guarantee than the raw scheduler provides and requires special handling in dequeue_task_scx().
enqueue_task_scx() → ops.runnable()
static void enqueue_task_scx(struct rq *rq, struct task_struct *p, int enq_flags)
{
...
rq->scx.nr_running++;
add_nr_running(rq, 1);
if (SCX_HAS_OP(runnable))
SCX_CALL_OP_TASK(SCX_KF_REST, runnable, p, enq_flags);
do_enqueue_task(rq, p, enq_flags, sticky_cpu);
}
Called before do_enqueue_task() so the BPF scheduler sees the task enter its runnable state before any dispatch decisions are made. enq_flags carries context: SCX_ENQ_WAKEUP if the task is waking from sleep, or nothing if it is being moved from another CPU.
dequeue_task_scx() → ops.stopping() (early) + ops.quiescent()
This is the trickiest part of the patch:
static void dequeue_task_scx(struct rq *rq, struct task_struct *p, int deq_flags)
{
...
ops_dequeue(p, deq_flags);
/*
* A currently running task which is going off @rq first gets dequeued
* and then stops running. As we want running <-> stopping transitions
* to be contained within runnable <-> quiescent transitions, trigger
* ->stopping() early here instead of in put_prev_task_scx().
*/
if (SCX_HAS_OP(stopping) && task_current(rq, p)) {
update_curr_scx(rq);
SCX_CALL_OP_TASK(SCX_KF_REST, stopping, p, false);
}
if (SCX_HAS_OP(quiescent))
SCX_CALL_OP_TASK(SCX_KF_REST, quiescent, p, deq_flags);
...
}
The kernel's normal ordering is: dequeue the task, then (when another task is selected) call put_prev_task(). For the state machine invariant, quiescent must come AFTER stopping. If the task is currently running (task_current(rq, p)), stopping must be fired here — in dequeue_task_scx() — before quiescent, not later in put_prev_task_scx(). This produces the correct sequence: stopping(false) → quiescent().
set_next_task_scx() → ops.running()
static void set_next_task_scx(struct rq *rq, struct task_struct *p, bool first)
{
...
/* see dequeue_task_scx() on why we skip when !QUEUED */
if (SCX_HAS_OP(running) && (p->scx.flags & SCX_TASK_QUEUED))
SCX_CALL_OP_TASK(SCX_KF_REST, running, p);
clr_task_runnable(p, true);
}
ops.running() is called when a task is about to start executing (selected as the next task). The SCX_TASK_QUEUED guard is explained by the dequeue early-stopping() logic: if a task was dequeued (and thus had stopping(false) fired) but the kernel still calls set_next_task_scx() for it during balance races, the !QUEUED guard prevents a spurious running() call after stopping().
put_prev_task_scx() → ops.stopping(runnable=true)
static void put_prev_task_scx(struct rq *rq, struct task_struct *p)
{
...
/* see dequeue_task_scx() on why we skip when !QUEUED */
if (SCX_HAS_OP(stopping) && (p->scx.flags & SCX_TASK_QUEUED))
SCX_CALL_OP_TASK(SCX_KF_REST, stopping, p, true);
...
}
ops.stopping(p, runnable=true) is fired when a task yields the CPU but is still runnable (will be re-enqueued). The runnable parameter tells the BPF scheduler whether to expect a follow-up quiescent() (when runnable=false) or not.
Stub functions
static void runnable_stub(struct task_struct *p, u64 enq_flags) {}
static void running_stub(struct task_struct *p) {}
static void stopping_stub(struct task_struct *p, bool runnable) {}
static void quiescent_stub(struct task_struct *p, u64 deq_flags) {}
Added to __bpf_ops_sched_ext_ops (the dummy ops used for BPF verifier type checking). These ensure the BPF verifier knows the correct function signatures for these callbacks.
Key Concepts
- State machine invariant:
running↔stoppingtransitions are always nested withinrunnable↔quiescent. This is enforced by triggeringstopping(false)early indequeue_task_scx()when a currently-running task is dequeued. stopping(p, runnable)semantics: Therunnablebool tells the BPF scheduler what comes next.runnable=truemeans the task is preempted or yielded but stays on the run queue — noquiescent()will follow.runnable=falsemeans the task is leaving the run queue —quiescent()will follow immediately.SCX_TASK_QUEUEDguard: Bothops.running()andops.stopping(runnable=true)checkSCX_TASK_QUEUED. If the task is not queued (was already dequeued, which triggered earlystopping(false)), these callbacks are skipped. This prevents duplicatestoppingcalls.- Decoupling from
enqueue/dequeue: The comment inops.runnable()documentation is explicit:runnable()andenqueue()are related but NOT coupled. A task can becomerunnable()without a subsequentenqueue()(e.g., dispatched to a remote CPU's local DSQ directly), and a task can beenqueue()d without a precedingrunnable()(e.g., after exhausting its slice, which doesn't re-enter the runnable notification path).
Locking and Concurrency Notes
ops.runnable()is called fromenqueue_task_scx()withrq->lockheld. The BPF callback runs inSCX_KF_RESTcontext.ops.quiescent()is called fromdequeue_task_scx()withrq->lockheld.ops.running()is called fromset_next_task_scx()withrq->lockheld.ops.stopping()is called from bothdequeue_task_scx()(the early path) andput_prev_task_scx(), both withrq->lockheld.- All four callbacks are called under
rq->lock. BPF schedulers using these callbacks for per-task accounting must use lock-free data structures (BPF spinlocks or per-CPU data) to avoid deadlock. - The
SCX_CALL_OP_TASK()macro (introduced by PATCH 22/30) is used here — it setscurrent->scx.kf_tasks[0] = pbefore the call, enabling kfuncs that require the task to be "in-flight" to verify the task argument.
Why Maintainers Need to Know This
- The early
stopping()indequeue_task_scx()is subtle: New contributors often miss the comment explaining whystopping(false)fires duringdequeuerather than duringput_prev_task. The reason is the invariant:quiescentmust come last. Ifstoppingfired input_prev_task_scxas it would naturally, the order would bequiescent→stopping, violating the contract. runnable()≠enqueue(): use both for accurate accounting: A task being migrated (pulled from another CPU) triggersrunnable()without triggeringenqueue(). A BPF scheduler tracking queue depth viaenqueue()/dequeue()will miscount tasks that are migrated. Userunnable()/quiescent()for task-count accounting andenqueue()/dequeue()for dispatch queue accounting.stopping(runnable=false)followed byquiescent()is the sleep notification: When a task callssleep()or blocks on I/O, the sequence isstopping(false)thenquiescent(SCX_DEQ_SLEEP). A BPF scheduler tracking sleeping tasks should checkdeq_flags & SCX_DEQ_SLEEPinquiescent().scx_centralin PATCH 21/30 usesrunning/stoppingfor timer-based preemption: Thecpu_started_at[]array records when each CPU's current task started running (inops.running()) and clears it when the task stops (inops.stopping()). The BPF timer comparesbpf_ktime_get_ns()againststarted_at + slice_nsto decide which CPUs to preempt.
Connection to Other Patches
- PATCH 22/30 replaces
SCX_CALL_OP()withSCX_CALL_OP_TASK()for all callbacks that take a task argument, including all four introduced here. This enables task-specific kfuncs to verify their input task. - PATCH 21/30 (
scx_centraltickless) is the first real user ofops.running()andops.stopping()in the example schedulers, using them to track per-CPU execution start times for timer-based slice enforcement. - The
enq_flagspassed toops.runnable()anddeq_flagspassed toops.quiescent()use the same flag namespaces (SCX_ENQ_*andSCX_DEQ_*) established in earlier patches forops.enqueue()andops.dequeue().
[PATCH 21/30] sched_ext: Implement tickless support
View on Lore: https://lore.kernel.org/all/20240618212056.2833381-22-tj@kernel.org
Commit Message
Allow BPF schedulers to indicate tickless operation by setting p->scx.slice
to SCX_SLICE_INF. A CPU whose current task has infinte slice goes into
tickless operation.
scx_central is updated to use tickless operations for all tasks and
instead use a BPF timer to expire slices. This also uses the SCX_ENQ_PREEMPT
and task state tracking added by the previous patches.
Currently, there is no way to pin the timer on the central CPU, so it may
end up on one of the worker CPUs; however, outside of that, the worker CPUs
can go tickless both while running sched_ext tasks and idling.
With schbench running, scx_central shows:
root@test ~# grep ^LOC /proc/interrupts; sleep 10; grep ^LOC /proc/interrupts
LOC: 142024 656 664 449 Local timer interrupts
LOC: 161663 663 665 449 Local timer interrupts
Without it:
root@test ~ [SIGINT]# grep ^LOC /proc/interrupts; sleep 10; grep ^LOC /proc/interrupts
LOC: 188778 3142 3793 3993 Local timer interrupts
LOC: 198993 5314 6323 6438 Local timer interrupts
While scx_central itself is too barebone to be useful as a
production scheduler, a more featureful central scheduler can be built using
the same approach. Google's experience shows that such an approach can have
significant benefits for certain applications such as VM hosting.
v4: Allow operation even if BPF_F_TIMER_CPU_PIN is not available.
v3: Pin the central scheduler's timer on the central_cpu using
BPF_F_TIMER_CPU_PIN.
v2: Convert to BPF inline iterators.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: David Vernet <dvernet@meta.com>
Acked-by: Josh Don <joshdon@google.com>
Acked-by: Hao Luo <haoluo@google.com>
Acked-by: Barret Rhoden <brho@google.com>
---
include/linux/sched/ext.h | 1 +
kernel/sched/core.c | 11 ++-
kernel/sched/ext.c | 52 +++++++++-
kernel/sched/ext.h | 2 +
kernel/sched/sched.h | 1 +
tools/sched_ext/scx_central.bpf.c | 159 ++++++++++++++++++++++++++++--
tools/sched_ext/scx_central.c | 29 +++++-
7 files changed, 242 insertions(+), 13 deletions(-)
diff --git a/include/linux/sched/ext.h b/include/linux/sched/ext.h
index 3b2809b980ac..6f1a4977e9f8 100644
--- a/include/linux/sched/ext.h
+++ b/include/linux/sched/ext.h
@@ -16,6 +16,7 @@ enum scx_public_consts {
SCX_OPS_NAME_LEN = 128,
SCX_SLICE_DFL = 20 * 1000000, /* 20ms */
+ SCX_SLICE_INF = U64_MAX, /* infinite, implies nohz */
};
/*
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 1a3144c80af8..d5eff4036be7 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -1256,11 +1256,14 @@ bool sched_can_stop_tick(struct rq *rq)
return true;
/*
- * If there are no DL,RR/FIFO tasks, there must only be CFS tasks left;
- * if there's more than one we need the tick for involuntary
- * preemption.
+ * If there are no DL,RR/FIFO tasks, there must only be CFS or SCX tasks
+ * left. For CFS, if there's more than one we need the tick for
+ * involuntary preemption. For SCX, ask.
*/
- if (rq->nr_running > 1)
+ if (!scx_switched_all() && rq->nr_running > 1)
+ return false;
+
+ if (scx_enabled() && !scx_can_stop_tick(rq))
return false;
/*
diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c
index 2e652f7b8f54..ce32fc6b05cd 100644
--- a/kernel/sched/ext.c
+++ b/kernel/sched/ext.c
@@ -1086,7 +1086,8 @@ static void update_curr_scx(struct rq *rq)
account_group_exec_runtime(curr, delta_exec);
cgroup_account_cputime(curr, delta_exec);
- curr->scx.slice -= min(curr->scx.slice, delta_exec);
+ if (curr->scx.slice != SCX_SLICE_INF)
+ curr->scx.slice -= min(curr->scx.slice, delta_exec);
}
static void dsq_mod_nr(struct scx_dispatch_q *dsq, s32 delta)
@@ -2093,6 +2094,28 @@ static void set_next_task_scx(struct rq *rq, struct task_struct *p, bool first)
SCX_CALL_OP(SCX_KF_REST, running, p);
clr_task_runnable(p, true);
+
+ /*
+ * @p is getting newly scheduled or got kicked after someone updated its
+ * slice. Refresh whether tick can be stopped. See scx_can_stop_tick().
+ */
+ if ((p->scx.slice == SCX_SLICE_INF) !=
+ (bool)(rq->scx.flags & SCX_RQ_CAN_STOP_TICK)) {
+ if (p->scx.slice == SCX_SLICE_INF)
+ rq->scx.flags |= SCX_RQ_CAN_STOP_TICK;
+ else
+ rq->scx.flags &= ~SCX_RQ_CAN_STOP_TICK;
+
+ sched_update_tick_dependency(rq);
+
+ /*
+ * For now, let's refresh the load_avgs just when transitioning
+ * in and out of nohz. In the future, we might want to add a
+ * mechanism which calls the following periodically on
+ * tick-stopped CPUs.
+ */
+ update_other_load_avgs(rq);
+ }
}
static void put_prev_task_scx(struct rq *rq, struct task_struct *p)
@@ -2818,6 +2841,26 @@ int scx_check_setscheduler(struct task_struct *p, int policy)
return 0;
}
+#ifdef CONFIG_NO_HZ_FULL
+bool scx_can_stop_tick(struct rq *rq)
+{
+ struct task_struct *p = rq->curr;
+
+ if (scx_ops_bypassing())
+ return false;
+
+ if (p->sched_class != &ext_sched_class)
+ return true;
+
+ /*
+ * @rq can dispatch from different DSQs, so we can't tell whether it
+ * needs the tick or not by looking at nr_running. Allow stopping ticks
+ * iff the BPF scheduler indicated so. See set_next_task_scx().
+ */
+ return rq->scx.flags & SCX_RQ_CAN_STOP_TICK;
+}
+#endif
+
/*
* Omitted operations:
*
@@ -3120,6 +3163,9 @@ static void scx_ops_bypass(bool bypass)
}
rq_unlock_irqrestore(rq, &rf);
+
+ /* kick to restore ticks */
+ resched_cpu(cpu);
}
}
@@ -4576,7 +4622,9 @@ __bpf_kfunc_start_defs();
* BPF locks (in the future when BPF introduces more flexible locking).
*
* @p is allowed to run for @slice. The scheduling path is triggered on slice
- * exhaustion. If zero, the current residual slice is maintained.
+ * exhaustion. If zero, the current residual slice is maintained. If
+ * %SCX_SLICE_INF, @p never expires and the BPF scheduler must kick the CPU with
+ * scx_bpf_kick_cpu() to trigger scheduling.
*/
__bpf_kfunc void scx_bpf_dispatch(struct task_struct *p, u64 dsq_id, u64 slice,
u64 enq_flags)
diff --git a/kernel/sched/ext.h b/kernel/sched/ext.h
index 33a9f7fe5832..6ed946f72489 100644
--- a/kernel/sched/ext.h
+++ b/kernel/sched/ext.h
@@ -35,6 +35,7 @@ void scx_pre_fork(struct task_struct *p);
int scx_fork(struct task_struct *p);
void scx_post_fork(struct task_struct *p);
void scx_cancel_fork(struct task_struct *p);
+bool scx_can_stop_tick(struct rq *rq);
int scx_check_setscheduler(struct task_struct *p, int policy);
bool task_should_scx(struct task_struct *p);
void init_sched_ext_class(void);
@@ -73,6 +74,7 @@ static inline void scx_pre_fork(struct task_struct *p) {}
static inline int scx_fork(struct task_struct *p) { return 0; }
static inline void scx_post_fork(struct task_struct *p) {}
static inline void scx_cancel_fork(struct task_struct *p) {}
+static inline bool scx_can_stop_tick(struct rq *rq) { return true; }
static inline int scx_check_setscheduler(struct task_struct *p, int policy) { return 0; }
static inline bool task_on_scx(const struct task_struct *p) { return false; }
static inline void init_sched_ext_class(void) {}
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index d9054eb4ba82..b3c578cb43cd 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -727,6 +727,7 @@ struct cfs_rq {
/* scx_rq->flags, protected by the rq lock */
enum scx_rq_flags {
SCX_RQ_BALANCING = 1 << 1,
+ SCX_RQ_CAN_STOP_TICK = 1 << 2,
};
struct scx_rq {
diff --git a/tools/sched_ext/scx_central.bpf.c b/tools/sched_ext/scx_central.bpf.c
index 428b2262faa3..1d8fd570eaa7 100644
--- a/tools/sched_ext/scx_central.bpf.c
+++ b/tools/sched_ext/scx_central.bpf.c
@@ -13,7 +13,26 @@
* through per-CPU BPF queues. The current design is chosen to maximally
* utilize and verify various SCX mechanisms such as LOCAL_ON dispatching.
*
- * b. Preemption
+ * b. Tickless operation
+ *
+ * All tasks are dispatched with the infinite slice which allows stopping the
+ * ticks on CONFIG_NO_HZ_FULL kernels running with the proper nohz_full
+ * parameter. The tickless operation can be observed through
+ * /proc/interrupts.
+ *
+ * Periodic switching is enforced by a periodic timer checking all CPUs and
+ * preempting them as necessary. Unfortunately, BPF timer currently doesn't
+ * have a way to pin to a specific CPU, so the periodic timer isn't pinned to
+ * the central CPU.
+ *
+ * c. Preemption
+ *
+ * Kthreads are unconditionally queued to the head of a matching local dsq
+ * and dispatched with SCX_DSQ_PREEMPT. This ensures that a kthread is always
+ * prioritized over user threads, which is required for ensuring forward
+ * progress as e.g. the periodic timer may run on a ksoftirqd and if the
+ * ksoftirqd gets starved by a user thread, there may not be anything else to
+ * vacate that user thread.
*
* SCX_KICK_PREEMPT is used to trigger scheduling and CPUs to move to the
* next tasks.
@@ -32,14 +51,17 @@ char _license[] SEC("license") = "GPL";
enum {
FALLBACK_DSQ_ID = 0,
+ MS_TO_NS = 1000LLU * 1000,
+ TIMER_INTERVAL_NS = 1 * MS_TO_NS,
};
const volatile s32 central_cpu;
const volatile u32 nr_cpu_ids = 1; /* !0 for veristat, set during init */
const volatile u64 slice_ns = SCX_SLICE_DFL;
+bool timer_pinned = true;
u64 nr_total, nr_locals, nr_queued, nr_lost_pids;
-u64 nr_dispatches, nr_mismatches, nr_retries;
+u64 nr_timers, nr_dispatches, nr_mismatches, nr_retries;
u64 nr_overflows;
UEI_DEFINE(uei);
@@ -52,6 +74,23 @@ struct {
/* can't use percpu map due to bad lookups */
bool RESIZABLE_ARRAY(data, cpu_gimme_task);
+u64 RESIZABLE_ARRAY(data, cpu_started_at);
+
+struct central_timer {
+ struct bpf_timer timer;
+};
+
+struct {
+ __uint(type, BPF_MAP_TYPE_ARRAY);
+ __uint(max_entries, 1);
+ __type(key, u32);
+ __type(value, struct central_timer);
+} central_timer SEC(".maps");
+
+static bool vtime_before(u64 a, u64 b)
+{
+ return (s64)(a - b) < 0;
+}
s32 BPF_STRUCT_OPS(central_select_cpu, struct task_struct *p,
s32 prev_cpu, u64 wake_flags)
@@ -71,9 +110,22 @@ void BPF_STRUCT_OPS(central_enqueue, struct task_struct *p, u64 enq_flags)
__sync_fetch_and_add(&nr_total, 1);
+ /*
+ * Push per-cpu kthreads at the head of local dsq's and preempt the
+ * corresponding CPU. This ensures that e.g. ksoftirqd isn't blocked
+ * behind other threads which is necessary for forward progress
+ * guarantee as we depend on the BPF timer which may run from ksoftirqd.
+ */
+ if ((p->flags & PF_KTHREAD) && p->nr_cpus_allowed == 1) {
+ __sync_fetch_and_add(&nr_locals, 1);
+ scx_bpf_dispatch(p, SCX_DSQ_LOCAL, SCX_SLICE_INF,
+ enq_flags | SCX_ENQ_PREEMPT);
+ return;
+ }
+
if (bpf_map_push_elem(¢ral_q, &pid, 0)) {
__sync_fetch_and_add(&nr_overflows, 1);
- scx_bpf_dispatch(p, FALLBACK_DSQ_ID, SCX_SLICE_DFL, enq_flags);
+ scx_bpf_dispatch(p, FALLBACK_DSQ_ID, SCX_SLICE_INF, enq_flags);
return;
}
@@ -106,7 +158,7 @@ static bool dispatch_to_cpu(s32 cpu)
*/
if (!bpf_cpumask_test_cpu(cpu, p->cpus_ptr)) {
__sync_fetch_and_add(&nr_mismatches, 1);
- scx_bpf_dispatch(p, FALLBACK_DSQ_ID, SCX_SLICE_DFL, 0);
+ scx_bpf_dispatch(p, FALLBACK_DSQ_ID, SCX_SLICE_INF, 0);
bpf_task_release(p);
/*
* We might run out of dispatch buffer slots if we continue dispatching
@@ -120,7 +172,7 @@ static bool dispatch_to_cpu(s32 cpu)
}
/* dispatch to local and mark that @cpu doesn't need more */
- scx_bpf_dispatch(p, SCX_DSQ_LOCAL_ON | cpu, SCX_SLICE_DFL, 0);
+ scx_bpf_dispatch(p, SCX_DSQ_LOCAL_ON | cpu, SCX_SLICE_INF, 0);
if (cpu != central_cpu)
scx_bpf_kick_cpu(cpu, SCX_KICK_IDLE);
@@ -188,9 +240,102 @@ void BPF_STRUCT_OPS(central_dispatch, s32 cpu, struct task_struct *prev)
}
}
+void BPF_STRUCT_OPS(central_running, struct task_struct *p)
+{
+ s32 cpu = scx_bpf_task_cpu(p);
+ u64 *started_at = ARRAY_ELEM_PTR(cpu_started_at, cpu, nr_cpu_ids);
+ if (started_at)
+ *started_at = bpf_ktime_get_ns() ?: 1; /* 0 indicates idle */
+}
+
+void BPF_STRUCT_OPS(central_stopping, struct task_struct *p, bool runnable)
+{
+ s32 cpu = scx_bpf_task_cpu(p);
+ u64 *started_at = ARRAY_ELEM_PTR(cpu_started_at, cpu, nr_cpu_ids);
+ if (started_at)
+ *started_at = 0;
+}
+
+static int central_timerfn(void *map, int *key, struct bpf_timer *timer)
+{
+ u64 now = bpf_ktime_get_ns();
+ u64 nr_to_kick = nr_queued;
+ s32 i, curr_cpu;
+
+ curr_cpu = bpf_get_smp_processor_id();
+ if (timer_pinned && (curr_cpu != central_cpu)) {
+ scx_bpf_error("Central timer ran on CPU %d, not central CPU %d",
+ curr_cpu, central_cpu);
+ return 0;
+ }
+
+ bpf_for(i, 0, nr_cpu_ids) {
+ s32 cpu = (nr_timers + i) % nr_cpu_ids;
+ u64 *started_at;
+
+ if (cpu == central_cpu)
+ continue;
+
+ /* kick iff the current one exhausted its slice */
+ started_at = ARRAY_ELEM_PTR(cpu_started_at, cpu, nr_cpu_ids);
+ if (started_at && *started_at &&
+ vtime_before(now, *started_at + slice_ns))
+ continue;
+
+ /* and there's something pending */
+ if (scx_bpf_dsq_nr_queued(FALLBACK_DSQ_ID) ||
+ scx_bpf_dsq_nr_queued(SCX_DSQ_LOCAL_ON | cpu))
+ ;
+ else if (nr_to_kick)
+ nr_to_kick--;
+ else
+ continue;
+
+ scx_bpf_kick_cpu(cpu, SCX_KICK_PREEMPT);
+ }
+
+ bpf_timer_start(timer, TIMER_INTERVAL_NS, BPF_F_TIMER_CPU_PIN);
+ __sync_fetch_and_add(&nr_timers, 1);
+ return 0;
+}
+
int BPF_STRUCT_OPS_SLEEPABLE(central_init)
{
- return scx_bpf_create_dsq(FALLBACK_DSQ_ID, -1);
+ u32 key = 0;
+ struct bpf_timer *timer;
+ int ret;
+
+ ret = scx_bpf_create_dsq(FALLBACK_DSQ_ID, -1);
+ if (ret)
+ return ret;
+
+ timer = bpf_map_lookup_elem(¢ral_timer, &key);
+ if (!timer)
+ return -ESRCH;
+
+ if (bpf_get_smp_processor_id() != central_cpu) {
+ scx_bpf_error("init from non-central CPU");
+ return -EINVAL;
+ }
+
+ bpf_timer_init(timer, ¢ral_timer, CLOCK_MONOTONIC);
+ bpf_timer_set_callback(timer, central_timerfn);
+
+ ret = bpf_timer_start(timer, TIMER_INTERVAL_NS, BPF_F_TIMER_CPU_PIN);
+ /*
+ * BPF_F_TIMER_CPU_PIN is pretty new (>=6.7). If we're running in a
+ * kernel which doesn't have it, bpf_timer_start() will return -EINVAL.
+ * Retry without the PIN. This would be the perfect use case for
+ * bpf_core_enum_value_exists() but the enum type doesn't have a name
+ * and can't be used with bpf_core_enum_value_exists(). Oh well...
+ */
+ if (ret == -EINVAL) {
+ timer_pinned = false;
+ ret = bpf_timer_start(timer, TIMER_INTERVAL_NS, 0);
+ }
+ if (ret)
+ scx_bpf_error("bpf_timer_start failed (%d)", ret);
+ return ret;
}
void BPF_STRUCT_OPS(central_exit, struct scx_exit_info *ei)
@@ -209,6 +354,8 @@ SCX_OPS_DEFINE(central_ops,
.select_cpu = (void *)central_select_cpu,
.enqueue = (void *)central_enqueue,
.dispatch = (void *)central_dispatch,
+ .running = (void *)central_running,
+ .stopping = (void *)central_stopping,
.init = (void *)central_init,
.exit = (void *)central_exit,
.name = "central");
diff --git a/tools/sched_ext/scx_central.c b/tools/sched_ext/scx_central.c
index 5f09fc666a63..fb3f50886552 100644
--- a/tools/sched_ext/scx_central.c
+++ b/tools/sched_ext/scx_central.c
@@ -48,6 +48,7 @@ int main(int argc, char **argv)
struct bpf_link *link;
__u64 seq = 0;
__s32 opt;
+ cpu_set_t *cpuset;
libbpf_set_print(libbpf_print_fn);
signal(SIGINT, sigint_handler);
@@ -77,10 +78,35 @@ int main(int argc, char **argv)
/* Resize arrays so their element count is equal to cpu count. */
RESIZE_ARRAY(skel, data, cpu_gimme_task, skel->rodata->nr_cpu_ids);
+ RESIZE_ARRAY(skel, data, cpu_started_at, skel->rodata->nr_cpu_ids);
SCX_OPS_LOAD(skel, central_ops, scx_central, uei);
+
+ /*
+ * Affinitize the loading thread to the central CPU, as:
+ * - That's where the BPF timer is first invoked in the BPF program.
+ * - We probably don't want this user space component to take up a core
+ * from a task that would benefit from avoiding preemption on one of
+ * the tickless cores.
+ *
+ * Until BPF supports pinning the timer, it's not guaranteed that it
+ * will always be invoked on the central CPU. In practice, this
+ * suffices the majority of the time.
+ */
+ cpuset = CPU_ALLOC(skel->rodata->nr_cpu_ids);
+ SCX_BUG_ON(!cpuset, "Failed to allocate cpuset");
+ CPU_ZERO(cpuset);
+ CPU_SET(skel->rodata->central_cpu, cpuset);
+ SCX_BUG_ON(sched_setaffinity(0, sizeof(cpuset), cpuset),
+ "Failed to affinitize to central CPU %d (max %d)",
+ skel->rodata->central_cpu, skel->rodata->nr_cpu_ids - 1);
+ CPU_FREE(cpuset);
+
link = SCX_OPS_ATTACH(skel, central_ops, scx_central);
+ if (!skel->data->timer_pinned)
+ printf("WARNING : BPF_F_TIMER_CPU_PIN not available, timer not pinned to central\n");
+
while (!exit_req && !UEI_EXITED(skel, uei)) {
printf("[SEQ %llu]\n", seq++);
printf("total :%10" PRIu64 " local:%10" PRIu64 " queued:%10" PRIu64 " lost:%10" PRIu64 "\n",
@@ -88,7 +114,8 @@ int main(int argc, char **argv)
skel->bss->nr_locals,
skel->bss->nr_queued,
skel->bss->nr_lost_pids);
- printf(" dispatch:%10" PRIu64 " mismatch:%10" PRIu64 " retry:%10" PRIu64 "\n",
+ printf("timer :%10" PRIu64 " dispatch:%10" PRIu64 " mismatch:%10" PRIu64 " retry:%10" PRIu64 "\n",
+ skel->bss->nr_timers,
skel->bss->nr_dispatches,
skel->bss->nr_mismatches,
skel->bss->nr_retries);
--
2.45.2
Diff
---
include/linux/sched/ext.h | 1 +
kernel/sched/core.c | 11 ++-
kernel/sched/ext.c | 52 +++++++++-
kernel/sched/ext.h | 2 +
kernel/sched/sched.h | 1 +
tools/sched_ext/scx_central.bpf.c | 159 ++++++++++++++++++++++++++++--
tools/sched_ext/scx_central.c | 29 +++++-
7 files changed, 242 insertions(+), 13 deletions(-)
diff --git a/include/linux/sched/ext.h b/include/linux/sched/ext.h
index 3b2809b980ac..6f1a4977e9f8 100644
--- a/include/linux/sched/ext.h
+++ b/include/linux/sched/ext.h
@@ -16,6 +16,7 @@ enum scx_public_consts {
SCX_OPS_NAME_LEN = 128,
SCX_SLICE_DFL = 20 * 1000000, /* 20ms */
+ SCX_SLICE_INF = U64_MAX, /* infinite, implies nohz */
};
/*
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 1a3144c80af8..d5eff4036be7 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -1256,11 +1256,14 @@ bool sched_can_stop_tick(struct rq *rq)
return true;
/*
- * If there are no DL,RR/FIFO tasks, there must only be CFS tasks left;
- * if there's more than one we need the tick for involuntary
- * preemption.
+ * If there are no DL,RR/FIFO tasks, there must only be CFS or sched_ext tasks
+ * left. For CFS, if there's more than one we need the tick for
+ * involuntary preemption. For sched_ext, ask.
*/
- if (rq->nr_running > 1)
+ if (!scx_switched_all() && rq->nr_running > 1)
+ return false;
+
+ if (scx_enabled() && !scx_can_stop_tick(rq))
return false;
/*
diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c
index 2e652f7b8f54..ce32fc6b05cd 100644
--- a/kernel/sched/ext.c
+++ b/kernel/sched/ext.c
@@ -1086,7 +1086,8 @@ static void update_curr_scx(struct rq *rq)
account_group_exec_runtime(curr, delta_exec);
cgroup_account_cputime(curr, delta_exec);
- curr->scx.slice -= min(curr->scx.slice, delta_exec);
+ if (curr->scx.slice != SCX_SLICE_INF)
+ curr->scx.slice -= min(curr->scx.slice, delta_exec);
}
static void dsq_mod_nr(struct scx_dispatch_q *dsq, s32 delta)
@@ -2093,6 +2094,28 @@ static void set_next_task_scx(struct rq *rq, struct task_struct *p, bool first)
SCX_CALL_OP(SCX_KF_REST, running, p);
clr_task_runnable(p, true);
+
+ /*
+ * @p is getting newly scheduled or got kicked after someone updated its
+ * slice. Refresh whether tick can be stopped. See scx_can_stop_tick().
+ */
+ if ((p->scx.slice == SCX_SLICE_INF) !=
+ (bool)(rq->scx.flags & SCX_RQ_CAN_STOP_TICK)) {
+ if (p->scx.slice == SCX_SLICE_INF)
+ rq->scx.flags |= SCX_RQ_CAN_STOP_TICK;
+ else
+ rq->scx.flags &= ~SCX_RQ_CAN_STOP_TICK;
+
+ sched_update_tick_dependency(rq);
+
+ /*
+ * For now, let's refresh the load_avgs just when transitioning
+ * in and out of nohz. In the future, we might want to add a
+ * mechanism which calls the following periodically on
+ * tick-stopped CPUs.
+ */
+ update_other_load_avgs(rq);
+ }
}
static void put_prev_task_scx(struct rq *rq, struct task_struct *p)
@@ -2818,6 +2841,26 @@ int scx_check_setscheduler(struct task_struct *p, int policy)
return 0;
}
+#ifdef CONFIG_NO_HZ_FULL
+bool scx_can_stop_tick(struct rq *rq)
+{
+ struct task_struct *p = rq->curr;
+
+ if (scx_ops_bypassing())
+ return false;
+
+ if (p->sched_class != &ext_sched_class)
+ return true;
+
+ /*
+ * @rq can dispatch from different DSQs, so we can't tell whether it
+ * needs the tick or not by looking at nr_running. Allow stopping ticks
+ * iff the BPF scheduler indicated so. See set_next_task_scx().
+ */
+ return rq->scx.flags & SCX_RQ_CAN_STOP_TICK;
+}
+#endif
+
/*
* Omitted operations:
*
@@ -3120,6 +3163,9 @@ static void scx_ops_bypass(bool bypass)
}
rq_unlock_irqrestore(rq, &rf);
+
+ /* kick to restore ticks */
+ resched_cpu(cpu);
}
}
@@ -4576,7 +4622,9 @@ __bpf_kfunc_start_defs();
* BPF locks (in the future when BPF introduces more flexible locking).
*
* @p is allowed to run for @slice. The scheduling path is triggered on slice
- * exhaustion. If zero, the current residual slice is maintained.
+ * exhaustion. If zero, the current residual slice is maintained. If
+ * %SCX_SLICE_INF, @p never expires and the BPF scheduler must kick the CPU with
+ * scx_bpf_kick_cpu() to trigger scheduling.
*/
__bpf_kfunc void scx_bpf_dispatch(struct task_struct *p, u64 dsq_id, u64 slice,
u64 enq_flags)
diff --git a/kernel/sched/ext.h b/kernel/sched/ext.h
index 33a9f7fe5832..6ed946f72489 100644
--- a/kernel/sched/ext.h
+++ b/kernel/sched/ext.h
@@ -35,6 +35,7 @@ void scx_pre_fork(struct task_struct *p);
int scx_fork(struct task_struct *p);
void scx_post_fork(struct task_struct *p);
void scx_cancel_fork(struct task_struct *p);
+bool scx_can_stop_tick(struct rq *rq);
int scx_check_setscheduler(struct task_struct *p, int policy);
bool task_should_scx(struct task_struct *p);
void init_sched_ext_class(void);
@@ -73,6 +74,7 @@ static inline void scx_pre_fork(struct task_struct *p) {}
static inline int scx_fork(struct task_struct *p) { return 0; }
static inline void scx_post_fork(struct task_struct *p) {}
static inline void scx_cancel_fork(struct task_struct *p) {}
+static inline bool scx_can_stop_tick(struct rq *rq) { return true; }
static inline int scx_check_setscheduler(struct task_struct *p, int policy) { return 0; }
static inline bool task_on_scx(const struct task_struct *p) { return false; }
static inline void init_sched_ext_class(void) {}
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index d9054eb4ba82..b3c578cb43cd 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -727,6 +727,7 @@ struct cfs_rq {
/* scx_rq->flags, protected by the rq lock */
enum scx_rq_flags {
SCX_RQ_BALANCING = 1 << 1,
+ SCX_RQ_CAN_STOP_TICK = 1 << 2,
};
struct scx_rq {
diff --git a/tools/sched_ext/scx_central.bpf.c b/tools/sched_ext/scx_central.bpf.c
index 428b2262faa3..1d8fd570eaa7 100644
--- a/tools/sched_ext/scx_central.bpf.c
+++ b/tools/sched_ext/scx_central.bpf.c
@@ -13,7 +13,26 @@
* through per-CPU BPF queues. The current design is chosen to maximally
* utilize and verify various sched_ext mechanisms such as LOCAL_ON dispatching.
*
- * b. Preemption
+ * b. Tickless operation
+ *
+ * All tasks are dispatched with the infinite slice which allows stopping the
+ * ticks on CONFIG_NO_HZ_FULL kernels running with the proper nohz_full
+ * parameter. The tickless operation can be observed through
+ * /proc/interrupts.
+ *
+ * Periodic switching is enforced by a periodic timer checking all CPUs and
+ * preempting them as necessary. Unfortunately, BPF timer currently doesn't
+ * have a way to pin to a specific CPU, so the periodic timer isn't pinned to
+ * the central CPU.
+ *
+ * c. Preemption
+ *
+ * Kthreads are unconditionally queued to the head of a matching local dsq
+ * and dispatched with SCX_DSQ_PREEMPT. This ensures that a kthread is always
+ * prioritized over user threads, which is required for ensuring forward
+ * progress as e.g. the periodic timer may run on a ksoftirqd and if the
+ * ksoftirqd gets starved by a user thread, there may not be anything else to
+ * vacate that user thread.
*
* SCX_KICK_PREEMPT is used to trigger scheduling and CPUs to move to the
* next tasks.
@@ -32,14 +51,17 @@ char _license[] SEC("license") = "GPL";
enum {
FALLBACK_DSQ_ID = 0,
+ MS_TO_NS = 1000LLU * 1000,
+ TIMER_INTERVAL_NS = 1 * MS_TO_NS,
};
const volatile s32 central_cpu;
const volatile u32 nr_cpu_ids = 1; /* !0 for veristat, set during init */
const volatile u64 slice_ns = SCX_SLICE_DFL;
+bool timer_pinned = true;
u64 nr_total, nr_locals, nr_queued, nr_lost_pids;
-u64 nr_dispatches, nr_mismatches, nr_retries;
+u64 nr_timers, nr_dispatches, nr_mismatches, nr_retries;
u64 nr_overflows;
UEI_DEFINE(uei);
@@ -52,6 +74,23 @@ struct {
/* can't use percpu map due to bad lookups */
bool RESIZABLE_ARRAY(data, cpu_gimme_task);
+u64 RESIZABLE_ARRAY(data, cpu_started_at);
+
+struct central_timer {
+ struct bpf_timer timer;
+};
+
+struct {
+ __uint(type, BPF_MAP_TYPE_ARRAY);
+ __uint(max_entries, 1);
+ __type(key, u32);
+ __type(value, struct central_timer);
+} central_timer SEC(".maps");
+
+static bool vtime_before(u64 a, u64 b)
+{
+ return (s64)(a - b) < 0;
+}
s32 BPF_STRUCT_OPS(central_select_cpu, struct task_struct *p,
s32 prev_cpu, u64 wake_flags)
@@ -71,9 +110,22 @@ void BPF_STRUCT_OPS(central_enqueue, struct task_struct *p, u64 enq_flags)
__sync_fetch_and_add(&nr_total, 1);
+ /*
+ * Push per-cpu kthreads at the head of local dsq's and preempt the
+ * corresponding CPU. This ensures that e.g. ksoftirqd isn't blocked
+ * behind other threads which is necessary for forward progress
+ * guarantee as we depend on the BPF timer which may run from ksoftirqd.
+ */
+ if ((p->flags & PF_KTHREAD) && p->nr_cpus_allowed == 1) {
+ __sync_fetch_and_add(&nr_locals, 1);
+ scx_bpf_dispatch(p, SCX_DSQ_LOCAL, SCX_SLICE_INF,
+ enq_flags | SCX_ENQ_PREEMPT);
+ return;
+ }
+
if (bpf_map_push_elem(¢ral_q, &pid, 0)) {
__sync_fetch_and_add(&nr_overflows, 1);
- scx_bpf_dispatch(p, FALLBACK_DSQ_ID, SCX_SLICE_DFL, enq_flags);
+ scx_bpf_dispatch(p, FALLBACK_DSQ_ID, SCX_SLICE_INF, enq_flags);
return;
}
@@ -106,7 +158,7 @@ static bool dispatch_to_cpu(s32 cpu)
*/
if (!bpf_cpumask_test_cpu(cpu, p->cpus_ptr)) {
__sync_fetch_and_add(&nr_mismatches, 1);
- scx_bpf_dispatch(p, FALLBACK_DSQ_ID, SCX_SLICE_DFL, 0);
+ scx_bpf_dispatch(p, FALLBACK_DSQ_ID, SCX_SLICE_INF, 0);
bpf_task_release(p);
/*
* We might run out of dispatch buffer slots if we continue dispatching
@@ -120,7 +172,7 @@ static bool dispatch_to_cpu(s32 cpu)
}
/* dispatch to local and mark that @cpu doesn't need more */
- scx_bpf_dispatch(p, SCX_DSQ_LOCAL_ON | cpu, SCX_SLICE_DFL, 0);
+ scx_bpf_dispatch(p, SCX_DSQ_LOCAL_ON | cpu, SCX_SLICE_INF, 0);
if (cpu != central_cpu)
scx_bpf_kick_cpu(cpu, SCX_KICK_IDLE);
@@ -188,9 +240,102 @@ void BPF_STRUCT_OPS(central_dispatch, s32 cpu, struct task_struct *prev)
}
}
+void BPF_STRUCT_OPS(central_running, struct task_struct *p)
+{
+ s32 cpu = scx_bpf_task_cpu(p);
+ u64 *started_at = ARRAY_ELEM_PTR(cpu_started_at, cpu, nr_cpu_ids);
+ if (started_at)
+ *started_at = bpf_ktime_get_ns() ?: 1; /* 0 indicates idle */
+}
+
+void BPF_STRUCT_OPS(central_stopping, struct task_struct *p, bool runnable)
+{
+ s32 cpu = scx_bpf_task_cpu(p);
+ u64 *started_at = ARRAY_ELEM_PTR(cpu_started_at, cpu, nr_cpu_ids);
+ if (started_at)
+ *started_at = 0;
+}
+
+static int central_timerfn(void *map, int *key, struct bpf_timer *timer)
+{
+ u64 now = bpf_ktime_get_ns();
+ u64 nr_to_kick = nr_queued;
+ s32 i, curr_cpu;
+
+ curr_cpu = bpf_get_smp_processor_id();
+ if (timer_pinned && (curr_cpu != central_cpu)) {
+ scx_bpf_error("Central timer ran on CPU %d, not central CPU %d",
+ curr_cpu, central_cpu);
+ return 0;
+ }
+
+ bpf_for(i, 0, nr_cpu_ids) {
+ s32 cpu = (nr_timers + i) % nr_cpu_ids;
+ u64 *started_at;
+
+ if (cpu == central_cpu)
+ continue;
+
+ /* kick iff the current one exhausted its slice */
+ started_at = ARRAY_ELEM_PTR(cpu_started_at, cpu, nr_cpu_ids);
+ if (started_at && *started_at &&
+ vtime_before(now, *started_at + slice_ns))
+ continue;
+
+ /* and there's something pending */
+ if (scx_bpf_dsq_nr_queued(FALLBACK_DSQ_ID) ||
+ scx_bpf_dsq_nr_queued(SCX_DSQ_LOCAL_ON | cpu))
+ ;
+ else if (nr_to_kick)
+ nr_to_kick--;
+ else
+ continue;
+
+ scx_bpf_kick_cpu(cpu, SCX_KICK_PREEMPT);
+ }
+
+ bpf_timer_start(timer, TIMER_INTERVAL_NS, BPF_F_TIMER_CPU_PIN);
+ __sync_fetch_and_add(&nr_timers, 1);
+ return 0;
+}
+
int BPF_STRUCT_OPS_SLEEPABLE(central_init)
{
- return scx_bpf_create_dsq(FALLBACK_DSQ_ID, -1);
+ u32 key = 0;
+ struct bpf_timer *timer;
+ int ret;
+
+ ret = scx_bpf_create_dsq(FALLBACK_DSQ_ID, -1);
+ if (ret)
+ return ret;
+
+ timer = bpf_map_lookup_elem(¢ral_timer, &key);
+ if (!timer)
+ return -ESRCH;
+
+ if (bpf_get_smp_processor_id() != central_cpu) {
+ scx_bpf_error("init from non-central CPU");
+ return -EINVAL;
+ }
+
+ bpf_timer_init(timer, ¢ral_timer, CLOCK_MONOTONIC);
+ bpf_timer_set_callback(timer, central_timerfn);
+
+ ret = bpf_timer_start(timer, TIMER_INTERVAL_NS, BPF_F_TIMER_CPU_PIN);
+ /*
+ * BPF_F_TIMER_CPU_PIN is pretty new (>=6.7). If we're running in a
+ * kernel which doesn't have it, bpf_timer_start() will return -EINVAL.
+ * Retry without the PIN. This would be the perfect use case for
+ * bpf_core_enum_value_exists() but the enum type doesn't have a name
+ * and can't be used with bpf_core_enum_value_exists(). Oh well...
+ */
+ if (ret == -EINVAL) {
+ timer_pinned = false;
+ ret = bpf_timer_start(timer, TIMER_INTERVAL_NS, 0);
+ }
+ if (ret)
+ scx_bpf_error("bpf_timer_start failed (%d)", ret);
+ return ret;
}
void BPF_STRUCT_OPS(central_exit, struct scx_exit_info *ei)
@@ -209,6 +354,8 @@ SCX_OPS_DEFINE(central_ops,
.select_cpu = (void *)central_select_cpu,
.enqueue = (void *)central_enqueue,
.dispatch = (void *)central_dispatch,
+ .running = (void *)central_running,
+ .stopping = (void *)central_stopping,
.init = (void *)central_init,
.exit = (void *)central_exit,
.name = "central");
diff --git a/tools/sched_ext/scx_central.c b/tools/sched_ext/scx_central.c
index 5f09fc666a63..fb3f50886552 100644
--- a/tools/sched_ext/scx_central.c
+++ b/tools/sched_ext/scx_central.c
@@ -48,6 +48,7 @@ int main(int argc, char **argv)
struct bpf_link *link;
__u64 seq = 0;
__s32 opt;
+ cpu_set_t *cpuset;
libbpf_set_print(libbpf_print_fn);
signal(SIGINT, sigint_handler);
@@ -77,10 +78,35 @@ int main(int argc, char **argv)
/* Resize arrays so their element count is equal to cpu count. */
RESIZE_ARRAY(skel, data, cpu_gimme_task, skel->rodata->nr_cpu_ids);
+ RESIZE_ARRAY(skel, data, cpu_started_at, skel->rodata->nr_cpu_ids);
SCX_OPS_LOAD(skel, central_ops, scx_central, uei);
+
+ /*
+ * Affinitize the loading thread to the central CPU, as:
+ * - That's where the BPF timer is first invoked in the BPF program.
+ * - We probably don't want this user space component to take up a core
+ * from a task that would benefit from avoiding preemption on one of
+ * the tickless cores.
+ *
+ * Until BPF supports pinning the timer, it's not guaranteed that it
+ * will always be invoked on the central CPU. In practice, this
+ * suffices the majority of the time.
+ */
+ cpuset = CPU_ALLOC(skel->rodata->nr_cpu_ids);
+ SCX_BUG_ON(!cpuset, "Failed to allocate cpuset");
+ CPU_ZERO(cpuset);
+ CPU_SET(skel->rodata->central_cpu, cpuset);
+ SCX_BUG_ON(sched_setaffinity(0, sizeof(cpuset), cpuset),
+ "Failed to affinitize to central CPU %d (max %d)",
+ skel->rodata->central_cpu, skel->rodata->nr_cpu_ids - 1);
+ CPU_FREE(cpuset);
+
link = SCX_OPS_ATTACH(skel, central_ops, scx_central);
+ if (!skel->data->timer_pinned)
+ printf("WARNING : BPF_F_TIMER_CPU_PIN not available, timer not pinned to central\n");
+
while (!exit_req && !UEI_EXITED(skel, uei)) {
printf("[SEQ %llu]\n", seq++);
printf("total :%10" PRIu64 " local:%10" PRIu64 " queued:%10" PRIu64 " lost:%10" PRIu64 "\n",
@@ -88,7 +114,8 @@ int main(int argc, char **argv)
skel->bss->nr_locals,
skel->bss->nr_queued,
skel->bss->nr_lost_pids);
- printf(" dispatch:%10" PRIu64 " mismatch:%10" PRIu64 " retry:%10" PRIu64 "\n",
+ printf("timer :%10" PRIu64 " dispatch:%10" PRIu64 " mismatch:%10" PRIu64 " retry:%10" PRIu64 "\n",
+ skel->bss->nr_timers,
skel->bss->nr_dispatches,
skel->bss->nr_mismatches,
skel->bss->nr_retries);
--
2.45.2
Implementation Analysis
Overview
Linux's CONFIG_NO_HZ_FULL (nohz_full) feature allows CPUs to run without periodic timer ticks, reducing latency and overhead for real-time and high-performance workloads. Before this patch, sched_ext tasks always required ticks (for slice accounting and involuntary preemption). This patch integrates sched_ext with nohz: a BPF scheduler can signal that a task should run without ticks by setting p->scx.slice = SCX_SLICE_INF (U64_MAX). When a CPU's current task has an infinite slice, the CPU can enter tickless mode. The scx_central example is updated to demonstrate full tickless operation using a BPF timer for preemption instead of the tick.
Code Walkthrough
include/linux/sched/ext.h — SCX_SLICE_INF
SCX_SLICE_INF = U64_MAX, /* infinite, implies nohz */
A new sentinel value for p->scx.slice. Setting this value tells sched_ext: "this task should run until the BPF scheduler explicitly kicks it off via scx_bpf_kick_cpu()." The kernel will not decrement the slice and will not trigger a scheduling event based on time expiration.
kernel/sched/sched.h — SCX_RQ_CAN_STOP_TICK
enum scx_rq_flags {
SCX_RQ_BALANCING = 1 << 1,
SCX_RQ_CAN_STOP_TICK = 1 << 2, // NEW
};
A per-rq flag set in set_next_task_scx() when the newly selected task has slice == SCX_SLICE_INF. Cleared when the slice is not infinite. This flag is the single source of truth that scx_can_stop_tick() checks.
kernel/sched/ext.c — update_curr_scx() guard
if (curr->scx.slice != SCX_SLICE_INF)
curr->scx.slice -= min(curr->scx.slice, delta_exec);
The slice accounting code is guarded by a SCX_SLICE_INF check. An infinite-slice task never has its slice decremented, so it will never naturally trigger a reschedule due to time expiration.
kernel/sched/ext.c — set_next_task_scx() tick dependency update
if ((p->scx.slice == SCX_SLICE_INF) !=
(bool)(rq->scx.flags & SCX_RQ_CAN_STOP_TICK)) {
if (p->scx.slice == SCX_SLICE_INF)
rq->scx.flags |= SCX_RQ_CAN_STOP_TICK;
else
rq->scx.flags &= ~SCX_RQ_CAN_STOP_TICK;
sched_update_tick_dependency(rq);
update_other_load_avgs(rq);
}
When a task starts running, the code checks if the tick-stop state has changed. If it has, sched_update_tick_dependency() notifies the nohz subsystem. update_other_load_avgs() refreshes load averages at the transition boundary, since tickless CPUs stop receiving the periodic load average updates that normally come from the tick.
kernel/sched/ext.c — scx_can_stop_tick() (CONFIG_NO_HZ_FULL)
#ifdef CONFIG_NO_HZ_FULL
bool scx_can_stop_tick(struct rq *rq)
{
struct task_struct *p = rq->curr;
if (scx_ops_bypassing()) return false;
if (p->sched_class != &ext_sched_class) return true;
return rq->scx.flags & SCX_RQ_CAN_STOP_TICK;
}
#endif
Called from sched_can_stop_tick() in kernel/sched/core.c. If the current task is not an SCX task, return true (defer to other scheduler classes). If it is an SCX task, return the SCX_RQ_CAN_STOP_TICK flag. During bypass, always return false.
kernel/sched/core.c — sched_can_stop_tick() modification
// Before:
if (rq->nr_running > 1) return false;
// After:
if (!scx_switched_all() && rq->nr_running > 1) return false;
if (scx_enabled() && !scx_can_stop_tick(rq)) return false;
The "more than one runnable task → need tick for preemption" check is relaxed for scx_switched_all() mode: when all tasks use SCHED_EXT, the BPF scheduler handles preemption and does not need the tick for involuntary preemption.
scx_ops_bypass() — kick to restore ticks
rq_unlock_irqrestore(rq, &rf);
resched_cpu(cpu); // NEW: kick to restore ticks
When bypass mode is exited, each CPU is kicked to force a scheduling cycle that re-evaluates tick dependency, restoring ticks where needed.
scx_central.bpf.c — tickless central scheduler
The example scheduler is substantially extended:
- All dispatches use
SCX_SLICE_INFinstead ofSCX_SLICE_DFL. - A BPF timer (
central_timerfn) fires every 1ms, checks which worker CPUs have exceededslice_ns, and kicks them withSCX_KICK_PREEMPT. ops.running()recordscpu_started_at[cpu] = bpf_ktime_get_ns()when a task starts.ops.stopping()clearscpu_started_at[cpu] = 0.- Per-CPU kthreads are dispatched with
SCX_ENQ_PREEMPTto prevent ksoftirqd starvation (which would break the BPF timer). - The userspace loader affinitizes itself to
central_cpuso the initial timer fires on the correct CPU.
The observed result: worker CPUs accumulate ~7 local timer interrupts over 10 seconds with tickless vs. ~2200 without.
Key Concepts
SCX_SLICE_INF= U64_MAX: The BPF scheduler's signal for "this task runs until I kick it." The kernel will not decrement the slice. The BPF scheduler takes responsibility for preemption viascx_bpf_kick_cpu()or BPF timers.SCX_RQ_CAN_STOP_TICK: A per-rq flag caching whether the current task hasSCX_SLICE_INF. Updated inset_next_task_scx()underrq->lock. Read without locking inscx_can_stop_tick()— treated as advisory.sched_update_tick_dependency(rq): The kernel function that tells the nohz subsystem whether a CPU can stop its tick. Must be called whenSCX_RQ_CAN_STOP_TICKchanges. Requiresrq->lockto be held.- BPF timer for preemption: When ticks are stopped, the kernel's tick-based involuntary preemption is gone. The BPF scheduler must substitute its own preemption mechanism.
scx_centraluses abpf_timerat 1ms intervals. BPF_F_TIMER_CPU_PIN: A BPF timer flag (kernel >= 6.7) that pins the timer to a specific CPU.scx_centraltries to pin tocentral_cpuwith a fallback for older kernels.
Locking and Concurrency Notes
scx_can_stop_tick()is called fromsched_can_stop_tick()which does NOT holdrq->lock. It readsrq->scx.flagswithout locking. This is intentional — the value is advisory.SCX_RQ_CAN_STOP_TICKis written underrq->lock(inset_next_task_scx()) and read without locking inscx_can_stop_tick(). The nohz subsystem tolerates this inconsistency.scx_ops_bypass()iterates all CPUs holding each CPU'srq->lockindividually. Theresched_cpu()call after unlocking wakes tickless CPUs back to a schedulable state.
Why Maintainers Need to Know This
- Tickless mode requires
SCX_SLICE_INFon every dispatch to that task: Tick suppression is per-scheduling-event. Switching to a finite-slice task immediately re-enables the tick for that CPU. - The BPF timer may not stay on
central_cpu: Even withBPF_F_TIMER_CPU_PIN, timer migration can occur.scx_centralerrors if this happens withtimer_pinned=true. Production schedulers need robust handling. - ksoftirqd starvation kills the timer: If a user task with infinite slice starves ksoftirqd, the BPF timer cannot fire. The per-CPU kthread priority boost in
central_enqueue()is the mitigation — single-CPU-affinity kthreads are dispatched withSCX_ENQ_PREEMPT. - Load averages go stale on tickless CPUs:
update_other_load_avgs()is called only at the tick transition boundary. Production schedulers using tickless mode should account for potentially stale load data.
Connection to Other Patches
- PATCH 20/30 added
ops.running()andops.stopping()— this patch'sscx_centralis their first significant consumer, using them to trackcpu_started_at[]for the BPF timer-based slice enforcement. - PATCH 17/30 introduced
SCX_KICK_PREEMPT— the BPF timer usesscx_bpf_kick_cpu(cpu, SCX_KICK_PREEMPT)to force-expire infinite-slice tasks. - PATCH 22/30 replaces
SCX_CALL_OP()withSCX_CALL_OP_TASK()inset_next_task_scx(), upgrading theops.running()call added here.
Detailed Walkthrough
- File:
include/linux/sched/ext.hHunk:@@ -16,6 +16,7 @@ enum scx_public_consts {Before
After(no notable removed lines in sampled hunk)SCX_SLICE_INF = U64_MAX, /* infinite, implies nohz */ - File:
kernel/sched/core.cHunk:@@ -1256,11 +1256,14 @@ bool sched_can_stop_tick(struct rq *rq)Before
After* If there are no DL,RR/FIFO tasks, there must only be CFS tasks left; * if there's more than one we need the tick for involuntary * preemption.* If there are no DL,RR/FIFO tasks, there must only be CFS or sched_ext tasks * left. For CFS, if there's more than one we need the tick for * involuntary preemption. For sched_ext, ask. - File:
kernel/sched/ext.cHunk:@@ -1086,7 +1086,8 @@ static void update_curr_scx(struct rq *rq)Before
Aftercurr->scx.slice -= min(curr->scx.slice, delta_exec);
Noteif (curr->scx.slice != SCX_SLICE_INF) curr->scx.slice -= min(curr->scx.slice, delta_exec);Additional hunks continue in this file (4 more section(s)). - File:
kernel/sched/ext.hHunk:@@ -35,6 +35,7 @@ void scx_pre_fork(struct task_struct *p);Before
After(no notable removed lines in sampled hunk)
Notebool scx_can_stop_tick(struct rq *rq);Additional hunks continue in this file (1 more section(s)). - File:
kernel/sched/sched.hHunk:@@ -727,6 +727,7 @@ struct cfs_rq {Before
After(no notable removed lines in sampled hunk)SCX_RQ_CAN_STOP_TICK = 1 << 2, - File:
tools/sched_ext/scx_central.bpf.cHunk:@@ -13,7 +13,26 @@Before
After* b. Preemption
Note* b. Tickless operation * * All tasks are dispatched with the infinite slice which allows stopping theAdditional hunks continue in this file (7 more section(s)). - File:
tools/sched_ext/scx_central.cHunk:@@ -48,6 +48,7 @@ int main(int argc, char **argv)Before
After(no notable removed lines in sampled hunk)
Notecpu_set_t *cpuset;Additional hunks continue in this file (2 more section(s)).
sched_ext Context
This patch directly expands sched_ext integration points and makes the scheduler core more extensible for BPF-defined policies.
[PATCH 22/30] sched_ext: Track tasks that are subjects of the in-flight SCX operation
View on Lore: https://lore.kernel.org/all/20240618212056.2833381-23-tj@kernel.org
Commit Message
When some SCX operations are in flight, it is known that the subject task's
rq lock is held throughout which makes it safe to access certain fields of
the task - e.g. its current task_group. We want to add SCX kfunc helpers
that can make use of this guarantee - e.g. to help determining the currently
associated CPU cgroup from the task's current task_group.
As it'd be dangerous call such a helper on a task which isn't rq lock
protected, the helper should be able to verify the input task and reject
accordingly. This patch adds sched_ext_entity.kf_tasks[] that track the
tasks which are currently being operated on by a terminal SCX operation. The
new SCX_CALL_OP_[2]TASK[_RET]() can be used when invoking SCX operations
which take tasks as arguments and the scx_kf_allowed_on_arg_tasks() can be
used by kfunc helpers to verify the input task status.
Note that as sched_ext_entity.kf_tasks[] can't handle nesting, the tracking
is currently only limited to terminal SCX operations. If needed in the
future, this restriction can be removed by moving the tracking to the task
side with a couple per-task counters.
v2: Updated to reflect the addition of SCX_KF_SELECT_CPU.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: David Vernet <dvernet@meta.com>
---
include/linux/sched/ext.h | 2 +
kernel/sched/ext.c | 91 +++++++++++++++++++++++++++++++--------
2 files changed, 76 insertions(+), 17 deletions(-)
diff --git a/include/linux/sched/ext.h b/include/linux/sched/ext.h
index 6f1a4977e9f8..74341dbc6a19 100644
--- a/include/linux/sched/ext.h
+++ b/include/linux/sched/ext.h
@@ -106,6 +106,7 @@ enum scx_kf_mask {
__SCX_KF_RQ_LOCKED = SCX_KF_DISPATCH |
SCX_KF_ENQUEUE | SCX_KF_SELECT_CPU | SCX_KF_REST,
+ __SCX_KF_TERMINAL = SCX_KF_ENQUEUE | SCX_KF_SELECT_CPU | SCX_KF_REST,
};
/*
@@ -120,6 +121,7 @@ struct sched_ext_entity {
s32 sticky_cpu;
s32 holding_cpu;
u32 kf_mask; /* see scx_kf_mask above */
+ struct task_struct *kf_tasks[2]; /* see SCX_CALL_OP_TASK() */
atomic_long_t ops_state;
struct list_head runnable_node; /* rq->scx.runnable_list */
diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c
index ce32fc6b05cd..838a96cb10ea 100644
--- a/kernel/sched/ext.c
+++ b/kernel/sched/ext.c
@@ -817,6 +817,47 @@ do { \
__ret; \
})
+/*
+ * Some kfuncs are allowed only on the tasks that are subjects of the
+ * in-progress scx_ops operation for, e.g., locking guarantees. To enforce such
+ * restrictions, the following SCX_CALL_OP_*() variants should be used when
+ * invoking scx_ops operations that take task arguments. These can only be used
+ * for non-nesting operations due to the way the tasks are tracked.
+ *
+ * kfuncs which can only operate on such tasks can in turn use
+ * scx_kf_allowed_on_arg_tasks() to test whether the invocation is allowed on
+ * the specific task.
+ */
+#define SCX_CALL_OP_TASK(mask, op, task, args...) \
+do { \
+ BUILD_BUG_ON((mask) & ~__SCX_KF_TERMINAL); \
+ current->scx.kf_tasks[0] = task; \
+ SCX_CALL_OP(mask, op, task, ##args); \
+ current->scx.kf_tasks[0] = NULL; \
+} while (0)
+
+#define SCX_CALL_OP_TASK_RET(mask, op, task, args...) \
+({ \
+ __typeof__(scx_ops.op(task, ##args)) __ret; \
+ BUILD_BUG_ON((mask) & ~__SCX_KF_TERMINAL); \
+ current->scx.kf_tasks[0] = task; \
+ __ret = SCX_CALL_OP_RET(mask, op, task, ##args); \
+ current->scx.kf_tasks[0] = NULL; \
+ __ret; \
+})
+
+#define SCX_CALL_OP_2TASKS_RET(mask, op, task0, task1, args...) \
+({ \
+ __typeof__(scx_ops.op(task0, task1, ##args)) __ret; \
+ BUILD_BUG_ON((mask) & ~__SCX_KF_TERMINAL); \
+ current->scx.kf_tasks[0] = task0; \
+ current->scx.kf_tasks[1] = task1; \
+ __ret = SCX_CALL_OP_RET(mask, op, task0, task1, ##args); \
+ current->scx.kf_tasks[0] = NULL; \
+ current->scx.kf_tasks[1] = NULL; \
+ __ret; \
+})
+
/* @mask is constant, always inline to cull unnecessary branches */
static __always_inline bool scx_kf_allowed(u32 mask)
{
@@ -846,6 +887,22 @@ static __always_inline bool scx_kf_allowed(u32 mask)
return true;
}
+/* see SCX_CALL_OP_TASK() */
+static __always_inline bool scx_kf_allowed_on_arg_tasks(u32 mask,
+ struct task_struct *p)
+{
+ if (!scx_kf_allowed(mask))
+ return false;
+
+ if (unlikely((p != current->scx.kf_tasks[0] &&
+ p != current->scx.kf_tasks[1]))) {
+ scx_ops_error("called on a task not being operated on");
+ return false;
+ }
+
+ return true;
+}
+
/*
* SCX task iterator.
@@ -1342,7 +1399,7 @@ static void do_enqueue_task(struct rq *rq, struct task_struct *p, u64 enq_flags,
WARN_ON_ONCE(*ddsp_taskp);
*ddsp_taskp = p;
- SCX_CALL_OP(SCX_KF_ENQUEUE, enqueue, p, enq_flags);
+ SCX_CALL_OP_TASK(SCX_KF_ENQUEUE, enqueue, p, enq_flags);
*ddsp_taskp = NULL;
if (p->scx.ddsp_dsq_id != SCX_DSQ_INVALID)
@@ -1427,7 +1484,7 @@ static void enqueue_task_scx(struct rq *rq, struct task_struct *p, int enq_flags
add_nr_running(rq, 1);
if (SCX_HAS_OP(runnable))
- SCX_CALL_OP(SCX_KF_REST, runnable, p, enq_flags);
+ SCX_CALL_OP_TASK(SCX_KF_REST, runnable, p, enq_flags);
do_enqueue_task(rq, p, enq_flags, sticky_cpu);
}
@@ -1453,7 +1510,7 @@ static void ops_dequeue(struct task_struct *p, u64 deq_flags)
BUG();
case SCX_OPSS_QUEUED:
if (SCX_HAS_OP(dequeue))
- SCX_CALL_OP(SCX_KF_REST, dequeue, p, deq_flags);
+ SCX_CALL_OP_TASK(SCX_KF_REST, dequeue, p, deq_flags);
if (atomic_long_try_cmpxchg(&p->scx.ops_state, &opss,
SCX_OPSS_NONE))
@@ -1502,11 +1559,11 @@ static void dequeue_task_scx(struct rq *rq, struct task_struct *p, int deq_flags
*/
if (SCX_HAS_OP(stopping) && task_current(rq, p)) {
update_curr_scx(rq);
- SCX_CALL_OP(SCX_KF_REST, stopping, p, false);
+ SCX_CALL_OP_TASK(SCX_KF_REST, stopping, p, false);
}
if (SCX_HAS_OP(quiescent))
- SCX_CALL_OP(SCX_KF_REST, quiescent, p, deq_flags);
+ SCX_CALL_OP_TASK(SCX_KF_REST, quiescent, p, deq_flags);
if (deq_flags & SCX_DEQ_SLEEP)
p->scx.flags |= SCX_TASK_DEQD_FOR_SLEEP;
@@ -1525,7 +1582,7 @@ static void yield_task_scx(struct rq *rq)
struct task_struct *p = rq->curr;
if (SCX_HAS_OP(yield))
- SCX_CALL_OP_RET(SCX_KF_REST, yield, p, NULL);
+ SCX_CALL_OP_2TASKS_RET(SCX_KF_REST, yield, p, NULL);
else
p->scx.slice = 0;
}
@@ -1535,7 +1592,7 @@ static bool yield_to_task_scx(struct rq *rq, struct task_struct *to)
struct task_struct *from = rq->curr;
if (SCX_HAS_OP(yield))
- return SCX_CALL_OP_RET(SCX_KF_REST, yield, from, to);
+ return SCX_CALL_OP_2TASKS_RET(SCX_KF_REST, yield, from, to);
else
return false;
}
@@ -2091,7 +2148,7 @@ static void set_next_task_scx(struct rq *rq, struct task_struct *p, bool first)
/* see dequeue_task_scx() on why we skip when !QUEUED */
if (SCX_HAS_OP(running) && (p->scx.flags & SCX_TASK_QUEUED))
- SCX_CALL_OP(SCX_KF_REST, running, p);
+ SCX_CALL_OP_TASK(SCX_KF_REST, running, p);
clr_task_runnable(p, true);
@@ -2155,7 +2212,7 @@ static void put_prev_task_scx(struct rq *rq, struct task_struct *p)
/* see dequeue_task_scx() on why we skip when !QUEUED */
if (SCX_HAS_OP(stopping) && (p->scx.flags & SCX_TASK_QUEUED))
- SCX_CALL_OP(SCX_KF_REST, stopping, p, true);
+ SCX_CALL_OP_TASK(SCX_KF_REST, stopping, p, true);
/*
* If we're being called from put_prev_task_balance(), balance_scx() may
@@ -2377,8 +2434,8 @@ static int select_task_rq_scx(struct task_struct *p, int prev_cpu, int wake_flag
WARN_ON_ONCE(*ddsp_taskp);
*ddsp_taskp = p;
- cpu = SCX_CALL_OP_RET(SCX_KF_ENQUEUE | SCX_KF_SELECT_CPU,
- select_cpu, p, prev_cpu, wake_flags);
+ cpu = SCX_CALL_OP_TASK_RET(SCX_KF_ENQUEUE | SCX_KF_SELECT_CPU,
+ select_cpu, p, prev_cpu, wake_flags);
*ddsp_taskp = NULL;
if (ops_cpu_valid(cpu, "from ops.select_cpu()"))
return cpu;
@@ -2411,8 +2468,8 @@ static void set_cpus_allowed_scx(struct task_struct *p,
* designation pointless. Cast it away when calling the operation.
*/
if (SCX_HAS_OP(set_cpumask))
- SCX_CALL_OP(SCX_KF_REST, set_cpumask, p,
- (struct cpumask *)p->cpus_ptr);
+ SCX_CALL_OP_TASK(SCX_KF_REST, set_cpumask, p,
+ (struct cpumask *)p->cpus_ptr);
}
static void reset_idle_masks(void)
@@ -2647,7 +2704,7 @@ static void scx_ops_enable_task(struct task_struct *p)
*/
set_task_scx_weight(p);
if (SCX_HAS_OP(enable))
- SCX_CALL_OP(SCX_KF_REST, enable, p);
+ SCX_CALL_OP_TASK(SCX_KF_REST, enable, p);
scx_set_task_state(p, SCX_TASK_ENABLED);
if (SCX_HAS_OP(set_weight))
@@ -2801,7 +2858,7 @@ static void reweight_task_scx(struct rq *rq, struct task_struct *p, int newprio)
set_task_scx_weight(p);
if (SCX_HAS_OP(set_weight))
- SCX_CALL_OP(SCX_KF_REST, set_weight, p, p->scx.weight);
+ SCX_CALL_OP_TASK(SCX_KF_REST, set_weight, p, p->scx.weight);
}
static void prio_changed_scx(struct rq *rq, struct task_struct *p, int oldprio)
@@ -2817,8 +2874,8 @@ static void switching_to_scx(struct rq *rq, struct task_struct *p)
* different scheduler class. Keep the BPF scheduler up-to-date.
*/
if (SCX_HAS_OP(set_cpumask))
- SCX_CALL_OP(SCX_KF_REST, set_cpumask, p,
- (struct cpumask *)p->cpus_ptr);
+ SCX_CALL_OP_TASK(SCX_KF_REST, set_cpumask, p,
+ (struct cpumask *)p->cpus_ptr);
}
static void switched_from_scx(struct rq *rq, struct task_struct *p)
--
2.45.2
Diff
---
include/linux/sched/ext.h | 2 +
kernel/sched/ext.c | 91 +++++++++++++++++++++++++++++++--------
2 files changed, 76 insertions(+), 17 deletions(-)
diff --git a/include/linux/sched/ext.h b/include/linux/sched/ext.h
index 6f1a4977e9f8..74341dbc6a19 100644
--- a/include/linux/sched/ext.h
+++ b/include/linux/sched/ext.h
@@ -106,6 +106,7 @@ enum scx_kf_mask {
__SCX_KF_RQ_LOCKED = SCX_KF_DISPATCH |
SCX_KF_ENQUEUE | SCX_KF_SELECT_CPU | SCX_KF_REST,
+ __SCX_KF_TERMINAL = SCX_KF_ENQUEUE | SCX_KF_SELECT_CPU | SCX_KF_REST,
};
/*
@@ -120,6 +121,7 @@ struct sched_ext_entity {
s32 sticky_cpu;
s32 holding_cpu;
u32 kf_mask; /* see scx_kf_mask above */
+ struct task_struct *kf_tasks[2]; /* see SCX_CALL_OP_TASK() */
atomic_long_t ops_state;
struct list_head runnable_node; /* rq->scx.runnable_list */
diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c
index ce32fc6b05cd..838a96cb10ea 100644
--- a/kernel/sched/ext.c
+++ b/kernel/sched/ext.c
@@ -817,6 +817,47 @@ do { \
__ret; \
})
+/*
+ * Some kfuncs are allowed only on the tasks that are subjects of the
+ * in-progress scx_ops operation for, e.g., locking guarantees. To enforce such
+ * restrictions, the following SCX_CALL_OP_*() variants should be used when
+ * invoking scx_ops operations that take task arguments. These can only be used
+ * for non-nesting operations due to the way the tasks are tracked.
+ *
+ * kfuncs which can only operate on such tasks can in turn use
+ * scx_kf_allowed_on_arg_tasks() to test whether the invocation is allowed on
+ * the specific task.
+ */
+#define SCX_CALL_OP_TASK(mask, op, task, args...) \
+do { \
+ BUILD_BUG_ON((mask) & ~__SCX_KF_TERMINAL); \
+ current->scx.kf_tasks[0] = task; \
+ SCX_CALL_OP(mask, op, task, ##args); \
+ current->scx.kf_tasks[0] = NULL; \
+} while (0)
+
+#define SCX_CALL_OP_TASK_RET(mask, op, task, args...) \
+({ \
+ __typeof__(scx_ops.op(task, ##args)) __ret; \
+ BUILD_BUG_ON((mask) & ~__SCX_KF_TERMINAL); \
+ current->scx.kf_tasks[0] = task; \
+ __ret = SCX_CALL_OP_RET(mask, op, task, ##args); \
+ current->scx.kf_tasks[0] = NULL; \
+ __ret; \
+})
+
+#define SCX_CALL_OP_2TASKS_RET(mask, op, task0, task1, args...) \
+({ \
+ __typeof__(scx_ops.op(task0, task1, ##args)) __ret; \
+ BUILD_BUG_ON((mask) & ~__SCX_KF_TERMINAL); \
+ current->scx.kf_tasks[0] = task0; \
+ current->scx.kf_tasks[1] = task1; \
+ __ret = SCX_CALL_OP_RET(mask, op, task0, task1, ##args); \
+ current->scx.kf_tasks[0] = NULL; \
+ current->scx.kf_tasks[1] = NULL; \
+ __ret; \
+})
+
/* @mask is constant, always inline to cull unnecessary branches */
static __always_inline bool scx_kf_allowed(u32 mask)
{
@@ -846,6 +887,22 @@ static __always_inline bool scx_kf_allowed(u32 mask)
return true;
}
+/* see SCX_CALL_OP_TASK() */
+static __always_inline bool scx_kf_allowed_on_arg_tasks(u32 mask,
+ struct task_struct *p)
+{
+ if (!scx_kf_allowed(mask))
+ return false;
+
+ if (unlikely((p != current->scx.kf_tasks[0] &&
+ p != current->scx.kf_tasks[1]))) {
+ scx_ops_error("called on a task not being operated on");
+ return false;
+ }
+
+ return true;
+}
+
/*
* sched_ext task iterator.
@@ -1342,7 +1399,7 @@ static void do_enqueue_task(struct rq *rq, struct task_struct *p, u64 enq_flags,
WARN_ON_ONCE(*ddsp_taskp);
*ddsp_taskp = p;
- SCX_CALL_OP(SCX_KF_ENQUEUE, enqueue, p, enq_flags);
+ SCX_CALL_OP_TASK(SCX_KF_ENQUEUE, enqueue, p, enq_flags);
*ddsp_taskp = NULL;
if (p->scx.ddsp_dsq_id != SCX_DSQ_INVALID)
@@ -1427,7 +1484,7 @@ static void enqueue_task_scx(struct rq *rq, struct task_struct *p, int enq_flags
add_nr_running(rq, 1);
if (SCX_HAS_OP(runnable))
- SCX_CALL_OP(SCX_KF_REST, runnable, p, enq_flags);
+ SCX_CALL_OP_TASK(SCX_KF_REST, runnable, p, enq_flags);
do_enqueue_task(rq, p, enq_flags, sticky_cpu);
}
@@ -1453,7 +1510,7 @@ static void ops_dequeue(struct task_struct *p, u64 deq_flags)
BUG();
case SCX_OPSS_QUEUED:
if (SCX_HAS_OP(dequeue))
- SCX_CALL_OP(SCX_KF_REST, dequeue, p, deq_flags);
+ SCX_CALL_OP_TASK(SCX_KF_REST, dequeue, p, deq_flags);
if (atomic_long_try_cmpxchg(&p->scx.ops_state, &opss,
SCX_OPSS_NONE))
@@ -1502,11 +1559,11 @@ static void dequeue_task_scx(struct rq *rq, struct task_struct *p, int deq_flags
*/
if (SCX_HAS_OP(stopping) && task_current(rq, p)) {
update_curr_scx(rq);
- SCX_CALL_OP(SCX_KF_REST, stopping, p, false);
+ SCX_CALL_OP_TASK(SCX_KF_REST, stopping, p, false);
}
if (SCX_HAS_OP(quiescent))
- SCX_CALL_OP(SCX_KF_REST, quiescent, p, deq_flags);
+ SCX_CALL_OP_TASK(SCX_KF_REST, quiescent, p, deq_flags);
if (deq_flags & SCX_DEQ_SLEEP)
p->scx.flags |= SCX_TASK_DEQD_FOR_SLEEP;
@@ -1525,7 +1582,7 @@ static void yield_task_scx(struct rq *rq)
struct task_struct *p = rq->curr;
if (SCX_HAS_OP(yield))
- SCX_CALL_OP_RET(SCX_KF_REST, yield, p, NULL);
+ SCX_CALL_OP_2TASKS_RET(SCX_KF_REST, yield, p, NULL);
else
p->scx.slice = 0;
}
@@ -1535,7 +1592,7 @@ static bool yield_to_task_scx(struct rq *rq, struct task_struct *to)
struct task_struct *from = rq->curr;
if (SCX_HAS_OP(yield))
- return SCX_CALL_OP_RET(SCX_KF_REST, yield, from, to);
+ return SCX_CALL_OP_2TASKS_RET(SCX_KF_REST, yield, from, to);
else
return false;
}
@@ -2091,7 +2148,7 @@ static void set_next_task_scx(struct rq *rq, struct task_struct *p, bool first)
/* see dequeue_task_scx() on why we skip when !QUEUED */
if (SCX_HAS_OP(running) && (p->scx.flags & SCX_TASK_QUEUED))
- SCX_CALL_OP(SCX_KF_REST, running, p);
+ SCX_CALL_OP_TASK(SCX_KF_REST, running, p);
clr_task_runnable(p, true);
@@ -2155,7 +2212,7 @@ static void put_prev_task_scx(struct rq *rq, struct task_struct *p)
/* see dequeue_task_scx() on why we skip when !QUEUED */
if (SCX_HAS_OP(stopping) && (p->scx.flags & SCX_TASK_QUEUED))
- SCX_CALL_OP(SCX_KF_REST, stopping, p, true);
+ SCX_CALL_OP_TASK(SCX_KF_REST, stopping, p, true);
/*
* If we're being called from put_prev_task_balance(), balance_scx() may
@@ -2377,8 +2434,8 @@ static int select_task_rq_scx(struct task_struct *p, int prev_cpu, int wake_flag
WARN_ON_ONCE(*ddsp_taskp);
*ddsp_taskp = p;
- cpu = SCX_CALL_OP_RET(SCX_KF_ENQUEUE | SCX_KF_SELECT_CPU,
- select_cpu, p, prev_cpu, wake_flags);
+ cpu = SCX_CALL_OP_TASK_RET(SCX_KF_ENQUEUE | SCX_KF_SELECT_CPU,
+ select_cpu, p, prev_cpu, wake_flags);
*ddsp_taskp = NULL;
if (ops_cpu_valid(cpu, "from ops.select_cpu()"))
return cpu;
@@ -2411,8 +2468,8 @@ static void set_cpus_allowed_scx(struct task_struct *p,
* designation pointless. Cast it away when calling the operation.
*/
if (SCX_HAS_OP(set_cpumask))
- SCX_CALL_OP(SCX_KF_REST, set_cpumask, p,
- (struct cpumask *)p->cpus_ptr);
+ SCX_CALL_OP_TASK(SCX_KF_REST, set_cpumask, p,
+ (struct cpumask *)p->cpus_ptr);
}
static void reset_idle_masks(void)
@@ -2647,7 +2704,7 @@ static void scx_ops_enable_task(struct task_struct *p)
*/
set_task_scx_weight(p);
if (SCX_HAS_OP(enable))
- SCX_CALL_OP(SCX_KF_REST, enable, p);
+ SCX_CALL_OP_TASK(SCX_KF_REST, enable, p);
scx_set_task_state(p, SCX_TASK_ENABLED);
if (SCX_HAS_OP(set_weight))
@@ -2801,7 +2858,7 @@ static void reweight_task_scx(struct rq *rq, struct task_struct *p, int newprio)
set_task_scx_weight(p);
if (SCX_HAS_OP(set_weight))
- SCX_CALL_OP(SCX_KF_REST, set_weight, p, p->scx.weight);
+ SCX_CALL_OP_TASK(SCX_KF_REST, set_weight, p, p->scx.weight);
}
static void prio_changed_scx(struct rq *rq, struct task_struct *p, int oldprio)
@@ -2817,8 +2874,8 @@ static void switching_to_scx(struct rq *rq, struct task_struct *p)
* different scheduler class. Keep the BPF scheduler up-to-date.
*/
if (SCX_HAS_OP(set_cpumask))
- SCX_CALL_OP(SCX_KF_REST, set_cpumask, p,
- (struct cpumask *)p->cpus_ptr);
+ SCX_CALL_OP_TASK(SCX_KF_REST, set_cpumask, p,
+ (struct cpumask *)p->cpus_ptr);
}
static void switched_from_scx(struct rq *rq, struct task_struct *p)
--
2.45.2
Implementation Analysis
Overview
Some sched_ext kfuncs need stronger guarantees than just "we're inside an SCX operation" — they need to know that a specific task's rq->lock is held. For example, a future kfunc that reads a task's current cgroup assignment needs to guarantee the task cannot be migrated between CPUs while it's being read. This requires the rq holding the task to remain locked during the operation.
This patch establishes an in-flight task tracking mechanism: when certain "terminal" SCX operations are running for a task, the task pointer is stored in current->scx.kf_tasks[]. A new set of SCX_CALL_OP_TASK() macros handle this bookkeeping. kfuncs that require this guarantee use scx_kf_allowed_on_arg_tasks() to verify their input task is currently tracked as in-flight.
Code Walkthrough
include/linux/sched/ext.h — new field and mask
// New internal kf_mask value
__SCX_KF_TERMINAL = SCX_KF_ENQUEUE | SCX_KF_SELECT_CPU | SCX_KF_REST,
// New field in sched_ext_entity
struct task_struct *kf_tasks[2]; /* see SCX_CALL_OP_TASK() */
__SCX_KF_TERMINAL identifies the set of operations where task-specific kfuncs are permitted. It includes enqueue, select_cpu, and all REST operations — these are the operations that take a task argument and where the rq lock is held. The kf_tasks[2] array holds up to two task pointers (for yield which takes two tasks).
kernel/sched/ext.c — three new call-op macros
#define SCX_CALL_OP_TASK(mask, op, task, args...)
do {
BUILD_BUG_ON((mask) & ~__SCX_KF_TERMINAL);
current->scx.kf_tasks[0] = task;
SCX_CALL_OP(mask, op, task, ##args);
current->scx.kf_tasks[0] = NULL;
} while (0)
#define SCX_CALL_OP_TASK_RET(mask, op, task, args...)
({
__typeof__(scx_ops.op(task, ##args)) __ret;
BUILD_BUG_ON((mask) & ~__SCX_KF_TERMINAL);
current->scx.kf_tasks[0] = task;
__ret = SCX_CALL_OP_RET(mask, op, task, ##args);
current->scx.kf_tasks[0] = NULL;
__ret;
})
#define SCX_CALL_OP_2TASKS_RET(mask, op, task0, task1, args...)
({
BUILD_BUG_ON((mask) & ~__SCX_KF_TERMINAL);
current->scx.kf_tasks[0] = task0;
current->scx.kf_tasks[1] = task1;
__ret = SCX_CALL_OP_RET(mask, op, task0, task1, ##args);
current->scx.kf_tasks[0] = NULL;
current->scx.kf_tasks[1] = NULL;
__ret;
})
These macros wrap SCX_CALL_OP() and SCX_CALL_OP_RET(). Before the BPF callback executes, the task pointer(s) are stored in current->scx.kf_tasks[]. After the callback returns, the slots are cleared. The BUILD_BUG_ON ensures these macros are only used with terminal operations.
kernel/sched/ext.c — scx_kf_allowed_on_arg_tasks()
static __always_inline bool scx_kf_allowed_on_arg_tasks(u32 mask,
struct task_struct *p)
{
if (!scx_kf_allowed(mask)) return false;
if (unlikely(p != current->scx.kf_tasks[0] &&
p != current->scx.kf_tasks[1])) {
scx_ops_error("called on a task not being operated on");
return false;
}
return true;
}
A kfunc uses this helper to validate that p is currently the subject of an in-flight operation. If not, scx_ops_error() is called (which eventually kills the BPF scheduler) and false is returned. This is the enforcement mechanism — kfuncs that need rq-lock protection call this before accessing protected fields.
Pervasive call-site updates
Every SCX_CALL_OP() and SCX_CALL_OP_RET() that passes a task argument is replaced with the corresponding SCX_CALL_OP_TASK() variant:
do_enqueue_task()→SCX_CALL_OP_TASK(SCX_KF_ENQUEUE, enqueue, p, enq_flags)enqueue_task_scx()→SCX_CALL_OP_TASK(SCX_KF_REST, runnable, p, enq_flags)ops_dequeue()→SCX_CALL_OP_TASK(SCX_KF_REST, dequeue, p, deq_flags)dequeue_task_scx()→SCX_CALL_OP_TASK(SCX_KF_REST, stopping, p, false/true)dequeue_task_scx()→SCX_CALL_OP_TASK(SCX_KF_REST, quiescent, p, deq_flags)yield_task_scx()→SCX_CALL_OP_2TASKS_RET(SCX_KF_REST, yield, p, NULL)yield_to_task_scx()→SCX_CALL_OP_2TASKS_RET(SCX_KF_REST, yield, from, to)set_next_task_scx()→SCX_CALL_OP_TASK(SCX_KF_REST, running, p)put_prev_task_scx()→SCX_CALL_OP_TASK(SCX_KF_REST, stopping, p, true)select_task_rq_scx()→SCX_CALL_OP_TASK_RET(SCX_KF_ENQUEUE | SCX_KF_SELECT_CPU, select_cpu, p, prev_cpu, wake_flags)set_cpus_allowed_scx()→SCX_CALL_OP_TASK(SCX_KF_REST, set_cpumask, p, ...)scx_ops_enable_task()→SCX_CALL_OP_TASK(SCX_KF_REST, enable, p)reweight_task_scx()→SCX_CALL_OP_TASK(SCX_KF_REST, set_weight, p, ...)switching_to_scx()→SCX_CALL_OP_TASK(SCX_KF_REST, set_cpumask, p, ...)
Key Concepts
kf_tasks[2]oncurrent: The tracking is stored on the calling CPU's current task (current), not on the subject taskp. This is because the operation runs on the current CPU's context, andcurrent->scx.kf_tasks[0]is only valid for the duration of the BPF callback execution on that CPU.- Non-nesting restriction: The comment in the code explicitly states these macros "can only be used for non-nesting operations." If two terminal operations could nest (e.g., one op calling into the scheduler which triggers another op),
kf_tasks[0]would be overwritten. For now, terminal operations do not nest. If they did, the tracking would need to move to per-task counters. __SCX_KF_TERMINALvs.__SCX_KF_RQ_LOCKED:__SCX_KF_RQ_LOCKEDincludesSCX_KF_DISPATCHbut__SCX_KF_TERMINALdoes not. Dispatch operations (ops.dispatch()) are called with the rq lock held but do not take a task as the primary argument, so they are not "terminal" in this sense.BUILD_BUG_ONas safety net: The compile-time check(mask) & ~__SCX_KF_TERMINALensures thatSCX_CALL_OP_TASK()is never mistakenly used for non-terminal operations (likeSCX_KF_DISPATCH). This prevents incorrectly signaling to kfuncs that a task is rq-lock protected when it is not.scx_ops_error()on violation: If a kfunc is called with a task that is notkf_tasks[0]orkf_tasks[1], the error kills the BPF scheduler. This is intentional — calling such a kfunc on an arbitrary task pointer (not the in-flight subject) would bypass the locking guarantee and is a scheduler bug.
Locking and Concurrency Notes
kf_tasks[]is written and read on the same CPU (current). No cross-CPU locking is required. The writes happen before the BPF callback (insideSCX_CALL_OP_TASK()) and the reads happen during the BPF callback (inside kfunc implementations). Since the BPF callback runs inline on the same CPU, there is no race.- The
kf_tasks[0] = NULLcleanup after the callback is not strictly needed for correctness (the nextSCX_CALL_OP_TASK()would overwrite it), but it is good defensive practice: a stale non-NULL pointer would incorrectly pass thescx_kf_allowed_on_arg_tasks()check in a subsequent unrelated operation. - The
kf_tasks[]array is part ofsched_ext_entitywhich is embedded intask_struct. This means it is automatically zeroed at task init viainit_scx_entity().
Why Maintainers Need to Know This
- Every new op that takes a task argument should use
SCX_CALL_OP_TASK(): This is now the standard. The oldSCX_CALL_OP()with a task argument bypasses the tracking. Future additions tosched_ext_opsthat take task arguments must use the TASK variants, or kfuncs that callscx_kf_allowed_on_arg_tasks()will incorrectly reject them. kf_tasks[2]limits tracking to two tasks: The array size of 2 is sufficient for all current operations (the maximum isyieldwhich has two task arguments). If a future op takes three or more task arguments, the array must be extended.- This enables future task-group/cgroup kfuncs: The commit message mentions "determining the currently associated CPU cgroup from the task's current task_group." This patch lays the groundwork for such kfuncs, but does not add them. Reviewers of future kfuncs that call
scx_kf_allowed_on_arg_tasks()should trace back to this patch to understand the guarantee. - Non-terminal ops (
SCX_KF_DISPATCH) cannot use this mechanism:ops.dispatch()does not have a specific task subject — it dispatches for a CPU, not a task. Ifdispatch()wants to look up task-specific protected state, it must use a different mechanism (e.g., looking up tasks from BPF maps with explicit reference counting).
Connection to Other Patches
- PATCH 20/30 added
ops.runnable(),ops.running(),ops.stopping(),ops.quiescent()— all of which are upgraded fromSCX_CALL_OP()toSCX_CALL_OP_TASK()here. - PATCH 21/30 (tickless) added the
ops.running()call inset_next_task_scx()— this patch upgrades that call toSCX_CALL_OP_TASK(). - This patch is a prerequisite for any kfunc that needs per-task rq-lock guarantees. Future patches adding cgroup-aware scheduling kfuncs (e.g., reading
task->task_groupsafely) depend onscx_kf_allowed_on_arg_tasks()established here.
[PATCH 23/30] sched_ext: Implement SCX_KICK_WAIT
View on Lore: https://lore.kernel.org/all/20240618212056.2833381-24-tj@kernel.org
Commit Message
From: David Vernet <dvernet@meta.com>
If set when calling scx_bpf_kick_cpu(), the invoking CPU will busy wait for
the kicked cpu to enter the scheduler. See the following for example usage:
https://github.com/sched-ext/scx/blob/main/scheds/c/scx_pair.bpf.c
v2: - Updated to fit the updated kick_cpus_irq_workfn() implementation.
- Include SCX_KICK_WAIT related information in debug dump.
Signed-off-by: David Vernet <dvernet@meta.com>
Reviewed-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
Acked-by: Josh Don <joshdon@google.com>
Acked-by: Hao Luo <haoluo@google.com>
Acked-by: Barret Rhoden <brho@google.com>
---
kernel/sched/core.c | 4 ++-
kernel/sched/ext.c | 82 ++++++++++++++++++++++++++++++++++++++++----
kernel/sched/ext.h | 4 +++
kernel/sched/sched.h | 2 ++
4 files changed, 85 insertions(+), 7 deletions(-)
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index d5eff4036be7..0e6ff33f34e4 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -5898,8 +5898,10 @@ __pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
for_each_active_class(class) {
p = class->pick_next_task(rq);
- if (p)
+ if (p) {
+ scx_next_task_picked(rq, p, class);
return p;
+ }
}
BUG(); /* The idle class should always have a runnable task. */
diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c
index 838a96cb10ea..1ca3067b4e0a 100644
--- a/kernel/sched/ext.c
+++ b/kernel/sched/ext.c
@@ -532,6 +532,12 @@ enum scx_kick_flags {
* task expires and the dispatch path is invoked.
*/
SCX_KICK_PREEMPT = 1LLU << 1,
+
+ /*
+ * Wait for the CPU to be rescheduled. The scx_bpf_kick_cpu() call will
+ * return after the target CPU finishes picking the next task.
+ */
+ SCX_KICK_WAIT = 1LLU << 2,
};
enum scx_ops_enable_state {
@@ -661,6 +667,9 @@ static struct {
#endif /* CONFIG_SMP */
+/* for %SCX_KICK_WAIT */
+static unsigned long __percpu *scx_kick_cpus_pnt_seqs;
+
/*
* Direct dispatch marker.
*
@@ -2288,6 +2297,23 @@ static struct task_struct *pick_next_task_scx(struct rq *rq)
return p;
}
+void scx_next_task_picked(struct rq *rq, struct task_struct *p,
+ const struct sched_class *active)
+{
+ lockdep_assert_rq_held(rq);
+
+ if (!scx_enabled())
+ return;
+#ifdef CONFIG_SMP
+ /*
+ * Pairs with the smp_load_acquire() issued by a CPU in
+ * kick_cpus_irq_workfn() who is waiting for this CPU to perform a
+ * resched.
+ */
+ smp_store_release(&rq->scx.pnt_seq, rq->scx.pnt_seq + 1);
+#endif
+}
+
#ifdef CONFIG_SMP
static bool test_and_clear_cpu_idle(int cpu)
@@ -3673,9 +3699,9 @@ static void scx_dump_state(struct scx_exit_info *ei, size_t dump_len)
seq_buf_init(&ns, buf, avail);
dump_newline(&ns);
- dump_line(&ns, "CPU %-4d: nr_run=%u flags=0x%x ops_qseq=%lu",
+ dump_line(&ns, "CPU %-4d: nr_run=%u flags=0x%x ops_qseq=%lu pnt_seq=%lu",
cpu, rq->scx.nr_running, rq->scx.flags,
- rq->scx.ops_qseq);
+ rq->scx.ops_qseq, rq->scx.pnt_seq);
dump_line(&ns, " curr=%s[%d] class=%ps",
rq->curr->comm, rq->curr->pid,
rq->curr->sched_class);
@@ -3688,6 +3714,9 @@ static void scx_dump_state(struct scx_exit_info *ei, size_t dump_len)
if (!cpumask_empty(rq->scx.cpus_to_preempt))
dump_line(&ns, " cpus_to_preempt: %*pb",
cpumask_pr_args(rq->scx.cpus_to_preempt));
+ if (!cpumask_empty(rq->scx.cpus_to_wait))
+ dump_line(&ns, " cpus_to_wait : %*pb",
+ cpumask_pr_args(rq->scx.cpus_to_wait));
used = seq_buf_used(&ns);
if (SCX_HAS_OP(dump_cpu)) {
@@ -4383,10 +4412,11 @@ static bool can_skip_idle_kick(struct rq *rq)
return !is_idle_task(rq->curr) && !(rq->scx.flags & SCX_RQ_BALANCING);
}
-static void kick_one_cpu(s32 cpu, struct rq *this_rq)
+static bool kick_one_cpu(s32 cpu, struct rq *this_rq, unsigned long *pseqs)
{
struct rq *rq = cpu_rq(cpu);
struct scx_rq *this_scx = &this_rq->scx;
+ bool should_wait = false;
unsigned long flags;
raw_spin_rq_lock_irqsave(rq, flags);
@@ -4402,12 +4432,20 @@ static void kick_one_cpu(s32 cpu, struct rq *this_rq)
cpumask_clear_cpu(cpu, this_scx->cpus_to_preempt);
}
+ if (cpumask_test_cpu(cpu, this_scx->cpus_to_wait)) {
+ pseqs[cpu] = rq->scx.pnt_seq;
+ should_wait = true;
+ }
+
resched_curr(rq);
} else {
cpumask_clear_cpu(cpu, this_scx->cpus_to_preempt);
+ cpumask_clear_cpu(cpu, this_scx->cpus_to_wait);
}
raw_spin_rq_unlock_irqrestore(rq, flags);
+
+ return should_wait;
}
static void kick_one_cpu_if_idle(s32 cpu, struct rq *this_rq)
@@ -4428,10 +4466,12 @@ static void kick_cpus_irq_workfn(struct irq_work *irq_work)
{
struct rq *this_rq = this_rq();
struct scx_rq *this_scx = &this_rq->scx;
+ unsigned long *pseqs = this_cpu_ptr(scx_kick_cpus_pnt_seqs);
+ bool should_wait = false;
s32 cpu;
for_each_cpu(cpu, this_scx->cpus_to_kick) {
- kick_one_cpu(cpu, this_rq);
+ should_wait |= kick_one_cpu(cpu, this_rq, pseqs);
cpumask_clear_cpu(cpu, this_scx->cpus_to_kick);
cpumask_clear_cpu(cpu, this_scx->cpus_to_kick_if_idle);
}
@@ -4440,6 +4480,28 @@ static void kick_cpus_irq_workfn(struct irq_work *irq_work)
kick_one_cpu_if_idle(cpu, this_rq);
cpumask_clear_cpu(cpu, this_scx->cpus_to_kick_if_idle);
}
+
+ if (!should_wait)
+ return;
+
+ for_each_cpu(cpu, this_scx->cpus_to_wait) {
+ unsigned long *wait_pnt_seq = &cpu_rq(cpu)->scx.pnt_seq;
+
+ if (cpu != cpu_of(this_rq)) {
+ /*
+ * Pairs with smp_store_release() issued by this CPU in
+ * scx_next_task_picked() on the resched path.
+ *
+ * We busy-wait here to guarantee that no other task can
+ * be scheduled on our core before the target CPU has
+ * entered the resched path.
+ */
+ while (smp_load_acquire(wait_pnt_seq) == pseqs[cpu])
+ cpu_relax();
+ }
+
+ cpumask_clear_cpu(cpu, this_scx->cpus_to_wait);
+ }
}
/**
@@ -4504,6 +4566,11 @@ void __init init_sched_ext_class(void)
BUG_ON(!alloc_cpumask_var(&idle_masks.cpu, GFP_KERNEL));
BUG_ON(!alloc_cpumask_var(&idle_masks.smt, GFP_KERNEL));
#endif
+ scx_kick_cpus_pnt_seqs =
+ __alloc_percpu(sizeof(scx_kick_cpus_pnt_seqs[0]) * nr_cpu_ids,
+ __alignof__(scx_kick_cpus_pnt_seqs[0]));
+ BUG_ON(!scx_kick_cpus_pnt_seqs);
+
for_each_possible_cpu(cpu) {
struct rq *rq = cpu_rq(cpu);
@@ -4513,6 +4580,7 @@ void __init init_sched_ext_class(void)
BUG_ON(!zalloc_cpumask_var(&rq->scx.cpus_to_kick, GFP_KERNEL));
BUG_ON(!zalloc_cpumask_var(&rq->scx.cpus_to_kick_if_idle, GFP_KERNEL));
BUG_ON(!zalloc_cpumask_var(&rq->scx.cpus_to_preempt, GFP_KERNEL));
+ BUG_ON(!zalloc_cpumask_var(&rq->scx.cpus_to_wait, GFP_KERNEL));
init_irq_work(&rq->scx.kick_cpus_irq_work, kick_cpus_irq_workfn);
}
@@ -4840,8 +4908,8 @@ __bpf_kfunc void scx_bpf_kick_cpu(s32 cpu, u64 flags)
if (flags & SCX_KICK_IDLE) {
struct rq *target_rq = cpu_rq(cpu);
- if (unlikely(flags & SCX_KICK_PREEMPT))
- scx_ops_error("PREEMPT cannot be used with SCX_KICK_IDLE");
+ if (unlikely(flags & (SCX_KICK_PREEMPT | SCX_KICK_WAIT)))
+ scx_ops_error("PREEMPT/WAIT cannot be used with SCX_KICK_IDLE");
if (raw_spin_rq_trylock(target_rq)) {
if (can_skip_idle_kick(target_rq)) {
@@ -4856,6 +4924,8 @@ __bpf_kfunc void scx_bpf_kick_cpu(s32 cpu, u64 flags)
if (flags & SCX_KICK_PREEMPT)
cpumask_set_cpu(cpu, this_rq->scx.cpus_to_preempt);
+ if (flags & SCX_KICK_WAIT)
+ cpumask_set_cpu(cpu, this_rq->scx.cpus_to_wait);
}
irq_work_queue(&this_rq->scx.kick_cpus_irq_work);
diff --git a/kernel/sched/ext.h b/kernel/sched/ext.h
index 6ed946f72489..0aeb1fda1794 100644
--- a/kernel/sched/ext.h
+++ b/kernel/sched/ext.h
@@ -29,6 +29,8 @@ static inline bool task_on_scx(const struct task_struct *p)
return scx_enabled() && p->sched_class == &ext_sched_class;
}
+void scx_next_task_picked(struct rq *rq, struct task_struct *p,
+ const struct sched_class *active);
void scx_tick(struct rq *rq);
void init_scx_entity(struct sched_ext_entity *scx);
void scx_pre_fork(struct task_struct *p);
@@ -69,6 +71,8 @@ static inline const struct sched_class *next_active_class(const struct sched_cla
#define scx_enabled() false
#define scx_switched_all() false
+static inline void scx_next_task_picked(struct rq *rq, struct task_struct *p,
+ const struct sched_class *active) {}
static inline void scx_tick(struct rq *rq) {}
static inline void scx_pre_fork(struct task_struct *p) {}
static inline int scx_fork(struct task_struct *p) { return 0; }
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index b3c578cb43cd..734206e13897 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -740,6 +740,8 @@ struct scx_rq {
cpumask_var_t cpus_to_kick;
cpumask_var_t cpus_to_kick_if_idle;
cpumask_var_t cpus_to_preempt;
+ cpumask_var_t cpus_to_wait;
+ unsigned long pnt_seq;
struct irq_work kick_cpus_irq_work;
};
#endif /* CONFIG_SCHED_CLASS_EXT */
--
2.45.2
Diff
---
kernel/sched/core.c | 4 ++-
kernel/sched/ext.c | 82 ++++++++++++++++++++++++++++++++++++++++----
kernel/sched/ext.h | 4 +++
kernel/sched/sched.h | 2 ++
4 files changed, 85 insertions(+), 7 deletions(-)
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index d5eff4036be7..0e6ff33f34e4 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -5898,8 +5898,10 @@ __pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
for_each_active_class(class) {
p = class->pick_next_task(rq);
- if (p)
+ if (p) {
+ scx_next_task_picked(rq, p, class);
return p;
+ }
}
BUG(); /* The idle class should always have a runnable task. */
diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c
index 838a96cb10ea..1ca3067b4e0a 100644
--- a/kernel/sched/ext.c
+++ b/kernel/sched/ext.c
@@ -532,6 +532,12 @@ enum scx_kick_flags {
* task expires and the dispatch path is invoked.
*/
SCX_KICK_PREEMPT = 1LLU << 1,
+
+ /*
+ * Wait for the CPU to be rescheduled. The scx_bpf_kick_cpu() call will
+ * return after the target CPU finishes picking the next task.
+ */
+ SCX_KICK_WAIT = 1LLU << 2,
};
enum scx_ops_enable_state {
@@ -661,6 +667,9 @@ static struct {
#endif /* CONFIG_SMP */
+/* for %SCX_KICK_WAIT */
+static unsigned long __percpu *scx_kick_cpus_pnt_seqs;
+
/*
* Direct dispatch marker.
*
@@ -2288,6 +2297,23 @@ static struct task_struct *pick_next_task_scx(struct rq *rq)
return p;
}
+void scx_next_task_picked(struct rq *rq, struct task_struct *p,
+ const struct sched_class *active)
+{
+ lockdep_assert_rq_held(rq);
+
+ if (!scx_enabled())
+ return;
+#ifdef CONFIG_SMP
+ /*
+ * Pairs with the smp_load_acquire() issued by a CPU in
+ * kick_cpus_irq_workfn() who is waiting for this CPU to perform a
+ * resched.
+ */
+ smp_store_release(&rq->scx.pnt_seq, rq->scx.pnt_seq + 1);
+#endif
+}
+
#ifdef CONFIG_SMP
static bool test_and_clear_cpu_idle(int cpu)
@@ -3673,9 +3699,9 @@ static void scx_dump_state(struct scx_exit_info *ei, size_t dump_len)
seq_buf_init(&ns, buf, avail);
dump_newline(&ns);
- dump_line(&ns, "CPU %-4d: nr_run=%u flags=0x%x ops_qseq=%lu",
+ dump_line(&ns, "CPU %-4d: nr_run=%u flags=0x%x ops_qseq=%lu pnt_seq=%lu",
cpu, rq->scx.nr_running, rq->scx.flags,
- rq->scx.ops_qseq);
+ rq->scx.ops_qseq, rq->scx.pnt_seq);
dump_line(&ns, " curr=%s[%d] class=%ps",
rq->curr->comm, rq->curr->pid,
rq->curr->sched_class);
@@ -3688,6 +3714,9 @@ static void scx_dump_state(struct scx_exit_info *ei, size_t dump_len)
if (!cpumask_empty(rq->scx.cpus_to_preempt))
dump_line(&ns, " cpus_to_preempt: %*pb",
cpumask_pr_args(rq->scx.cpus_to_preempt));
+ if (!cpumask_empty(rq->scx.cpus_to_wait))
+ dump_line(&ns, " cpus_to_wait : %*pb",
+ cpumask_pr_args(rq->scx.cpus_to_wait));
used = seq_buf_used(&ns);
if (SCX_HAS_OP(dump_cpu)) {
@@ -4383,10 +4412,11 @@ static bool can_skip_idle_kick(struct rq *rq)
return !is_idle_task(rq->curr) && !(rq->scx.flags & SCX_RQ_BALANCING);
}
-static void kick_one_cpu(s32 cpu, struct rq *this_rq)
+static bool kick_one_cpu(s32 cpu, struct rq *this_rq, unsigned long *pseqs)
{
struct rq *rq = cpu_rq(cpu);
struct scx_rq *this_scx = &this_rq->scx;
+ bool should_wait = false;
unsigned long flags;
raw_spin_rq_lock_irqsave(rq, flags);
@@ -4402,12 +4432,20 @@ static void kick_one_cpu(s32 cpu, struct rq *this_rq)
cpumask_clear_cpu(cpu, this_scx->cpus_to_preempt);
}
+ if (cpumask_test_cpu(cpu, this_scx->cpus_to_wait)) {
+ pseqs[cpu] = rq->scx.pnt_seq;
+ should_wait = true;
+ }
+
resched_curr(rq);
} else {
cpumask_clear_cpu(cpu, this_scx->cpus_to_preempt);
+ cpumask_clear_cpu(cpu, this_scx->cpus_to_wait);
}
raw_spin_rq_unlock_irqrestore(rq, flags);
+
+ return should_wait;
}
static void kick_one_cpu_if_idle(s32 cpu, struct rq *this_rq)
@@ -4428,10 +4466,12 @@ static void kick_cpus_irq_workfn(struct irq_work *irq_work)
{
struct rq *this_rq = this_rq();
struct scx_rq *this_scx = &this_rq->scx;
+ unsigned long *pseqs = this_cpu_ptr(scx_kick_cpus_pnt_seqs);
+ bool should_wait = false;
s32 cpu;
for_each_cpu(cpu, this_scx->cpus_to_kick) {
- kick_one_cpu(cpu, this_rq);
+ should_wait |= kick_one_cpu(cpu, this_rq, pseqs);
cpumask_clear_cpu(cpu, this_scx->cpus_to_kick);
cpumask_clear_cpu(cpu, this_scx->cpus_to_kick_if_idle);
}
@@ -4440,6 +4480,28 @@ static void kick_cpus_irq_workfn(struct irq_work *irq_work)
kick_one_cpu_if_idle(cpu, this_rq);
cpumask_clear_cpu(cpu, this_scx->cpus_to_kick_if_idle);
}
+
+ if (!should_wait)
+ return;
+
+ for_each_cpu(cpu, this_scx->cpus_to_wait) {
+ unsigned long *wait_pnt_seq = &cpu_rq(cpu)->scx.pnt_seq;
+
+ if (cpu != cpu_of(this_rq)) {
+ /*
+ * Pairs with smp_store_release() issued by this CPU in
+ * scx_next_task_picked() on the resched path.
+ *
+ * We busy-wait here to guarantee that no other task can
+ * be scheduled on our core before the target CPU has
+ * entered the resched path.
+ */
+ while (smp_load_acquire(wait_pnt_seq) == pseqs[cpu])
+ cpu_relax();
+ }
+
+ cpumask_clear_cpu(cpu, this_scx->cpus_to_wait);
+ }
}
/**
@@ -4504,6 +4566,11 @@ void __init init_sched_ext_class(void)
BUG_ON(!alloc_cpumask_var(&idle_masks.cpu, GFP_KERNEL));
BUG_ON(!alloc_cpumask_var(&idle_masks.smt, GFP_KERNEL));
#endif
+ scx_kick_cpus_pnt_seqs =
+ __alloc_percpu(sizeof(scx_kick_cpus_pnt_seqs[0]) * nr_cpu_ids,
+ __alignof__(scx_kick_cpus_pnt_seqs[0]));
+ BUG_ON(!scx_kick_cpus_pnt_seqs);
+
for_each_possible_cpu(cpu) {
struct rq *rq = cpu_rq(cpu);
@@ -4513,6 +4580,7 @@ void __init init_sched_ext_class(void)
BUG_ON(!zalloc_cpumask_var(&rq->scx.cpus_to_kick, GFP_KERNEL));
BUG_ON(!zalloc_cpumask_var(&rq->scx.cpus_to_kick_if_idle, GFP_KERNEL));
BUG_ON(!zalloc_cpumask_var(&rq->scx.cpus_to_preempt, GFP_KERNEL));
+ BUG_ON(!zalloc_cpumask_var(&rq->scx.cpus_to_wait, GFP_KERNEL));
init_irq_work(&rq->scx.kick_cpus_irq_work, kick_cpus_irq_workfn);
}
@@ -4840,8 +4908,8 @@ __bpf_kfunc void scx_bpf_kick_cpu(s32 cpu, u64 flags)
if (flags & SCX_KICK_IDLE) {
struct rq *target_rq = cpu_rq(cpu);
- if (unlikely(flags & SCX_KICK_PREEMPT))
- scx_ops_error("PREEMPT cannot be used with SCX_KICK_IDLE");
+ if (unlikely(flags & (SCX_KICK_PREEMPT | SCX_KICK_WAIT)))
+ scx_ops_error("PREEMPT/WAIT cannot be used with SCX_KICK_IDLE");
if (raw_spin_rq_trylock(target_rq)) {
if (can_skip_idle_kick(target_rq)) {
@@ -4856,6 +4924,8 @@ __bpf_kfunc void scx_bpf_kick_cpu(s32 cpu, u64 flags)
if (flags & SCX_KICK_PREEMPT)
cpumask_set_cpu(cpu, this_rq->scx.cpus_to_preempt);
+ if (flags & SCX_KICK_WAIT)
+ cpumask_set_cpu(cpu, this_rq->scx.cpus_to_wait);
}
irq_work_queue(&this_rq->scx.kick_cpus_irq_work);
diff --git a/kernel/sched/ext.h b/kernel/sched/ext.h
index 6ed946f72489..0aeb1fda1794 100644
--- a/kernel/sched/ext.h
+++ b/kernel/sched/ext.h
@@ -29,6 +29,8 @@ static inline bool task_on_scx(const struct task_struct *p)
return scx_enabled() && p->sched_class == &ext_sched_class;
}
+void scx_next_task_picked(struct rq *rq, struct task_struct *p,
+ const struct sched_class *active);
void scx_tick(struct rq *rq);
void init_scx_entity(struct sched_ext_entity *scx);
void scx_pre_fork(struct task_struct *p);
@@ -69,6 +71,8 @@ static inline const struct sched_class *next_active_class(const struct sched_cla
#define scx_enabled() false
#define scx_switched_all() false
+static inline void scx_next_task_picked(struct rq *rq, struct task_struct *p,
+ const struct sched_class *active) {}
static inline void scx_tick(struct rq *rq) {}
static inline void scx_pre_fork(struct task_struct *p) {}
static inline int scx_fork(struct task_struct *p) { return 0; }
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index b3c578cb43cd..734206e13897 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -740,6 +740,8 @@ struct scx_rq {
cpumask_var_t cpus_to_kick;
cpumask_var_t cpus_to_kick_if_idle;
cpumask_var_t cpus_to_preempt;
+ cpumask_var_t cpus_to_wait;
+ unsigned long pnt_seq;
struct irq_work kick_cpus_irq_work;
};
#endif /* CONFIG_SCHED_CLASS_EXT */
--
2.45.2
Implementation Analysis
Overview
This patch extends scx_bpf_kick_cpu() with SCX_KICK_WAIT, a flag that causes the calling CPU to busy-wait until the target CPU completes one full scheduling cycle (i.e., picks its next task). Without this flag, scx_bpf_kick_cpu() merely schedules a reschedule interrupt on the target CPU and returns immediately — there is no guarantee the target CPU has actually acted on it before the caller continues.
The motivating use case is the scx_pair scheduler: it dispatches a task to CPU A's local DSQ from CPU B's context, then needs a hard guarantee that CPU A has consumed that task before CPU B proceeds. Without SCX_KICK_WAIT, CPU B would race forward and potentially dispatch again before CPU A had a chance to pick up the first task.
The implementation uses a sequence counter (pnt_seq) incremented by the target CPU in __pick_next_task(), combined with an smp_store_release/smp_load_acquire barrier pair for correct cross-CPU memory ordering. The waiting CPU spins in kick_cpus_irq_workfn() — an irq_work context — which is a carefully chosen location because irq_work runs with all prior irq_work on this CPU complete, avoiding re-entrancy issues.
Code Walkthrough
New pnt_seq counter in scx_rq (sched.h)
struct scx_rq {
cpumask_var_t cpus_to_kick;
cpumask_var_t cpus_to_kick_if_idle;
cpumask_var_t cpus_to_preempt;
+ cpumask_var_t cpus_to_wait;
+ unsigned long pnt_seq;
struct irq_work kick_cpus_irq_work;
};
pnt_seq (pick-next-task sequence counter) is a monotonically increasing counter embedded in scx_rq. It starts at zero and is incremented by one each time this CPU picks a next task while sched_ext is enabled. The cpus_to_wait cpumask parallels cpus_to_preempt — it accumulates the set of CPUs the current CPU needs to wait for before the irq_work handler returns.
scx_next_task_picked() hook in core.c
+if (p) {
+ scx_next_task_picked(rq, p, class);
return p;
+}
The hook is inserted inside __pick_next_task() at the exact point where a task has been selected. scx_next_task_picked() is called unconditionally for any scheduler class (not just SCX), so the sequence counter advances whether or not the picked task was scheduled by the BPF program. This is intentional: a SCX_KICK_WAIT caller only cares that the target CPU went through a full scheduling decision, regardless of which task won.
scx_next_task_picked() implementation (ext.c)
void scx_next_task_picked(struct rq *rq, struct task_struct *p,
const struct sched_class *active)
{
lockdep_assert_rq_held(rq);
if (!scx_enabled())
return;
#ifdef CONFIG_SMP
smp_store_release(&rq->scx.pnt_seq, rq->scx.pnt_seq + 1);
#endif
}
smp_store_release() ensures that all prior memory writes on this CPU (including any modifications the BPF program or sched class made when dequeuing/running the picked task) are visible to any other CPU that observes the new pnt_seq value via smp_load_acquire(). This is the release side of the acquire/release pair.
scx_kick_cpus_pnt_seqs per-CPU snapshot array
static unsigned long __percpu *scx_kick_cpus_pnt_seqs;
This is a per-CPU array of size nr_cpu_ids. When CPU A's irq_work handler is about to wait for CPU B, it first snapshots CPU B's current pnt_seq into scx_kick_cpus_pnt_seqs[B]. After all kicks are issued, it waits until CPU B's pnt_seq has advanced beyond that snapshot value.
The per-CPU allocation in init_sched_ext_class():
scx_kick_cpus_pnt_seqs =
__alloc_percpu(sizeof(scx_kick_cpus_pnt_seqs[0]) * nr_cpu_ids,
__alignof__(scx_kick_cpus_pnt_seqs[0]));
BUG_ON(!scx_kick_cpus_pnt_seqs);
Per-CPU allocation avoids false sharing: if two different CPUs are simultaneously running kick_cpus_irq_workfn() and waiting for different targets, they each have their own snapshot array with no cache line contention.
kick_one_cpu() changes
-static void kick_one_cpu(s32 cpu, struct rq *this_rq)
+static bool kick_one_cpu(s32 cpu, struct rq *this_rq, unsigned long *pseqs)
The function now returns bool indicating whether a wait is needed. The pseqs parameter is the caller's snapshot array. Inside, after acquiring the target rq lock:
if (cpumask_test_cpu(cpu, this_scx->cpus_to_wait)) {
pseqs[cpu] = rq->scx.pnt_seq; /* snapshot before resched */
should_wait = true;
}
resched_curr(rq);
The snapshot must be taken under the target rq lock, before calling resched_curr(). This ordering is critical: if the snapshot were taken after resched_curr(), the target CPU could have already advanced pnt_seq before the snapshot, causing the waiter to see an already-advanced counter and immediately stop waiting — it would not have actually waited for the resched triggered by this kick.
When the target CPU is not running (i.e., can_skip_idle_kick() path falls through to the else branch), cpus_to_wait is cleared immediately since there is nothing to wait for.
Busy-wait loop in kick_cpus_irq_workfn()
for_each_cpu(cpu, this_scx->cpus_to_wait) {
unsigned long *wait_pnt_seq = &cpu_rq(cpu)->scx.pnt_seq;
if (cpu != cpu_of(this_rq)) {
while (smp_load_acquire(wait_pnt_seq) == pseqs[cpu])
cpu_relax();
}
cpumask_clear_cpu(cpu, this_scx->cpus_to_wait);
}
The self-CPU check (cpu != cpu_of(this_rq)) prevents deadlock: you cannot wait for yourself to pick a task because you are currently in irq_work on that CPU and cannot schedule. smp_load_acquire() is the acquire side of the barrier pair — it ensures that once the waiter observes pnt_seq advance, all writes made by the target CPU before its smp_store_release() are also visible to the waiter.
cpu_relax() inserts a pause/hint instruction (x86: PAUSE, ARM: YIELD) that reduces power consumption and avoids memory ordering hazards in tight spin loops.
Flag registration in scx_bpf_kick_cpu()
if (flags & SCX_KICK_PREEMPT)
cpumask_set_cpu(cpu, this_rq->scx.cpus_to_preempt);
+if (flags & SCX_KICK_WAIT)
+ cpumask_set_cpu(cpu, this_rq->scx.cpus_to_wait);
SCX_KICK_WAIT is incompatible with SCX_KICK_IDLE (enforced by the existing error check):
-if (unlikely(flags & SCX_KICK_PREEMPT))
- scx_ops_error("PREEMPT cannot be used with SCX_KICK_IDLE");
+if (unlikely(flags & (SCX_KICK_PREEMPT | SCX_KICK_WAIT)))
+ scx_ops_error("PREEMPT/WAIT cannot be used with SCX_KICK_IDLE");
SCX_KICK_IDLE uses a trylock path that returns early without guaranteed delivery if the CPU is not idle — waiting would be undefined. SCX_KICK_WAIT can be combined with SCX_KICK_PREEMPT to mean "force a reschedule now and wait for it to complete".
Debug dump additions
pnt_seq is added to the per-CPU line in scx_dump_state():
dump_line(&ns, "CPU %-4d: nr_run=%u flags=0x%x ops_qseq=%lu pnt_seq=%lu", ...);
cpus_to_wait is conditionally printed when non-empty, paralleling the existing cpus_to_preempt dump. A non-empty cpus_to_wait in a crash dump indicates the crash occurred while a CPU was in the middle of waiting for another CPU to reschedule — useful for diagnosing wait-related deadlocks or scheduler stalls.
Key Concepts
pnt_seq — pick-next-task sequence counter
A per-rq monotonically increasing counter that serves as a generation number for scheduling decisions. It is incremented exactly once per task selection, unconditionally (for all sched classes, not just SCX). The counter lives in rq->scx.pnt_seq rather than rq itself because it is an SCX-specific concern — adding it to the core rq would pollute the struct for non-SCX kernels.
acquire/release memory ordering
The smp_store_release in scx_next_task_picked() and smp_load_acquire in kick_cpus_irq_workfn() form a synchronization pair. Release ensures all prior stores are visible before the counter update; acquire ensures all subsequent loads see memory written before the release. Without this pair, the waiter could observe the advanced counter but still see stale task state.
Why irq_work context for busy-waiting
kick_cpus_irq_workfn() runs in irq_work context (softirq-level, with local IRQs disabled). Busy-waiting here is safe for short durations (one scheduling cycle), but it holds IRQs off on the waiting CPU for that period. This is intentional and acceptable for the scx_pair use case where the wait is bounded by the target CPU's time to schedule. Schedulers that use SCX_KICK_WAIT must be aware they are introducing latency on the waiting CPU.
scx_kick_cpus_pnt_seqs — why per-CPU
If a single CPU could kick and wait for multiple targets simultaneously (via for_each_cpu over cpus_to_kick), a shared array would require locking. Per-CPU allocation gives each kicker its own snapshot array, trading memory (N CPUs × N CPU_IDS × 8 bytes) for lock-free access.
Locking and Concurrency Notes
pnt_seqis read and written without any spinlock, relying entirely onsmp_store_release/smp_load_acquirefor ordering. This is correct because it is only written by the owning CPU (inscx_next_task_picked(), called from__pick_next_task()which holds the rq lock) and only read by remote CPUs spinning in irq_work.- The
pseqs[cpu]snapshot inkick_one_cpu()is taken underraw_spin_rq_lock_irqsave(rq)on the target rq. This ensures atomicity: the snapshot and theresched_curr()call are both under the same lock, so the target CPU cannot have advancedpnt_seqbetween snapshot and resched. cpus_to_waitis modified only from the owning CPU's context (BPF program call toscx_bpf_kick_cpu()sets it; irq_work handler clears it), so no additional locking is needed for the cpumask itself.- The self-CPU skip in the wait loop prevents a deadlock where a CPU waits for its own
pnt_seqto advance while holding irq_work context — which prevents the very scheduling that would advance it.
Why Maintainers Need to Know This
Ordering invariant for dispatch + wait: The entire SCX_KICK_WAIT mechanism is built on the invariant that pnt_seq is snapshotted under the target rq lock, before resched_curr(). Any future refactoring that changes this ordering (e.g., moving the snapshot outside the lock) would silently break the wait guarantee and introduce hard-to-reproduce races where the waiter returns too early.
Bounded busy-wait assumption: The wait loop spins with IRQs disabled (irq_work context). It assumes the target CPU will call __pick_next_task() in bounded time. If the target CPU is stuck in a long non-preemptible section (e.g., a spinlock), the waiting CPU will spin for the duration, potentially causing RCU stall warnings or watchdog timeouts. BPF schedulers using SCX_KICK_WAIT must ensure the kicked CPU is free to schedule promptly.
SCX_KICK_WAIT does not guarantee task pickup: The wait ensures the target CPU went through __pick_next_task(), but does not guarantee it picked the specific task you dispatched. If the target CPU picks a higher-priority non-SCX task, your dispatched task remains in the local DSQ. SCX_KICK_PREEMPT | SCX_KICK_WAIT together provide the strongest guarantee: force preemption and wait for it to complete, but even then the task selection is subject to normal priority ordering.
Debug signal: A non-empty cpus_to_wait in a post-mortem dump means a CPU died mid-wait. Cross-reference the pnt_seq values — if the waited-for CPU's pnt_seq equals the snapshot value, the target CPU never completed the scheduling cycle, pointing to a hang or panic on the target CPU during or before __pick_next_task().
Per-CPU snapshot array sizing: scx_kick_cpus_pnt_seqs is allocated as nr_cpu_ids entries per CPU. On systems with many CPUs, this is a significant allocation (e.g., 1024 CPUs × 1024 entries × 8 bytes = 8 MB). If nr_cpu_ids is ever changed after boot or hotplug changes the CPU layout, the array would be stale. Currently this is safe because init_sched_ext_class() uses nr_cpu_ids at boot, but maintainers should watch for any dynamic CPU topology changes.
Connection to Other Patches
-
PATCH 17/30 (scx_bpf_kick_cpu infrastructure): This patch builds directly on the kick infrastructure from PATCH 17, which introduced
cpus_to_kick,cpus_to_preempt,kick_cpus_irq_workfn(), andkick_one_cpu().SCX_KICK_WAITadds the fourth cpumask (cpus_to_wait) to the same pattern and extendskick_one_cpu()with snapshot logic. -
PATCH 19/30 (dispatch loop watchdog): Both patches deal with bounded execution in
balance_scx()/kick_cpus_irq_workfn(). The dispatch loop watchdog limits iterations withSCX_DSP_MAX_LOOPS;SCX_KICK_WAITintroduces bounded spinning in irq_work. Together they represent the two places where BPF scheduler misbehavior can stall the kernel, and both have mitigations. -
PATCH 18/30 (post-mortem dump): The
pnt_seqandcpus_to_waitadditions toscx_dump_state()make this patch's debugging artifacts visible in the dump infrastructure introduced in PATCH 18. A complete dump now shows whether a CPU was actively waiting when the scheduler exited. -
scx_pairexample scheduler: The reference implementation forSCX_KICK_WAITisscx_pair.bpf.cin the scx repository. It pairs two CPUs and usesSCX_KICK_PREEMPT | SCX_KICK_WAITto synchronize task placement across the pair, ensuring both CPUs of a pair always run tasks from the same cgroup simultaneously.
System Integration (Patches 24–28)
Overview
A production scheduler must coexist with the full complexity of a real Linux system: CPUs can be hot-plugged in and out, power management events suspend the machine, SMT siblings share execution resources with security implications, and some workloads demand weighted fair scheduling rather than simple FIFO. This group of five patches integrates sched_ext with these system-level concerns.
Patches 24–25 handle CPU topology changes. Patch 26 handles power management. Patch 27 integrates with Linux's core scheduling (CONFIG_SCHED_CORE) for SMT security. Patch 28 adds virtual time ordering to DSQs, enabling weighted fair scheduling in BPF.
These patches are the most "outward-facing" in the sched_ext series: each one connects sched_ext to a pre-existing kernel subsystem with its own conventions, locks, and lifecycle rules. For a maintainer, understanding how sched_ext integrates with each subsystem is as important as understanding sched_ext itself.
Why These Patches Are Needed
CPU Topology Changes
Linux supports CPU hotplug: CPUs can be brought online and offline at runtime (for power saving, hardware maintenance, or virtualization). A BPF scheduler that maintains a bitmask of online CPUs, or builds per-CPU data structures at init time, will malfunction if CPUs are added or removed without notification.
Additionally, even among always-present CPUs, the question of "which CPUs are owned by sched_ext vs CFS" changes dynamically. When CFS and RT have no runnable tasks on a CPU, that CPU's capacity is available exclusively to sched_ext. When CFS or RT gains runnable tasks, they reclaim the CPU.
Patches 24 and 25 provide BPF programs with notifications for both of these changes.
Power Management
Linux's power management (PM) subsystem can suspend the entire system. During suspend, PM callbacks are called from process context with interrupts disabled or in unusual lock states. If a BPF scheduler is active during these callbacks, it may attempt to schedule tasks in contexts where scheduling is unsafe, causing deadlocks or kernel panics.
The correct response is to bypass the BPF scheduler during PM transitions and fall back to direct CFS dispatch. Patch 26 implements this bypass.
Core Scheduling (SMT Security)
Simultaneous Multi-Threading (SMT) allows multiple hardware threads to share a physical core's execution resources. This is a security concern: a malicious thread on one hardware thread can potentially observe microarchitectural state (caches, branch predictor) from a thread running on the sibling hardware thread. Linux's "core scheduling" (CONFIG_SCHED_CORE) feature addresses this by ensuring that hardware thread siblings run tasks that trust each other (same "core tag").
Without sched_ext integration, a BPF scheduler could dispatch tasks to SMT sibling threads without respecting core scheduling constraints, undermining the security model.
Virtual Time Ordering
The base DSQ implementation (patch 09) is strictly FIFO: tasks are dispatched in the order they were enqueued. FIFO is simple and fair in the sense that every task waits its turn, but it does not respect task weights. A high-weight task (high CPU priority) should get more CPU time than a low-weight task, not just equal turns in the queue.
CFS implements weighted fair scheduling using a virtual time algorithm: each task's virtual clock advances at a rate inversely proportional to its weight. Tasks are ordered by virtual time, so lower-weight tasks advance their virtual time faster and are placed later in the queue. Patch 28 brings this same mechanism to sched_ext DSQs.
Key Concepts
PATCH 24 — ops.cpu_acquire() and ops.cpu_release()
These callbacks notify the BPF scheduler when a CPU transitions between "shared" (CFS/RT also have runnable tasks) and "exclusive to SCX" states.
ops.cpu_acquire(cpu, reason): Called when a CPU becomes exclusively available to the SCX
scheduler. This happens when:
- CFS and RT both have no runnable tasks on this CPU.
- The CPU was previously running a CFS/RT task that has now gone idle.
The reason argument encodes why the CPU became available (e.g., SCX_CPU_ACQUIRE_HOTPLUG
for a newly onlined CPU vs. SCX_CPU_ACQUIRE_IDLE for a CPU whose non-SCX tasks have drained).
ops.cpu_release(cpu, reason): Called when a CPU is being taken away from exclusive SCX
use. This happens when:
- A CFS or RT task becomes runnable on this CPU (it outranks SCX in priority).
- The CPU is being offlined (hotplug).
The BPF scheduler uses these callbacks to maintain a bitmask of "CPUs available to SCX" and to
make CPU selection decisions in ops.select_cpu(). A BPF scheduler that does not implement
these callbacks will have a static view of CPU availability and may dispatch tasks to CPUs that
CFS has reclaimed, causing tasks to run later than expected.
Implementation: The callbacks are called from sched_class->balance() and from the idle
loop when a CPU transitions between idle states. The lock ordering requires that these callbacks
be called with the runqueue lock held, so BPF programs implementing these callbacks must not
acquire any lock that nests inside the runqueue lock.
PATCH 25 — ops.cpu_online() and ops.cpu_offline()
While ops.cpu_acquire/release track whether a CPU is available to SCX at a given moment,
ops.cpu_online/offline track the fundamental presence of the CPU in the system.
ops.cpu_online(cpu): Called when a CPU is brought online via hotplug. The BPF scheduler
should update any CPU topology data structures (bitmasks, per-CPU arrays) to include this CPU.
ops.cpu_offline(cpu): Called when a CPU is being taken offline. The BPF scheduler must:
- Stop dispatching tasks to this CPU.
- Drain any tasks in this CPU's local DSQ —
scx_bpf_consume()must not be called for an offline CPU after this callback returns. - Free any per-CPU BPF state associated with this CPU.
Ordering with cpu_release: ops.cpu_offline() is always preceded by ops.cpu_release()
(the CPU is taken from SCX before it is removed from the system). A well-written BPF scheduler
uses cpu_release() to stop new dispatches and cpu_offline() to perform final cleanup.
Implementation challenges: CPU hotplug callbacks in the kernel follow a strict ordering
protocol (CPUHP states). sched_ext registers its own hotplug state (CPUHP_AP_SCX_ONLINE) at
a specific point in the hotplug sequence, ensuring that ops.cpu_online/offline are called at
the right time relative to other subsystem hotplug handlers. Reviewing future changes to the
hotplug registration point requires understanding the CPUHP state ordering.
PATCH 26 — PM Event Bypass
When the system enters suspend, pm_notifier callbacks fire. At this point:
- Interrupts may be disabled or in an unusual state.
- The PM code holds PM-specific locks.
- Normal task scheduling is supposed to be quiescent.
If a BPF scheduler receives an ops.enqueue() or ops.dispatch() call during PM transitions,
it may access BPF maps that are locked (because another CPU is in the BPF runtime), call
scx_bpf_kick_cpu() that sends IPIs to CPUs that are in the process of being halted, or
allocate memory in a context where allocation is forbidden.
Patch 26 implements a bypass mode triggered by PM notifiers:
- On
PM_SUSPEND_PREPARE(system is about to suspend),scx_pm_handler()setsscx_ops_bypassing()to true. - While bypassing,
enqueue_task_scx()andops.dispatch()skip the BPF callbacks entirely and fall back to CFS-like behavior: tasks are dispatched directly to the global DSQ in FIFO order, and the global DSQ is served by the idle CPU selection logic. - On
PM_POST_SUSPEND(system has resumed), bypass mode is cleared.
The bypass mechanism is the same one used during scx_ops_disable_workfn() (the error exit
path). This reuse is intentional: both situations require "safe scheduling without BPF", and
consolidating them in one bypass path reduces the number of special cases in the enqueue/dispatch
hot paths.
Why not just call scx_ops_disable() on suspend? Disabling the BPF scheduler on every suspend/resume cycle would force the user to reload the BPF scheduler after every resume, which is expensive and inconvenient. Bypass mode preserves the scheduler's registration and BPF state, allowing it to resume operation immediately after the system wakes up.
PATCH 27 — Core Scheduling Integration
Linux's core scheduling (CONFIG_SCHED_CORE) ensures that when SMT siblings run tasks, those tasks have the same "core tag" (indicating they trust each other). Without SCX integration:
- A BPF scheduler could dispatch an untrusted task to a CPU whose SMT sibling is running a sensitive task.
- The microarchitectural side-channel isolation provided by core scheduling would be violated.
Patch 27 integrates sched_ext with the core scheduling framework:
sched_core_pick(): When the kernel selects the next task for a CPU that has SMT siblings,
it calls sched_core_pick() to verify that the chosen task has a compatible core tag with the
task running on sibling threads. For sched_ext, this means verifying that the task the BPF
scheduler wants to run is core-tag compatible.
scx_pick_task_cookie(): A sched_ext-specific function that the core scheduling framework
calls to get the core tag ("cookie" in core scheduling terminology) of the task at the head of
a CPU's local DSQ. This allows the core scheduler to evaluate compatibility before committing
to running the task.
BPF program implications: A BPF scheduler that does not set core tags on tasks will behave
as if all tasks have the same core tag (the default tag 0), which means they are all considered
compatible. This is correct for environments where core scheduling is not needed. For
environments that require core scheduling for security, the BPF scheduler must use
sched_core_put_cookie()/sched_core_get_cookie() BPF helpers (added alongside this patch)
to manage task core tags.
Lock ordering: Core scheduling uses its own locks (sched_core_lock) that nest under the
runqueue lock. The sched_ext integration must respect this nesting. Any future change to how
sched_ext interacts with sched_core_pick() must verify the lock ordering.
PATCH 28 — DSQ Virtual Time Ordering
Patch 28 adds scx_bpf_dispatch_vtime(p, dsq_id, vtime, slice, enq_flags), a variant of
scx_bpf_dispatch() that associates a virtual time key with the dispatch. Within a DSQ, tasks
are ordered by virtual time: tasks with lower vtime values are dispatched first.
Virtual time semantics: vtime is a u64 value. The BPF scheduler is responsible for
computing meaningful vtime values. A common pattern (implementing weighted fair scheduling):
new_vtime = current_vtime + (time_slice / task_weight)
Where task_weight is proportional to the task's nice value or cgroup weight. High-weight tasks
advance their vtime slowly (they are always near the front of the queue); low-weight tasks
advance it quickly (they are pushed toward the back).
scx_bpf_now(): Also added in this patch, this helper returns the current monotonic time
as a u64, giving BPF programs a time source for vtime calculations without requiring a
full ktime_get() call.
DSQ ordering invariant: Within a single DSQ, a BPF program must use either all-FIFO
dispatches or all-vtime dispatches — mixing them is undefined behavior. The kernel enforces
this by checking whether the DSQ has existing FIFO entries when a vtime dispatch arrives and
vice versa. Violating this invariant triggers scx_ops_error().
Relationship to CFS vruntime: CFS's vruntime and sched_ext's DSQ vtime are conceptually
similar but technically independent. CFS's vruntime is managed by the kernel; DSQ vtime is
managed entirely by the BPF program. The BPF program can use scx_bpf_task_cgroup_weight() to
get a task's CFS-compatible weight for computing vtime values that match CFS's fairness model.
Per-DSQ vtime tracking: Each DSQ has a min_vtime field that tracks the minimum vtime
of any task in the DSQ. This allows BPF programs to initialize new tasks' vtimes to min_vtime
(preventing new tasks from immediately jumping to the front of the queue with vtime = 0).
scx_bpf_dsq_vtime_anchor(dsq_id) returns the current min_vtime for a DSQ.
Connections Between Patches
PATCH 24 (cpu_acquire/release)
└─→ Interacts with PATCH 25: acquire/release happen around online/offline transitions
└─→ Calls must respect PATCH 17's IPI mechanism (notifying CPUs of state change)
PATCH 25 (cpu_online/offline)
└─→ Pairs with PATCH 24: offline always preceded by release
└─→ Interacts with PATCH 21 (tickless): offline CPUs don't participate in tick logic
PATCH 26 (PM bypass)
└─→ Reuses the bypass mechanism from PATCH 09's scx_ops_disable path
└─→ Interacts with PATCH 22 (in_op_task): in-flight operations must complete before
bypass mode activates
PATCH 27 (core scheduling)
└─→ Affects PATCH 09's pick_next_task_scx(): adds core tag compatibility check
└─→ Interacts with PATCH 24: core scheduling constraints apply on acquired CPUs too
PATCH 28 (vtime DSQ)
└─→ Extends PATCH 09's DSQ implementation with a new ordering mode
└─→ Interacts with PATCH 20 (ops.running): vtime accounting typically happens in
ops.running/stopping to update the task's vtime key
What to Focus On
For a maintainer, the critical lessons from this group:
-
CPU hotplug ordering and CPUHP states. The
ops.cpu_online/offlinecallbacks are registered at a specific CPUHP state. The exact state matters: if registered too early, the CPU's per-CPU data structures may not be initialized; too late, and the CPU may start running tasks before the BPF scheduler knows about it. When reviewing changes to the hotplug registration point, verify against the full CPUHP state list and the ordering requirements of other subsystems that sched_ext depends on. -
Bypass mode as a safety invariant. Bypass mode (patch 26) is the mechanism by which sched_ext guarantees safe operation during system transitions. It is used in three places: PM suspend,
scx_ops_disable_workfn(), and brief transition periods duringscx_ops_enable(). When reviewing changes to any of these three paths, verify that bypass mode is entered before BPF callbacks become unsafe and exited only after they become safe again. -
Core scheduling and the cookie model. Core scheduling in sched_ext uses the same cookie model as CFS (same
struct sched_core_cookie). A BPF scheduler does not need to understand the details of core scheduling to be correct — it just needs to not prevent the kernel from performing the compatibility check. The risk is in changes toscx_pick_task_cookie()or the DSQ iteration code that might inadvertently skip the compatibility check for some tasks. -
vtime overflow. DSQ vtime is a
u64. BPF programs that compute vtime asaccumulated_runtime / weightwill eventually overflow if the BPF scheduler runs long enough. The conventional approach is to use relative vtime (difference frommin_vtimerather than absolute elapsed time) and to periodically reset the anchor. When reviewing BPF schedulers that use vtime DSQs, check for overflow handling. -
The acquire/release and online/offline orthogonality.
cpu_acquire/releaseandcpu_online/offlineare conceptually distinct and should not be confused. A CPU can be online but released (CFS has taken it back). A CPU can be offline (not available at all). BPF programs must track both states independently. When reviewing the sched_ext hotplug and CPU state code, verify that the four combinations (online+acquired, online+released, offline+acquired-during-transition, offline) are all handled correctly.
[PATCH 24/30] sched_ext: Implement sched_ext_ops.cpu_acquire/release()
View on Lore: https://lore.kernel.org/all/20240618212056.2833381-25-tj@kernel.org
Commit Message
From: David Vernet <dvernet@meta.com>
Scheduler classes are strictly ordered and when a higher priority class has
tasks to run, the lower priority ones lose access to the CPU. Being able to
monitor and act on these events are necessary for use cases includling
strict core-scheduling and latency management.
This patch adds two operations ops.cpu_acquire() and .cpu_release(). The
former is invoked when a CPU becomes available to the BPF scheduler and the
opposite for the latter. This patch also implements
scx_bpf_reenqueue_local() which can be called from .cpu_release() to trigger
requeueing of all tasks in the local dsq of the CPU so that the tasks can be
reassigned to other available CPUs.
scx_pair is updated to use .cpu_acquire/release() along with
%SCX_KICK_WAIT to make the pair scheduling guarantee strict even when a CPU
is preempted by a higher priority scheduler class.
scx_qmap is updated to use .cpu_acquire/release() to empty the local
dsq of a preempted CPU. A similar approach can be adopted by BPF schedulers
that want to have a tight control over latency.
v4: Use the new SCX_KICK_IDLE to wake up a CPU after re-enqueueing.
v3: Drop the const qualifier from scx_cpu_release_args.task. BPF enforces
access control through the verifier, so the qualifier isn't actually
operative and only gets in the way when interacting with various
helpers.
v2: Add p->scx.kf_mask annotation to allow calling scx_bpf_reenqueue_local()
from ops.cpu_release() nested inside ops.init() and other sleepable
operations.
Signed-off-by: David Vernet <dvernet@meta.com>
Reviewed-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
Acked-by: Josh Don <joshdon@google.com>
Acked-by: Hao Luo <haoluo@google.com>
Acked-by: Barret Rhoden <brho@google.com>
---
include/linux/sched/ext.h | 4 +-
kernel/sched/ext.c | 198 ++++++++++++++++++++++-
kernel/sched/ext.h | 2 +
kernel/sched/sched.h | 1 +
tools/sched_ext/include/scx/common.bpf.h | 1 +
tools/sched_ext/scx_qmap.bpf.c | 37 ++++-
tools/sched_ext/scx_qmap.c | 4 +-
7 files changed, 240 insertions(+), 7 deletions(-)
diff --git a/include/linux/sched/ext.h b/include/linux/sched/ext.h
index 74341dbc6a19..21c627337e01 100644
--- a/include/linux/sched/ext.h
+++ b/include/linux/sched/ext.h
@@ -98,13 +98,15 @@ enum scx_kf_mask {
SCX_KF_UNLOCKED = 0, /* not sleepable, not rq locked */
/* all non-sleepables may be nested inside SLEEPABLE */
SCX_KF_SLEEPABLE = 1 << 0, /* sleepable init operations */
+ /* ENQUEUE and DISPATCH may be nested inside CPU_RELEASE */
+ SCX_KF_CPU_RELEASE = 1 << 1, /* ops.cpu_release() */
/* ops.dequeue (in REST) may be nested inside DISPATCH */
SCX_KF_DISPATCH = 1 << 2, /* ops.dispatch() */
SCX_KF_ENQUEUE = 1 << 3, /* ops.enqueue() and ops.select_cpu() */
SCX_KF_SELECT_CPU = 1 << 4, /* ops.select_cpu() */
SCX_KF_REST = 1 << 5, /* other rq-locked operations */
- __SCX_KF_RQ_LOCKED = SCX_KF_DISPATCH |
+ __SCX_KF_RQ_LOCKED = SCX_KF_CPU_RELEASE | SCX_KF_DISPATCH |
SCX_KF_ENQUEUE | SCX_KF_SELECT_CPU | SCX_KF_REST,
__SCX_KF_TERMINAL = SCX_KF_ENQUEUE | SCX_KF_SELECT_CPU | SCX_KF_REST,
};
diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c
index 1ca3067b4e0a..686dab6ab592 100644
--- a/kernel/sched/ext.c
+++ b/kernel/sched/ext.c
@@ -110,6 +110,32 @@ struct scx_exit_task_args {
bool cancelled;
};
+enum scx_cpu_preempt_reason {
+ /* next task is being scheduled by &sched_class_rt */
+ SCX_CPU_PREEMPT_RT,
+ /* next task is being scheduled by &sched_class_dl */
+ SCX_CPU_PREEMPT_DL,
+ /* next task is being scheduled by &sched_class_stop */
+ SCX_CPU_PREEMPT_STOP,
+ /* unknown reason for SCX being preempted */
+ SCX_CPU_PREEMPT_UNKNOWN,
+};
+
+/*
+ * Argument container for ops->cpu_acquire(). Currently empty, but may be
+ * expanded in the future.
+ */
+struct scx_cpu_acquire_args {};
+
+/* argument container for ops->cpu_release() */
+struct scx_cpu_release_args {
+ /* the reason the CPU was preempted */
+ enum scx_cpu_preempt_reason reason;
+
+ /* the task that's going to be scheduled on the CPU */
+ struct task_struct *task;
+};
+
/*
* Informational context provided to dump operations.
*/
@@ -335,6 +361,28 @@ struct sched_ext_ops {
*/
void (*update_idle)(s32 cpu, bool idle);
+ /**
+ * cpu_acquire - A CPU is becoming available to the BPF scheduler
+ * @cpu: The CPU being acquired by the BPF scheduler.
+ * @args: Acquire arguments, see the struct definition.
+ *
+ * A CPU that was previously released from the BPF scheduler is now once
+ * again under its control.
+ */
+ void (*cpu_acquire)(s32 cpu, struct scx_cpu_acquire_args *args);
+
+ /**
+ * cpu_release - A CPU is taken away from the BPF scheduler
+ * @cpu: The CPU being released by the BPF scheduler.
+ * @args: Release arguments, see the struct definition.
+ *
+ * The specified CPU is no longer under the control of the BPF
+ * scheduler. This could be because it was preempted by a higher
+ * priority sched_class, though there may be other reasons as well. The
+ * caller should consult @args->reason to determine the cause.
+ */
+ void (*cpu_release)(s32 cpu, struct scx_cpu_release_args *args);
+
/**
* init_task - Initialize a task to run in a BPF scheduler
* @p: task to initialize for BPF scheduling
@@ -487,6 +535,17 @@ enum scx_enq_flags {
*/
SCX_ENQ_PREEMPT = 1LLU << 32,
+ /*
+ * The task being enqueued was previously enqueued on the current CPU's
+ * %SCX_DSQ_LOCAL, but was removed from it in a call to the
+ * bpf_scx_reenqueue_local() kfunc. If bpf_scx_reenqueue_local() was
+ * invoked in a ->cpu_release() callback, and the task is again
+ * dispatched back to %SCX_LOCAL_DSQ by this current ->enqueue(), the
+ * task will not be scheduled on the CPU until at least the next invocation
+ * of the ->cpu_acquire() callback.
+ */
+ SCX_ENQ_REENQ = 1LLU << 40,
+
/*
* The task being enqueued is the only task available for the cpu. By
* default, ext core keeps executing such tasks but when
@@ -625,6 +684,7 @@ static bool scx_warned_zero_slice;
static DEFINE_STATIC_KEY_FALSE(scx_ops_enq_last);
static DEFINE_STATIC_KEY_FALSE(scx_ops_enq_exiting);
+DEFINE_STATIC_KEY_FALSE(scx_ops_cpu_preempt);
static DEFINE_STATIC_KEY_FALSE(scx_builtin_idle_enabled);
struct static_key_false scx_has_op[SCX_OPI_END] =
@@ -887,6 +947,12 @@ static __always_inline bool scx_kf_allowed(u32 mask)
* inside ops.dispatch(). We don't need to check the SCX_KF_SLEEPABLE
* boundary thanks to the above in_interrupt() check.
*/
+ if (unlikely(highest_bit(mask) == SCX_KF_CPU_RELEASE &&
+ (current->scx.kf_mask & higher_bits(SCX_KF_CPU_RELEASE)))) {
+ scx_ops_error("cpu_release kfunc called from a nested operation");
+ return false;
+ }
+
if (unlikely(highest_bit(mask) == SCX_KF_DISPATCH &&
(current->scx.kf_mask & higher_bits(SCX_KF_DISPATCH)))) {
scx_ops_error("dispatch kfunc called from a nested operation");
@@ -2070,6 +2136,19 @@ static int balance_scx(struct rq *rq, struct task_struct *prev,
lockdep_assert_rq_held(rq);
rq->scx.flags |= SCX_RQ_BALANCING;
+ if (static_branch_unlikely(&scx_ops_cpu_preempt) &&
+ unlikely(rq->scx.cpu_released)) {
+ /*
+ * If the previous sched_class for the current CPU was not SCX,
+ * notify the BPF scheduler that it again has control of the
+ * core. This callback complements ->cpu_release(), which is
+ * emitted in scx_next_task_picked().
+ */
+ if (SCX_HAS_OP(cpu_acquire))
+ SCX_CALL_OP(0, cpu_acquire, cpu_of(rq), NULL);
+ rq->scx.cpu_released = false;
+ }
+
if (prev_on_scx) {
WARN_ON_ONCE(prev->scx.flags & SCX_TASK_BAL_KEEP);
update_curr_scx(rq);
@@ -2077,7 +2156,9 @@ static int balance_scx(struct rq *rq, struct task_struct *prev,
/*
* If @prev is runnable & has slice left, it has priority and
* fetching more just increases latency for the fetched tasks.
- * Tell put_prev_task_scx() to put @prev on local_dsq.
+ * Tell put_prev_task_scx() to put @prev on local_dsq. If the
+ * BPF scheduler wants to handle this explicitly, it should
+ * implement ->cpu_released().
*
* See scx_ops_disable_workfn() for the explanation on the
* bypassing test.
@@ -2297,6 +2378,20 @@ static struct task_struct *pick_next_task_scx(struct rq *rq)
return p;
}
+static enum scx_cpu_preempt_reason
+preempt_reason_from_class(const struct sched_class *class)
+{
+#ifdef CONFIG_SMP
+ if (class == &stop_sched_class)
+ return SCX_CPU_PREEMPT_STOP;
+#endif
+ if (class == &dl_sched_class)
+ return SCX_CPU_PREEMPT_DL;
+ if (class == &rt_sched_class)
+ return SCX_CPU_PREEMPT_RT;
+ return SCX_CPU_PREEMPT_UNKNOWN;
+}
+
void scx_next_task_picked(struct rq *rq, struct task_struct *p,
const struct sched_class *active)
{
@@ -2312,6 +2407,40 @@ void scx_next_task_picked(struct rq *rq, struct task_struct *p,
*/
smp_store_release(&rq->scx.pnt_seq, rq->scx.pnt_seq + 1);
#endif
+ if (!static_branch_unlikely(&scx_ops_cpu_preempt))
+ return;
+
+ /*
+ * The callback is conceptually meant to convey that the CPU is no
+ * longer under the control of SCX. Therefore, don't invoke the
+ * callback if the CPU is is staying on SCX, or going idle (in which
+ * case the SCX scheduler has actively decided not to schedule any
+ * tasks on the CPU).
+ */
+ if (likely(active >= &ext_sched_class))
+ return;
+
+ /*
+ * At this point we know that SCX was preempted by a higher priority
+ * sched_class, so invoke the ->cpu_release() callback if we have not
+ * done so already. We only send the callback once between SCX being
+ * preempted, and it regaining control of the CPU.
+ *
+ * ->cpu_release() complements ->cpu_acquire(), which is emitted the
+ * next time that balance_scx() is invoked.
+ */
+ if (!rq->scx.cpu_released) {
+ if (SCX_HAS_OP(cpu_release)) {
+ struct scx_cpu_release_args args = {
+ .reason = preempt_reason_from_class(active),
+ .task = p,
+ };
+
+ SCX_CALL_OP(SCX_KF_CPU_RELEASE,
+ cpu_release, cpu_of(rq), &args);
+ }
+ rq->scx.cpu_released = true;
+ }
}
#ifdef CONFIG_SMP
@@ -3398,6 +3527,7 @@ static void scx_ops_disable_workfn(struct kthread_work *work)
static_branch_disable_cpuslocked(&scx_has_op[i]);
static_branch_disable_cpuslocked(&scx_ops_enq_last);
static_branch_disable_cpuslocked(&scx_ops_enq_exiting);
+ static_branch_disable_cpuslocked(&scx_ops_cpu_preempt);
static_branch_disable_cpuslocked(&scx_builtin_idle_enabled);
synchronize_rcu();
@@ -3699,9 +3829,10 @@ static void scx_dump_state(struct scx_exit_info *ei, size_t dump_len)
seq_buf_init(&ns, buf, avail);
dump_newline(&ns);
- dump_line(&ns, "CPU %-4d: nr_run=%u flags=0x%x ops_qseq=%lu pnt_seq=%lu",
+ dump_line(&ns, "CPU %-4d: nr_run=%u flags=0x%x cpu_rel=%d ops_qseq=%lu pnt_seq=%lu",
cpu, rq->scx.nr_running, rq->scx.flags,
- rq->scx.ops_qseq, rq->scx.pnt_seq);
+ rq->scx.cpu_released, rq->scx.ops_qseq,
+ rq->scx.pnt_seq);
dump_line(&ns, " curr=%s[%d] class=%ps",
rq->curr->comm, rq->curr->pid,
rq->curr->sched_class);
@@ -3942,6 +4073,8 @@ static int scx_ops_enable(struct sched_ext_ops *ops, struct bpf_link *link)
if (ops->flags & SCX_OPS_ENQ_EXITING)
static_branch_enable_cpuslocked(&scx_ops_enq_exiting);
+ if (scx_ops.cpu_acquire || scx_ops.cpu_release)
+ static_branch_enable_cpuslocked(&scx_ops_cpu_preempt);
if (!ops->update_idle || (ops->flags & SCX_OPS_KEEP_BUILTIN_IDLE)) {
reset_idle_masks();
@@ -4318,6 +4451,8 @@ static bool yield_stub(struct task_struct *from, struct task_struct *to) { retur
static void set_weight_stub(struct task_struct *p, u32 weight) {}
static void set_cpumask_stub(struct task_struct *p, const struct cpumask *mask) {}
static void update_idle_stub(s32 cpu, bool idle) {}
+static void cpu_acquire_stub(s32 cpu, struct scx_cpu_acquire_args *args) {}
+static void cpu_release_stub(s32 cpu, struct scx_cpu_release_args *args) {}
static s32 init_task_stub(struct task_struct *p, struct scx_init_task_args *args) { return -EINVAL; }
static void exit_task_stub(struct task_struct *p, struct scx_exit_task_args *args) {}
static void enable_stub(struct task_struct *p) {}
@@ -4338,6 +4473,8 @@ static struct sched_ext_ops __bpf_ops_sched_ext_ops = {
.set_weight = set_weight_stub,
.set_cpumask = set_cpumask_stub,
.update_idle = update_idle_stub,
+ .cpu_acquire = cpu_acquire_stub,
+ .cpu_release = cpu_release_stub,
.init_task = init_task_stub,
.exit_task = exit_task_stub,
.enable = enable_stub,
@@ -4870,6 +5007,59 @@ static const struct btf_kfunc_id_set scx_kfunc_set_dispatch = {
__bpf_kfunc_start_defs();
+/**
+ * scx_bpf_reenqueue_local - Re-enqueue tasks on a local DSQ
+ *
+ * Iterate over all of the tasks currently enqueued on the local DSQ of the
+ * caller's CPU, and re-enqueue them in the BPF scheduler. Returns the number of
+ * processed tasks. Can only be called from ops.cpu_release().
+ */
+__bpf_kfunc u32 scx_bpf_reenqueue_local(void)
+{
+ u32 nr_enqueued, i;
+ struct rq *rq;
+
+ if (!scx_kf_allowed(SCX_KF_CPU_RELEASE))
+ return 0;
+
+ rq = cpu_rq(smp_processor_id());
+ lockdep_assert_rq_held(rq);
+
+ /*
+ * Get the number of tasks on the local DSQ before iterating over it to
+ * pull off tasks. The enqueue callback below can signal that it wants
+ * the task to stay on the local DSQ, and we want to prevent the BPF
+ * scheduler from causing us to loop indefinitely.
+ */
+ nr_enqueued = rq->scx.local_dsq.nr;
+ for (i = 0; i < nr_enqueued; i++) {
+ struct task_struct *p;
+
+ p = first_local_task(rq);
+ WARN_ON_ONCE(atomic_long_read(&p->scx.ops_state) !=
+ SCX_OPSS_NONE);
+ WARN_ON_ONCE(!(p->scx.flags & SCX_TASK_QUEUED));
+ WARN_ON_ONCE(p->scx.holding_cpu != -1);
+ dispatch_dequeue(rq, p);
+ do_enqueue_task(rq, p, SCX_ENQ_REENQ, -1);
+ }
+
+ return nr_enqueued;
+}
+
+__bpf_kfunc_end_defs();
+
+BTF_KFUNCS_START(scx_kfunc_ids_cpu_release)
+BTF_ID_FLAGS(func, scx_bpf_reenqueue_local)
+BTF_KFUNCS_END(scx_kfunc_ids_cpu_release)
+
+static const struct btf_kfunc_id_set scx_kfunc_set_cpu_release = {
+ .owner = THIS_MODULE,
+ .set = &scx_kfunc_ids_cpu_release,
+};
+
+__bpf_kfunc_start_defs();
+
/**
* scx_bpf_kick_cpu - Trigger reschedule on a CPU
* @cpu: cpu to kick
@@ -5379,6 +5569,8 @@ static int __init scx_init(void)
&scx_kfunc_set_enqueue_dispatch)) ||
(ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_STRUCT_OPS,
&scx_kfunc_set_dispatch)) ||
+ (ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_STRUCT_OPS,
+ &scx_kfunc_set_cpu_release)) ||
(ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_STRUCT_OPS,
&scx_kfunc_set_any)) ||
(ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_TRACING,
diff --git a/kernel/sched/ext.h b/kernel/sched/ext.h
index 0aeb1fda1794..4ebd1c2478f1 100644
--- a/kernel/sched/ext.h
+++ b/kernel/sched/ext.h
@@ -24,6 +24,8 @@ DECLARE_STATIC_KEY_FALSE(__scx_switched_all);
#define scx_enabled() static_branch_unlikely(&__scx_ops_enabled)
#define scx_switched_all() static_branch_unlikely(&__scx_switched_all)
+DECLARE_STATIC_KEY_FALSE(scx_ops_cpu_preempt);
+
static inline bool task_on_scx(const struct task_struct *p)
{
return scx_enabled() && p->sched_class == &ext_sched_class;
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 734206e13897..147d18cf01ce 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -737,6 +737,7 @@ struct scx_rq {
u64 extra_enq_flags; /* see move_task_to_local_dsq() */
u32 nr_running;
u32 flags;
+ bool cpu_released;
cpumask_var_t cpus_to_kick;
cpumask_var_t cpus_to_kick_if_idle;
cpumask_var_t cpus_to_preempt;
diff --git a/tools/sched_ext/include/scx/common.bpf.h b/tools/sched_ext/include/scx/common.bpf.h
index 421118bc56ff..8686f84497db 100644
--- a/tools/sched_ext/include/scx/common.bpf.h
+++ b/tools/sched_ext/include/scx/common.bpf.h
@@ -34,6 +34,7 @@ void scx_bpf_dispatch(struct task_struct *p, u64 dsq_id, u64 slice, u64 enq_flag
u32 scx_bpf_dispatch_nr_slots(void) __ksym;
void scx_bpf_dispatch_cancel(void) __ksym;
bool scx_bpf_consume(u64 dsq_id) __ksym;
+u32 scx_bpf_reenqueue_local(void) __ksym;
void scx_bpf_kick_cpu(s32 cpu, u64 flags) __ksym;
s32 scx_bpf_dsq_nr_queued(u64 dsq_id) __ksym;
void scx_bpf_destroy_dsq(u64 dsq_id) __ksym;
diff --git a/tools/sched_ext/scx_qmap.bpf.c b/tools/sched_ext/scx_qmap.bpf.c
index 879fc9c788e5..4a87377558c8 100644
--- a/tools/sched_ext/scx_qmap.bpf.c
+++ b/tools/sched_ext/scx_qmap.bpf.c
@@ -11,6 +11,8 @@
*
* - BPF-side queueing using PIDs.
* - Sleepable per-task storage allocation using ops.prep_enable().
+ * - Using ops.cpu_release() to handle a higher priority scheduling class taking
+ * the CPU away.
*
* This scheduler is primarily for demonstration and testing of sched_ext
* features and unlikely to be useful for actual workloads.
@@ -90,7 +92,7 @@ struct {
} cpu_ctx_stor SEC(".maps");
/* Statistics */
-u64 nr_enqueued, nr_dispatched, nr_dequeued;
+u64 nr_enqueued, nr_dispatched, nr_reenqueued, nr_dequeued;
s32 BPF_STRUCT_OPS(qmap_select_cpu, struct task_struct *p,
s32 prev_cpu, u64 wake_flags)
@@ -164,6 +166,22 @@ void BPF_STRUCT_OPS(qmap_enqueue, struct task_struct *p, u64 enq_flags)
return;
}
+ /*
+ * If the task was re-enqueued due to the CPU being preempted by a
+ * higher priority scheduling class, just re-enqueue the task directly
+ * on the global DSQ. As we want another CPU to pick it up, find and
+ * kick an idle CPU.
+ */
+ if (enq_flags & SCX_ENQ_REENQ) {
+ s32 cpu;
+
+ scx_bpf_dispatch(p, SHARED_DSQ, 0, enq_flags);
+ cpu = scx_bpf_pick_idle_cpu(p->cpus_ptr, 0);
+ if (cpu >= 0)
+ scx_bpf_kick_cpu(cpu, SCX_KICK_IDLE);
+ return;
+ }
+
ring = bpf_map_lookup_elem(&queue_arr, &idx);
if (!ring) {
scx_bpf_error("failed to find ring %d", idx);
@@ -257,6 +275,22 @@ void BPF_STRUCT_OPS(qmap_dispatch, s32 cpu, struct task_struct *prev)
}
}
+void BPF_STRUCT_OPS(qmap_cpu_release, s32 cpu, struct scx_cpu_release_args *args)
+{
+ u32 cnt;
+
+ /*
+ * Called when @cpu is taken by a higher priority scheduling class. This
+ * makes @cpu no longer available for executing sched_ext tasks. As we
+ * don't want the tasks in @cpu's local dsq to sit there until @cpu
+ * becomes available again, re-enqueue them into the global dsq. See
+ * %SCX_ENQ_REENQ handling in qmap_enqueue().
+ */
+ cnt = scx_bpf_reenqueue_local();
+ if (cnt)
+ __sync_fetch_and_add(&nr_reenqueued, cnt);
+}
+
s32 BPF_STRUCT_OPS(qmap_init_task, struct task_struct *p,
struct scx_init_task_args *args)
{
@@ -339,6 +373,7 @@ SCX_OPS_DEFINE(qmap_ops,
.enqueue = (void *)qmap_enqueue,
.dequeue = (void *)qmap_dequeue,
.dispatch = (void *)qmap_dispatch,
+ .cpu_release = (void *)qmap_cpu_release,
.init_task = (void *)qmap_init_task,
.dump = (void *)qmap_dump,
.dump_cpu = (void *)qmap_dump_cpu,
diff --git a/tools/sched_ext/scx_qmap.c b/tools/sched_ext/scx_qmap.c
index 594147a710a8..2a97421afe9a 100644
--- a/tools/sched_ext/scx_qmap.c
+++ b/tools/sched_ext/scx_qmap.c
@@ -112,9 +112,9 @@ int main(int argc, char **argv)
long nr_enqueued = skel->bss->nr_enqueued;
long nr_dispatched = skel->bss->nr_dispatched;
- printf("stats : enq=%lu dsp=%lu delta=%ld deq=%"PRIu64"\n",
+ printf("stats : enq=%lu dsp=%lu delta=%ld reenq=%"PRIu64" deq=%"PRIu64"\n",
nr_enqueued, nr_dispatched, nr_enqueued - nr_dispatched,
- skel->bss->nr_dequeued);
+ skel->bss->nr_reenqueued, skel->bss->nr_dequeued);
fflush(stdout);
sleep(1);
}
--
2.45.2
Diff
---
include/linux/sched/ext.h | 4 +-
kernel/sched/ext.c | 198 ++++++++++++++++++++++-
kernel/sched/ext.h | 2 +
kernel/sched/sched.h | 1 +
tools/sched_ext/include/scx/common.bpf.h | 1 +
tools/sched_ext/scx_qmap.bpf.c | 37 ++++-
tools/sched_ext/scx_qmap.c | 4 +-
7 files changed, 240 insertions(+), 7 deletions(-)
diff --git a/include/linux/sched/ext.h b/include/linux/sched/ext.h
index 74341dbc6a19..21c627337e01 100644
--- a/include/linux/sched/ext.h
+++ b/include/linux/sched/ext.h
@@ -98,13 +98,15 @@ enum scx_kf_mask {
SCX_KF_UNLOCKED = 0, /* not sleepable, not rq locked */
/* all non-sleepables may be nested inside SLEEPABLE */
SCX_KF_SLEEPABLE = 1 << 0, /* sleepable init operations */
+ /* ENQUEUE and DISPATCH may be nested inside CPU_RELEASE */
+ SCX_KF_CPU_RELEASE = 1 << 1, /* ops.cpu_release() */
/* ops.dequeue (in REST) may be nested inside DISPATCH */
SCX_KF_DISPATCH = 1 << 2, /* ops.dispatch() */
SCX_KF_ENQUEUE = 1 << 3, /* ops.enqueue() and ops.select_cpu() */
SCX_KF_SELECT_CPU = 1 << 4, /* ops.select_cpu() */
SCX_KF_REST = 1 << 5, /* other rq-locked operations */
- __SCX_KF_RQ_LOCKED = SCX_KF_DISPATCH |
+ __SCX_KF_RQ_LOCKED = SCX_KF_CPU_RELEASE | SCX_KF_DISPATCH |
SCX_KF_ENQUEUE | SCX_KF_SELECT_CPU | SCX_KF_REST,
__SCX_KF_TERMINAL = SCX_KF_ENQUEUE | SCX_KF_SELECT_CPU | SCX_KF_REST,
};
diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c
index 1ca3067b4e0a..686dab6ab592 100644
--- a/kernel/sched/ext.c
+++ b/kernel/sched/ext.c
@@ -110,6 +110,32 @@ struct scx_exit_task_args {
bool cancelled;
};
+enum scx_cpu_preempt_reason {
+ /* next task is being scheduled by &sched_class_rt */
+ SCX_CPU_PREEMPT_RT,
+ /* next task is being scheduled by &sched_class_dl */
+ SCX_CPU_PREEMPT_DL,
+ /* next task is being scheduled by &sched_class_stop */
+ SCX_CPU_PREEMPT_STOP,
+ /* unknown reason for sched_ext being preempted */
+ SCX_CPU_PREEMPT_UNKNOWN,
+};
+
+/*
+ * Argument container for ops->cpu_acquire(). Currently empty, but may be
+ * expanded in the future.
+ */
+struct scx_cpu_acquire_args {};
+
+/* argument container for ops->cpu_release() */
+struct scx_cpu_release_args {
+ /* the reason the CPU was preempted */
+ enum scx_cpu_preempt_reason reason;
+
+ /* the task that's going to be scheduled on the CPU */
+ struct task_struct *task;
+};
+
/*
* Informational context provided to dump operations.
*/
@@ -335,6 +361,28 @@ struct sched_ext_ops {
*/
void (*update_idle)(s32 cpu, bool idle);
+ /**
+ * cpu_acquire - A CPU is becoming available to the BPF scheduler
+ * @cpu: The CPU being acquired by the BPF scheduler.
+ * @args: Acquire arguments, see the struct definition.
+ *
+ * A CPU that was previously released from the BPF scheduler is now once
+ * again under its control.
+ */
+ void (*cpu_acquire)(s32 cpu, struct scx_cpu_acquire_args *args);
+
+ /**
+ * cpu_release - A CPU is taken away from the BPF scheduler
+ * @cpu: The CPU being released by the BPF scheduler.
+ * @args: Release arguments, see the struct definition.
+ *
+ * The specified CPU is no longer under the control of the BPF
+ * scheduler. This could be because it was preempted by a higher
+ * priority sched_class, though there may be other reasons as well. The
+ * caller should consult @args->reason to determine the cause.
+ */
+ void (*cpu_release)(s32 cpu, struct scx_cpu_release_args *args);
+
/**
* init_task - Initialize a task to run in a BPF scheduler
* @p: task to initialize for BPF scheduling
@@ -487,6 +535,17 @@ enum scx_enq_flags {
*/
SCX_ENQ_PREEMPT = 1LLU << 32,
+ /*
+ * The task being enqueued was previously enqueued on the current CPU's
+ * %SCX_DSQ_LOCAL, but was removed from it in a call to the
+ * bpf_scx_reenqueue_local() kfunc. If bpf_scx_reenqueue_local() was
+ * invoked in a ->cpu_release() callback, and the task is again
+ * dispatched back to %SCX_LOCAL_DSQ by this current ->enqueue(), the
+ * task will not be scheduled on the CPU until at least the next invocation
+ * of the ->cpu_acquire() callback.
+ */
+ SCX_ENQ_REENQ = 1LLU << 40,
+
/*
* The task being enqueued is the only task available for the cpu. By
* default, ext core keeps executing such tasks but when
@@ -625,6 +684,7 @@ static bool scx_warned_zero_slice;
static DEFINE_STATIC_KEY_FALSE(scx_ops_enq_last);
static DEFINE_STATIC_KEY_FALSE(scx_ops_enq_exiting);
+DEFINE_STATIC_KEY_FALSE(scx_ops_cpu_preempt);
static DEFINE_STATIC_KEY_FALSE(scx_builtin_idle_enabled);
struct static_key_false scx_has_op[SCX_OPI_END] =
@@ -887,6 +947,12 @@ static __always_inline bool scx_kf_allowed(u32 mask)
* inside ops.dispatch(). We don't need to check the SCX_KF_SLEEPABLE
* boundary thanks to the above in_interrupt() check.
*/
+ if (unlikely(highest_bit(mask) == SCX_KF_CPU_RELEASE &&
+ (current->scx.kf_mask & higher_bits(SCX_KF_CPU_RELEASE)))) {
+ scx_ops_error("cpu_release kfunc called from a nested operation");
+ return false;
+ }
+
if (unlikely(highest_bit(mask) == SCX_KF_DISPATCH &&
(current->scx.kf_mask & higher_bits(SCX_KF_DISPATCH)))) {
scx_ops_error("dispatch kfunc called from a nested operation");
@@ -2070,6 +2136,19 @@ static int balance_scx(struct rq *rq, struct task_struct *prev,
lockdep_assert_rq_held(rq);
rq->scx.flags |= SCX_RQ_BALANCING;
+ if (static_branch_unlikely(&scx_ops_cpu_preempt) &&
+ unlikely(rq->scx.cpu_released)) {
+ /*
+ * If the previous sched_class for the current CPU was not sched_ext,
+ * notify the BPF scheduler that it again has control of the
+ * core. This callback complements ->cpu_release(), which is
+ * emitted in scx_next_task_picked().
+ */
+ if (SCX_HAS_OP(cpu_acquire))
+ SCX_CALL_OP(0, cpu_acquire, cpu_of(rq), NULL);
+ rq->scx.cpu_released = false;
+ }
+
if (prev_on_scx) {
WARN_ON_ONCE(prev->scx.flags & SCX_TASK_BAL_KEEP);
update_curr_scx(rq);
@@ -2077,7 +2156,9 @@ static int balance_scx(struct rq *rq, struct task_struct *prev,
/*
* If @prev is runnable & has slice left, it has priority and
* fetching more just increases latency for the fetched tasks.
- * Tell put_prev_task_scx() to put @prev on local_dsq.
+ * Tell put_prev_task_scx() to put @prev on local_dsq. If the
+ * BPF scheduler wants to handle this explicitly, it should
+ * implement ->cpu_released().
*
* See scx_ops_disable_workfn() for the explanation on the
* bypassing test.
@@ -2297,6 +2378,20 @@ static struct task_struct *pick_next_task_scx(struct rq *rq)
return p;
}
+static enum scx_cpu_preempt_reason
+preempt_reason_from_class(const struct sched_class *class)
+{
+#ifdef CONFIG_SMP
+ if (class == &stop_sched_class)
+ return SCX_CPU_PREEMPT_STOP;
+#endif
+ if (class == &dl_sched_class)
+ return SCX_CPU_PREEMPT_DL;
+ if (class == &rt_sched_class)
+ return SCX_CPU_PREEMPT_RT;
+ return SCX_CPU_PREEMPT_UNKNOWN;
+}
+
void scx_next_task_picked(struct rq *rq, struct task_struct *p,
const struct sched_class *active)
{
@@ -2312,6 +2407,40 @@ void scx_next_task_picked(struct rq *rq, struct task_struct *p,
*/
smp_store_release(&rq->scx.pnt_seq, rq->scx.pnt_seq + 1);
#endif
+ if (!static_branch_unlikely(&scx_ops_cpu_preempt))
+ return;
+
+ /*
+ * The callback is conceptually meant to convey that the CPU is no
+ * longer under the control of sched_ext. Therefore, don't invoke the
+ * callback if the CPU is is staying on sched_ext, or going idle (in which
+ * case the sched_ext scheduler has actively decided not to schedule any
+ * tasks on the CPU).
+ */
+ if (likely(active >= &ext_sched_class))
+ return;
+
+ /*
+ * At this point we know that sched_ext was preempted by a higher priority
+ * sched_class, so invoke the ->cpu_release() callback if we have not
+ * done so already. We only send the callback once between sched_ext being
+ * preempted, and it regaining control of the CPU.
+ *
+ * ->cpu_release() complements ->cpu_acquire(), which is emitted the
+ * next time that balance_scx() is invoked.
+ */
+ if (!rq->scx.cpu_released) {
+ if (SCX_HAS_OP(cpu_release)) {
+ struct scx_cpu_release_args args = {
+ .reason = preempt_reason_from_class(active),
+ .task = p,
+ };
+
+ SCX_CALL_OP(SCX_KF_CPU_RELEASE,
+ cpu_release, cpu_of(rq), &args);
+ }
+ rq->scx.cpu_released = true;
+ }
}
#ifdef CONFIG_SMP
@@ -3398,6 +3527,7 @@ static void scx_ops_disable_workfn(struct kthread_work *work)
static_branch_disable_cpuslocked(&scx_has_op[i]);
static_branch_disable_cpuslocked(&scx_ops_enq_last);
static_branch_disable_cpuslocked(&scx_ops_enq_exiting);
+ static_branch_disable_cpuslocked(&scx_ops_cpu_preempt);
static_branch_disable_cpuslocked(&scx_builtin_idle_enabled);
synchronize_rcu();
@@ -3699,9 +3829,10 @@ static void scx_dump_state(struct scx_exit_info *ei, size_t dump_len)
seq_buf_init(&ns, buf, avail);
dump_newline(&ns);
- dump_line(&ns, "CPU %-4d: nr_run=%u flags=0x%x ops_qseq=%lu pnt_seq=%lu",
+ dump_line(&ns, "CPU %-4d: nr_run=%u flags=0x%x cpu_rel=%d ops_qseq=%lu pnt_seq=%lu",
cpu, rq->scx.nr_running, rq->scx.flags,
- rq->scx.ops_qseq, rq->scx.pnt_seq);
+ rq->scx.cpu_released, rq->scx.ops_qseq,
+ rq->scx.pnt_seq);
dump_line(&ns, " curr=%s[%d] class=%ps",
rq->curr->comm, rq->curr->pid,
rq->curr->sched_class);
@@ -3942,6 +4073,8 @@ static int scx_ops_enable(struct sched_ext_ops *ops, struct bpf_link *link)
if (ops->flags & SCX_OPS_ENQ_EXITING)
static_branch_enable_cpuslocked(&scx_ops_enq_exiting);
+ if (scx_ops.cpu_acquire || scx_ops.cpu_release)
+ static_branch_enable_cpuslocked(&scx_ops_cpu_preempt);
if (!ops->update_idle || (ops->flags & SCX_OPS_KEEP_BUILTIN_IDLE)) {
reset_idle_masks();
@@ -4318,6 +4451,8 @@ static bool yield_stub(struct task_struct *from, struct task_struct *to) { retur
static void set_weight_stub(struct task_struct *p, u32 weight) {}
static void set_cpumask_stub(struct task_struct *p, const struct cpumask *mask) {}
static void update_idle_stub(s32 cpu, bool idle) {}
+static void cpu_acquire_stub(s32 cpu, struct scx_cpu_acquire_args *args) {}
+static void cpu_release_stub(s32 cpu, struct scx_cpu_release_args *args) {}
static s32 init_task_stub(struct task_struct *p, struct scx_init_task_args *args) { return -EINVAL; }
static void exit_task_stub(struct task_struct *p, struct scx_exit_task_args *args) {}
static void enable_stub(struct task_struct *p) {}
@@ -4338,6 +4473,8 @@ static struct sched_ext_ops __bpf_ops_sched_ext_ops = {
.set_weight = set_weight_stub,
.set_cpumask = set_cpumask_stub,
.update_idle = update_idle_stub,
+ .cpu_acquire = cpu_acquire_stub,
+ .cpu_release = cpu_release_stub,
.init_task = init_task_stub,
.exit_task = exit_task_stub,
.enable = enable_stub,
@@ -4870,6 +5007,59 @@ static const struct btf_kfunc_id_set scx_kfunc_set_dispatch = {
__bpf_kfunc_start_defs();
+/**
+ * scx_bpf_reenqueue_local - Re-enqueue tasks on a local DSQ
+ *
+ * Iterate over all of the tasks currently enqueued on the local DSQ of the
+ * caller's CPU, and re-enqueue them in the BPF scheduler. Returns the number of
+ * processed tasks. Can only be called from ops.cpu_release().
+ */
+__bpf_kfunc u32 scx_bpf_reenqueue_local(void)
+{
+ u32 nr_enqueued, i;
+ struct rq *rq;
+
+ if (!scx_kf_allowed(SCX_KF_CPU_RELEASE))
+ return 0;
+
+ rq = cpu_rq(smp_processor_id());
+ lockdep_assert_rq_held(rq);
+
+ /*
+ * Get the number of tasks on the local DSQ before iterating over it to
+ * pull off tasks. The enqueue callback below can signal that it wants
+ * the task to stay on the local DSQ, and we want to prevent the BPF
+ * scheduler from causing us to loop indefinitely.
+ */
+ nr_enqueued = rq->scx.local_dsq.nr;
+ for (i = 0; i < nr_enqueued; i++) {
+ struct task_struct *p;
+
+ p = first_local_task(rq);
+ WARN_ON_ONCE(atomic_long_read(&p->scx.ops_state) !=
+ SCX_OPSS_NONE);
+ WARN_ON_ONCE(!(p->scx.flags & SCX_TASK_QUEUED));
+ WARN_ON_ONCE(p->scx.holding_cpu != -1);
+ dispatch_dequeue(rq, p);
+ do_enqueue_task(rq, p, SCX_ENQ_REENQ, -1);
+ }
+
+ return nr_enqueued;
+}
+
+__bpf_kfunc_end_defs();
+
+BTF_KFUNCS_START(scx_kfunc_ids_cpu_release)
+BTF_ID_FLAGS(func, scx_bpf_reenqueue_local)
+BTF_KFUNCS_END(scx_kfunc_ids_cpu_release)
+
+static const struct btf_kfunc_id_set scx_kfunc_set_cpu_release = {
+ .owner = THIS_MODULE,
+ .set = &scx_kfunc_ids_cpu_release,
+};
+
+__bpf_kfunc_start_defs();
+
/**
* scx_bpf_kick_cpu - Trigger reschedule on a CPU
* @cpu: cpu to kick
@@ -5379,6 +5569,8 @@ static int __init scx_init(void)
&scx_kfunc_set_enqueue_dispatch)) ||
(ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_STRUCT_OPS,
&scx_kfunc_set_dispatch)) ||
+ (ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_STRUCT_OPS,
+ &scx_kfunc_set_cpu_release)) ||
(ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_STRUCT_OPS,
&scx_kfunc_set_any)) ||
(ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_TRACING,
diff --git a/kernel/sched/ext.h b/kernel/sched/ext.h
index 0aeb1fda1794..4ebd1c2478f1 100644
--- a/kernel/sched/ext.h
+++ b/kernel/sched/ext.h
@@ -24,6 +24,8 @@ DECLARE_STATIC_KEY_FALSE(__scx_switched_all);
#define scx_enabled() static_branch_unlikely(&__scx_ops_enabled)
#define scx_switched_all() static_branch_unlikely(&__scx_switched_all)
+DECLARE_STATIC_KEY_FALSE(scx_ops_cpu_preempt);
+
static inline bool task_on_scx(const struct task_struct *p)
{
return scx_enabled() && p->sched_class == &ext_sched_class;
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 734206e13897..147d18cf01ce 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -737,6 +737,7 @@ struct scx_rq {
u64 extra_enq_flags; /* see move_task_to_local_dsq() */
u32 nr_running;
u32 flags;
+ bool cpu_released;
cpumask_var_t cpus_to_kick;
cpumask_var_t cpus_to_kick_if_idle;
cpumask_var_t cpus_to_preempt;
diff --git a/tools/sched_ext/include/scx/common.bpf.h b/tools/sched_ext/include/scx/common.bpf.h
index 421118bc56ff..8686f84497db 100644
--- a/tools/sched_ext/include/scx/common.bpf.h
+++ b/tools/sched_ext/include/scx/common.bpf.h
@@ -34,6 +34,7 @@ void scx_bpf_dispatch(struct task_struct *p, u64 dsq_id, u64 slice, u64 enq_flag
u32 scx_bpf_dispatch_nr_slots(void) __ksym;
void scx_bpf_dispatch_cancel(void) __ksym;
bool scx_bpf_consume(u64 dsq_id) __ksym;
+u32 scx_bpf_reenqueue_local(void) __ksym;
void scx_bpf_kick_cpu(s32 cpu, u64 flags) __ksym;
s32 scx_bpf_dsq_nr_queued(u64 dsq_id) __ksym;
void scx_bpf_destroy_dsq(u64 dsq_id) __ksym;
diff --git a/tools/sched_ext/scx_qmap.bpf.c b/tools/sched_ext/scx_qmap.bpf.c
index 879fc9c788e5..4a87377558c8 100644
--- a/tools/sched_ext/scx_qmap.bpf.c
+++ b/tools/sched_ext/scx_qmap.bpf.c
@@ -11,6 +11,8 @@
*
* - BPF-side queueing using PIDs.
* - Sleepable per-task storage allocation using ops.prep_enable().
+ * - Using ops.cpu_release() to handle a higher priority scheduling class taking
+ * the CPU away.
*
* This scheduler is primarily for demonstration and testing of sched_ext
* features and unlikely to be useful for actual workloads.
@@ -90,7 +92,7 @@ struct {
} cpu_ctx_stor SEC(".maps");
/* Statistics */
-u64 nr_enqueued, nr_dispatched, nr_dequeued;
+u64 nr_enqueued, nr_dispatched, nr_reenqueued, nr_dequeued;
s32 BPF_STRUCT_OPS(qmap_select_cpu, struct task_struct *p,
s32 prev_cpu, u64 wake_flags)
@@ -164,6 +166,22 @@ void BPF_STRUCT_OPS(qmap_enqueue, struct task_struct *p, u64 enq_flags)
return;
}
+ /*
+ * If the task was re-enqueued due to the CPU being preempted by a
+ * higher priority scheduling class, just re-enqueue the task directly
+ * on the global DSQ. As we want another CPU to pick it up, find and
+ * kick an idle CPU.
+ */
+ if (enq_flags & SCX_ENQ_REENQ) {
+ s32 cpu;
+
+ scx_bpf_dispatch(p, SHARED_DSQ, 0, enq_flags);
+ cpu = scx_bpf_pick_idle_cpu(p->cpus_ptr, 0);
+ if (cpu >= 0)
+ scx_bpf_kick_cpu(cpu, SCX_KICK_IDLE);
+ return;
+ }
+
ring = bpf_map_lookup_elem(&queue_arr, &idx);
if (!ring) {
scx_bpf_error("failed to find ring %d", idx);
@@ -257,6 +275,22 @@ void BPF_STRUCT_OPS(qmap_dispatch, s32 cpu, struct task_struct *prev)
}
}
+void BPF_STRUCT_OPS(qmap_cpu_release, s32 cpu, struct scx_cpu_release_args *args)
+{
+ u32 cnt;
+
+ /*
+ * Called when @cpu is taken by a higher priority scheduling class. This
+ * makes @cpu no longer available for executing sched_ext tasks. As we
+ * don't want the tasks in @cpu's local dsq to sit there until @cpu
+ * becomes available again, re-enqueue them into the global dsq. See
+ * %SCX_ENQ_REENQ handling in qmap_enqueue().
+ */
+ cnt = scx_bpf_reenqueue_local();
+ if (cnt)
+ __sync_fetch_and_add(&nr_reenqueued, cnt);
+}
+
s32 BPF_STRUCT_OPS(qmap_init_task, struct task_struct *p,
struct scx_init_task_args *args)
{
@@ -339,6 +373,7 @@ SCX_OPS_DEFINE(qmap_ops,
.enqueue = (void *)qmap_enqueue,
.dequeue = (void *)qmap_dequeue,
.dispatch = (void *)qmap_dispatch,
+ .cpu_release = (void *)qmap_cpu_release,
.init_task = (void *)qmap_init_task,
.dump = (void *)qmap_dump,
.dump_cpu = (void *)qmap_dump_cpu,
diff --git a/tools/sched_ext/scx_qmap.c b/tools/sched_ext/scx_qmap.c
index 594147a710a8..2a97421afe9a 100644
--- a/tools/sched_ext/scx_qmap.c
+++ b/tools/sched_ext/scx_qmap.c
@@ -112,9 +112,9 @@ int main(int argc, char **argv)
long nr_enqueued = skel->bss->nr_enqueued;
long nr_dispatched = skel->bss->nr_dispatched;
- printf("stats : enq=%lu dsp=%lu delta=%ld deq=%"PRIu64"\n",
+ printf("stats : enq=%lu dsp=%lu delta=%ld reenq=%"PRIu64" deq=%"PRIu64"\n",
nr_enqueued, nr_dispatched, nr_enqueued - nr_dispatched,
- skel->bss->nr_dequeued);
+ skel->bss->nr_reenqueued, skel->bss->nr_dequeued);
fflush(stdout);
sleep(1);
}
--
2.45.2
Implementation Analysis
Overview
This patch (authored by David Vernet) introduces ops.cpu_acquire() and ops.cpu_release() — a pair of callbacks that let the BPF scheduler track when it gains and loses exclusive use of a CPU. When a higher-priority sched_class (RT, DL, stop) preempts sched_ext on a CPU, cpu_release() is called with the reason and the preempting task. When sched_ext regains the CPU, cpu_acquire() is called. It also introduces scx_bpf_reenqueue_local(), callable only from cpu_release(), which ejects all tasks from the local DSQ back into the BPF scheduler so they can be picked up by other CPUs rather than waiting.
Code Walkthrough
New argument structures (kernel/sched/ext.c):
enum scx_cpu_preempt_reason {
SCX_CPU_PREEMPT_RT,
SCX_CPU_PREEMPT_DL,
SCX_CPU_PREEMPT_STOP,
SCX_CPU_PREEMPT_UNKNOWN,
};
struct scx_cpu_acquire_args {}; /* empty, extensible */
struct scx_cpu_release_args {
enum scx_cpu_preempt_reason reason;
struct task_struct *task; /* the preempting task */
};
The scx_cpu_acquire_args struct is intentionally empty now but reserved for future expansion. The release args tell the BPF scheduler both why it lost the CPU and which task is taking it.
Ops definitions added to struct sched_ext_ops:
void (*cpu_acquire)(s32 cpu, struct scx_cpu_acquire_args *args);
void (*cpu_release)(s32 cpu, struct scx_cpu_release_args *args);
SCX_KF_CPU_RELEASE added to enum scx_kf_mask (include/linux/sched/ext.h):
SCX_KF_CPU_RELEASE = 1 << 1, /* ops.cpu_release() */
__SCX_KF_RQ_LOCKED = SCX_KF_CPU_RELEASE | SCX_KF_DISPATCH | ...
This gives cpu_release() its own kfunc permission slot. It is rq-locked and allows ENQUEUE and DISPATCH to be called nested inside it, but not other CPU_RELEASE calls.
rq->scx.cpu_released flag (kernel/sched/sched.h):
bool cpu_released;
Per-rq boolean tracking whether sched_ext has already fired cpu_release() for this preemption event. This prevents duplicate callbacks between preemption and the next balance_scx() call.
Where cpu_release() fires — scx_next_task_picked() in ext.c:
if (!rq->scx.cpu_released) {
if (SCX_HAS_OP(cpu_release)) {
struct scx_cpu_release_args args = {
.reason = preempt_reason_from_class(active),
.task = p,
};
SCX_CALL_OP(SCX_KF_CPU_RELEASE, cpu_release, cpu_of(rq), &args);
}
rq->scx.cpu_released = true;
}
This fires only when active < &ext_sched_class (a higher-priority class is taking over), not when sched_ext continues or the CPU goes idle.
Where cpu_acquire() fires — balance_scx():
if (static_branch_unlikely(&scx_ops_cpu_preempt) &&
unlikely(rq->scx.cpu_released)) {
if (SCX_HAS_OP(cpu_acquire))
SCX_CALL_OP(0, cpu_acquire, cpu_of(rq), NULL);
rq->scx.cpu_released = false;
}
scx_bpf_reenqueue_local() — the companion kfunc:
__bpf_kfunc u32 scx_bpf_reenqueue_local(void)
{
nr_enqueued = rq->scx.local_dsq.nr;
for (i = 0; i < nr_enqueued; i++) {
p = first_local_task(rq);
dispatch_dequeue(rq, p);
do_enqueue_task(rq, p, SCX_ENQ_REENQ, -1);
}
return nr_enqueued;
}
The snapshot of nr before the loop prevents infinite looping if a BPF scheduler re-dispatches tasks back to local DSQ. Tasks re-enqueued this way are tagged with SCX_ENQ_REENQ so the BPF scheduler knows they were ejected by a preemption.
Static key scx_ops_cpu_preempt (kernel/sched/ext.h):
DEFINE_STATIC_KEY_FALSE(scx_ops_cpu_preempt);
Enabled only when either cpu_acquire or cpu_release is set. Guards the hot-path preemption check so schedulers that don't implement these ops pay zero overhead.
scx_qmap example (tools/sched_ext/scx_qmap.bpf.c):
void BPF_STRUCT_OPS(qmap_cpu_release, s32 cpu, struct scx_cpu_release_args *args)
{
cnt = scx_bpf_reenqueue_local();
if (cnt)
__sync_fetch_and_add(&nr_reenqueued, cnt);
}
In qmap_enqueue(), tasks with SCX_ENQ_REENQ are dispatched to the global SHARED_DSQ and an idle CPU is kicked to pick them up immediately.
Debugging: cpu_rel=%d added to scx_dump_state() output so the cpu_released field is visible in crash dumps.
Key Concepts
ops.cpu_acquire(cpu, args): Called frombalance_scx()when sched_ext resumes control of a CPU that had been preempted. The args struct is currently empty.ops.cpu_release(cpu, args): Called fromscx_next_task_picked()when a higher-priority class takes the CPU.args->reasonis one ofSCX_CPU_PREEMPT_RT/DL/STOP/UNKNOWNandargs->taskis the preempting task.SCX_ENQ_REENQ(1LLU << 40): Flag on tasks that were ejected from local DSQ viascx_bpf_reenqueue_local(). If a scheduler dispatches such a task back to local DSQ, the kernel will not schedule it until the nextcpu_acquire().scx_ops_cpu_preemptstatic key: Zero-overhead guard for the preemption tracking code path. Only active when the BPF scheduler implements either callback.preempt_reason_from_class(): Maps the activesched_class*pointer to ascx_cpu_preempt_reasonenum value by comparing against&stop_sched_class,&dl_sched_class, and&rt_sched_class.
Locking and Concurrency Notes
cpu_release()is called under rq->lock (it is in__SCX_KF_RQ_LOCKED). The kfunc setscx_kfunc_set_cpu_releaseis registered withBPF_PROG_TYPE_STRUCT_OPSspecifically for this context.cpu_acquire()is called frombalance_scx()which also holds rq->lock.scx_bpf_reenqueue_local()asserts rq is held vialockdep_assert_rq_held(rq). It can only be called fromops.cpu_release()— calling it from any other context fails thescx_kf_allowed(SCX_KF_CPU_RELEASE)check.- The
cpu_releasedflag is per-rq and protected by rq->lock. Only set to true once per preemption event to prevent duplicatecpu_release()calls (e.g., if the preempting class yields and regains control multiple times before sched_ext runs again). SCX_KF_CPU_RELEASEhas a nesting check: attempting to call a CPU_RELEASE kfunc from within a CPU_RELEASE context fails with an error.
Integration with Kernel Subsystems
The hook point for cpu_release() is scx_next_task_picked(), which sits in core.c's pick_next_task() path immediately after the winning sched_class has been determined. The active parameter is the class that won, not the class of the picked task. This is a clean integration point because it is called exactly once per scheduling decision, after all class ordering is resolved.
The cpu_acquire() hook fires at the top of balance_scx(), which is called when sched_ext is the active class again and begins its balance-and-dispatch cycle. This pairing (release on yield, acquire on balance) ensures the BPF scheduler is never surprised by tasks running on CPUs it thinks it owns.
What Maintainers Need to Know
- The acquire/release pair is opt-in: neither callback is required. The static key means non-implementing schedulers pay zero cost. Enable the key by implementing either
cpu_acquireorcpu_releasein your ops. - Do not dispatch to a CPU's local DSQ from
cpu_release()without also callingscx_bpf_reenqueue_local()first — tasks left in the local DSQ will sit until the CPU is re-acquired, which may cause latency spikes. - The
SCX_ENQ_REENQflag inops.enqueue()is your signal that the task arrived viascx_bpf_reenqueue_local(). Dispatch it to a global or shared DSQ and kick an idle CPU to avoid it being stranded. scx_bpf_reenqueue_local()is restricted tocpu_release(). Attempting to call it fromops.enqueue(),ops.dispatch(), or elsewhere will fail the kfunc permission check and trigger an ops error.- The
argspointer passed tocpu_acquire()is currentlyNULL(passed asNULLin theSCX_CALL_OPinvocation). This is intentional: the struct is empty. Future patches may populate it. - Inspect
cpu_rel=%din debug dumps to verify whethercpu_releasedis stuck true, which would indicatebalance_scx()is not being called after a preemption event.
Connection to Other Patches
- Patch 23/30 (cpu_online/offline): Those callbacks cover permanent CPU topology changes (hotplug). This patch covers transient CPU ownership changes (higher-priority preemption). Together they give BPF schedulers complete visibility into CPU availability.
- Patch 27/30 (core-sched): Core scheduling can also take a CPU away from a sched_ext task. That path goes through
set_next_task_scx()withSCX_DEQ_CORE_SCHED_EXEC, not throughcpu_release(). - Patch 28/30 (vtime DSQs): A latency-sensitive BPF scheduler using vtime ordering may want to use
cpu_release()to drain local DSQs so vtime-ordered tasks are picked up with minimal delay by other CPUs.
[PATCH 25/30] sched_ext: Implement sched_ext_ops.cpu_online/offline()
View on Lore: https://lore.kernel.org/all/20240618212056.2833381-26-tj@kernel.org
Commit Message
Add ops.cpu_online/offline() which are invoked when CPUs come online and
offline respectively. As the enqueue path already automatically bypasses
tasks to the local dsq on a deactivated CPU, BPF schedulers are guaranteed
to see tasks only on CPUs which are between online() and offline().
If the BPF scheduler doesn't implement ops.cpu_online/offline(), the
scheduler is automatically exited with SCX_ECODE_RESTART |
SCX_ECODE_RSN_HOTPLUG. Userspace can implement CPU hotpplug support
trivially by simply reinitializing and reloading the scheduler.
scx_qmap is updated to print out online CPUs on hotplug events. Other
schedulers are updated to restart based on ecode.
v3: - The previous implementation added @reason to
sched_class.rq_on/offline() to distinguish between CPU hotplug events
and topology updates. This was buggy and fragile as the methods are
skipped if the current state equals the target state. Instead, add
scx_rq_[de]activate() which are directly called from
sched_cpu_de/activate(). This also allows ops.cpu_on/offline() to
sleep which can be useful.
- ops.dispatch() could be called on a CPU that the BPF scheduler was
told to be offline. The dispatch patch is updated to bypass in such
cases.
v2: - To accommodate lock ordering change between scx_cgroup_rwsem and
cpus_read_lock(), CPU hotplug operations are put into its own SCX_OPI
block and enabled eariler during scx_ope_enable() so that
cpus_read_lock() can be dropped before acquiring scx_cgroup_rwsem.
- Auto exit with ECODE added.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: David Vernet <dvernet@meta.com>
Acked-by: Josh Don <joshdon@google.com>
Acked-by: Hao Luo <haoluo@google.com>
Acked-by: Barret Rhoden <brho@google.com>
---
kernel/sched/core.c | 4 +
kernel/sched/ext.c | 156 ++++++++++++++++++-
kernel/sched/ext.h | 4 +
kernel/sched/sched.h | 6 +
tools/sched_ext/include/scx/compat.h | 26 ++++
tools/sched_ext/include/scx/user_exit_info.h | 28 ++++
tools/sched_ext/scx_central.c | 9 +-
tools/sched_ext/scx_qmap.bpf.c | 57 +++++++
tools/sched_ext/scx_qmap.c | 4 +
tools/sched_ext/scx_simple.c | 8 +-
10 files changed, 290 insertions(+), 12 deletions(-)
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 0e6ff33f34e4..c798c847d57e 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -7984,6 +7984,8 @@ int sched_cpu_activate(unsigned int cpu)
cpuset_cpu_active();
}
+ scx_rq_activate(rq);
+
/*
* Put the rq online, if not already. This happens:
*
@@ -8044,6 +8046,8 @@ int sched_cpu_deactivate(unsigned int cpu)
}
rq_unlock_irqrestore(rq, &rf);
+ scx_rq_deactivate(rq);
+
#ifdef CONFIG_SCHED_SMT
/*
* When going down, decrement the number of cores with SMT present.
diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c
index 686dab6ab592..7c2f2a542b32 100644
--- a/kernel/sched/ext.c
+++ b/kernel/sched/ext.c
@@ -30,6 +30,29 @@ enum scx_exit_kind {
SCX_EXIT_ERROR_STALL, /* watchdog detected stalled runnable tasks */
};
+/*
+ * An exit code can be specified when exiting with scx_bpf_exit() or
+ * scx_ops_exit(), corresponding to exit_kind UNREG_BPF and UNREG_KERN
+ * respectively. The codes are 64bit of the format:
+ *
+ * Bits: [63 .. 48 47 .. 32 31 .. 0]
+ * [ SYS ACT ] [ SYS RSN ] [ USR ]
+ *
+ * SYS ACT: System-defined exit actions
+ * SYS RSN: System-defined exit reasons
+ * USR : User-defined exit codes and reasons
+ *
+ * Using the above, users may communicate intention and context by ORing system
+ * actions and/or system reasons with a user-defined exit code.
+ */
+enum scx_exit_code {
+ /* Reasons */
+ SCX_ECODE_RSN_HOTPLUG = 1LLU << 32,
+
+ /* Actions */
+ SCX_ECODE_ACT_RESTART = 1LLU << 48,
+};
+
/*
* scx_exit_info is passed to ops.exit() to describe why the BPF scheduler is
* being disabled.
@@ -457,7 +480,29 @@ struct sched_ext_ops {
void (*dump_task)(struct scx_dump_ctx *ctx, struct task_struct *p);
/*
- * All online ops must come before ops.init().
+ * All online ops must come before ops.cpu_online().
+ */
+
+ /**
+ * cpu_online - A CPU became online
+ * @cpu: CPU which just came up
+ *
+ * @cpu just came online. @cpu will not call ops.enqueue() or
+ * ops.dispatch(), nor run tasks associated with other CPUs beforehand.
+ */
+ void (*cpu_online)(s32 cpu);
+
+ /**
+ * cpu_offline - A CPU is going offline
+ * @cpu: CPU which is going offline
+ *
+ * @cpu is going offline. @cpu will not call ops.enqueue() or
+ * ops.dispatch(), nor run tasks associated with other CPUs afterwards.
+ */
+ void (*cpu_offline)(s32 cpu);
+
+ /*
+ * All CPU hotplug ops must come before ops.init().
*/
/**
@@ -496,6 +541,15 @@ struct sched_ext_ops {
*/
u32 exit_dump_len;
+ /**
+ * hotplug_seq - A sequence number that may be set by the scheduler to
+ * detect when a hotplug event has occurred during the loading process.
+ * If 0, no detection occurs. Otherwise, the scheduler will fail to
+ * load if the sequence number does not match @scx_hotplug_seq on the
+ * enable path.
+ */
+ u64 hotplug_seq;
+
/**
* name - BPF scheduler's name
*
@@ -509,7 +563,9 @@ struct sched_ext_ops {
enum scx_opi {
SCX_OPI_BEGIN = 0,
SCX_OPI_NORMAL_BEGIN = 0,
- SCX_OPI_NORMAL_END = SCX_OP_IDX(init),
+ SCX_OPI_NORMAL_END = SCX_OP_IDX(cpu_online),
+ SCX_OPI_CPU_HOTPLUG_BEGIN = SCX_OP_IDX(cpu_online),
+ SCX_OPI_CPU_HOTPLUG_END = SCX_OP_IDX(init),
SCX_OPI_END = SCX_OP_IDX(init),
};
@@ -694,6 +750,7 @@ static atomic_t scx_exit_kind = ATOMIC_INIT(SCX_EXIT_DONE);
static struct scx_exit_info *scx_exit_info;
static atomic_long_t scx_nr_rejected = ATOMIC_LONG_INIT(0);
+static atomic_long_t scx_hotplug_seq = ATOMIC_LONG_INIT(0);
/*
* The maximum amount of time in jiffies that a task may be runnable without
@@ -1419,11 +1476,7 @@ static void direct_dispatch(struct task_struct *p, u64 enq_flags)
static bool scx_rq_online(struct rq *rq)
{
-#ifdef CONFIG_SMP
- return likely(rq->online);
-#else
- return true;
-#endif
+ return likely(rq->scx.flags & SCX_RQ_ONLINE);
}
static void do_enqueue_task(struct rq *rq, struct task_struct *p, u64 enq_flags,
@@ -1438,6 +1491,11 @@ static void do_enqueue_task(struct rq *rq, struct task_struct *p, u64 enq_flags,
if (sticky_cpu == cpu_of(rq))
goto local_norefill;
+ /*
+ * If !scx_rq_online(), we already told the BPF scheduler that the CPU
+ * is offline and are just running the hotplug path. Don't bother the
+ * BPF scheduler.
+ */
if (!scx_rq_online(rq))
goto local;
@@ -2673,6 +2731,42 @@ void __scx_update_idle(struct rq *rq, bool idle)
#endif
}
+static void handle_hotplug(struct rq *rq, bool online)
+{
+ int cpu = cpu_of(rq);
+
+ atomic_long_inc(&scx_hotplug_seq);
+
+ if (online && SCX_HAS_OP(cpu_online))
+ SCX_CALL_OP(SCX_KF_SLEEPABLE, cpu_online, cpu);
+ else if (!online && SCX_HAS_OP(cpu_offline))
+ SCX_CALL_OP(SCX_KF_SLEEPABLE, cpu_offline, cpu);
+ else
+ scx_ops_exit(SCX_ECODE_ACT_RESTART | SCX_ECODE_RSN_HOTPLUG,
+ "cpu %d going %s, exiting scheduler", cpu,
+ online ? "online" : "offline");
+}
+
+void scx_rq_activate(struct rq *rq)
+{
+ handle_hotplug(rq, true);
+}
+
+void scx_rq_deactivate(struct rq *rq)
+{
+ handle_hotplug(rq, false);
+}
+
+static void rq_online_scx(struct rq *rq)
+{
+ rq->scx.flags |= SCX_RQ_ONLINE;
+}
+
+static void rq_offline_scx(struct rq *rq)
+{
+ rq->scx.flags &= ~SCX_RQ_ONLINE;
+}
+
#else /* CONFIG_SMP */
static bool test_and_clear_cpu_idle(int cpu) { return false; }
@@ -3104,6 +3198,9 @@ DEFINE_SCHED_CLASS(ext) = {
.balance = balance_scx,
.select_task_rq = select_task_rq_scx,
.set_cpus_allowed = set_cpus_allowed_scx,
+
+ .rq_online = rq_online_scx,
+ .rq_offline = rq_offline_scx,
#endif
.task_tick = task_tick_scx,
@@ -3235,10 +3332,18 @@ static ssize_t scx_attr_nr_rejected_show(struct kobject *kobj,
}
SCX_ATTR(nr_rejected);
+static ssize_t scx_attr_hotplug_seq_show(struct kobject *kobj,
+ struct kobj_attribute *ka, char *buf)
+{
+ return sysfs_emit(buf, "%ld\n", atomic_long_read(&scx_hotplug_seq));
+}
+SCX_ATTR(hotplug_seq);
+
static struct attribute *scx_global_attrs[] = {
&scx_attr_state.attr,
&scx_attr_switch_all.attr,
&scx_attr_nr_rejected.attr,
+ &scx_attr_hotplug_seq.attr,
NULL,
};
@@ -3941,6 +4046,25 @@ static struct kthread_worker *scx_create_rt_helper(const char *name)
return helper;
}
+static void check_hotplug_seq(const struct sched_ext_ops *ops)
+{
+ unsigned long long global_hotplug_seq;
+
+ /*
+ * If a hotplug event has occurred between when a scheduler was
+ * initialized, and when we were able to attach, exit and notify user
+ * space about it.
+ */
+ if (ops->hotplug_seq) {
+ global_hotplug_seq = atomic_long_read(&scx_hotplug_seq);
+ if (ops->hotplug_seq != global_hotplug_seq) {
+ scx_ops_exit(SCX_ECODE_ACT_RESTART | SCX_ECODE_RSN_HOTPLUG,
+ "expected hotplug seq %llu did not match actual %llu",
+ ops->hotplug_seq, global_hotplug_seq);
+ }
+ }
+}
+
static int validate_ops(const struct sched_ext_ops *ops)
{
/*
@@ -4023,6 +4147,10 @@ static int scx_ops_enable(struct sched_ext_ops *ops, struct bpf_link *link)
}
}
+ for (i = SCX_OPI_CPU_HOTPLUG_BEGIN; i < SCX_OPI_CPU_HOTPLUG_END; i++)
+ if (((void (**)(void))ops)[i])
+ static_branch_enable_cpuslocked(&scx_has_op[i]);
+
cpus_read_unlock();
ret = validate_ops(ops);
@@ -4064,6 +4192,8 @@ static int scx_ops_enable(struct sched_ext_ops *ops, struct bpf_link *link)
percpu_down_write(&scx_fork_rwsem);
cpus_read_lock();
+ check_hotplug_seq(ops);
+
for (i = SCX_OPI_NORMAL_BEGIN; i < SCX_OPI_NORMAL_END; i++)
if (((void (**)(void))ops)[i])
static_branch_enable_cpuslocked(&scx_has_op[i]);
@@ -4374,6 +4504,9 @@ static int bpf_scx_init_member(const struct btf_type *t,
ops->exit_dump_len =
*(u32 *)(udata + moff) ?: SCX_EXIT_DUMP_DFL_LEN;
return 1;
+ case offsetof(struct sched_ext_ops, hotplug_seq):
+ ops->hotplug_seq = *(u64 *)(udata + moff);
+ return 1;
}
return 0;
@@ -4387,6 +4520,8 @@ static int bpf_scx_check_member(const struct btf_type *t,
switch (moff) {
case offsetof(struct sched_ext_ops, init_task):
+ case offsetof(struct sched_ext_ops, cpu_online):
+ case offsetof(struct sched_ext_ops, cpu_offline):
case offsetof(struct sched_ext_ops, init):
case offsetof(struct sched_ext_ops, exit):
break;
@@ -4457,6 +4592,8 @@ static s32 init_task_stub(struct task_struct *p, struct scx_init_task_args *args
static void exit_task_stub(struct task_struct *p, struct scx_exit_task_args *args) {}
static void enable_stub(struct task_struct *p) {}
static void disable_stub(struct task_struct *p) {}
+static void cpu_online_stub(s32 cpu) {}
+static void cpu_offline_stub(s32 cpu) {}
static s32 init_stub(void) { return -EINVAL; }
static void exit_stub(struct scx_exit_info *info) {}
@@ -4479,6 +4616,8 @@ static struct sched_ext_ops __bpf_ops_sched_ext_ops = {
.exit_task = exit_task_stub,
.enable = enable_stub,
.disable = disable_stub,
+ .cpu_online = cpu_online_stub,
+ .cpu_offline = cpu_offline_stub,
.init = init_stub,
.exit = exit_stub,
};
@@ -4719,6 +4858,9 @@ void __init init_sched_ext_class(void)
BUG_ON(!zalloc_cpumask_var(&rq->scx.cpus_to_preempt, GFP_KERNEL));
BUG_ON(!zalloc_cpumask_var(&rq->scx.cpus_to_wait, GFP_KERNEL));
init_irq_work(&rq->scx.kick_cpus_irq_work, kick_cpus_irq_workfn);
+
+ if (cpu_online(cpu))
+ cpu_rq(cpu)->scx.flags |= SCX_RQ_ONLINE;
}
register_sysrq_key('S', &sysrq_sched_ext_reset_op);
diff --git a/kernel/sched/ext.h b/kernel/sched/ext.h
index 4ebd1c2478f1..037f9acdf443 100644
--- a/kernel/sched/ext.h
+++ b/kernel/sched/ext.h
@@ -40,6 +40,8 @@ int scx_fork(struct task_struct *p);
void scx_post_fork(struct task_struct *p);
void scx_cancel_fork(struct task_struct *p);
bool scx_can_stop_tick(struct rq *rq);
+void scx_rq_activate(struct rq *rq);
+void scx_rq_deactivate(struct rq *rq);
int scx_check_setscheduler(struct task_struct *p, int policy);
bool task_should_scx(struct task_struct *p);
void init_sched_ext_class(void);
@@ -81,6 +83,8 @@ static inline int scx_fork(struct task_struct *p) { return 0; }
static inline void scx_post_fork(struct task_struct *p) {}
static inline void scx_cancel_fork(struct task_struct *p) {}
static inline bool scx_can_stop_tick(struct rq *rq) { return true; }
+static inline void scx_rq_activate(struct rq *rq) {}
+static inline void scx_rq_deactivate(struct rq *rq) {}
static inline int scx_check_setscheduler(struct task_struct *p, int policy) { return 0; }
static inline bool task_on_scx(const struct task_struct *p) { return false; }
static inline void init_sched_ext_class(void) {}
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 147d18cf01ce..c0d6e42c99cc 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -726,6 +726,12 @@ struct cfs_rq {
#ifdef CONFIG_SCHED_CLASS_EXT
/* scx_rq->flags, protected by the rq lock */
enum scx_rq_flags {
+ /*
+ * A hotplugged CPU starts scheduling before rq_online_scx(). Track
+ * ops.cpu_on/offline() state so that ops.enqueue/dispatch() are called
+ * only while the BPF scheduler considers the CPU to be online.
+ */
+ SCX_RQ_ONLINE = 1 << 0,
SCX_RQ_BALANCING = 1 << 1,
SCX_RQ_CAN_STOP_TICK = 1 << 2,
};
diff --git a/tools/sched_ext/include/scx/compat.h b/tools/sched_ext/include/scx/compat.h
index c58024c980c8..cc56ff9aa252 100644
--- a/tools/sched_ext/include/scx/compat.h
+++ b/tools/sched_ext/include/scx/compat.h
@@ -8,6 +8,9 @@
#define __SCX_COMPAT_H
#include <bpf/btf.h>
+#include <fcntl.h>
+#include <stdlib.h>
+#include <unistd.h>
struct btf *__COMPAT_vmlinux_btf __attribute__((weak));
@@ -106,6 +109,28 @@ static inline bool __COMPAT_struct_has_field(const char *type, const char *field
#define SCX_OPS_SWITCH_PARTIAL \
__COMPAT_ENUM_OR_ZERO("scx_ops_flags", "SCX_OPS_SWITCH_PARTIAL")
+static inline long scx_hotplug_seq(void)
+{
+ int fd;
+ char buf[32];
+ ssize_t len;
+ long val;
+
+ fd = open("/sys/kernel/sched_ext/hotplug_seq", O_RDONLY);
+ if (fd < 0)
+ return -ENOENT;
+
+ len = read(fd, buf, sizeof(buf) - 1);
+ SCX_BUG_ON(len <= 0, "read failed (%ld)", len);
+ buf[len] = 0;
+ close(fd);
+
+ val = strtoul(buf, NULL, 10);
+ SCX_BUG_ON(val < 0, "invalid num hotplug events: %lu", val);
+
+ return val;
+}
+
/*
* struct sched_ext_ops can change over time. If compat.bpf.h::SCX_OPS_DEFINE()
* is used to define ops and compat.h::SCX_OPS_LOAD/ATTACH() are used to load
@@ -123,6 +148,7 @@ static inline bool __COMPAT_struct_has_field(const char *type, const char *field
\
__skel = __scx_name##__open(); \
SCX_BUG_ON(!__skel, "Could not open " #__scx_name); \
+ __skel->struct_ops.__ops_name->hotplug_seq = scx_hotplug_seq(); \
__skel; \
})
diff --git a/tools/sched_ext/include/scx/user_exit_info.h b/tools/sched_ext/include/scx/user_exit_info.h
index c2ef85c645e1..891693ee604e 100644
--- a/tools/sched_ext/include/scx/user_exit_info.h
+++ b/tools/sched_ext/include/scx/user_exit_info.h
@@ -77,7 +77,35 @@ struct user_exit_info {
if (__uei->msg[0] != '\0') \
fprintf(stderr, " (%s)", __uei->msg); \
fputs("\n", stderr); \
+ __uei->exit_code; \
})
+/*
+ * We can't import vmlinux.h while compiling user C code. Let's duplicate
+ * scx_exit_code definition.
+ */
+enum scx_exit_code {
+ /* Reasons */
+ SCX_ECODE_RSN_HOTPLUG = 1LLU << 32,
+
+ /* Actions */
+ SCX_ECODE_ACT_RESTART = 1LLU << 48,
+};
+
+enum uei_ecode_mask {
+ UEI_ECODE_USER_MASK = ((1LLU << 32) - 1),
+ UEI_ECODE_SYS_RSN_MASK = ((1LLU << 16) - 1) << 32,
+ UEI_ECODE_SYS_ACT_MASK = ((1LLU << 16) - 1) << 48,
+};
+
+/*
+ * These macro interpret the ecode returned from UEI_REPORT().
+ */
+#define UEI_ECODE_USER(__ecode) ((__ecode) & UEI_ECODE_USER_MASK)
+#define UEI_ECODE_SYS_RSN(__ecode) ((__ecode) & UEI_ECODE_SYS_RSN_MASK)
+#define UEI_ECODE_SYS_ACT(__ecode) ((__ecode) & UEI_ECODE_SYS_ACT_MASK)
+
+#define UEI_ECODE_RESTART(__ecode) (UEI_ECODE_SYS_ACT((__ecode)) == SCX_ECODE_ACT_RESTART)
+
#endif /* __bpf__ */
#endif /* __USER_EXIT_INFO_H */
diff --git a/tools/sched_ext/scx_central.c b/tools/sched_ext/scx_central.c
index fb3f50886552..21deea320bd7 100644
--- a/tools/sched_ext/scx_central.c
+++ b/tools/sched_ext/scx_central.c
@@ -46,14 +46,14 @@ int main(int argc, char **argv)
{
struct scx_central *skel;
struct bpf_link *link;
- __u64 seq = 0;
+ __u64 seq = 0, ecode;
__s32 opt;
cpu_set_t *cpuset;
libbpf_set_print(libbpf_print_fn);
signal(SIGINT, sigint_handler);
signal(SIGTERM, sigint_handler);
-
+restart:
skel = SCX_OPS_OPEN(central_ops, scx_central);
skel->rodata->central_cpu = 0;
@@ -126,7 +126,10 @@ int main(int argc, char **argv)
}
bpf_link__destroy(link);
- UEI_REPORT(skel, uei);
+ ecode = UEI_REPORT(skel, uei);
scx_central__destroy(skel);
+
+ if (UEI_ECODE_RESTART(ecode))
+ goto restart;
return 0;
}
diff --git a/tools/sched_ext/scx_qmap.bpf.c b/tools/sched_ext/scx_qmap.bpf.c
index 4a87377558c8..619078355bf5 100644
--- a/tools/sched_ext/scx_qmap.bpf.c
+++ b/tools/sched_ext/scx_qmap.bpf.c
@@ -358,8 +358,63 @@ void BPF_STRUCT_OPS(qmap_dump_task, struct scx_dump_ctx *dctx, struct task_struc
taskc->force_local);
}
+/*
+ * Print out the online and possible CPU map using bpf_printk() as a
+ * demonstration of using the cpumask kfuncs and ops.cpu_on/offline().
+ */
+static void print_cpus(void)
+{
+ const struct cpumask *possible, *online;
+ s32 cpu;
+ char buf[128] = "", *p;
+ int idx;
+
+ possible = scx_bpf_get_possible_cpumask();
+ online = scx_bpf_get_online_cpumask();
+
+ idx = 0;
+ bpf_for(cpu, 0, scx_bpf_nr_cpu_ids()) {
+ if (!(p = MEMBER_VPTR(buf, [idx++])))
+ break;
+ if (bpf_cpumask_test_cpu(cpu, online))
+ *p++ = 'O';
+ else if (bpf_cpumask_test_cpu(cpu, possible))
+ *p++ = 'X';
+ else
+ *p++ = ' ';
+
+ if ((cpu & 7) == 7) {
+ if (!(p = MEMBER_VPTR(buf, [idx++])))
+ break;
+ *p++ = '|';
+ }
+ }
+ buf[sizeof(buf) - 1] = '\0';
+
+ scx_bpf_put_cpumask(online);
+ scx_bpf_put_cpumask(possible);
+
+ bpf_printk("CPUS: |%s", buf);
+}
+
+void BPF_STRUCT_OPS(qmap_cpu_online, s32 cpu)
+{
+ bpf_printk("CPU %d coming online", cpu);
+ /* @cpu is already online at this point */
+ print_cpus();
+}
+
+void BPF_STRUCT_OPS(qmap_cpu_offline, s32 cpu)
+{
+ bpf_printk("CPU %d going offline", cpu);
+ /* @cpu is still online at this point */
+ print_cpus();
+}
+
s32 BPF_STRUCT_OPS_SLEEPABLE(qmap_init)
{
+ print_cpus();
+
return scx_bpf_create_dsq(SHARED_DSQ, -1);
}
@@ -378,6 +433,8 @@ SCX_OPS_DEFINE(qmap_ops,
.dump = (void *)qmap_dump,
.dump_cpu = (void *)qmap_dump_cpu,
.dump_task = (void *)qmap_dump_task,
+ .cpu_online = (void *)qmap_cpu_online,
+ .cpu_offline = (void *)qmap_cpu_offline,
.init = (void *)qmap_init,
.exit = (void *)qmap_exit,
.timeout_ms = 5000U,
diff --git a/tools/sched_ext/scx_qmap.c b/tools/sched_ext/scx_qmap.c
index 2a97421afe9a..920fb54f9c77 100644
--- a/tools/sched_ext/scx_qmap.c
+++ b/tools/sched_ext/scx_qmap.c
@@ -122,5 +122,9 @@ int main(int argc, char **argv)
bpf_link__destroy(link);
UEI_REPORT(skel, uei);
scx_qmap__destroy(skel);
+ /*
+ * scx_qmap implements ops.cpu_on/offline() and doesn't need to restart
+ * on CPU hotplug events.
+ */
return 0;
}
diff --git a/tools/sched_ext/scx_simple.c b/tools/sched_ext/scx_simple.c
index 7f500d1d56ac..bead482e1383 100644
--- a/tools/sched_ext/scx_simple.c
+++ b/tools/sched_ext/scx_simple.c
@@ -62,11 +62,12 @@ int main(int argc, char **argv)
struct scx_simple *skel;
struct bpf_link *link;
__u32 opt;
+ __u64 ecode;
libbpf_set_print(libbpf_print_fn);
signal(SIGINT, sigint_handler);
signal(SIGTERM, sigint_handler);
-
+restart:
skel = SCX_OPS_OPEN(simple_ops, scx_simple);
while ((opt = getopt(argc, argv, "vh")) != -1) {
@@ -93,7 +94,10 @@ int main(int argc, char **argv)
}
bpf_link__destroy(link);
- UEI_REPORT(skel, uei);
+ ecode = UEI_REPORT(skel, uei);
scx_simple__destroy(skel);
+
+ if (UEI_ECODE_RESTART(ecode))
+ goto restart;
return 0;
}
--
2.45.2
Diff
---
kernel/sched/core.c | 4 +
kernel/sched/ext.c | 156 ++++++++++++++++++-
kernel/sched/ext.h | 4 +
kernel/sched/sched.h | 6 +
tools/sched_ext/include/scx/compat.h | 26 ++++
tools/sched_ext/include/scx/user_exit_info.h | 28 ++++
tools/sched_ext/scx_central.c | 9 +-
tools/sched_ext/scx_qmap.bpf.c | 57 +++++++
tools/sched_ext/scx_qmap.c | 4 +
tools/sched_ext/scx_simple.c | 8 +-
10 files changed, 290 insertions(+), 12 deletions(-)
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 0e6ff33f34e4..c798c847d57e 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -7984,6 +7984,8 @@ int sched_cpu_activate(unsigned int cpu)
cpuset_cpu_active();
}
+ scx_rq_activate(rq);
+
/*
* Put the rq online, if not already. This happens:
*
@@ -8044,6 +8046,8 @@ int sched_cpu_deactivate(unsigned int cpu)
}
rq_unlock_irqrestore(rq, &rf);
+ scx_rq_deactivate(rq);
+
#ifdef CONFIG_SCHED_SMT
/*
* When going down, decrement the number of cores with SMT present.
diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c
index 686dab6ab592..7c2f2a542b32 100644
--- a/kernel/sched/ext.c
+++ b/kernel/sched/ext.c
@@ -30,6 +30,29 @@ enum scx_exit_kind {
SCX_EXIT_ERROR_STALL, /* watchdog detected stalled runnable tasks */
};
+/*
+ * An exit code can be specified when exiting with scx_bpf_exit() or
+ * scx_ops_exit(), corresponding to exit_kind UNREG_BPF and UNREG_KERN
+ * respectively. The codes are 64bit of the format:
+ *
+ * Bits: [63 .. 48 47 .. 32 31 .. 0]
+ * [ SYS ACT ] [ SYS RSN ] [ USR ]
+ *
+ * SYS ACT: System-defined exit actions
+ * SYS RSN: System-defined exit reasons
+ * USR : User-defined exit codes and reasons
+ *
+ * Using the above, users may communicate intention and context by ORing system
+ * actions and/or system reasons with a user-defined exit code.
+ */
+enum scx_exit_code {
+ /* Reasons */
+ SCX_ECODE_RSN_HOTPLUG = 1LLU << 32,
+
+ /* Actions */
+ SCX_ECODE_ACT_RESTART = 1LLU << 48,
+};
+
/*
* scx_exit_info is passed to ops.exit() to describe why the BPF scheduler is
* being disabled.
@@ -457,7 +480,29 @@ struct sched_ext_ops {
void (*dump_task)(struct scx_dump_ctx *ctx, struct task_struct *p);
/*
- * All online ops must come before ops.init().
+ * All online ops must come before ops.cpu_online().
+ */
+
+ /**
+ * cpu_online - A CPU became online
+ * @cpu: CPU which just came up
+ *
+ * @cpu just came online. @cpu will not call ops.enqueue() or
+ * ops.dispatch(), nor run tasks associated with other CPUs beforehand.
+ */
+ void (*cpu_online)(s32 cpu);
+
+ /**
+ * cpu_offline - A CPU is going offline
+ * @cpu: CPU which is going offline
+ *
+ * @cpu is going offline. @cpu will not call ops.enqueue() or
+ * ops.dispatch(), nor run tasks associated with other CPUs afterwards.
+ */
+ void (*cpu_offline)(s32 cpu);
+
+ /*
+ * All CPU hotplug ops must come before ops.init().
*/
/**
@@ -496,6 +541,15 @@ struct sched_ext_ops {
*/
u32 exit_dump_len;
+ /**
+ * hotplug_seq - A sequence number that may be set by the scheduler to
+ * detect when a hotplug event has occurred during the loading process.
+ * If 0, no detection occurs. Otherwise, the scheduler will fail to
+ * load if the sequence number does not match @scx_hotplug_seq on the
+ * enable path.
+ */
+ u64 hotplug_seq;
+
/**
* name - BPF scheduler's name
*
@@ -509,7 +563,9 @@ struct sched_ext_ops {
enum scx_opi {
SCX_OPI_BEGIN = 0,
SCX_OPI_NORMAL_BEGIN = 0,
- SCX_OPI_NORMAL_END = SCX_OP_IDX(init),
+ SCX_OPI_NORMAL_END = SCX_OP_IDX(cpu_online),
+ SCX_OPI_CPU_HOTPLUG_BEGIN = SCX_OP_IDX(cpu_online),
+ SCX_OPI_CPU_HOTPLUG_END = SCX_OP_IDX(init),
SCX_OPI_END = SCX_OP_IDX(init),
};
@@ -694,6 +750,7 @@ static atomic_t scx_exit_kind = ATOMIC_INIT(SCX_EXIT_DONE);
static struct scx_exit_info *scx_exit_info;
static atomic_long_t scx_nr_rejected = ATOMIC_LONG_INIT(0);
+static atomic_long_t scx_hotplug_seq = ATOMIC_LONG_INIT(0);
/*
* The maximum amount of time in jiffies that a task may be runnable without
@@ -1419,11 +1476,7 @@ static void direct_dispatch(struct task_struct *p, u64 enq_flags)
static bool scx_rq_online(struct rq *rq)
{
-#ifdef CONFIG_SMP
- return likely(rq->online);
-#else
- return true;
-#endif
+ return likely(rq->scx.flags & SCX_RQ_ONLINE);
}
static void do_enqueue_task(struct rq *rq, struct task_struct *p, u64 enq_flags,
@@ -1438,6 +1491,11 @@ static void do_enqueue_task(struct rq *rq, struct task_struct *p, u64 enq_flags,
if (sticky_cpu == cpu_of(rq))
goto local_norefill;
+ /*
+ * If !scx_rq_online(), we already told the BPF scheduler that the CPU
+ * is offline and are just running the hotplug path. Don't bother the
+ * BPF scheduler.
+ */
if (!scx_rq_online(rq))
goto local;
@@ -2673,6 +2731,42 @@ void __scx_update_idle(struct rq *rq, bool idle)
#endif
}
+static void handle_hotplug(struct rq *rq, bool online)
+{
+ int cpu = cpu_of(rq);
+
+ atomic_long_inc(&scx_hotplug_seq);
+
+ if (online && SCX_HAS_OP(cpu_online))
+ SCX_CALL_OP(SCX_KF_SLEEPABLE, cpu_online, cpu);
+ else if (!online && SCX_HAS_OP(cpu_offline))
+ SCX_CALL_OP(SCX_KF_SLEEPABLE, cpu_offline, cpu);
+ else
+ scx_ops_exit(SCX_ECODE_ACT_RESTART | SCX_ECODE_RSN_HOTPLUG,
+ "cpu %d going %s, exiting scheduler", cpu,
+ online ? "online" : "offline");
+}
+
+void scx_rq_activate(struct rq *rq)
+{
+ handle_hotplug(rq, true);
+}
+
+void scx_rq_deactivate(struct rq *rq)
+{
+ handle_hotplug(rq, false);
+}
+
+static void rq_online_scx(struct rq *rq)
+{
+ rq->scx.flags |= SCX_RQ_ONLINE;
+}
+
+static void rq_offline_scx(struct rq *rq)
+{
+ rq->scx.flags &= ~SCX_RQ_ONLINE;
+}
+
#else /* CONFIG_SMP */
static bool test_and_clear_cpu_idle(int cpu) { return false; }
@@ -3104,6 +3198,9 @@ DEFINE_SCHED_CLASS(ext) = {
.balance = balance_scx,
.select_task_rq = select_task_rq_scx,
.set_cpus_allowed = set_cpus_allowed_scx,
+
+ .rq_online = rq_online_scx,
+ .rq_offline = rq_offline_scx,
#endif
.task_tick = task_tick_scx,
@@ -3235,10 +3332,18 @@ static ssize_t scx_attr_nr_rejected_show(struct kobject *kobj,
}
SCX_ATTR(nr_rejected);
+static ssize_t scx_attr_hotplug_seq_show(struct kobject *kobj,
+ struct kobj_attribute *ka, char *buf)
+{
+ return sysfs_emit(buf, "%ld\n", atomic_long_read(&scx_hotplug_seq));
+}
+SCX_ATTR(hotplug_seq);
+
static struct attribute *scx_global_attrs[] = {
&scx_attr_state.attr,
&scx_attr_switch_all.attr,
&scx_attr_nr_rejected.attr,
+ &scx_attr_hotplug_seq.attr,
NULL,
};
@@ -3941,6 +4046,25 @@ static struct kthread_worker *scx_create_rt_helper(const char *name)
return helper;
}
+static void check_hotplug_seq(const struct sched_ext_ops *ops)
+{
+ unsigned long long global_hotplug_seq;
+
+ /*
+ * If a hotplug event has occurred between when a scheduler was
+ * initialized, and when we were able to attach, exit and notify user
+ * space about it.
+ */
+ if (ops->hotplug_seq) {
+ global_hotplug_seq = atomic_long_read(&scx_hotplug_seq);
+ if (ops->hotplug_seq != global_hotplug_seq) {
+ scx_ops_exit(SCX_ECODE_ACT_RESTART | SCX_ECODE_RSN_HOTPLUG,
+ "expected hotplug seq %llu did not match actual %llu",
+ ops->hotplug_seq, global_hotplug_seq);
+ }
+ }
+}
+
static int validate_ops(const struct sched_ext_ops *ops)
{
/*
@@ -4023,6 +4147,10 @@ static int scx_ops_enable(struct sched_ext_ops *ops, struct bpf_link *link)
}
}
+ for (i = SCX_OPI_CPU_HOTPLUG_BEGIN; i < SCX_OPI_CPU_HOTPLUG_END; i++)
+ if (((void (**)(void))ops)[i])
+ static_branch_enable_cpuslocked(&scx_has_op[i]);
+
cpus_read_unlock();
ret = validate_ops(ops);
@@ -4064,6 +4192,8 @@ static int scx_ops_enable(struct sched_ext_ops *ops, struct bpf_link *link)
percpu_down_write(&scx_fork_rwsem);
cpus_read_lock();
+ check_hotplug_seq(ops);
+
for (i = SCX_OPI_NORMAL_BEGIN; i < SCX_OPI_NORMAL_END; i++)
if (((void (**)(void))ops)[i])
static_branch_enable_cpuslocked(&scx_has_op[i]);
@@ -4374,6 +4504,9 @@ static int bpf_scx_init_member(const struct btf_type *t,
ops->exit_dump_len =
*(u32 *)(udata + moff) ?: SCX_EXIT_DUMP_DFL_LEN;
return 1;
+ case offsetof(struct sched_ext_ops, hotplug_seq):
+ ops->hotplug_seq = *(u64 *)(udata + moff);
+ return 1;
}
return 0;
@@ -4387,6 +4520,8 @@ static int bpf_scx_check_member(const struct btf_type *t,
switch (moff) {
case offsetof(struct sched_ext_ops, init_task):
+ case offsetof(struct sched_ext_ops, cpu_online):
+ case offsetof(struct sched_ext_ops, cpu_offline):
case offsetof(struct sched_ext_ops, init):
case offsetof(struct sched_ext_ops, exit):
break;
@@ -4457,6 +4592,8 @@ static s32 init_task_stub(struct task_struct *p, struct scx_init_task_args *args
static void exit_task_stub(struct task_struct *p, struct scx_exit_task_args *args) {}
static void enable_stub(struct task_struct *p) {}
static void disable_stub(struct task_struct *p) {}
+static void cpu_online_stub(s32 cpu) {}
+static void cpu_offline_stub(s32 cpu) {}
static s32 init_stub(void) { return -EINVAL; }
static void exit_stub(struct scx_exit_info *info) {}
@@ -4479,6 +4616,8 @@ static struct sched_ext_ops __bpf_ops_sched_ext_ops = {
.exit_task = exit_task_stub,
.enable = enable_stub,
.disable = disable_stub,
+ .cpu_online = cpu_online_stub,
+ .cpu_offline = cpu_offline_stub,
.init = init_stub,
.exit = exit_stub,
};
@@ -4719,6 +4858,9 @@ void __init init_sched_ext_class(void)
BUG_ON(!zalloc_cpumask_var(&rq->scx.cpus_to_preempt, GFP_KERNEL));
BUG_ON(!zalloc_cpumask_var(&rq->scx.cpus_to_wait, GFP_KERNEL));
init_irq_work(&rq->scx.kick_cpus_irq_work, kick_cpus_irq_workfn);
+
+ if (cpu_online(cpu))
+ cpu_rq(cpu)->scx.flags |= SCX_RQ_ONLINE;
}
register_sysrq_key('S', &sysrq_sched_ext_reset_op);
diff --git a/kernel/sched/ext.h b/kernel/sched/ext.h
index 4ebd1c2478f1..037f9acdf443 100644
--- a/kernel/sched/ext.h
+++ b/kernel/sched/ext.h
@@ -40,6 +40,8 @@ int scx_fork(struct task_struct *p);
void scx_post_fork(struct task_struct *p);
void scx_cancel_fork(struct task_struct *p);
bool scx_can_stop_tick(struct rq *rq);
+void scx_rq_activate(struct rq *rq);
+void scx_rq_deactivate(struct rq *rq);
int scx_check_setscheduler(struct task_struct *p, int policy);
bool task_should_scx(struct task_struct *p);
void init_sched_ext_class(void);
@@ -81,6 +83,8 @@ static inline int scx_fork(struct task_struct *p) { return 0; }
static inline void scx_post_fork(struct task_struct *p) {}
static inline void scx_cancel_fork(struct task_struct *p) {}
static inline bool scx_can_stop_tick(struct rq *rq) { return true; }
+static inline void scx_rq_activate(struct rq *rq) {}
+static inline void scx_rq_deactivate(struct rq *rq) {}
static inline int scx_check_setscheduler(struct task_struct *p, int policy) { return 0; }
static inline bool task_on_scx(const struct task_struct *p) { return false; }
static inline void init_sched_ext_class(void) {}
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 147d18cf01ce..c0d6e42c99cc 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -726,6 +726,12 @@ struct cfs_rq {
#ifdef CONFIG_SCHED_CLASS_EXT
/* scx_rq->flags, protected by the rq lock */
enum scx_rq_flags {
+ /*
+ * A hotplugged CPU starts scheduling before rq_online_scx(). Track
+ * ops.cpu_on/offline() state so that ops.enqueue/dispatch() are called
+ * only while the BPF scheduler considers the CPU to be online.
+ */
+ SCX_RQ_ONLINE = 1 << 0,
SCX_RQ_BALANCING = 1 << 1,
SCX_RQ_CAN_STOP_TICK = 1 << 2,
};
diff --git a/tools/sched_ext/include/scx/compat.h b/tools/sched_ext/include/scx/compat.h
index c58024c980c8..cc56ff9aa252 100644
--- a/tools/sched_ext/include/scx/compat.h
+++ b/tools/sched_ext/include/scx/compat.h
@@ -8,6 +8,9 @@
#define __SCX_COMPAT_H
#include <bpf/btf.h>
+#include <fcntl.h>
+#include <stdlib.h>
+#include <unistd.h>
struct btf *__COMPAT_vmlinux_btf __attribute__((weak));
@@ -106,6 +109,28 @@ static inline bool __COMPAT_struct_has_field(const char *type, const char *field
#define SCX_OPS_SWITCH_PARTIAL \
__COMPAT_ENUM_OR_ZERO("scx_ops_flags", "SCX_OPS_SWITCH_PARTIAL")
+static inline long scx_hotplug_seq(void)
+{
+ int fd;
+ char buf[32];
+ ssize_t len;
+ long val;
+
+ fd = open("/sys/kernel/sched_ext/hotplug_seq", O_RDONLY);
+ if (fd < 0)
+ return -ENOENT;
+
+ len = read(fd, buf, sizeof(buf) - 1);
+ SCX_BUG_ON(len <= 0, "read failed (%ld)", len);
+ buf[len] = 0;
+ close(fd);
+
+ val = strtoul(buf, NULL, 10);
+ SCX_BUG_ON(val < 0, "invalid num hotplug events: %lu", val);
+
+ return val;
+}
+
/*
* struct sched_ext_ops can change over time. If compat.bpf.h::SCX_OPS_DEFINE()
* is used to define ops and compat.h::SCX_OPS_LOAD/ATTACH() are used to load
@@ -123,6 +148,7 @@ static inline bool __COMPAT_struct_has_field(const char *type, const char *field
\
__skel = __scx_name##__open(); \
SCX_BUG_ON(!__skel, "Could not open " #__scx_name); \
+ __skel->struct_ops.__ops_name->hotplug_seq = scx_hotplug_seq(); \
__skel; \
})
diff --git a/tools/sched_ext/include/scx/user_exit_info.h b/tools/sched_ext/include/scx/user_exit_info.h
index c2ef85c645e1..891693ee604e 100644
--- a/tools/sched_ext/include/scx/user_exit_info.h
+++ b/tools/sched_ext/include/scx/user_exit_info.h
@@ -77,7 +77,35 @@ struct user_exit_info {
if (__uei->msg[0] != '\0') \
fprintf(stderr, " (%s)", __uei->msg); \
fputs("\n", stderr); \
+ __uei->exit_code; \
})
+/*
+ * We can't import vmlinux.h while compiling user C code. Let's duplicate
+ * scx_exit_code definition.
+ */
+enum scx_exit_code {
+ /* Reasons */
+ SCX_ECODE_RSN_HOTPLUG = 1LLU << 32,
+
+ /* Actions */
+ SCX_ECODE_ACT_RESTART = 1LLU << 48,
+};
+
+enum uei_ecode_mask {
+ UEI_ECODE_USER_MASK = ((1LLU << 32) - 1),
+ UEI_ECODE_SYS_RSN_MASK = ((1LLU << 16) - 1) << 32,
+ UEI_ECODE_SYS_ACT_MASK = ((1LLU << 16) - 1) << 48,
+};
+
+/*
+ * These macro interpret the ecode returned from UEI_REPORT().
+ */
+#define UEI_ECODE_USER(__ecode) ((__ecode) & UEI_ECODE_USER_MASK)
+#define UEI_ECODE_SYS_RSN(__ecode) ((__ecode) & UEI_ECODE_SYS_RSN_MASK)
+#define UEI_ECODE_SYS_ACT(__ecode) ((__ecode) & UEI_ECODE_SYS_ACT_MASK)
+
+#define UEI_ECODE_RESTART(__ecode) (UEI_ECODE_SYS_ACT((__ecode)) == SCX_ECODE_ACT_RESTART)
+
#endif /* __bpf__ */
#endif /* __USER_EXIT_INFO_H */
diff --git a/tools/sched_ext/scx_central.c b/tools/sched_ext/scx_central.c
index fb3f50886552..21deea320bd7 100644
--- a/tools/sched_ext/scx_central.c
+++ b/tools/sched_ext/scx_central.c
@@ -46,14 +46,14 @@ int main(int argc, char **argv)
{
struct scx_central *skel;
struct bpf_link *link;
- __u64 seq = 0;
+ __u64 seq = 0, ecode;
__s32 opt;
cpu_set_t *cpuset;
libbpf_set_print(libbpf_print_fn);
signal(SIGINT, sigint_handler);
signal(SIGTERM, sigint_handler);
-
+restart:
skel = SCX_OPS_OPEN(central_ops, scx_central);
skel->rodata->central_cpu = 0;
@@ -126,7 +126,10 @@ int main(int argc, char **argv)
}
bpf_link__destroy(link);
- UEI_REPORT(skel, uei);
+ ecode = UEI_REPORT(skel, uei);
scx_central__destroy(skel);
+
+ if (UEI_ECODE_RESTART(ecode))
+ goto restart;
return 0;
}
diff --git a/tools/sched_ext/scx_qmap.bpf.c b/tools/sched_ext/scx_qmap.bpf.c
index 4a87377558c8..619078355bf5 100644
--- a/tools/sched_ext/scx_qmap.bpf.c
+++ b/tools/sched_ext/scx_qmap.bpf.c
@@ -358,8 +358,63 @@ void BPF_STRUCT_OPS(qmap_dump_task, struct scx_dump_ctx *dctx, struct task_struc
taskc->force_local);
}
+/*
+ * Print out the online and possible CPU map using bpf_printk() as a
+ * demonstration of using the cpumask kfuncs and ops.cpu_on/offline().
+ */
+static void print_cpus(void)
+{
+ const struct cpumask *possible, *online;
+ s32 cpu;
+ char buf[128] = "", *p;
+ int idx;
+
+ possible = scx_bpf_get_possible_cpumask();
+ online = scx_bpf_get_online_cpumask();
+
+ idx = 0;
+ bpf_for(cpu, 0, scx_bpf_nr_cpu_ids()) {
+ if (!(p = MEMBER_VPTR(buf, [idx++])))
+ break;
+ if (bpf_cpumask_test_cpu(cpu, online))
+ *p++ = 'O';
+ else if (bpf_cpumask_test_cpu(cpu, possible))
+ *p++ = 'X';
+ else
+ *p++ = ' ';
+
+ if ((cpu & 7) == 7) {
+ if (!(p = MEMBER_VPTR(buf, [idx++])))
+ break;
+ *p++ = '|';
+ }
+ }
+ buf[sizeof(buf) - 1] = '\0';
+
+ scx_bpf_put_cpumask(online);
+ scx_bpf_put_cpumask(possible);
+
+ bpf_printk("CPUS: |%s", buf);
+}
+
+void BPF_STRUCT_OPS(qmap_cpu_online, s32 cpu)
+{
+ bpf_printk("CPU %d coming online", cpu);
+ /* @cpu is already online at this point */
+ print_cpus();
+}
+
+void BPF_STRUCT_OPS(qmap_cpu_offline, s32 cpu)
+{
+ bpf_printk("CPU %d going offline", cpu);
+ /* @cpu is still online at this point */
+ print_cpus();
+}
+
s32 BPF_STRUCT_OPS_SLEEPABLE(qmap_init)
{
+ print_cpus();
+
return scx_bpf_create_dsq(SHARED_DSQ, -1);
}
@@ -378,6 +433,8 @@ SCX_OPS_DEFINE(qmap_ops,
.dump = (void *)qmap_dump,
.dump_cpu = (void *)qmap_dump_cpu,
.dump_task = (void *)qmap_dump_task,
+ .cpu_online = (void *)qmap_cpu_online,
+ .cpu_offline = (void *)qmap_cpu_offline,
.init = (void *)qmap_init,
.exit = (void *)qmap_exit,
.timeout_ms = 5000U,
diff --git a/tools/sched_ext/scx_qmap.c b/tools/sched_ext/scx_qmap.c
index 2a97421afe9a..920fb54f9c77 100644
--- a/tools/sched_ext/scx_qmap.c
+++ b/tools/sched_ext/scx_qmap.c
@@ -122,5 +122,9 @@ int main(int argc, char **argv)
bpf_link__destroy(link);
UEI_REPORT(skel, uei);
scx_qmap__destroy(skel);
+ /*
+ * scx_qmap implements ops.cpu_on/offline() and doesn't need to restart
+ * on CPU hotplug events.
+ */
return 0;
}
diff --git a/tools/sched_ext/scx_simple.c b/tools/sched_ext/scx_simple.c
index 7f500d1d56ac..bead482e1383 100644
--- a/tools/sched_ext/scx_simple.c
+++ b/tools/sched_ext/scx_simple.c
@@ -62,11 +62,12 @@ int main(int argc, char **argv)
struct scx_simple *skel;
struct bpf_link *link;
__u32 opt;
+ __u64 ecode;
libbpf_set_print(libbpf_print_fn);
signal(SIGINT, sigint_handler);
signal(SIGTERM, sigint_handler);
-
+restart:
skel = SCX_OPS_OPEN(simple_ops, scx_simple);
while ((opt = getopt(argc, argv, "vh")) != -1) {
@@ -93,7 +94,10 @@ int main(int argc, char **argv)
}
bpf_link__destroy(link);
- UEI_REPORT(skel, uei);
+ ecode = UEI_REPORT(skel, uei);
scx_simple__destroy(skel);
+
+ if (UEI_ECODE_RESTART(ecode))
+ goto restart;
return 0;
}
--
2.45.2
Implementation Analysis
Overview
This patch adds ops.cpu_online() and ops.cpu_offline() callbacks to let BPF schedulers react to CPU hotplug events. These are permanent state changes — a CPU joining or leaving the system — as opposed to the transient preemption events covered by cpu_acquire/release. The patch also introduces a structured exit code system (scx_exit_code) with action and reason fields, and a hotplug_seq mechanism that lets BPF schedulers detect hotplug events that occurred during their initialization window.
If a BPF scheduler does not implement cpu_online/offline and a hotplug event occurs, the scheduler is automatically exited with SCX_ECODE_ACT_RESTART | SCX_ECODE_RSN_HOTPLUG, allowing userspace to simply reload the scheduler.
Code Walkthrough
Hotplug hooks into sched_cpu_activate/deactivate (kernel/sched/core.c):
int sched_cpu_activate(unsigned int cpu) {
...
scx_rq_activate(rq); /* added here */
...
}
int sched_cpu_deactivate(unsigned int cpu) {
...
scx_rq_deactivate(rq); /* added here */
...
}
These are called directly from the CPU hotplug state machine, not from sched_class.rq_online/offline. This was a deliberate design choice (noted in the commit's v3 annotation): the class callbacks are skipped if the state is already at target, but the direct calls are always invoked. It also allows cpu_online/offline to sleep.
handle_hotplug() (kernel/sched/ext.c):
static void handle_hotplug(struct rq *rq, bool online)
{
atomic_long_inc(&scx_hotplug_seq);
if (online && SCX_HAS_OP(cpu_online))
SCX_CALL_OP(SCX_KF_SLEEPABLE, cpu_online, cpu);
else if (!online && SCX_HAS_OP(cpu_offline))
SCX_CALL_OP(SCX_KF_SLEEPABLE, cpu_offline, cpu);
else
scx_ops_exit(SCX_ECODE_ACT_RESTART | SCX_ECODE_RSN_HOTPLUG,
"cpu %d going %s, exiting scheduler", ...);
}
The scx_hotplug_seq counter is incremented unconditionally on every hotplug event. If the BPF scheduler implements neither callback, the scheduler exits with a restart exit code so userspace can reinitialize.
SCX_RQ_ONLINE flag in enum scx_rq_flags (kernel/sched/sched.h):
SCX_RQ_ONLINE = 1 << 0,
Tracks whether ops.cpu_online() has been called for this CPU (and ops.cpu_offline() has not yet been called). The scx_rq_online() helper was previously checking rq->online (the generic kernel field), but is now changed to check the SCX-specific flag:
static bool scx_rq_online(struct rq *rq)
{
return likely(rq->scx.flags & SCX_RQ_ONLINE);
}
This change is critical: a hotplugged CPU begins running and may reach do_enqueue_task before rq_online_scx() has been called (which sets the generic rq->online flag). The comment in do_enqueue_task explains: "If !scx_rq_online(), we already told the BPF scheduler that the CPU is offline and are just running the hotplug path. Don't bother the BPF scheduler."
rq_online_scx / rq_offline_scx hooked into ext_sched_class:
.rq_online = rq_online_scx, /* sets SCX_RQ_ONLINE */
.rq_offline = rq_offline_scx, /* clears SCX_RQ_ONLINE */
Exit code format (enum scx_exit_code):
Bits: [63 .. 48] [47 .. 32] [31 .. 0]
[ SYS ACT ] [ SYS RSN ] [ USR ]
SCX_ECODE_RSN_HOTPLUG = 1LLU << 32, /* reason: hotplug */
SCX_ECODE_ACT_RESTART = 1LLU << 48, /* action: restart */
Userspace uses UEI_ECODE_RESTART(ecode) to detect whether the exit was a restart request.
hotplug_seq race detection (kernel/sched/ext.c):
static void check_hotplug_seq(const struct sched_ext_ops *ops)
{
if (ops->hotplug_seq) {
global_hotplug_seq = atomic_long_read(&scx_hotplug_seq);
if (ops->hotplug_seq != global_hotplug_seq)
scx_ops_exit(SCX_ECODE_ACT_RESTART | SCX_ECODE_RSN_HOTPLUG, ...);
}
}
Called during scx_ops_enable(). If the scheduler read hotplug_seq at open time (via /sys/kernel/sched_ext/hotplug_seq) and a hotplug event happened before it attached, the seqcount will differ and the load fails gracefully. The SCX_OPS_OPEN() macro in compat.h automatically populates hotplug_seq.
Separate SCX_OPI_CPU_HOTPLUG_BEGIN/END range in enum scx_opi:
SCX_OPI_NORMAL_END = SCX_OP_IDX(cpu_online),
SCX_OPI_CPU_HOTPLUG_BEGIN = SCX_OP_IDX(cpu_online),
SCX_OPI_CPU_HOTPLUG_END = SCX_OP_IDX(init),
CPU hotplug ops are enabled earlier during scx_ops_enable() (before cpus_read_unlock()) so the lock ordering between scx_cgroup_rwsem and cpus_read_lock() is preserved.
/sys/kernel/sched_ext/hotplug_seq sysfs attribute: Exposes the current hotplug sequence number to userspace for the race detection mechanism.
scx_qmap example: Implements qmap_cpu_online and qmap_cpu_offline, which both call print_cpus() to log the current online/possible CPU bitmap via bpf_printk() using scx_bpf_get_online_cpumask() and scx_bpf_get_possible_cpumask().
scx_simple and scx_central: Updated to use the goto restart pattern with UEI_ECODE_RESTART(), since they do not implement cpu_online/offline and must restart on hotplug.
Key Concepts
ops.cpu_online(cpu): Called withSCX_KF_SLEEPABLE— it can sleep. The CPU is already online and can run tasks when this callback fires. TheSCX_RQ_ONLINEflag is set byrq_online_scx()which fires slightly later (when the sched_class.rq_onlinemethod is called), creating a brief window.ops.cpu_offline(cpu): Called while the CPU is still online but being deactivated. TheSCX_RQ_ONLINEflag is cleared inrq_offline_scx().SCX_ECODE_ACT_RESTART: A system-defined exit action telling userspace to reinitialize and reload the scheduler. This is not a failure — it is a clean restart protocol.hotplug_seq: A monotonically-increasing counter exposed via sysfs. BPF schedulers that don't implementcpu_online/offlineshould use this to detect if they missed a hotplug event during initialization. Populated automatically bySCX_OPS_OPEN().SCX_RQ_ONLINEvsrq->online: The distinction matters for the narrow window between when the hotplug CPU starts scheduling and whenrq_online_scx()fires. The SCX flag is conservative: enqueue/dispatch only reach BPF while the flag is set.
Locking and Concurrency Notes
ops.cpu_online/offline()are called withSCX_KF_SLEEPABLE— they are called without rq->lock held and may sleep. This enables BPF schedulers to do per-CPU memory allocation or complex initialization in these callbacks.scx_hotplug_seqis anatomic_long_tincremented withatomic_long_inc(), safe from concurrent hotplug events.- The
SCX_RQ_ONLINEflag is inscx_rq->flagswhich is "protected by the rq lock" per the enum comment. Setting/clearing it inrq_online_scx/rq_offline_scxhappens under rq lock. - The hotplug op static keys are enabled under
cpus_read_lock()in a separate pass before the normal ops, satisfying the lock ordering constraint documented in the v2 commit note. check_hotplug_seq()is called insidepercpu_down_write(&scx_fork_rwsem)andcpus_read_lock(), which prevents concurrent CPU topology changes during the check window.
Integration with Kernel Subsystems
The integration is with the CPU hotplug subsystem. The sched_cpu_activate() and sched_cpu_deactivate() functions in kernel/sched/core.c are the canonical hotplug callbacks for the scheduler. Adding scx_rq_activate/deactivate() calls here is the correct integration point — it parallels how cpuset_cpu_active() is called in the same path.
The sysfs attribute hotplug_seq lives under /sys/kernel/sched_ext/ alongside the existing state, switch_all, and nr_rejected attributes, giving userspace a coherent interface.
What Maintainers Need to Know
- If you implement neither
cpu_onlinenorcpu_offline: your scheduler will exit withSCX_ECODE_ACT_RESTART | SCX_ECODE_RSN_HOTPLUGon any hotplug event. CheckUEI_ECODE_RESTART(ecode)in your userspace loop and usegoto restartto reinitialize. TheSCX_OPS_OPEN()macro populateshotplug_seqautomatically to detect races during load. - If you implement both callbacks: you are responsible for updating any per-CPU data structures (available CPU bitmasks, etc.) in the callbacks. You do not need
hotplug_seqbecause you handle events directly. ops.cpu_online/offline()are NOT the same asops.cpu_acquire/release(): online/offline are permanent topology changes; acquire/release are transient preemption events. A newly-online CPU will callcpu_acquire()later when sched_ext first runs balance on it.- Tasks on a CPU going offline are migrated by the kernel before
ops.cpu_offline()fires. The comment indo_enqueue_taskconfirms that onceSCX_RQ_ONLINEis cleared, enqueue/dispatch will bypass the BPF scheduler and use the local DSQ directly. - Sleepability matters: because
cpu_online/offlineare sleepable, you can usebpf_map_update_elem()withGFP_KERNELallocations and other sleeping kfuncs inside these callbacks. This is unusual for scheduler callbacks — most are non-sleepable.
Connection to Other Patches
- Patch 24/30 (cpu_acquire/release): Covers transient CPU ownership. Together with cpu_online/offline, BPF schedulers have complete coverage of all CPU availability state changes.
- Patch 26/30 (PM bypass): The bypass mechanism (
scx_ops_bypass(true/false)) is a complementary way to temporarily suspend BPF scheduling without hotplug. The PM patch and this patch both deal with "the BPF scheduler should not be involved right now" scenarios but with different mechanisms. - Patch 30/30 (selftests): The
hotplugselftest directly exercises this patch's functionality, verifying thatcpu_online/offlinecallbacks fire correctly and that schedulers without these callbacks restart cleanly.
[PATCH 26/30] sched_ext: Bypass BPF scheduler while PM events are in progress
View on Lore: https://lore.kernel.org/all/20240618212056.2833381-27-tj@kernel.org
Commit Message
PM operations freeze userspace. Some BPF schedulers have active userspace
component and may misbehave as expected across PM events. While the system
is frozen, nothing too interesting is happening in terms of scheduling and
we can get by just fine with the fallback FIFO behavior. Let's make things
easier by always bypassing the BPF scheduler while PM events are in
progress.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: David Vernet <dvernet@meta.com>
---
kernel/sched/ext.c | 34 ++++++++++++++++++++++++++++++++++
1 file changed, 34 insertions(+)
diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c
index 7c2f2a542b32..26616cd0c5df 100644
--- a/kernel/sched/ext.c
+++ b/kernel/sched/ext.c
@@ -4825,6 +4825,34 @@ void print_scx_info(const char *log_lvl, struct task_struct *p)
runnable_at_buf);
}
+static int scx_pm_handler(struct notifier_block *nb, unsigned long event, void *ptr)
+{
+ /*
+ * SCX schedulers often have userspace components which are sometimes
+ * involved in critial scheduling paths. PM operations involve freezing
+ * userspace which can lead to scheduling misbehaviors including stalls.
+ * Let's bypass while PM operations are in progress.
+ */
+ switch (event) {
+ case PM_HIBERNATION_PREPARE:
+ case PM_SUSPEND_PREPARE:
+ case PM_RESTORE_PREPARE:
+ scx_ops_bypass(true);
+ break;
+ case PM_POST_HIBERNATION:
+ case PM_POST_SUSPEND:
+ case PM_POST_RESTORE:
+ scx_ops_bypass(false);
+ break;
+ }
+
+ return NOTIFY_OK;
+}
+
+static struct notifier_block scx_pm_notifier = {
+ .notifier_call = scx_pm_handler,
+};
+
void __init init_sched_ext_class(void)
{
s32 cpu, v;
@@ -5729,6 +5757,12 @@ static int __init scx_init(void)
return ret;
}
+ ret = register_pm_notifier(&scx_pm_notifier);
+ if (ret) {
+ pr_err("sched_ext: Failed to register PM notifier (%d)\n", ret);
+ return ret;
+ }
+
scx_kset = kset_create_and_add("sched_ext", &scx_uevent_ops, kernel_kobj);
if (!scx_kset) {
pr_err("sched_ext: Failed to create /sys/kernel/sched_ext\n");
--
2.45.2
Diff
---
kernel/sched/ext.c | 34 ++++++++++++++++++++++++++++++++++
1 file changed, 34 insertions(+)
diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c
index 7c2f2a542b32..26616cd0c5df 100644
--- a/kernel/sched/ext.c
+++ b/kernel/sched/ext.c
@@ -4825,6 +4825,34 @@ void print_scx_info(const char *log_lvl, struct task_struct *p)
runnable_at_buf);
}
+static int scx_pm_handler(struct notifier_block *nb, unsigned long event, void *ptr)
+{
+ /*
+ * sched_ext schedulers often have userspace components which are sometimes
+ * involved in critial scheduling paths. PM operations involve freezing
+ * userspace which can lead to scheduling misbehaviors including stalls.
+ * Let's bypass while PM operations are in progress.
+ */
+ switch (event) {
+ case PM_HIBERNATION_PREPARE:
+ case PM_SUSPEND_PREPARE:
+ case PM_RESTORE_PREPARE:
+ scx_ops_bypass(true);
+ break;
+ case PM_POST_HIBERNATION:
+ case PM_POST_SUSPEND:
+ case PM_POST_RESTORE:
+ scx_ops_bypass(false);
+ break;
+ }
+
+ return NOTIFY_OK;
+}
+
+static struct notifier_block scx_pm_notifier = {
+ .notifier_call = scx_pm_handler,
+};
+
void __init init_sched_ext_class(void)
{
s32 cpu, v;
@@ -5729,6 +5757,12 @@ static int __init scx_init(void)
return ret;
}
+ ret = register_pm_notifier(&scx_pm_notifier);
+ if (ret) {
+ pr_err("sched_ext: Failed to register PM notifier (%d)\n", ret);
+ return ret;
+ }
+
scx_kset = kset_create_and_add("sched_ext", &scx_uevent_ops, kernel_kobj);
if (!scx_kset) {
pr_err("sched_ext: Failed to create /sys/kernel/sched_ext\n");
--
2.45.2
Implementation Analysis
Overview
This patch (PATCH 26/30) adds a power management notifier to sched_ext. When a PM event (suspend, hibernate, restore) is in progress, userspace is frozen. Many BPF schedulers have active userspace components — daemons that provide scheduling hints, load balancing decisions, or topology information. With userspace frozen, these components cannot respond, which can cause scheduling stalls. The fix is simple: bypass the BPF scheduler during PM events, falling back to FIFO behavior, and re-enable it when PM completes.
This is a bypass, not a disable. The BPF program stays loaded; it is just temporarily skipped via the scx_ops_bypass() mechanism.
Code Walkthrough
The PM notifier handler (kernel/sched/ext.c):
static int scx_pm_handler(struct notifier_block *nb, unsigned long event, void *ptr)
{
switch (event) {
case PM_HIBERNATION_PREPARE:
case PM_SUSPEND_PREPARE:
case PM_RESTORE_PREPARE:
scx_ops_bypass(true);
break;
case PM_POST_HIBERNATION:
case PM_POST_SUSPEND:
case PM_POST_RESTORE:
scx_ops_bypass(false);
break;
}
return NOTIFY_OK;
}
static struct notifier_block scx_pm_notifier = {
.notifier_call = scx_pm_handler,
};
Three PM event types are handled on each side. PM_RESTORE_PREPARE covers the case where a hibernation image is being restored (which also freezes tasks). The POST variants undo the bypass after the PM transition is complete.
Registration in scx_init():
ret = register_pm_notifier(&scx_pm_notifier);
if (ret) {
pr_err("sched_ext: Failed to register PM notifier (%d)\n", ret);
return ret;
}
This is called from the module __init function. If registration fails, sched_ext initialization fails entirely. The notifier block is global and static — there is one registration for the lifetime of the kernel module.
What scx_ops_bypass(true) does (documented in the bypass function's comment in ext.c, populated by prior patches):
- Increments
bypass_depth - Sets
scx_switching_all = falsesoops.select_cpu()returns-EBUSYand tasks fall back to CFS dispatch - Tasks already on SCX local DSQs are rotated out at every tick
scx_bpf_kick_cpu()is disabled to avoid irq_work malfunction during PM operationsscx_prio_less()reverts to defaultcore_sched_atordering (from the core-sched patch)
bypass_depth is a counter, not a boolean, so nested bypass calls are safe (e.g., if a future patch adds another bypass condition that can overlap with PM).
Key Concepts
- Bypass vs. disable: Bypass is temporary and leaves the BPF scheduler loaded. The BPF program is not unloaded, BPF maps retain their state, and userspace does not need to reinitialize. When the PM event ends, scheduling resumes exactly where it left off.
PM_RESTORE_PREPARE: This is the hibernation restore case, distinct fromPM_HIBERNATION_PREPARE(saving the image). Both require userspace to be frozen, hence both trigger bypass.NOTIFY_OK: The notifier returnsNOTIFY_OKunconditionally — if the bypass fails for some reason (whichscx_ops_bypass()handles internally), the PM event is not blocked.register_pm_notifier(): Part of the kernel's power management notifier chain (kernel/power/main.c). Notifiers in this chain are called before and after PM state transitions. The notifier block must remain valid for the lifetime of the module.
Locking and Concurrency Notes
scx_ops_bypass(true/false)acquirescpus_read_lock()internally to safely modify the bypass depth and static keys across all CPUs. PM notifiers are called from process context during the PM transition, so sleeping is allowed.- The PM notifier fires before userspace is frozen (in the
PREPAREphase), ensuring that the bypass is active before any userspace scheduler component stops responding. NOTIFY_OK(notNOTIFY_STOPorNOTIFY_BAD) means sched_ext does not block or veto the PM transition. It is a pure observer that adjusts its own behavior.
Integration with Kernel Subsystems
This patch integrates with the kernel's pm_notifier chain, registered via register_pm_notifier(). The PM notifier infrastructure is in kernel/power/. The six event codes handled (PM_{HIBERNATION,SUSPEND,RESTORE}_PREPARE and their POST_ counterparts) cover all major PM transitions that involve freezing userspace tasks.
The choice of scx_init() (the module init function) as the registration point ensures the notifier is active for the entire lifetime of the sched_ext module, including before any BPF scheduler is loaded.
What Maintainers Need to Know
- This is a transparent bypass — BPF schedulers do not need to do anything to benefit from it. All schedulers automatically get PM safety.
- The bypass uses
bypass_depth(a counter), so it is safe to add additional bypass conditions in the future without breaking PM bypass. Eachscx_ops_bypass(true)must be paired with exactly onescx_ops_bypass(false). - BPF schedulers with userspace components (daemons, agents) do not need special PM handling code on the BPF side. The bypass handles it at the kernel level.
- cpufreq transitions are NOT covered by this patch. The commit message refers specifically to "PM operations" which freeze userspace. cpufreq governor transitions do not freeze userspace and are not intercepted here.
- If
register_pm_notifier()fails duringscx_init(), sched_ext will not load at all. This is intentional — failing to register the PM notifier would leave the system potentially vulnerable to PM-induced scheduling stalls. - During bypass, the
scx_show_state.pydrgn script will showbypass_depth > 0.
Connection to Other Patches
- Patch 25/30 (cpu_online/offline): CPU hotplug also suspends/resumes CPUs. PM events interact with hotplug but are separate kernel mechanisms. The bypass here covers the scheduling correctness problem during PM; hotplug callbacks handle the CPU topology changes.
- Patch 27/30 (core-sched): The bypass disables
ops.core_sched_before()by makingscx_prio_less()fall back to defaultcore_sched_atordering. This is explicitly noted as bullet pointfin the bypass function's comment. - The bypass mechanism itself was introduced in earlier patches in this series. This patch is a consumer of that existing infrastructure.
[PATCH 27/30] sched_ext: Implement core-sched support
View on Lore: https://lore.kernel.org/all/20240618212056.2833381-28-tj@kernel.org
Commit Message
The core-sched support is composed of the following parts:
- task_struct->scx.core_sched_at is added. This is a timestamp which can be
used to order tasks. Depending on whether the BPF scheduler implements
custom ordering, it tracks either global FIFO ordering of all tasks or
local-DSQ ordering within the dispatched tasks on a CPU.
- prio_less() is updated to call scx_prio_less() when comparing SCX tasks.
scx_prio_less() calls ops.core_sched_before() if available or uses the
core_sched_at timestamp. For global FIFO ordering, the BPF scheduler
doesn't need to do anything. Otherwise, it should implement
ops.core_sched_before() which reflects the ordering.
- When core-sched is enabled, balance_scx() balances all SMT siblings so
that they all have tasks dispatched if necessary before pick_task_scx() is
called. pick_task_scx() picks between the current task and the first
dispatched task on the local DSQ based on availability and the
core_sched_at timestamps. Note that FIFO ordering is expected among the
already dispatched tasks whether running or on the local DSQ, so this path
always compares core_sched_at instead of calling into
ops.core_sched_before().
qmap_core_sched_before() is added to scx_qmap. It scales the
distances from the heads of the queues to compare the tasks across different
priority queues and seems to behave as expected.
v3: Fixed build error when !CONFIG_SCHED_SMT reported by Andrea Righi.
v2: Sched core added the const qualifiers to prio_less task arguments.
Explicitly drop them for ops.core_sched_before() task arguments. BPF
enforces access control through the verifier, so the qualifier isn't
actually operative and only gets in the way when interacting with
various helpers.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: David Vernet <dvernet@meta.com>
Reviewed-by: Josh Don <joshdon@google.com>
Cc: Andrea Righi <andrea.righi@canonical.com>
---
include/linux/sched/ext.h | 3 +
kernel/Kconfig.preempt | 2 +-
kernel/sched/core.c | 10 +-
kernel/sched/ext.c | 250 +++++++++++++++++++++++++++++++--
kernel/sched/ext.h | 5 +
tools/sched_ext/scx_qmap.bpf.c | 91 +++++++++++-
tools/sched_ext/scx_qmap.c | 5 +-
7 files changed, 346 insertions(+), 20 deletions(-)
diff --git a/include/linux/sched/ext.h b/include/linux/sched/ext.h
index 21c627337e01..3db7b32b2d1d 100644
--- a/include/linux/sched/ext.h
+++ b/include/linux/sched/ext.h
@@ -129,6 +129,9 @@ struct sched_ext_entity {
struct list_head runnable_node; /* rq->scx.runnable_list */
unsigned long runnable_at;
+#ifdef CONFIG_SCHED_CORE
+ u64 core_sched_at; /* see scx_prio_less() */
+#endif
u64 ddsp_dsq_id;
u64 ddsp_enq_flags;
diff --git a/kernel/Kconfig.preempt b/kernel/Kconfig.preempt
index 39ecfc2b5a1c..7dde5e424ac3 100644
--- a/kernel/Kconfig.preempt
+++ b/kernel/Kconfig.preempt
@@ -135,7 +135,7 @@ config SCHED_CORE
config SCHED_CLASS_EXT
bool "Extensible Scheduling Class"
- depends on BPF_SYSCALL && BPF_JIT && !SCHED_CORE
+ depends on BPF_SYSCALL && BPF_JIT
help
This option enables a new scheduler class sched_ext (SCX), which
allows scheduling policies to be implemented as BPF programs to
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index c798c847d57e..5eec6639773b 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -169,7 +169,10 @@ static inline int __task_prio(const struct task_struct *p)
if (p->sched_class == &idle_sched_class)
return MAX_RT_PRIO + NICE_WIDTH; /* 140 */
- return MAX_RT_PRIO + MAX_NICE; /* 120, squash fair */
+ if (task_on_scx(p))
+ return MAX_RT_PRIO + MAX_NICE + 1; /* 120, squash ext */
+
+ return MAX_RT_PRIO + MAX_NICE; /* 119, squash fair */
}
/*
@@ -198,6 +201,11 @@ static inline bool prio_less(const struct task_struct *a,
if (pa == MAX_RT_PRIO + MAX_NICE) /* fair */
return cfs_prio_less(a, b, in_fi);
+#ifdef CONFIG_SCHED_CLASS_EXT
+ if (pa == MAX_RT_PRIO + MAX_NICE + 1) /* ext */
+ return scx_prio_less(a, b, in_fi);
+#endif
+
return false;
}
diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c
index 26616cd0c5df..1feb690be9d8 100644
--- a/kernel/sched/ext.c
+++ b/kernel/sched/ext.c
@@ -344,6 +344,24 @@ struct sched_ext_ops {
*/
bool (*yield)(struct task_struct *from, struct task_struct *to);
+ /**
+ * core_sched_before - Task ordering for core-sched
+ * @a: task A
+ * @b: task B
+ *
+ * Used by core-sched to determine the ordering between two tasks. See
+ * Documentation/admin-guide/hw-vuln/core-scheduling.rst for details on
+ * core-sched.
+ *
+ * Both @a and @b are runnable and may or may not currently be queued on
+ * the BPF scheduler. Should return %true if @a should run before @b.
+ * %false if there's no required ordering or @b should run before @a.
+ *
+ * If not specified, the default is ordering them according to when they
+ * became runnable.
+ */
+ bool (*core_sched_before)(struct task_struct *a, struct task_struct *b);
+
/**
* set_weight - Set task weight
* @p: task to set weight for
@@ -625,6 +643,14 @@ enum scx_enq_flags {
enum scx_deq_flags {
/* expose select DEQUEUE_* flags as enums */
SCX_DEQ_SLEEP = DEQUEUE_SLEEP,
+
+ /* high 32bits are SCX specific */
+
+ /*
+ * The generic core-sched layer decided to execute the task even though
+ * it hasn't been dispatched yet. Dequeue from the BPF side.
+ */
+ SCX_DEQ_CORE_SCHED_EXEC = 1LLU << 32,
};
enum scx_pick_idle_cpu_flags {
@@ -1260,6 +1286,49 @@ static int ops_sanitize_err(const char *ops_name, s32 err)
return -EPROTO;
}
+/**
+ * touch_core_sched - Update timestamp used for core-sched task ordering
+ * @rq: rq to read clock from, must be locked
+ * @p: task to update the timestamp for
+ *
+ * Update @p->scx.core_sched_at timestamp. This is used by scx_prio_less() to
+ * implement global or local-DSQ FIFO ordering for core-sched. Should be called
+ * when a task becomes runnable and its turn on the CPU ends (e.g. slice
+ * exhaustion).
+ */
+static void touch_core_sched(struct rq *rq, struct task_struct *p)
+{
+#ifdef CONFIG_SCHED_CORE
+ /*
+ * It's okay to update the timestamp spuriously. Use
+ * sched_core_disabled() which is cheaper than enabled().
+ */
+ if (!sched_core_disabled())
+ p->scx.core_sched_at = rq_clock_task(rq);
+#endif
+}
+
+/**
+ * touch_core_sched_dispatch - Update core-sched timestamp on dispatch
+ * @rq: rq to read clock from, must be locked
+ * @p: task being dispatched
+ *
+ * If the BPF scheduler implements custom core-sched ordering via
+ * ops.core_sched_before(), @p->scx.core_sched_at is used to implement FIFO
+ * ordering within each local DSQ. This function is called from dispatch paths
+ * and updates @p->scx.core_sched_at if custom core-sched ordering is in effect.
+ */
+static void touch_core_sched_dispatch(struct rq *rq, struct task_struct *p)
+{
+ lockdep_assert_rq_held(rq);
+ assert_clock_updated(rq);
+
+#ifdef CONFIG_SCHED_CORE
+ if (SCX_HAS_OP(core_sched_before))
+ touch_core_sched(rq, p);
+#endif
+}
+
static void update_curr_scx(struct rq *rq)
{
struct task_struct *curr = rq->curr;
@@ -1275,8 +1344,11 @@ static void update_curr_scx(struct rq *rq)
account_group_exec_runtime(curr, delta_exec);
cgroup_account_cputime(curr, delta_exec);
- if (curr->scx.slice != SCX_SLICE_INF)
+ if (curr->scx.slice != SCX_SLICE_INF) {
curr->scx.slice -= min(curr->scx.slice, delta_exec);
+ if (!curr->scx.slice)
+ touch_core_sched(rq, curr);
+ }
}
static void dsq_mod_nr(struct scx_dispatch_q *dsq, s32 delta)
@@ -1469,6 +1541,8 @@ static void direct_dispatch(struct task_struct *p, u64 enq_flags)
{
struct scx_dispatch_q *dsq;
+ touch_core_sched_dispatch(task_rq(p), p);
+
enq_flags |= (p->scx.ddsp_enq_flags | SCX_ENQ_CLEAR_OPSS);
dsq = find_dsq_for_dispatch(task_rq(p), p->scx.ddsp_dsq_id, p);
dispatch_enqueue(dsq, p, enq_flags);
@@ -1550,12 +1624,19 @@ static void do_enqueue_task(struct rq *rq, struct task_struct *p, u64 enq_flags,
return;
local:
+ /*
+ * For task-ordering, slice refill must be treated as implying the end
+ * of the current slice. Otherwise, the longer @p stays on the CPU, the
+ * higher priority it becomes from scx_prio_less()'s POV.
+ */
+ touch_core_sched(rq, p);
p->scx.slice = SCX_SLICE_DFL;
local_norefill:
dispatch_enqueue(&rq->scx.local_dsq, p, enq_flags);
return;
global:
+ touch_core_sched(rq, p); /* see the comment in local: */
p->scx.slice = SCX_SLICE_DFL;
dispatch_enqueue(&scx_dsq_global, p, enq_flags);
}
@@ -1619,6 +1700,9 @@ static void enqueue_task_scx(struct rq *rq, struct task_struct *p, int enq_flags
if (SCX_HAS_OP(runnable))
SCX_CALL_OP_TASK(SCX_KF_REST, runnable, p, enq_flags);
+ if (enq_flags & SCX_ENQ_WAKEUP)
+ touch_core_sched(rq, p);
+
do_enqueue_task(rq, p, enq_flags, sticky_cpu);
}
@@ -2106,6 +2190,7 @@ static void finish_dispatch(struct rq *rq, struct rq_flags *rf,
struct scx_dispatch_q *dsq;
unsigned long opss;
+ touch_core_sched_dispatch(rq, p);
retry:
/*
* No need for _acquire here. @p is accessed only after a successful
@@ -2183,8 +2268,8 @@ static void flush_dispatch_buf(struct rq *rq, struct rq_flags *rf)
dspc->cursor = 0;
}
-static int balance_scx(struct rq *rq, struct task_struct *prev,
- struct rq_flags *rf)
+static int balance_one(struct rq *rq, struct task_struct *prev,
+ struct rq_flags *rf, bool local)
{
struct scx_dsp_ctx *dspc = this_cpu_ptr(scx_dsp_ctx);
bool prev_on_scx = prev->sched_class == &ext_sched_class;
@@ -2208,7 +2293,7 @@ static int balance_scx(struct rq *rq, struct task_struct *prev,
}
if (prev_on_scx) {
- WARN_ON_ONCE(prev->scx.flags & SCX_TASK_BAL_KEEP);
+ WARN_ON_ONCE(local && (prev->scx.flags & SCX_TASK_BAL_KEEP));
update_curr_scx(rq);
/*
@@ -2220,10 +2305,16 @@ static int balance_scx(struct rq *rq, struct task_struct *prev,
*
* See scx_ops_disable_workfn() for the explanation on the
* bypassing test.
+ *
+ * When balancing a remote CPU for core-sched, there won't be a
+ * following put_prev_task_scx() call and we don't own
+ * %SCX_TASK_BAL_KEEP. Instead, pick_task_scx() will test the
+ * same conditions later and pick @rq->curr accordingly.
*/
if ((prev->scx.flags & SCX_TASK_QUEUED) &&
prev->scx.slice && !scx_ops_bypassing()) {
- prev->scx.flags |= SCX_TASK_BAL_KEEP;
+ if (local)
+ prev->scx.flags |= SCX_TASK_BAL_KEEP;
goto has_tasks;
}
}
@@ -2285,10 +2376,56 @@ static int balance_scx(struct rq *rq, struct task_struct *prev,
return has_tasks;
}
+static int balance_scx(struct rq *rq, struct task_struct *prev,
+ struct rq_flags *rf)
+{
+ int ret;
+
+ ret = balance_one(rq, prev, rf, true);
+
+#ifdef CONFIG_SCHED_SMT
+ /*
+ * When core-sched is enabled, this ops.balance() call will be followed
+ * by put_prev_scx() and pick_task_scx() on this CPU and pick_task_scx()
+ * on the SMT siblings. Balance the siblings too.
+ */
+ if (sched_core_enabled(rq)) {
+ const struct cpumask *smt_mask = cpu_smt_mask(cpu_of(rq));
+ int scpu;
+
+ for_each_cpu_andnot(scpu, smt_mask, cpumask_of(cpu_of(rq))) {
+ struct rq *srq = cpu_rq(scpu);
+ struct rq_flags srf;
+ struct task_struct *sprev = srq->curr;
+
+ /*
+ * While core-scheduling, rq lock is shared among
+ * siblings but the debug annotations and rq clock
+ * aren't. Do pinning dance to transfer the ownership.
+ */
+ WARN_ON_ONCE(__rq_lockp(rq) != __rq_lockp(srq));
+ rq_unpin_lock(rq, rf);
+ rq_pin_lock(srq, &srf);
+
+ update_rq_clock(srq);
+ balance_one(srq, sprev, &srf, false);
+
+ rq_unpin_lock(srq, &srf);
+ rq_repin_lock(rq, rf);
+ }
+ }
+#endif
+ return ret;
+}
+
static void set_next_task_scx(struct rq *rq, struct task_struct *p, bool first)
{
if (p->scx.flags & SCX_TASK_QUEUED) {
- WARN_ON_ONCE(atomic_long_read(&p->scx.ops_state) != SCX_OPSS_NONE);
+ /*
+ * Core-sched might decide to execute @p before it is
+ * dispatched. Call ops_dequeue() to notify the BPF scheduler.
+ */
+ ops_dequeue(p, SCX_DEQ_CORE_SCHED_EXEC);
dispatch_dequeue(rq, p);
}
@@ -2379,7 +2516,8 @@ static void put_prev_task_scx(struct rq *rq, struct task_struct *p)
/*
* If @p has slice left and balance_scx() didn't tag it for
* keeping, @p is getting preempted by a higher priority
- * scheduler class. Leave it at the head of the local DSQ.
+ * scheduler class or core-sched forcing a different task. Leave
+ * it at the head of the local DSQ.
*/
if (p->scx.slice && !scx_ops_bypassing()) {
dispatch_enqueue(&rq->scx.local_dsq, p, SCX_ENQ_HEAD);
@@ -2436,6 +2574,84 @@ static struct task_struct *pick_next_task_scx(struct rq *rq)
return p;
}
+#ifdef CONFIG_SCHED_CORE
+/**
+ * scx_prio_less - Task ordering for core-sched
+ * @a: task A
+ * @b: task B
+ *
+ * Core-sched is implemented as an additional scheduling layer on top of the
+ * usual sched_class'es and needs to find out the expected task ordering. For
+ * SCX, core-sched calls this function to interrogate the task ordering.
+ *
+ * Unless overridden by ops.core_sched_before(), @p->scx.core_sched_at is used
+ * to implement the default task ordering. The older the timestamp, the higher
+ * prority the task - the global FIFO ordering matching the default scheduling
+ * behavior.
+ *
+ * When ops.core_sched_before() is enabled, @p->scx.core_sched_at is used to
+ * implement FIFO ordering within each local DSQ. See pick_task_scx().
+ */
+bool scx_prio_less(const struct task_struct *a, const struct task_struct *b,
+ bool in_fi)
+{
+ /*
+ * The const qualifiers are dropped from task_struct pointers when
+ * calling ops.core_sched_before(). Accesses are controlled by the
+ * verifier.
+ */
+ if (SCX_HAS_OP(core_sched_before) && !scx_ops_bypassing())
+ return SCX_CALL_OP_2TASKS_RET(SCX_KF_REST, core_sched_before,
+ (struct task_struct *)a,
+ (struct task_struct *)b);
+ else
+ return time_after64(a->scx.core_sched_at, b->scx.core_sched_at);
+}
+
+/**
+ * pick_task_scx - Pick a candidate task for core-sched
+ * @rq: rq to pick the candidate task from
+ *
+ * Core-sched calls this function on each SMT sibling to determine the next
+ * tasks to run on the SMT siblings. balance_one() has been called on all
+ * siblings and put_prev_task_scx() has been called only for the current CPU.
+ *
+ * As put_prev_task_scx() hasn't been called on remote CPUs, we can't just look
+ * at the first task in the local dsq. @rq->curr has to be considered explicitly
+ * to mimic %SCX_TASK_BAL_KEEP.
+ */
+static struct task_struct *pick_task_scx(struct rq *rq)
+{
+ struct task_struct *curr = rq->curr;
+ struct task_struct *first = first_local_task(rq);
+
+ if (curr->scx.flags & SCX_TASK_QUEUED) {
+ /* is curr the only runnable task? */
+ if (!first)
+ return curr;
+
+ /*
+ * Does curr trump first? We can always go by core_sched_at for
+ * this comparison as it represents global FIFO ordering when
+ * the default core-sched ordering is used and local-DSQ FIFO
+ * ordering otherwise.
+ *
+ * We can have a task with an earlier timestamp on the DSQ. For
+ * example, when a current task is preempted by a sibling
+ * picking a different cookie, the task would be requeued at the
+ * head of the local DSQ with an earlier timestamp than the
+ * core-sched picked next task. Besides, the BPF scheduler may
+ * dispatch any tasks to the local DSQ anytime.
+ */
+ if (curr->scx.slice && time_before64(curr->scx.core_sched_at,
+ first->scx.core_sched_at))
+ return curr;
+ }
+
+ return first; /* this may be %NULL */
+}
+#endif /* CONFIG_SCHED_CORE */
+
static enum scx_cpu_preempt_reason
preempt_reason_from_class(const struct sched_class *class)
{
@@ -2843,13 +3059,15 @@ static void task_tick_scx(struct rq *rq, struct task_struct *curr, int queued)
update_curr_scx(rq);
/*
- * While bypassing, always resched as we can't trust the slice
- * management.
+ * While disabling, always resched and refresh core-sched timestamp as
+ * we can't trust the slice management or ops.core_sched_before().
*/
- if (scx_ops_bypassing())
+ if (scx_ops_bypassing()) {
curr->scx.slice = 0;
- else if (SCX_HAS_OP(tick))
+ touch_core_sched(rq, curr);
+ } else if (SCX_HAS_OP(tick)) {
SCX_CALL_OP(SCX_KF_REST, tick, curr);
+ }
if (!curr->scx.slice)
resched_curr(rq);
@@ -3203,6 +3421,10 @@ DEFINE_SCHED_CLASS(ext) = {
.rq_offline = rq_offline_scx,
#endif
+#ifdef CONFIG_SCHED_CORE
+ .pick_task = pick_task_scx,
+#endif
+
.task_tick = task_tick_scx,
.switching_to = switching_to_scx,
@@ -3416,12 +3638,14 @@ bool task_should_scx(struct task_struct *p)
*
* c. balance_scx() never sets %SCX_TASK_BAL_KEEP as the slice value can't be
* trusted. Whenever a tick triggers, the running task is rotated to the tail
- * of the queue.
+ * of the queue with core_sched_at touched.
*
* d. pick_next_task() suppresses zero slice warning.
*
* e. scx_bpf_kick_cpu() is disabled to avoid irq_work malfunction during PM
* operations.
+ *
+ * f. scx_prio_less() reverts to the default core_sched_at order.
*/
static void scx_ops_bypass(bool bypass)
{
@@ -4583,6 +4807,7 @@ static void running_stub(struct task_struct *p) {}
static void stopping_stub(struct task_struct *p, bool runnable) {}
static void quiescent_stub(struct task_struct *p, u64 deq_flags) {}
static bool yield_stub(struct task_struct *from, struct task_struct *to) { return false; }
+static bool core_sched_before_stub(struct task_struct *a, struct task_struct *b) { return false; }
static void set_weight_stub(struct task_struct *p, u32 weight) {}
static void set_cpumask_stub(struct task_struct *p, const struct cpumask *mask) {}
static void update_idle_stub(s32 cpu, bool idle) {}
@@ -4607,6 +4832,7 @@ static struct sched_ext_ops __bpf_ops_sched_ext_ops = {
.stopping = stopping_stub,
.quiescent = quiescent_stub,
.yield = yield_stub,
+ .core_sched_before = core_sched_before_stub,
.set_weight = set_weight_stub,
.set_cpumask = set_cpumask_stub,
.update_idle = update_idle_stub,
diff --git a/kernel/sched/ext.h b/kernel/sched/ext.h
index 037f9acdf443..6555878c5da3 100644
--- a/kernel/sched/ext.h
+++ b/kernel/sched/ext.h
@@ -70,6 +70,11 @@ static inline const struct sched_class *next_active_class(const struct sched_cla
for_active_class_range(class, (prev_class) > &ext_sched_class ? \
&ext_sched_class : (prev_class), (end_class))
+#ifdef CONFIG_SCHED_CORE
+bool scx_prio_less(const struct task_struct *a, const struct task_struct *b,
+ bool in_fi);
+#endif
+
#else /* CONFIG_SCHED_CLASS_EXT */
#define scx_enabled() false
diff --git a/tools/sched_ext/scx_qmap.bpf.c b/tools/sched_ext/scx_qmap.bpf.c
index 619078355bf5..c75c70d6a8eb 100644
--- a/tools/sched_ext/scx_qmap.bpf.c
+++ b/tools/sched_ext/scx_qmap.bpf.c
@@ -13,6 +13,7 @@
* - Sleepable per-task storage allocation using ops.prep_enable().
* - Using ops.cpu_release() to handle a higher priority scheduling class taking
* the CPU away.
+ * - Core-sched support.
*
* This scheduler is primarily for demonstration and testing of sched_ext
* features and unlikely to be useful for actual workloads.
@@ -67,9 +68,21 @@ struct {
},
};
+/*
+ * Per-queue sequence numbers to implement core-sched ordering.
+ *
+ * Tail seq is assigned to each queued task and incremented. Head seq tracks the
+ * sequence number of the latest dispatched task. The distance between the a
+ * task's seq and the associated queue's head seq is called the queue distance
+ * and used when comparing two tasks for ordering. See qmap_core_sched_before().
+ */
+static u64 core_sched_head_seqs[5];
+static u64 core_sched_tail_seqs[5];
+
/* Per-task scheduling context */
struct task_ctx {
bool force_local; /* Dispatch directly to local_dsq */
+ u64 core_sched_seq;
};
struct {
@@ -93,6 +106,7 @@ struct {
/* Statistics */
u64 nr_enqueued, nr_dispatched, nr_reenqueued, nr_dequeued;
+u64 nr_core_sched_execed;
s32 BPF_STRUCT_OPS(qmap_select_cpu, struct task_struct *p,
s32 prev_cpu, u64 wake_flags)
@@ -159,8 +173,18 @@ void BPF_STRUCT_OPS(qmap_enqueue, struct task_struct *p, u64 enq_flags)
return;
}
- /* Is select_cpu() is telling us to enqueue locally? */
- if (tctx->force_local) {
+ /*
+ * All enqueued tasks must have their core_sched_seq updated for correct
+ * core-sched ordering, which is why %SCX_OPS_ENQ_LAST is specified in
+ * qmap_ops.flags.
+ */
+ tctx->core_sched_seq = core_sched_tail_seqs[idx]++;
+
+ /*
+ * If qmap_select_cpu() is telling us to or this is the last runnable
+ * task on the CPU, enqueue locally.
+ */
+ if (tctx->force_local || (enq_flags & SCX_ENQ_LAST)) {
tctx->force_local = false;
scx_bpf_dispatch(p, SCX_DSQ_LOCAL, slice_ns, enq_flags);
return;
@@ -204,6 +228,19 @@ void BPF_STRUCT_OPS(qmap_enqueue, struct task_struct *p, u64 enq_flags)
void BPF_STRUCT_OPS(qmap_dequeue, struct task_struct *p, u64 deq_flags)
{
__sync_fetch_and_add(&nr_dequeued, 1);
+ if (deq_flags & SCX_DEQ_CORE_SCHED_EXEC)
+ __sync_fetch_and_add(&nr_core_sched_execed, 1);
+}
+
+static void update_core_sched_head_seq(struct task_struct *p)
+{
+ struct task_ctx *tctx = bpf_task_storage_get(&task_ctx_stor, p, 0, 0);
+ int idx = weight_to_idx(p->scx.weight);
+
+ if (tctx)
+ core_sched_head_seqs[idx] = tctx->core_sched_seq;
+ else
+ scx_bpf_error("task_ctx lookup failed");
}
void BPF_STRUCT_OPS(qmap_dispatch, s32 cpu, struct task_struct *prev)
@@ -258,6 +295,7 @@ void BPF_STRUCT_OPS(qmap_dispatch, s32 cpu, struct task_struct *prev)
if (!p)
continue;
+ update_core_sched_head_seq(p);
__sync_fetch_and_add(&nr_dispatched, 1);
scx_bpf_dispatch(p, SHARED_DSQ, slice_ns, 0);
bpf_task_release(p);
@@ -275,6 +313,49 @@ void BPF_STRUCT_OPS(qmap_dispatch, s32 cpu, struct task_struct *prev)
}
}
+/*
+ * The distance from the head of the queue scaled by the weight of the queue.
+ * The lower the number, the older the task and the higher the priority.
+ */
+static s64 task_qdist(struct task_struct *p)
+{
+ int idx = weight_to_idx(p->scx.weight);
+ struct task_ctx *tctx;
+ s64 qdist;
+
+ tctx = bpf_task_storage_get(&task_ctx_stor, p, 0, 0);
+ if (!tctx) {
+ scx_bpf_error("task_ctx lookup failed");
+ return 0;
+ }
+
+ qdist = tctx->core_sched_seq - core_sched_head_seqs[idx];
+
+ /*
+ * As queue index increments, the priority doubles. The queue w/ index 3
+ * is dispatched twice more frequently than 2. Reflect the difference by
+ * scaling qdists accordingly. Note that the shift amount needs to be
+ * flipped depending on the sign to avoid flipping priority direction.
+ */
+ if (qdist >= 0)
+ return qdist << (4 - idx);
+ else
+ return qdist << idx;
+}
+
+/*
+ * This is called to determine the task ordering when core-sched is picking
+ * tasks to execute on SMT siblings and should encode about the same ordering as
+ * the regular scheduling path. Use the priority-scaled distances from the head
+ * of the queues to compare the two tasks which should be consistent with the
+ * dispatch path behavior.
+ */
+bool BPF_STRUCT_OPS(qmap_core_sched_before,
+ struct task_struct *a, struct task_struct *b)
+{
+ return task_qdist(a) > task_qdist(b);
+}
+
void BPF_STRUCT_OPS(qmap_cpu_release, s32 cpu, struct scx_cpu_release_args *args)
{
u32 cnt;
@@ -354,8 +435,8 @@ void BPF_STRUCT_OPS(qmap_dump_task, struct scx_dump_ctx *dctx, struct task_struc
if (!(taskc = bpf_task_storage_get(&task_ctx_stor, p, 0, 0)))
return;
- scx_bpf_dump("QMAP: force_local=%d",
- taskc->force_local);
+ scx_bpf_dump("QMAP: force_local=%d core_sched_seq=%llu",
+ taskc->force_local, taskc->core_sched_seq);
}
/*
@@ -428,6 +509,7 @@ SCX_OPS_DEFINE(qmap_ops,
.enqueue = (void *)qmap_enqueue,
.dequeue = (void *)qmap_dequeue,
.dispatch = (void *)qmap_dispatch,
+ .core_sched_before = (void *)qmap_core_sched_before,
.cpu_release = (void *)qmap_cpu_release,
.init_task = (void *)qmap_init_task,
.dump = (void *)qmap_dump,
@@ -437,5 +519,6 @@ SCX_OPS_DEFINE(qmap_ops,
.cpu_offline = (void *)qmap_cpu_offline,
.init = (void *)qmap_init,
.exit = (void *)qmap_exit,
+ .flags = SCX_OPS_ENQ_LAST,
.timeout_ms = 5000U,
.name = "qmap");
diff --git a/tools/sched_ext/scx_qmap.c b/tools/sched_ext/scx_qmap.c
index 920fb54f9c77..bc36ec4f88a7 100644
--- a/tools/sched_ext/scx_qmap.c
+++ b/tools/sched_ext/scx_qmap.c
@@ -112,9 +112,10 @@ int main(int argc, char **argv)
long nr_enqueued = skel->bss->nr_enqueued;
long nr_dispatched = skel->bss->nr_dispatched;
- printf("stats : enq=%lu dsp=%lu delta=%ld reenq=%"PRIu64" deq=%"PRIu64"\n",
+ printf("stats : enq=%lu dsp=%lu delta=%ld reenq=%"PRIu64" deq=%"PRIu64" core=%"PRIu64"\n",
nr_enqueued, nr_dispatched, nr_enqueued - nr_dispatched,
- skel->bss->nr_reenqueued, skel->bss->nr_dequeued);
+ skel->bss->nr_reenqueued, skel->bss->nr_dequeued,
+ skel->bss->nr_core_sched_execed);
fflush(stdout);
sleep(1);
}
--
2.45.2
Diff
---
include/linux/sched/ext.h | 3 +
kernel/Kconfig.preempt | 2 +-
kernel/sched/core.c | 10 +-
kernel/sched/ext.c | 250 +++++++++++++++++++++++++++++++--
kernel/sched/ext.h | 5 +
tools/sched_ext/scx_qmap.bpf.c | 91 +++++++++++-
tools/sched_ext/scx_qmap.c | 5 +-
7 files changed, 346 insertions(+), 20 deletions(-)
diff --git a/include/linux/sched/ext.h b/include/linux/sched/ext.h
index 21c627337e01..3db7b32b2d1d 100644
--- a/include/linux/sched/ext.h
+++ b/include/linux/sched/ext.h
@@ -129,6 +129,9 @@ struct sched_ext_entity {
struct list_head runnable_node; /* rq->scx.runnable_list */
unsigned long runnable_at;
+#ifdef CONFIG_SCHED_CORE
+ u64 core_sched_at; /* see scx_prio_less() */
+#endif
u64 ddsp_dsq_id;
u64 ddsp_enq_flags;
diff --git a/kernel/Kconfig.preempt b/kernel/Kconfig.preempt
index 39ecfc2b5a1c..7dde5e424ac3 100644
--- a/kernel/Kconfig.preempt
+++ b/kernel/Kconfig.preempt
@@ -135,7 +135,7 @@ config SCHED_CORE
config SCHED_CLASS_EXT
bool "Extensible Scheduling Class"
- depends on BPF_SYSCALL && BPF_JIT && !SCHED_CORE
+ depends on BPF_SYSCALL && BPF_JIT
help
This option enables a new scheduler class sched_ext (sched_ext), which
allows scheduling policies to be implemented as BPF programs to
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index c798c847d57e..5eec6639773b 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -169,7 +169,10 @@ static inline int __task_prio(const struct task_struct *p)
if (p->sched_class == &idle_sched_class)
return MAX_RT_PRIO + NICE_WIDTH; /* 140 */
- return MAX_RT_PRIO + MAX_NICE; /* 120, squash fair */
+ if (task_on_scx(p))
+ return MAX_RT_PRIO + MAX_NICE + 1; /* 120, squash ext */
+
+ return MAX_RT_PRIO + MAX_NICE; /* 119, squash fair */
}
/*
@@ -198,6 +201,11 @@ static inline bool prio_less(const struct task_struct *a,
if (pa == MAX_RT_PRIO + MAX_NICE) /* fair */
return cfs_prio_less(a, b, in_fi);
+#ifdef CONFIG_SCHED_CLASS_EXT
+ if (pa == MAX_RT_PRIO + MAX_NICE + 1) /* ext */
+ return scx_prio_less(a, b, in_fi);
+#endif
+
return false;
}
diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c
index 26616cd0c5df..1feb690be9d8 100644
--- a/kernel/sched/ext.c
+++ b/kernel/sched/ext.c
@@ -344,6 +344,24 @@ struct sched_ext_ops {
*/
bool (*yield)(struct task_struct *from, struct task_struct *to);
+ /**
+ * core_sched_before - Task ordering for core-sched
+ * @a: task A
+ * @b: task B
+ *
+ * Used by core-sched to determine the ordering between two tasks. See
+ * Documentation/admin-guide/hw-vuln/core-scheduling.rst for details on
+ * core-sched.
+ *
+ * Both @a and @b are runnable and may or may not currently be queued on
+ * the BPF scheduler. Should return %true if @a should run before @b.
+ * %false if there's no required ordering or @b should run before @a.
+ *
+ * If not specified, the default is ordering them according to when they
+ * became runnable.
+ */
+ bool (*core_sched_before)(struct task_struct *a, struct task_struct *b);
+
/**
* set_weight - Set task weight
* @p: task to set weight for
@@ -625,6 +643,14 @@ enum scx_enq_flags {
enum scx_deq_flags {
/* expose select DEQUEUE_* flags as enums */
SCX_DEQ_SLEEP = DEQUEUE_SLEEP,
+
+ /* high 32bits are sched_ext specific */
+
+ /*
+ * The generic core-sched layer decided to execute the task even though
+ * it hasn't been dispatched yet. Dequeue from the BPF side.
+ */
+ SCX_DEQ_CORE_SCHED_EXEC = 1LLU << 32,
};
enum scx_pick_idle_cpu_flags {
@@ -1260,6 +1286,49 @@ static int ops_sanitize_err(const char *ops_name, s32 err)
return -EPROTO;
}
+/**
+ * touch_core_sched - Update timestamp used for core-sched task ordering
+ * @rq: rq to read clock from, must be locked
+ * @p: task to update the timestamp for
+ *
+ * Update @p->scx.core_sched_at timestamp. This is used by scx_prio_less() to
+ * implement global or local-DSQ FIFO ordering for core-sched. Should be called
+ * when a task becomes runnable and its turn on the CPU ends (e.g. slice
+ * exhaustion).
+ */
+static void touch_core_sched(struct rq *rq, struct task_struct *p)
+{
+#ifdef CONFIG_SCHED_CORE
+ /*
+ * It's okay to update the timestamp spuriously. Use
+ * sched_core_disabled() which is cheaper than enabled().
+ */
+ if (!sched_core_disabled())
+ p->scx.core_sched_at = rq_clock_task(rq);
+#endif
+}
+
+/**
+ * touch_core_sched_dispatch - Update core-sched timestamp on dispatch
+ * @rq: rq to read clock from, must be locked
+ * @p: task being dispatched
+ *
+ * If the BPF scheduler implements custom core-sched ordering via
+ * ops.core_sched_before(), @p->scx.core_sched_at is used to implement FIFO
+ * ordering within each local DSQ. This function is called from dispatch paths
+ * and updates @p->scx.core_sched_at if custom core-sched ordering is in effect.
+ */
+static void touch_core_sched_dispatch(struct rq *rq, struct task_struct *p)
+{
+ lockdep_assert_rq_held(rq);
+ assert_clock_updated(rq);
+
+#ifdef CONFIG_SCHED_CORE
+ if (SCX_HAS_OP(core_sched_before))
+ touch_core_sched(rq, p);
+#endif
+}
+
static void update_curr_scx(struct rq *rq)
{
struct task_struct *curr = rq->curr;
@@ -1275,8 +1344,11 @@ static void update_curr_scx(struct rq *rq)
account_group_exec_runtime(curr, delta_exec);
cgroup_account_cputime(curr, delta_exec);
- if (curr->scx.slice != SCX_SLICE_INF)
+ if (curr->scx.slice != SCX_SLICE_INF) {
curr->scx.slice -= min(curr->scx.slice, delta_exec);
+ if (!curr->scx.slice)
+ touch_core_sched(rq, curr);
+ }
}
static void dsq_mod_nr(struct scx_dispatch_q *dsq, s32 delta)
@@ -1469,6 +1541,8 @@ static void direct_dispatch(struct task_struct *p, u64 enq_flags)
{
struct scx_dispatch_q *dsq;
+ touch_core_sched_dispatch(task_rq(p), p);
+
enq_flags |= (p->scx.ddsp_enq_flags | SCX_ENQ_CLEAR_OPSS);
dsq = find_dsq_for_dispatch(task_rq(p), p->scx.ddsp_dsq_id, p);
dispatch_enqueue(dsq, p, enq_flags);
@@ -1550,12 +1624,19 @@ static void do_enqueue_task(struct rq *rq, struct task_struct *p, u64 enq_flags,
return;
local:
+ /*
+ * For task-ordering, slice refill must be treated as implying the end
+ * of the current slice. Otherwise, the longer @p stays on the CPU, the
+ * higher priority it becomes from scx_prio_less()'s POV.
+ */
+ touch_core_sched(rq, p);
p->scx.slice = SCX_SLICE_DFL;
local_norefill:
dispatch_enqueue(&rq->scx.local_dsq, p, enq_flags);
return;
global:
+ touch_core_sched(rq, p); /* see the comment in local: */
p->scx.slice = SCX_SLICE_DFL;
dispatch_enqueue(&scx_dsq_global, p, enq_flags);
}
@@ -1619,6 +1700,9 @@ static void enqueue_task_scx(struct rq *rq, struct task_struct *p, int enq_flags
if (SCX_HAS_OP(runnable))
SCX_CALL_OP_TASK(SCX_KF_REST, runnable, p, enq_flags);
+ if (enq_flags & SCX_ENQ_WAKEUP)
+ touch_core_sched(rq, p);
+
do_enqueue_task(rq, p, enq_flags, sticky_cpu);
}
@@ -2106,6 +2190,7 @@ static void finish_dispatch(struct rq *rq, struct rq_flags *rf,
struct scx_dispatch_q *dsq;
unsigned long opss;
+ touch_core_sched_dispatch(rq, p);
retry:
/*
* No need for _acquire here. @p is accessed only after a successful
@@ -2183,8 +2268,8 @@ static void flush_dispatch_buf(struct rq *rq, struct rq_flags *rf)
dspc->cursor = 0;
}
-static int balance_scx(struct rq *rq, struct task_struct *prev,
- struct rq_flags *rf)
+static int balance_one(struct rq *rq, struct task_struct *prev,
+ struct rq_flags *rf, bool local)
{
struct scx_dsp_ctx *dspc = this_cpu_ptr(scx_dsp_ctx);
bool prev_on_scx = prev->sched_class == &ext_sched_class;
@@ -2208,7 +2293,7 @@ static int balance_scx(struct rq *rq, struct task_struct *prev,
}
if (prev_on_scx) {
- WARN_ON_ONCE(prev->scx.flags & SCX_TASK_BAL_KEEP);
+ WARN_ON_ONCE(local && (prev->scx.flags & SCX_TASK_BAL_KEEP));
update_curr_scx(rq);
/*
@@ -2220,10 +2305,16 @@ static int balance_scx(struct rq *rq, struct task_struct *prev,
*
* See scx_ops_disable_workfn() for the explanation on the
* bypassing test.
+ *
+ * When balancing a remote CPU for core-sched, there won't be a
+ * following put_prev_task_scx() call and we don't own
+ * %SCX_TASK_BAL_KEEP. Instead, pick_task_scx() will test the
+ * same conditions later and pick @rq->curr accordingly.
*/
if ((prev->scx.flags & SCX_TASK_QUEUED) &&
prev->scx.slice && !scx_ops_bypassing()) {
- prev->scx.flags |= SCX_TASK_BAL_KEEP;
+ if (local)
+ prev->scx.flags |= SCX_TASK_BAL_KEEP;
goto has_tasks;
}
}
@@ -2285,10 +2376,56 @@ static int balance_scx(struct rq *rq, struct task_struct *prev,
return has_tasks;
}
+static int balance_scx(struct rq *rq, struct task_struct *prev,
+ struct rq_flags *rf)
+{
+ int ret;
+
+ ret = balance_one(rq, prev, rf, true);
+
+#ifdef CONFIG_SCHED_SMT
+ /*
+ * When core-sched is enabled, this ops.balance() call will be followed
+ * by put_prev_scx() and pick_task_scx() on this CPU and pick_task_scx()
+ * on the SMT siblings. Balance the siblings too.
+ */
+ if (sched_core_enabled(rq)) {
+ const struct cpumask *smt_mask = cpu_smt_mask(cpu_of(rq));
+ int scpu;
+
+ for_each_cpu_andnot(scpu, smt_mask, cpumask_of(cpu_of(rq))) {
+ struct rq *srq = cpu_rq(scpu);
+ struct rq_flags srf;
+ struct task_struct *sprev = srq->curr;
+
+ /*
+ * While core-scheduling, rq lock is shared among
+ * siblings but the debug annotations and rq clock
+ * aren't. Do pinning dance to transfer the ownership.
+ */
+ WARN_ON_ONCE(__rq_lockp(rq) != __rq_lockp(srq));
+ rq_unpin_lock(rq, rf);
+ rq_pin_lock(srq, &srf);
+
+ update_rq_clock(srq);
+ balance_one(srq, sprev, &srf, false);
+
+ rq_unpin_lock(srq, &srf);
+ rq_repin_lock(rq, rf);
+ }
+ }
+#endif
+ return ret;
+}
+
static void set_next_task_scx(struct rq *rq, struct task_struct *p, bool first)
{
if (p->scx.flags & SCX_TASK_QUEUED) {
- WARN_ON_ONCE(atomic_long_read(&p->scx.ops_state) != SCX_OPSS_NONE);
+ /*
+ * Core-sched might decide to execute @p before it is
+ * dispatched. Call ops_dequeue() to notify the BPF scheduler.
+ */
+ ops_dequeue(p, SCX_DEQ_CORE_SCHED_EXEC);
dispatch_dequeue(rq, p);
}
@@ -2379,7 +2516,8 @@ static void put_prev_task_scx(struct rq *rq, struct task_struct *p)
/*
* If @p has slice left and balance_scx() didn't tag it for
* keeping, @p is getting preempted by a higher priority
- * scheduler class. Leave it at the head of the local DSQ.
+ * scheduler class or core-sched forcing a different task. Leave
+ * it at the head of the local DSQ.
*/
if (p->scx.slice && !scx_ops_bypassing()) {
dispatch_enqueue(&rq->scx.local_dsq, p, SCX_ENQ_HEAD);
@@ -2436,6 +2574,84 @@ static struct task_struct *pick_next_task_scx(struct rq *rq)
return p;
}
+#ifdef CONFIG_SCHED_CORE
+/**
+ * scx_prio_less - Task ordering for core-sched
+ * @a: task A
+ * @b: task B
+ *
+ * Core-sched is implemented as an additional scheduling layer on top of the
+ * usual sched_class'es and needs to find out the expected task ordering. For
+ * sched_ext, core-sched calls this function to interrogate the task ordering.
+ *
+ * Unless overridden by ops.core_sched_before(), @p->scx.core_sched_at is used
+ * to implement the default task ordering. The older the timestamp, the higher
+ * prority the task - the global FIFO ordering matching the default scheduling
+ * behavior.
+ *
+ * When ops.core_sched_before() is enabled, @p->scx.core_sched_at is used to
+ * implement FIFO ordering within each local DSQ. See pick_task_scx().
+ */
+bool scx_prio_less(const struct task_struct *a, const struct task_struct *b,
+ bool in_fi)
+{
+ /*
+ * The const qualifiers are dropped from task_struct pointers when
+ * calling ops.core_sched_before(). Accesses are controlled by the
+ * verifier.
+ */
+ if (SCX_HAS_OP(core_sched_before) && !scx_ops_bypassing())
+ return SCX_CALL_OP_2TASKS_RET(SCX_KF_REST, core_sched_before,
+ (struct task_struct *)a,
+ (struct task_struct *)b);
+ else
+ return time_after64(a->scx.core_sched_at, b->scx.core_sched_at);
+}
+
+/**
+ * pick_task_scx - Pick a candidate task for core-sched
+ * @rq: rq to pick the candidate task from
+ *
+ * Core-sched calls this function on each SMT sibling to determine the next
+ * tasks to run on the SMT siblings. balance_one() has been called on all
+ * siblings and put_prev_task_scx() has been called only for the current CPU.
+ *
+ * As put_prev_task_scx() hasn't been called on remote CPUs, we can't just look
+ * at the first task in the local dsq. @rq->curr has to be considered explicitly
+ * to mimic %SCX_TASK_BAL_KEEP.
+ */
+static struct task_struct *pick_task_scx(struct rq *rq)
+{
+ struct task_struct *curr = rq->curr;
+ struct task_struct *first = first_local_task(rq);
+
+ if (curr->scx.flags & SCX_TASK_QUEUED) {
+ /* is curr the only runnable task? */
+ if (!first)
+ return curr;
+
+ /*
+ * Does curr trump first? We can always go by core_sched_at for
+ * this comparison as it represents global FIFO ordering when
+ * the default core-sched ordering is used and local-DSQ FIFO
+ * ordering otherwise.
+ *
+ * We can have a task with an earlier timestamp on the DSQ. For
+ * example, when a current task is preempted by a sibling
+ * picking a different cookie, the task would be requeued at the
+ * head of the local DSQ with an earlier timestamp than the
+ * core-sched picked next task. Besides, the BPF scheduler may
+ * dispatch any tasks to the local DSQ anytime.
+ */
+ if (curr->scx.slice && time_before64(curr->scx.core_sched_at,
+ first->scx.core_sched_at))
+ return curr;
+ }
+
+ return first; /* this may be %NULL */
+}
+#endif /* CONFIG_SCHED_CORE */
+
static enum scx_cpu_preempt_reason
preempt_reason_from_class(const struct sched_class *class)
{
@@ -2843,13 +3059,15 @@ static void task_tick_scx(struct rq *rq, struct task_struct *curr, int queued)
update_curr_scx(rq);
/*
- * While bypassing, always resched as we can't trust the slice
- * management.
+ * While disabling, always resched and refresh core-sched timestamp as
+ * we can't trust the slice management or ops.core_sched_before().
*/
- if (scx_ops_bypassing())
+ if (scx_ops_bypassing()) {
curr->scx.slice = 0;
- else if (SCX_HAS_OP(tick))
+ touch_core_sched(rq, curr);
+ } else if (SCX_HAS_OP(tick)) {
SCX_CALL_OP(SCX_KF_REST, tick, curr);
+ }
if (!curr->scx.slice)
resched_curr(rq);
@@ -3203,6 +3421,10 @@ DEFINE_SCHED_CLASS(ext) = {
.rq_offline = rq_offline_scx,
#endif
+#ifdef CONFIG_SCHED_CORE
+ .pick_task = pick_task_scx,
+#endif
+
.task_tick = task_tick_scx,
.switching_to = switching_to_scx,
@@ -3416,12 +3638,14 @@ bool task_should_scx(struct task_struct *p)
*
* c. balance_scx() never sets %SCX_TASK_BAL_KEEP as the slice value can't be
* trusted. Whenever a tick triggers, the running task is rotated to the tail
- * of the queue.
+ * of the queue with core_sched_at touched.
*
* d. pick_next_task() suppresses zero slice warning.
*
* e. scx_bpf_kick_cpu() is disabled to avoid irq_work malfunction during PM
* operations.
+ *
+ * f. scx_prio_less() reverts to the default core_sched_at order.
*/
static void scx_ops_bypass(bool bypass)
{
@@ -4583,6 +4807,7 @@ static void running_stub(struct task_struct *p) {}
static void stopping_stub(struct task_struct *p, bool runnable) {}
static void quiescent_stub(struct task_struct *p, u64 deq_flags) {}
static bool yield_stub(struct task_struct *from, struct task_struct *to) { return false; }
+static bool core_sched_before_stub(struct task_struct *a, struct task_struct *b) { return false; }
static void set_weight_stub(struct task_struct *p, u32 weight) {}
static void set_cpumask_stub(struct task_struct *p, const struct cpumask *mask) {}
static void update_idle_stub(s32 cpu, bool idle) {}
@@ -4607,6 +4832,7 @@ static struct sched_ext_ops __bpf_ops_sched_ext_ops = {
.stopping = stopping_stub,
.quiescent = quiescent_stub,
.yield = yield_stub,
+ .core_sched_before = core_sched_before_stub,
.set_weight = set_weight_stub,
.set_cpumask = set_cpumask_stub,
.update_idle = update_idle_stub,
diff --git a/kernel/sched/ext.h b/kernel/sched/ext.h
index 037f9acdf443..6555878c5da3 100644
--- a/kernel/sched/ext.h
+++ b/kernel/sched/ext.h
@@ -70,6 +70,11 @@ static inline const struct sched_class *next_active_class(const struct sched_cla
for_active_class_range(class, (prev_class) > &ext_sched_class ? \
&ext_sched_class : (prev_class), (end_class))
+#ifdef CONFIG_SCHED_CORE
+bool scx_prio_less(const struct task_struct *a, const struct task_struct *b,
+ bool in_fi);
+#endif
+
#else /* CONFIG_SCHED_CLASS_EXT */
#define scx_enabled() false
diff --git a/tools/sched_ext/scx_qmap.bpf.c b/tools/sched_ext/scx_qmap.bpf.c
index 619078355bf5..c75c70d6a8eb 100644
--- a/tools/sched_ext/scx_qmap.bpf.c
+++ b/tools/sched_ext/scx_qmap.bpf.c
@@ -13,6 +13,7 @@
* - Sleepable per-task storage allocation using ops.prep_enable().
* - Using ops.cpu_release() to handle a higher priority scheduling class taking
* the CPU away.
+ * - Core-sched support.
*
* This scheduler is primarily for demonstration and testing of sched_ext
* features and unlikely to be useful for actual workloads.
@@ -67,9 +68,21 @@ struct {
},
};
+/*
+ * Per-queue sequence numbers to implement core-sched ordering.
+ *
+ * Tail seq is assigned to each queued task and incremented. Head seq tracks the
+ * sequence number of the latest dispatched task. The distance between the a
+ * task's seq and the associated queue's head seq is called the queue distance
+ * and used when comparing two tasks for ordering. See qmap_core_sched_before().
+ */
+static u64 core_sched_head_seqs[5];
+static u64 core_sched_tail_seqs[5];
+
/* Per-task scheduling context */
struct task_ctx {
bool force_local; /* Dispatch directly to local_dsq */
+ u64 core_sched_seq;
};
struct {
@@ -93,6 +106,7 @@ struct {
/* Statistics */
u64 nr_enqueued, nr_dispatched, nr_reenqueued, nr_dequeued;
+u64 nr_core_sched_execed;
s32 BPF_STRUCT_OPS(qmap_select_cpu, struct task_struct *p,
s32 prev_cpu, u64 wake_flags)
@@ -159,8 +173,18 @@ void BPF_STRUCT_OPS(qmap_enqueue, struct task_struct *p, u64 enq_flags)
return;
}
- /* Is select_cpu() is telling us to enqueue locally? */
- if (tctx->force_local) {
+ /*
+ * All enqueued tasks must have their core_sched_seq updated for correct
+ * core-sched ordering, which is why %SCX_OPS_ENQ_LAST is specified in
+ * qmap_ops.flags.
+ */
+ tctx->core_sched_seq = core_sched_tail_seqs[idx]++;
+
+ /*
+ * If qmap_select_cpu() is telling us to or this is the last runnable
+ * task on the CPU, enqueue locally.
+ */
+ if (tctx->force_local || (enq_flags & SCX_ENQ_LAST)) {
tctx->force_local = false;
scx_bpf_dispatch(p, SCX_DSQ_LOCAL, slice_ns, enq_flags);
return;
@@ -204,6 +228,19 @@ void BPF_STRUCT_OPS(qmap_enqueue, struct task_struct *p, u64 enq_flags)
void BPF_STRUCT_OPS(qmap_dequeue, struct task_struct *p, u64 deq_flags)
{
__sync_fetch_and_add(&nr_dequeued, 1);
+ if (deq_flags & SCX_DEQ_CORE_SCHED_EXEC)
+ __sync_fetch_and_add(&nr_core_sched_execed, 1);
+}
+
+static void update_core_sched_head_seq(struct task_struct *p)
+{
+ struct task_ctx *tctx = bpf_task_storage_get(&task_ctx_stor, p, 0, 0);
+ int idx = weight_to_idx(p->scx.weight);
+
+ if (tctx)
+ core_sched_head_seqs[idx] = tctx->core_sched_seq;
+ else
+ scx_bpf_error("task_ctx lookup failed");
}
void BPF_STRUCT_OPS(qmap_dispatch, s32 cpu, struct task_struct *prev)
@@ -258,6 +295,7 @@ void BPF_STRUCT_OPS(qmap_dispatch, s32 cpu, struct task_struct *prev)
if (!p)
continue;
+ update_core_sched_head_seq(p);
__sync_fetch_and_add(&nr_dispatched, 1);
scx_bpf_dispatch(p, SHARED_DSQ, slice_ns, 0);
bpf_task_release(p);
@@ -275,6 +313,49 @@ void BPF_STRUCT_OPS(qmap_dispatch, s32 cpu, struct task_struct *prev)
}
}
+/*
+ * The distance from the head of the queue scaled by the weight of the queue.
+ * The lower the number, the older the task and the higher the priority.
+ */
+static s64 task_qdist(struct task_struct *p)
+{
+ int idx = weight_to_idx(p->scx.weight);
+ struct task_ctx *tctx;
+ s64 qdist;
+
+ tctx = bpf_task_storage_get(&task_ctx_stor, p, 0, 0);
+ if (!tctx) {
+ scx_bpf_error("task_ctx lookup failed");
+ return 0;
+ }
+
+ qdist = tctx->core_sched_seq - core_sched_head_seqs[idx];
+
+ /*
+ * As queue index increments, the priority doubles. The queue w/ index 3
+ * is dispatched twice more frequently than 2. Reflect the difference by
+ * scaling qdists accordingly. Note that the shift amount needs to be
+ * flipped depending on the sign to avoid flipping priority direction.
+ */
+ if (qdist >= 0)
+ return qdist << (4 - idx);
+ else
+ return qdist << idx;
+}
+
+/*
+ * This is called to determine the task ordering when core-sched is picking
+ * tasks to execute on SMT siblings and should encode about the same ordering as
+ * the regular scheduling path. Use the priority-scaled distances from the head
+ * of the queues to compare the two tasks which should be consistent with the
+ * dispatch path behavior.
+ */
+bool BPF_STRUCT_OPS(qmap_core_sched_before,
+ struct task_struct *a, struct task_struct *b)
+{
+ return task_qdist(a) > task_qdist(b);
+}
+
void BPF_STRUCT_OPS(qmap_cpu_release, s32 cpu, struct scx_cpu_release_args *args)
{
u32 cnt;
@@ -354,8 +435,8 @@ void BPF_STRUCT_OPS(qmap_dump_task, struct scx_dump_ctx *dctx, struct task_struc
if (!(taskc = bpf_task_storage_get(&task_ctx_stor, p, 0, 0)))
return;
- scx_bpf_dump("QMAP: force_local=%d",
- taskc->force_local);
+ scx_bpf_dump("QMAP: force_local=%d core_sched_seq=%llu",
+ taskc->force_local, taskc->core_sched_seq);
}
/*
@@ -428,6 +509,7 @@ SCX_OPS_DEFINE(qmap_ops,
.enqueue = (void *)qmap_enqueue,
.dequeue = (void *)qmap_dequeue,
.dispatch = (void *)qmap_dispatch,
+ .core_sched_before = (void *)qmap_core_sched_before,
.cpu_release = (void *)qmap_cpu_release,
.init_task = (void *)qmap_init_task,
.dump = (void *)qmap_dump,
@@ -437,5 +519,6 @@ SCX_OPS_DEFINE(qmap_ops,
.cpu_offline = (void *)qmap_cpu_offline,
.init = (void *)qmap_init,
.exit = (void *)qmap_exit,
+ .flags = SCX_OPS_ENQ_LAST,
.timeout_ms = 5000U,
.name = "qmap");
diff --git a/tools/sched_ext/scx_qmap.c b/tools/sched_ext/scx_qmap.c
index 920fb54f9c77..bc36ec4f88a7 100644
--- a/tools/sched_ext/scx_qmap.c
+++ b/tools/sched_ext/scx_qmap.c
@@ -112,9 +112,10 @@ int main(int argc, char **argv)
long nr_enqueued = skel->bss->nr_enqueued;
long nr_dispatched = skel->bss->nr_dispatched;
- printf("stats : enq=%lu dsp=%lu delta=%ld reenq=%"PRIu64" deq=%"PRIu64"\n",
+ printf("stats : enq=%lu dsp=%lu delta=%ld reenq=%"PRIu64" deq=%"PRIu64" core=%"PRIu64"\n",
nr_enqueued, nr_dispatched, nr_enqueued - nr_dispatched,
- skel->bss->nr_reenqueued, skel->bss->nr_dequeued);
+ skel->bss->nr_reenqueued, skel->bss->nr_dequeued,
+ skel->bss->nr_core_sched_execed);
fflush(stdout);
sleep(1);
}
--
2.45.2
Implementation Analysis
Overview
Linux core scheduling (CONFIG_SCHED_CORE) is a security feature that prevents side-channel attacks (Spectre MDS variants) by ensuring that SMT sibling hyperthreads only run tasks from the same trust group (identified by a "cookie"). Prior to this patch, SCHED_CLASS_EXT had !SCHED_CORE as a Kconfig dependency — they were mutually exclusive. This patch lifts that restriction and integrates sched_ext fully with core scheduling.
The integration has three parts: (1) a new core_sched_at timestamp per task for ordering, (2) an optional ops.core_sched_before() callback for BPF-custom ordering, and (3) a modified balance_scx() that also balances SMT siblings and a new pick_task_scx() hook for the core-sched pick path.
Code Walkthrough
Kconfig dependency change (kernel/Kconfig.preempt):
- depends on BPF_SYSCALL && BPF_JIT && !SCHED_CORE
+ depends on BPF_SYSCALL && BPF_JIT
The mutual exclusion is removed. Both features can now be enabled simultaneously.
core_sched_at timestamp (include/linux/sched/ext.h):
#ifdef CONFIG_SCHED_CORE
u64 core_sched_at; /* see scx_prio_less() */
#endif
Added to struct sched_ext_entity. This timestamp records when a task last "renewed" its position in the scheduling order. Older timestamps mean higher priority in the default ordering.
touch_core_sched() and touch_core_sched_dispatch(): Two helper functions that update core_sched_at. touch_core_sched() is called at task wakeup (SCX_ENQ_WAKEUP), slice expiration, and when a task falls back to the local or global DSQ. touch_core_sched_dispatch() is called when a task is dispatched, but only if ops.core_sched_before() is implemented — in that case, dispatched tasks get a fresh timestamp to maintain local-DSQ FIFO ordering.
__task_prio() extension (kernel/sched/core.c):
if (task_on_scx(p))
return MAX_RT_PRIO + MAX_NICE + 1; /* squash ext */
return MAX_RT_PRIO + MAX_NICE; /* squash fair */
sched_ext tasks get their own priority bucket (120+1=121) distinct from CFS tasks (120). This allows prio_less() to dispatch to scx_prio_less() specifically for SCX tasks:
if (pa == MAX_RT_PRIO + MAX_NICE + 1) /* ext */
return scx_prio_less(a, b, in_fi);
ops.core_sched_before() (struct sched_ext_ops):
bool (*core_sched_before)(struct task_struct *a, struct task_struct *b);
Optional. If not set, the default ordering uses core_sched_at timestamps (global FIFO — older tasks have higher priority). If set, the BPF scheduler provides custom ordering. The const qualifiers on the task pointers are explicitly dropped in scx_prio_less() because BPF access is controlled by the verifier, not C qualifiers.
scx_prio_less() (kernel/sched/ext.c):
bool scx_prio_less(const struct task_struct *a, const struct task_struct *b,
bool in_fi)
{
if (SCX_HAS_OP(core_sched_before) && !scx_ops_bypassing())
return SCX_CALL_OP_2TASKS_RET(SCX_KF_REST, core_sched_before, ...);
else
return time_after64(a->scx.core_sched_at, b->scx.core_sched_at);
}
time_after64(a->ts, b->ts) returns true if a's timestamp is newer than b's — meaning a has less priority. So if a's core_sched_at is larger (more recent), a yields to b.
balance_scx() split — the old balance_scx() becomes balance_one() with a local boolean parameter, and a new balance_scx() wrapper calls balance_one() then balances SMT siblings:
if (sched_core_enabled(rq)) {
for_each_cpu_andnot(scpu, smt_mask, cpumask_of(cpu_of(rq))) {
rq_unpin_lock(rq, rf);
rq_pin_lock(srq, &srf);
balance_one(srq, sprev, &srf, false);
rq_unpin_lock(srq, &srf);
rq_repin_lock(rq, rf);
}
}
The rq lock is shared among SMT siblings in core-sched mode. The rq pin/unpin dance transfers debug annotations and clock ownership without dropping the lock.
pick_task_scx() — the new core-sched hook:
static struct task_struct *pick_task_scx(struct rq *rq)
{
struct task_struct *curr = rq->curr;
struct task_struct *first = first_local_task(rq);
if (curr->scx.flags & SCX_TASK_QUEUED) {
if (!first)
return curr;
if (curr->scx.slice && time_before64(curr->scx.core_sched_at,
first->scx.core_sched_at))
return curr;
}
return first;
}
Core-sched calls this on each sibling CPU to get a candidate task, then compares cookie groups across siblings. put_prev_task_scx() has only been called on the current CPU, so for remote CPUs curr is still running and must be considered explicitly (mimicking SCX_TASK_BAL_KEEP).
SCX_DEQ_CORE_SCHED_EXEC added to enum scx_deq_flags:
SCX_DEQ_CORE_SCHED_EXEC = 1LLU << 32,
When core-sched decides to execute a task that hasn't been dispatched yet (still in the BPF scheduler's queues), set_next_task_scx() calls ops_dequeue(p, SCX_DEQ_CORE_SCHED_EXEC) to notify the BPF scheduler. The qmap example counts these events in nr_core_sched_execed.
qmap core-sched implementation: Uses per-queue sequence numbers (core_sched_head_seqs, core_sched_tail_seqs) to compute a priority-scaled "queue distance" in task_qdist(). The qmap_core_sched_before() callback returns true if task a's queue distance is greater than task b's — meaning a is further from the head and has lower priority.
Key Concepts
core_sched_attimestamp: Updated on wakeup, slice expiry, and dispatch (if custom ordering). For default ordering: older timestamp = higher priority = runs first. For custom ordering: used only for FIFO ordering within the local DSQ after dispatch.- Two ordering modes: (1) Default — global FIFO using
core_sched_at. No BPF change needed. (2) Custom viaops.core_sched_before()— BPF defines the ordering.core_sched_atis then used only for local-DSQ FIFO within each CPU after dispatch. - SMT sibling balancing:
balance_scx()now balances all SMT siblings when core-sched is active. This is necessary because core-sched callspick_task_scx()on all siblings simultaneously to find compatible cookie pairs. - Cookie matching: sched_ext itself does not handle cookie assignment or matching. That is the core-sched layer's responsibility. sched_ext only provides ordering via
scx_prio_less()and candidate task selection viapick_task_scx().
Locking and Concurrency Notes
- In core-sched mode, SMT siblings share the same rq spinlock (the lock of the "master" CPU in the SMT group). The
rq_unpin_lock/rq_pin_lockdance inbalance_scx()is required to safely transfer lock ownership annotations between the per-rq debug state while holding the same underlying lock. scx_prio_less()is called fromprio_less()inside the core-sched scheduling path, under rq lock. TheSCX_KF_RESTkfunc set is used forcore_sched_before, so it runs under rq lock.touch_core_sched()only updates the timestamp when!sched_core_disabled(), using the cheapersched_core_disabled()check (a static key) rather thansched_core_enabled(). The comment notes "It's okay to update the timestamp spuriously" — a spurious update causes no harm, just a slightly different ordering.- The
localboolean inbalance_one()gatesSCX_TASK_BAL_KEEP: only the local CPU sets this flag, because remote CPUs won't haveput_prev_task_scx()called on them, sopick_task_scx()handles their current-task decision instead.
Integration with Kernel Subsystems
The integration is with kernel/sched/core.c's __task_prio() and prio_less() functions, which are the entry points for the core-sched task comparison logic. By assigning sched_ext tasks a unique priority bucket and dispatching through scx_prio_less(), sched_ext plugs into the existing core-sched comparison framework without requiring modifications to the generic code beyond those two functions.
The pick_task sched_class method (.pick_task = pick_task_scx) is the other integration point — this is the core-sched API for "give me your best candidate for this CPU."
What Maintainers Need to Know
- Opting in to custom ordering: If your BPF scheduler implements
ops.core_sched_before(), it must correctly reflect your scheduling order. Inconsistency betweenops.enqueue/dispatchordering andcore_sched_before()ordering will cause SMT imbalance — one sibling will stall waiting for a matching cookie partner. - Default ordering (no
core_sched_before): Global FIFO ordering based on when tasks became runnable. This is correct for simple schedulers and requires no BPF code changes. SCX_DEQ_CORE_SCHED_EXEC: When you receive this flag inops.dequeue(), core-sched has decided to run the task even though it was not yet dispatched from your queue. Update any per-queue accounting (e.g., head sequence numbers in qmap) appropriately.SCX_OPS_ENQ_LASTin qmap: The qmap example adds this flag to ensureops.enqueue()is called even for the last runnable task on a CPU. This is required for correctcore_sched_seqassignment — all tasks must go through enqueue to get a sequence number.- Bypass during core-sched: When
scx_ops_bypass()is active (PM events, etc.),scx_prio_less()falls back to defaultcore_sched_atordering regardless of whetherops.core_sched_before()is set. The bypass updatescore_sched_atat every tick to maintain a consistent ordering.
Connection to Other Patches
- Patch 26/30 (PM bypass): The bypass explicitly notes that
scx_prio_less()reverts to default ordering (pointfin the bypass comment). This patch is what that comment refers to. - Patch 24/30 (cpu_acquire/release):
cpu_release()can fire during core-sched scheduling decisions if a higher-priority class preempts. These are orthogonal paths:cpu_release()fires inscx_next_task_picked(), while core-sched operates inbalance/pick_task. - Patch 25/30 (cpu_online/offline): Both the
rq_online_scx/rq_offline_scxhooks from that patch and thepick_task_scxhook from this patch are registered in theext_sched_classvtable.
[PATCH 28/30] sched_ext: Add vtime-ordered priority queue to dispatch_q's
View on Lore: https://lore.kernel.org/all/20240618212056.2833381-29-tj@kernel.org
Commit Message
Currently, a dsq is always a FIFO. A task which is dispatched earlier gets
consumed or executed earlier. While this is sufficient when dsq's are used
for simple staging areas for tasks which are ready to execute, it'd make
dsq's a lot more useful if they can implement custom ordering.
This patch adds a vtime-ordered priority queue to dsq's. When the BPF
scheduler dispatches a task with the new scx_bpf_dispatch_vtime() helper, it
can specify the vtime tha the task should be inserted at and the task is
inserted into the priority queue in the dsq which is ordered according to
time_before64() comparison of the vtime values.
A DSQ can either be a FIFO or priority queue and automatically switches
between the two depending on whether scx_bpf_dispatch() or
scx_bpf_dispatch_vtime() is used. Using the wrong variant while the DSQ
already has the other type queued is not allowed and triggers an ops error.
Built-in DSQs must always be FIFOs.
This makes it very easy for the BPF schedulers to implement proper vtime
based scheduling within each dsq very easy and efficient at a negligible
cost in terms of code complexity and overhead.
scx_simple and scx_example_flatcg are updated to default to weighted
vtime scheduling (the latter within each cgroup). FIFO scheduling can be
selected with -f option.
v4: - As allowing mixing priority queue and FIFO on the same DSQ sometimes
led to unexpected starvations, DSQs now error out if both modes are
used at the same time and the built-in DSQs are no longer allowed to
be priority queues.
- Explicit type struct scx_dsq_node added to contain fields needed to be
linked on DSQs. This will be used to implement stateful iterator.
- Tasks are now always linked on dsq->list whether the DSQ is in FIFO or
PRIQ mode. This confines PRIQ related complexities to the enqueue and
dequeue paths. Other paths only need to look at dsq->list. This will
also ease implementing BPF iterator.
- Print p->scx.dsq_flags in debug dump.
v3: - SCX_TASK_DSQ_ON_PRIQ flag is moved from p->scx.flags into its own
p->scx.dsq_flags. The flag is protected with the dsq lock unlike other
flags in p->scx.flags. This led to flag corruption in some cases.
- Add comments explaining the interaction between using consumption of
p->scx.slice to determine vtime progress and yielding.
v2: - p->scx.dsq_vtime was not initialized on load or across cgroup
migrations leading to some tasks being stalled for extended period of
time depending on how saturated the machine is. Fixed.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: David Vernet <dvernet@meta.com>
---
include/linux/sched/ext.h | 29 ++++-
init/init_task.c | 2 +-
kernel/sched/ext.c | 156 ++++++++++++++++++++---
tools/sched_ext/include/scx/common.bpf.h | 1 +
tools/sched_ext/scx_simple.bpf.c | 97 +++++++++++++-
tools/sched_ext/scx_simple.c | 8 +-
6 files changed, 267 insertions(+), 26 deletions(-)
diff --git a/include/linux/sched/ext.h b/include/linux/sched/ext.h
index 3db7b32b2d1d..9cee193dab19 100644
--- a/include/linux/sched/ext.h
+++ b/include/linux/sched/ext.h
@@ -49,12 +49,15 @@ enum scx_dsq_id_flags {
};
/*
- * Dispatch queue (dsq) is a simple FIFO which is used to buffer between the
- * scheduler core and the BPF scheduler. See the documentation for more details.
+ * A dispatch queue (DSQ) can be either a FIFO or p->scx.dsq_vtime ordered
+ * queue. A built-in DSQ is always a FIFO. The built-in local DSQs are used to
+ * buffer between the scheduler core and the BPF scheduler. See the
+ * documentation for more details.
*/
struct scx_dispatch_q {
raw_spinlock_t lock;
struct list_head list; /* tasks in dispatch order */
+ struct rb_root priq; /* used to order by p->scx.dsq_vtime */
u32 nr;
u64 id;
struct rhash_head hash_node;
@@ -86,6 +89,11 @@ enum scx_task_state {
SCX_TASK_NR_STATES,
};
+/* scx_entity.dsq_flags */
+enum scx_ent_dsq_flags {
+ SCX_TASK_DSQ_ON_PRIQ = 1 << 0, /* task is queued on the priority queue of a dsq */
+};
+
/*
* Mask bits for scx_entity.kf_mask. Not all kfuncs can be called from
* everywhere and the following bits track which kfunc sets are currently
@@ -111,13 +119,19 @@ enum scx_kf_mask {
__SCX_KF_TERMINAL = SCX_KF_ENQUEUE | SCX_KF_SELECT_CPU | SCX_KF_REST,
};
+struct scx_dsq_node {
+ struct list_head list; /* dispatch order */
+ struct rb_node priq; /* p->scx.dsq_vtime order */
+ u32 flags; /* SCX_TASK_DSQ_* flags */
+};
+
/*
* The following is embedded in task_struct and contains all fields necessary
* for a task to be scheduled by SCX.
*/
struct sched_ext_entity {
struct scx_dispatch_q *dsq;
- struct list_head dsq_node;
+ struct scx_dsq_node dsq_node; /* protected by dsq lock */
u32 flags; /* protected by rq lock */
u32 weight;
s32 sticky_cpu;
@@ -149,6 +163,15 @@ struct sched_ext_entity {
*/
u64 slice;
+ /*
+ * Used to order tasks when dispatching to the vtime-ordered priority
+ * queue of a dsq. This is usually set through scx_bpf_dispatch_vtime()
+ * but can also be modified directly by the BPF scheduler. Modifying it
+ * while a task is queued on a dsq may mangle the ordering and is not
+ * recommended.
+ */
+ u64 dsq_vtime;
+
/*
* If set, reject future sched_setscheduler(2) calls updating the policy
* to %SCHED_EXT with -%EACCES.
diff --git a/init/init_task.c b/init/init_task.c
index 8a44c932d10f..5726b3a0eea9 100644
--- a/init/init_task.c
+++ b/init/init_task.c
@@ -102,7 +102,7 @@ struct task_struct init_task __aligned(L1_CACHE_BYTES) = {
#endif
#ifdef CONFIG_SCHED_CLASS_EXT
.scx = {
- .dsq_node = LIST_HEAD_INIT(init_task.scx.dsq_node),
+ .dsq_node.list = LIST_HEAD_INIT(init_task.scx.dsq_node.list),
.sticky_cpu = -1,
.holding_cpu = -1,
.runnable_node = LIST_HEAD_INIT(init_task.scx.runnable_node),
diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c
index 1feb690be9d8..f186c576e7d9 100644
--- a/kernel/sched/ext.c
+++ b/kernel/sched/ext.c
@@ -638,6 +638,7 @@ enum scx_enq_flags {
__SCX_ENQ_INTERNAL_MASK = 0xffLLU << 56,
SCX_ENQ_CLEAR_OPSS = 1LLU << 56,
+ SCX_ENQ_DSQ_PRIQ = 1LLU << 57,
};
enum scx_deq_flags {
@@ -1351,6 +1352,17 @@ static void update_curr_scx(struct rq *rq)
}
}
+static bool scx_dsq_priq_less(struct rb_node *node_a,
+ const struct rb_node *node_b)
+{
+ const struct task_struct *a =
+ container_of(node_a, struct task_struct, scx.dsq_node.priq);
+ const struct task_struct *b =
+ container_of(node_b, struct task_struct, scx.dsq_node.priq);
+
+ return time_before64(a->scx.dsq_vtime, b->scx.dsq_vtime);
+}
+
static void dsq_mod_nr(struct scx_dispatch_q *dsq, s32 delta)
{
/* scx_bpf_dsq_nr_queued() reads ->nr without locking, use WRITE_ONCE() */
@@ -1362,7 +1374,9 @@ static void dispatch_enqueue(struct scx_dispatch_q *dsq, struct task_struct *p,
{
bool is_local = dsq->id == SCX_DSQ_LOCAL;
- WARN_ON_ONCE(p->scx.dsq || !list_empty(&p->scx.dsq_node));
+ WARN_ON_ONCE(p->scx.dsq || !list_empty(&p->scx.dsq_node.list));
+ WARN_ON_ONCE((p->scx.dsq_node.flags & SCX_TASK_DSQ_ON_PRIQ) ||
+ !RB_EMPTY_NODE(&p->scx.dsq_node.priq));
if (!is_local) {
raw_spin_lock(&dsq->lock);
@@ -1375,10 +1389,59 @@ static void dispatch_enqueue(struct scx_dispatch_q *dsq, struct task_struct *p,
}
}
- if (enq_flags & (SCX_ENQ_HEAD | SCX_ENQ_PREEMPT))
- list_add(&p->scx.dsq_node, &dsq->list);
- else
- list_add_tail(&p->scx.dsq_node, &dsq->list);
+ if (unlikely((dsq->id & SCX_DSQ_FLAG_BUILTIN) &&
+ (enq_flags & SCX_ENQ_DSQ_PRIQ))) {
+ /*
+ * SCX_DSQ_LOCAL and SCX_DSQ_GLOBAL DSQs always consume from
+ * their FIFO queues. To avoid confusion and accidentally
+ * starving vtime-dispatched tasks by FIFO-dispatched tasks, we
+ * disallow any internal DSQ from doing vtime ordering of
+ * tasks.
+ */
+ scx_ops_error("cannot use vtime ordering for built-in DSQs");
+ enq_flags &= ~SCX_ENQ_DSQ_PRIQ;
+ }
+
+ if (enq_flags & SCX_ENQ_DSQ_PRIQ) {
+ struct rb_node *rbp;
+
+ /*
+ * A PRIQ DSQ shouldn't be using FIFO enqueueing. As tasks are
+ * linked to both the rbtree and list on PRIQs, this can only be
+ * tested easily when adding the first task.
+ */
+ if (unlikely(RB_EMPTY_ROOT(&dsq->priq) &&
+ !list_empty(&dsq->list)))
+ scx_ops_error("DSQ ID 0x%016llx already had FIFO-enqueued tasks",
+ dsq->id);
+
+ p->scx.dsq_node.flags |= SCX_TASK_DSQ_ON_PRIQ;
+ rb_add(&p->scx.dsq_node.priq, &dsq->priq, scx_dsq_priq_less);
+
+ /*
+ * Find the previous task and insert after it on the list so
+ * that @dsq->list is vtime ordered.
+ */
+ rbp = rb_prev(&p->scx.dsq_node.priq);
+ if (rbp) {
+ struct task_struct *prev =
+ container_of(rbp, struct task_struct,
+ scx.dsq_node.priq);
+ list_add(&p->scx.dsq_node.list, &prev->scx.dsq_node.list);
+ } else {
+ list_add(&p->scx.dsq_node.list, &dsq->list);
+ }
+ } else {
+ /* a FIFO DSQ shouldn't be using PRIQ enqueuing */
+ if (unlikely(!RB_EMPTY_ROOT(&dsq->priq)))
+ scx_ops_error("DSQ ID 0x%016llx already had PRIQ-enqueued tasks",
+ dsq->id);
+
+ if (enq_flags & (SCX_ENQ_HEAD | SCX_ENQ_PREEMPT))
+ list_add(&p->scx.dsq_node.list, &dsq->list);
+ else
+ list_add_tail(&p->scx.dsq_node.list, &dsq->list);
+ }
dsq_mod_nr(dsq, 1);
p->scx.dsq = dsq;
@@ -1417,13 +1480,30 @@ static void dispatch_enqueue(struct scx_dispatch_q *dsq, struct task_struct *p,
}
}
+static void task_unlink_from_dsq(struct task_struct *p,
+ struct scx_dispatch_q *dsq)
+{
+ if (p->scx.dsq_node.flags & SCX_TASK_DSQ_ON_PRIQ) {
+ rb_erase(&p->scx.dsq_node.priq, &dsq->priq);
+ RB_CLEAR_NODE(&p->scx.dsq_node.priq);
+ p->scx.dsq_node.flags &= ~SCX_TASK_DSQ_ON_PRIQ;
+ }
+
+ list_del_init(&p->scx.dsq_node.list);
+}
+
+static bool task_linked_on_dsq(struct task_struct *p)
+{
+ return !list_empty(&p->scx.dsq_node.list);
+}
+
static void dispatch_dequeue(struct rq *rq, struct task_struct *p)
{
struct scx_dispatch_q *dsq = p->scx.dsq;
bool is_local = dsq == &rq->scx.local_dsq;
if (!dsq) {
- WARN_ON_ONCE(!list_empty(&p->scx.dsq_node));
+ WARN_ON_ONCE(task_linked_on_dsq(p));
/*
* When dispatching directly from the BPF scheduler to a local
* DSQ, the task isn't associated with any DSQ but
@@ -1444,8 +1524,8 @@ static void dispatch_dequeue(struct rq *rq, struct task_struct *p)
*/
if (p->scx.holding_cpu < 0) {
/* @p must still be on @dsq, dequeue */
- WARN_ON_ONCE(list_empty(&p->scx.dsq_node));
- list_del_init(&p->scx.dsq_node);
+ WARN_ON_ONCE(!task_linked_on_dsq(p));
+ task_unlink_from_dsq(p, dsq);
dsq_mod_nr(dsq, -1);
} else {
/*
@@ -1454,7 +1534,7 @@ static void dispatch_dequeue(struct rq *rq, struct task_struct *p)
* holding_cpu which tells dispatch_to_local_dsq() that it lost
* the race.
*/
- WARN_ON_ONCE(!list_empty(&p->scx.dsq_node));
+ WARN_ON_ONCE(task_linked_on_dsq(p));
p->scx.holding_cpu = -1;
}
p->scx.dsq = NULL;
@@ -1949,7 +2029,8 @@ static void consume_local_task(struct rq *rq, struct scx_dispatch_q *dsq,
/* @dsq is locked and @p is on this rq */
WARN_ON_ONCE(p->scx.holding_cpu >= 0);
- list_move_tail(&p->scx.dsq_node, &rq->scx.local_dsq.list);
+ task_unlink_from_dsq(p, dsq);
+ list_add_tail(&p->scx.dsq_node.list, &rq->scx.local_dsq.list);
dsq_mod_nr(dsq, -1);
dsq_mod_nr(&rq->scx.local_dsq, 1);
p->scx.dsq = &rq->scx.local_dsq;
@@ -1992,7 +2073,7 @@ static bool consume_remote_task(struct rq *rq, struct rq_flags *rf,
* move_task_to_local_dsq().
*/
WARN_ON_ONCE(p->scx.holding_cpu >= 0);
- list_del_init(&p->scx.dsq_node);
+ task_unlink_from_dsq(p, dsq);
dsq_mod_nr(dsq, -1);
p->scx.holding_cpu = raw_smp_processor_id();
raw_spin_unlock(&dsq->lock);
@@ -2024,7 +2105,7 @@ static bool consume_dispatch_q(struct rq *rq, struct rq_flags *rf,
raw_spin_lock(&dsq->lock);
- list_for_each_entry(p, &dsq->list, scx.dsq_node) {
+ list_for_each_entry(p, &dsq->list, scx.dsq_node.list) {
struct rq *task_rq = task_rq(p);
if (rq == task_rq) {
@@ -2543,7 +2624,7 @@ static void put_prev_task_scx(struct rq *rq, struct task_struct *p)
static struct task_struct *first_local_task(struct rq *rq)
{
return list_first_entry_or_null(&rq->scx.local_dsq.list,
- struct task_struct, scx.dsq_node);
+ struct task_struct, scx.dsq_node.list);
}
static struct task_struct *pick_next_task_scx(struct rq *rq)
@@ -3225,7 +3306,8 @@ void init_scx_entity(struct sched_ext_entity *scx)
*/
memset(scx, 0, offsetof(struct sched_ext_entity, tasks_node));
- INIT_LIST_HEAD(&scx->dsq_node);
+ INIT_LIST_HEAD(&scx->dsq_node.list);
+ RB_CLEAR_NODE(&scx->dsq_node.priq);
scx->sticky_cpu = -1;
scx->holding_cpu = -1;
INIT_LIST_HEAD(&scx->runnable_node);
@@ -4070,12 +4152,13 @@ static void scx_dump_task(struct seq_buf *s, struct scx_dump_ctx *dctx,
dump_line(s, " %c%c %s[%d] %+ldms",
marker, task_state_to_char(p), p->comm, p->pid,
jiffies_delta_msecs(p->scx.runnable_at, dctx->at_jiffies));
- dump_line(s, " scx_state/flags=%u/0x%x ops_state/qseq=%lu/%lu",
+ dump_line(s, " scx_state/flags=%u/0x%x dsq_flags=0x%x ops_state/qseq=%lu/%lu",
scx_get_task_state(p), p->scx.flags & ~SCX_TASK_STATE_MASK,
- ops_state & SCX_OPSS_STATE_MASK,
+ p->scx.dsq_node.flags, ops_state & SCX_OPSS_STATE_MASK,
ops_state >> SCX_OPSS_QSEQ_SHIFT);
- dump_line(s, " sticky/holding_cpu=%d/%d dsq_id=%s",
- p->scx.sticky_cpu, p->scx.holding_cpu, dsq_id_buf);
+ dump_line(s, " sticky/holding_cpu=%d/%d dsq_id=%s dsq_vtime=%llu",
+ p->scx.sticky_cpu, p->scx.holding_cpu, dsq_id_buf,
+ p->scx.dsq_vtime);
dump_line(s, " cpus=%*pb", cpumask_pr_args(p->cpus_ptr));
if (SCX_HAS_OP(dump_task)) {
@@ -4663,6 +4746,9 @@ static int bpf_scx_btf_struct_access(struct bpf_verifier_log *log,
if (off >= offsetof(struct task_struct, scx.slice) &&
off + size <= offsetofend(struct task_struct, scx.slice))
return SCALAR_VALUE;
+ if (off >= offsetof(struct task_struct, scx.dsq_vtime) &&
+ off + size <= offsetofend(struct task_struct, scx.dsq_vtime))
+ return SCALAR_VALUE;
if (off >= offsetof(struct task_struct, scx.disallow) &&
off + size <= offsetofend(struct task_struct, scx.disallow))
return SCALAR_VALUE;
@@ -5298,10 +5384,44 @@ __bpf_kfunc void scx_bpf_dispatch(struct task_struct *p, u64 dsq_id, u64 slice,
scx_dispatch_commit(p, dsq_id, enq_flags);
}
+/**
+ * scx_bpf_dispatch_vtime - Dispatch a task into the vtime priority queue of a DSQ
+ * @p: task_struct to dispatch
+ * @dsq_id: DSQ to dispatch to
+ * @slice: duration @p can run for in nsecs
+ * @vtime: @p's ordering inside the vtime-sorted queue of the target DSQ
+ * @enq_flags: SCX_ENQ_*
+ *
+ * Dispatch @p into the vtime priority queue of the DSQ identified by @dsq_id.
+ * Tasks queued into the priority queue are ordered by @vtime and always
+ * consumed after the tasks in the FIFO queue. All other aspects are identical
+ * to scx_bpf_dispatch().
+ *
+ * @vtime ordering is according to time_before64() which considers wrapping. A
+ * numerically larger vtime may indicate an earlier position in the ordering and
+ * vice-versa.
+ */
+__bpf_kfunc void scx_bpf_dispatch_vtime(struct task_struct *p, u64 dsq_id,
+ u64 slice, u64 vtime, u64 enq_flags)
+{
+ if (!scx_dispatch_preamble(p, enq_flags))
+ return;
+
+ if (slice)
+ p->scx.slice = slice;
+ else
+ p->scx.slice = p->scx.slice ?: 1;
+
+ p->scx.dsq_vtime = vtime;
+
+ scx_dispatch_commit(p, dsq_id, enq_flags | SCX_ENQ_DSQ_PRIQ);
+}
+
__bpf_kfunc_end_defs();
BTF_KFUNCS_START(scx_kfunc_ids_enqueue_dispatch)
BTF_ID_FLAGS(func, scx_bpf_dispatch, KF_RCU)
+BTF_ID_FLAGS(func, scx_bpf_dispatch_vtime, KF_RCU)
BTF_KFUNCS_END(scx_kfunc_ids_enqueue_dispatch)
static const struct btf_kfunc_id_set scx_kfunc_set_enqueue_dispatch = {
diff --git a/tools/sched_ext/include/scx/common.bpf.h b/tools/sched_ext/include/scx/common.bpf.h
index 8686f84497db..3fa87084cf17 100644
--- a/tools/sched_ext/include/scx/common.bpf.h
+++ b/tools/sched_ext/include/scx/common.bpf.h
@@ -31,6 +31,7 @@ static inline void ___vmlinux_h_sanity_check___(void)
s32 scx_bpf_create_dsq(u64 dsq_id, s32 node) __ksym;
s32 scx_bpf_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags, bool *is_idle) __ksym;
void scx_bpf_dispatch(struct task_struct *p, u64 dsq_id, u64 slice, u64 enq_flags) __ksym;
+void scx_bpf_dispatch_vtime(struct task_struct *p, u64 dsq_id, u64 slice, u64 vtime, u64 enq_flags) __ksym;
u32 scx_bpf_dispatch_nr_slots(void) __ksym;
void scx_bpf_dispatch_cancel(void) __ksym;
bool scx_bpf_consume(u64 dsq_id) __ksym;
diff --git a/tools/sched_ext/scx_simple.bpf.c b/tools/sched_ext/scx_simple.bpf.c
index 6bb13a3c801b..ed7e8d535fc5 100644
--- a/tools/sched_ext/scx_simple.bpf.c
+++ b/tools/sched_ext/scx_simple.bpf.c
@@ -2,11 +2,20 @@
/*
* A simple scheduler.
*
- * A simple global FIFO scheduler. It also demonstrates the following niceties.
+ * By default, it operates as a simple global weighted vtime scheduler and can
+ * be switched to FIFO scheduling. It also demonstrates the following niceties.
*
* - Statistics tracking how many tasks are queued to local and global dsq's.
* - Termination notification for userspace.
*
+ * While very simple, this scheduler should work reasonably well on CPUs with a
+ * uniform L3 cache topology. While preemption is not implemented, the fact that
+ * the scheduling queue is shared across all CPUs means that whatever is at the
+ * front of the queue is likely to be executed fairly quickly given enough
+ * number of CPUs. The FIFO scheduling mode may be beneficial to some workloads
+ * but comes with the usual problems with FIFO scheduling where saturating
+ * threads can easily drown out interactive ones.
+ *
* Copyright (c) 2022 Meta Platforms, Inc. and affiliates.
* Copyright (c) 2022 Tejun Heo <tj@kernel.org>
* Copyright (c) 2022 David Vernet <dvernet@meta.com>
@@ -15,8 +24,20 @@
char _license[] SEC("license") = "GPL";
+const volatile bool fifo_sched;
+
+static u64 vtime_now;
UEI_DEFINE(uei);
+/*
+ * Built-in DSQs such as SCX_DSQ_GLOBAL cannot be used as priority queues
+ * (meaning, cannot be dispatched to with scx_bpf_dispatch_vtime()). We
+ * therefore create a separate DSQ with ID 0 that we dispatch to and consume
+ * from. If scx_simple only supported global FIFO scheduling, then we could
+ * just use SCX_DSQ_GLOBAL.
+ */
+#define SHARED_DSQ 0
+
struct {
__uint(type, BPF_MAP_TYPE_PERCPU_ARRAY);
__uint(key_size, sizeof(u32));
@@ -31,6 +52,11 @@ static void stat_inc(u32 idx)
(*cnt_p)++;
}
+static inline bool vtime_before(u64 a, u64 b)
+{
+ return (s64)(a - b) < 0;
+}
+
s32 BPF_STRUCT_OPS(simple_select_cpu, struct task_struct *p, s32 prev_cpu, u64 wake_flags)
{
bool is_idle = false;
@@ -48,7 +74,69 @@ s32 BPF_STRUCT_OPS(simple_select_cpu, struct task_struct *p, s32 prev_cpu, u64 w
void BPF_STRUCT_OPS(simple_enqueue, struct task_struct *p, u64 enq_flags)
{
stat_inc(1); /* count global queueing */
- scx_bpf_dispatch(p, SCX_DSQ_GLOBAL, SCX_SLICE_DFL, enq_flags);
+
+ if (fifo_sched) {
+ scx_bpf_dispatch(p, SHARED_DSQ, SCX_SLICE_DFL, enq_flags);
+ } else {
+ u64 vtime = p->scx.dsq_vtime;
+
+ /*
+ * Limit the amount of budget that an idling task can accumulate
+ * to one slice.
+ */
+ if (vtime_before(vtime, vtime_now - SCX_SLICE_DFL))
+ vtime = vtime_now - SCX_SLICE_DFL;
+
+ scx_bpf_dispatch_vtime(p, SHARED_DSQ, SCX_SLICE_DFL, vtime,
+ enq_flags);
+ }
+}
+
+void BPF_STRUCT_OPS(simple_dispatch, s32 cpu, struct task_struct *prev)
+{
+ scx_bpf_consume(SHARED_DSQ);
+}
+
+void BPF_STRUCT_OPS(simple_running, struct task_struct *p)
+{
+ if (fifo_sched)
+ return;
+
+ /*
+ * Global vtime always progresses forward as tasks start executing. The
+ * test and update can be performed concurrently from multiple CPUs and
+ * thus racy. Any error should be contained and temporary. Let's just
+ * live with it.
+ */
+ if (vtime_before(vtime_now, p->scx.dsq_vtime))
+ vtime_now = p->scx.dsq_vtime;
+}
+
+void BPF_STRUCT_OPS(simple_stopping, struct task_struct *p, bool runnable)
+{
+ if (fifo_sched)
+ return;
+
+ /*
+ * Scale the execution time by the inverse of the weight and charge.
+ *
+ * Note that the default yield implementation yields by setting
+ * @p->scx.slice to zero and the following would treat the yielding task
+ * as if it has consumed all its slice. If this penalizes yielding tasks
+ * too much, determine the execution time by taking explicit timestamps
+ * instead of depending on @p->scx.slice.
+ */
+ p->scx.dsq_vtime += (SCX_SLICE_DFL - p->scx.slice) * 100 / p->scx.weight;
+}
+
+void BPF_STRUCT_OPS(simple_enable, struct task_struct *p)
+{
+ p->scx.dsq_vtime = vtime_now;
+}
+
+s32 BPF_STRUCT_OPS_SLEEPABLE(simple_init)
+{
+ return scx_bpf_create_dsq(SHARED_DSQ, -1);
}
void BPF_STRUCT_OPS(simple_exit, struct scx_exit_info *ei)
@@ -59,5 +147,10 @@ void BPF_STRUCT_OPS(simple_exit, struct scx_exit_info *ei)
SCX_OPS_DEFINE(simple_ops,
.select_cpu = (void *)simple_select_cpu,
.enqueue = (void *)simple_enqueue,
+ .dispatch = (void *)simple_dispatch,
+ .running = (void *)simple_running,
+ .stopping = (void *)simple_stopping,
+ .enable = (void *)simple_enable,
+ .init = (void *)simple_init,
.exit = (void *)simple_exit,
.name = "simple");
diff --git a/tools/sched_ext/scx_simple.c b/tools/sched_ext/scx_simple.c
index bead482e1383..76d83199545c 100644
--- a/tools/sched_ext/scx_simple.c
+++ b/tools/sched_ext/scx_simple.c
@@ -17,8 +17,9 @@ const char help_fmt[] =
"\n"
"See the top-level comment in .bpf.c for more details.\n"
"\n"
-"Usage: %s [-v]\n"
+"Usage: %s [-f] [-v]\n"
"\n"
+" -f Use FIFO scheduling instead of weighted vtime scheduling\n"
" -v Print libbpf debug messages\n"
" -h Display this help and exit\n";
@@ -70,8 +71,11 @@ int main(int argc, char **argv)
restart:
skel = SCX_OPS_OPEN(simple_ops, scx_simple);
- while ((opt = getopt(argc, argv, "vh")) != -1) {
+ while ((opt = getopt(argc, argv, "fvh")) != -1) {
switch (opt) {
+ case 'f':
+ skel->rodata->fifo_sched = true;
+ break;
case 'v':
verbose = true;
break;
--
2.45.2
Diff
---
include/linux/sched/ext.h | 29 ++++-
init/init_task.c | 2 +-
kernel/sched/ext.c | 156 ++++++++++++++++++++---
tools/sched_ext/include/scx/common.bpf.h | 1 +
tools/sched_ext/scx_simple.bpf.c | 97 +++++++++++++-
tools/sched_ext/scx_simple.c | 8 +-
6 files changed, 267 insertions(+), 26 deletions(-)
diff --git a/include/linux/sched/ext.h b/include/linux/sched/ext.h
index 3db7b32b2d1d..9cee193dab19 100644
--- a/include/linux/sched/ext.h
+++ b/include/linux/sched/ext.h
@@ -49,12 +49,15 @@ enum scx_dsq_id_flags {
};
/*
- * Dispatch queue (dsq) is a simple FIFO which is used to buffer between the
- * scheduler core and the BPF scheduler. See the documentation for more details.
+ * A dispatch queue (DSQ) can be either a FIFO or p->scx.dsq_vtime ordered
+ * queue. A built-in DSQ is always a FIFO. The built-in local DSQs are used to
+ * buffer between the scheduler core and the BPF scheduler. See the
+ * documentation for more details.
*/
struct scx_dispatch_q {
raw_spinlock_t lock;
struct list_head list; /* tasks in dispatch order */
+ struct rb_root priq; /* used to order by p->scx.dsq_vtime */
u32 nr;
u64 id;
struct rhash_head hash_node;
@@ -86,6 +89,11 @@ enum scx_task_state {
SCX_TASK_NR_STATES,
};
+/* scx_entity.dsq_flags */
+enum scx_ent_dsq_flags {
+ SCX_TASK_DSQ_ON_PRIQ = 1 << 0, /* task is queued on the priority queue of a dsq */
+};
+
/*
* Mask bits for scx_entity.kf_mask. Not all kfuncs can be called from
* everywhere and the following bits track which kfunc sets are currently
@@ -111,13 +119,19 @@ enum scx_kf_mask {
__SCX_KF_TERMINAL = SCX_KF_ENQUEUE | SCX_KF_SELECT_CPU | SCX_KF_REST,
};
+struct scx_dsq_node {
+ struct list_head list; /* dispatch order */
+ struct rb_node priq; /* p->scx.dsq_vtime order */
+ u32 flags; /* SCX_TASK_DSQ_* flags */
+};
+
/*
* The following is embedded in task_struct and contains all fields necessary
* for a task to be scheduled by sched_ext.
*/
struct sched_ext_entity {
struct scx_dispatch_q *dsq;
- struct list_head dsq_node;
+ struct scx_dsq_node dsq_node; /* protected by dsq lock */
u32 flags; /* protected by rq lock */
u32 weight;
s32 sticky_cpu;
@@ -149,6 +163,15 @@ struct sched_ext_entity {
*/
u64 slice;
+ /*
+ * Used to order tasks when dispatching to the vtime-ordered priority
+ * queue of a dsq. This is usually set through scx_bpf_dispatch_vtime()
+ * but can also be modified directly by the BPF scheduler. Modifying it
+ * while a task is queued on a dsq may mangle the ordering and is not
+ * recommended.
+ */
+ u64 dsq_vtime;
+
/*
* If set, reject future sched_setscheduler(2) calls updating the policy
* to %SCHED_EXT with -%EACCES.
diff --git a/init/init_task.c b/init/init_task.c
index 8a44c932d10f..5726b3a0eea9 100644
--- a/init/init_task.c
+++ b/init/init_task.c
@@ -102,7 +102,7 @@ struct task_struct init_task __aligned(L1_CACHE_BYTES) = {
#endif
#ifdef CONFIG_SCHED_CLASS_EXT
.scx = {
- .dsq_node = LIST_HEAD_INIT(init_task.scx.dsq_node),
+ .dsq_node.list = LIST_HEAD_INIT(init_task.scx.dsq_node.list),
.sticky_cpu = -1,
.holding_cpu = -1,
.runnable_node = LIST_HEAD_INIT(init_task.scx.runnable_node),
diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c
index 1feb690be9d8..f186c576e7d9 100644
--- a/kernel/sched/ext.c
+++ b/kernel/sched/ext.c
@@ -638,6 +638,7 @@ enum scx_enq_flags {
__SCX_ENQ_INTERNAL_MASK = 0xffLLU << 56,
SCX_ENQ_CLEAR_OPSS = 1LLU << 56,
+ SCX_ENQ_DSQ_PRIQ = 1LLU << 57,
};
enum scx_deq_flags {
@@ -1351,6 +1352,17 @@ static void update_curr_scx(struct rq *rq)
}
}
+static bool scx_dsq_priq_less(struct rb_node *node_a,
+ const struct rb_node *node_b)
+{
+ const struct task_struct *a =
+ container_of(node_a, struct task_struct, scx.dsq_node.priq);
+ const struct task_struct *b =
+ container_of(node_b, struct task_struct, scx.dsq_node.priq);
+
+ return time_before64(a->scx.dsq_vtime, b->scx.dsq_vtime);
+}
+
static void dsq_mod_nr(struct scx_dispatch_q *dsq, s32 delta)
{
/* scx_bpf_dsq_nr_queued() reads ->nr without locking, use WRITE_ONCE() */
@@ -1362,7 +1374,9 @@ static void dispatch_enqueue(struct scx_dispatch_q *dsq, struct task_struct *p,
{
bool is_local = dsq->id == SCX_DSQ_LOCAL;
- WARN_ON_ONCE(p->scx.dsq || !list_empty(&p->scx.dsq_node));
+ WARN_ON_ONCE(p->scx.dsq || !list_empty(&p->scx.dsq_node.list));
+ WARN_ON_ONCE((p->scx.dsq_node.flags & SCX_TASK_DSQ_ON_PRIQ) ||
+ !RB_EMPTY_NODE(&p->scx.dsq_node.priq));
if (!is_local) {
raw_spin_lock(&dsq->lock);
@@ -1375,10 +1389,59 @@ static void dispatch_enqueue(struct scx_dispatch_q *dsq, struct task_struct *p,
}
}
- if (enq_flags & (SCX_ENQ_HEAD | SCX_ENQ_PREEMPT))
- list_add(&p->scx.dsq_node, &dsq->list);
- else
- list_add_tail(&p->scx.dsq_node, &dsq->list);
+ if (unlikely((dsq->id & SCX_DSQ_FLAG_BUILTIN) &&
+ (enq_flags & SCX_ENQ_DSQ_PRIQ))) {
+ /*
+ * SCX_DSQ_LOCAL and SCX_DSQ_GLOBAL DSQs always consume from
+ * their FIFO queues. To avoid confusion and accidentally
+ * starving vtime-dispatched tasks by FIFO-dispatched tasks, we
+ * disallow any internal DSQ from doing vtime ordering of
+ * tasks.
+ */
+ scx_ops_error("cannot use vtime ordering for built-in DSQs");
+ enq_flags &= ~SCX_ENQ_DSQ_PRIQ;
+ }
+
+ if (enq_flags & SCX_ENQ_DSQ_PRIQ) {
+ struct rb_node *rbp;
+
+ /*
+ * A PRIQ DSQ shouldn't be using FIFO enqueueing. As tasks are
+ * linked to both the rbtree and list on PRIQs, this can only be
+ * tested easily when adding the first task.
+ */
+ if (unlikely(RB_EMPTY_ROOT(&dsq->priq) &&
+ !list_empty(&dsq->list)))
+ scx_ops_error("DSQ ID 0x%016llx already had FIFO-enqueued tasks",
+ dsq->id);
+
+ p->scx.dsq_node.flags |= SCX_TASK_DSQ_ON_PRIQ;
+ rb_add(&p->scx.dsq_node.priq, &dsq->priq, scx_dsq_priq_less);
+
+ /*
+ * Find the previous task and insert after it on the list so
+ * that @dsq->list is vtime ordered.
+ */
+ rbp = rb_prev(&p->scx.dsq_node.priq);
+ if (rbp) {
+ struct task_struct *prev =
+ container_of(rbp, struct task_struct,
+ scx.dsq_node.priq);
+ list_add(&p->scx.dsq_node.list, &prev->scx.dsq_node.list);
+ } else {
+ list_add(&p->scx.dsq_node.list, &dsq->list);
+ }
+ } else {
+ /* a FIFO DSQ shouldn't be using PRIQ enqueuing */
+ if (unlikely(!RB_EMPTY_ROOT(&dsq->priq)))
+ scx_ops_error("DSQ ID 0x%016llx already had PRIQ-enqueued tasks",
+ dsq->id);
+
+ if (enq_flags & (SCX_ENQ_HEAD | SCX_ENQ_PREEMPT))
+ list_add(&p->scx.dsq_node.list, &dsq->list);
+ else
+ list_add_tail(&p->scx.dsq_node.list, &dsq->list);
+ }
dsq_mod_nr(dsq, 1);
p->scx.dsq = dsq;
@@ -1417,13 +1480,30 @@ static void dispatch_enqueue(struct scx_dispatch_q *dsq, struct task_struct *p,
}
}
+static void task_unlink_from_dsq(struct task_struct *p,
+ struct scx_dispatch_q *dsq)
+{
+ if (p->scx.dsq_node.flags & SCX_TASK_DSQ_ON_PRIQ) {
+ rb_erase(&p->scx.dsq_node.priq, &dsq->priq);
+ RB_CLEAR_NODE(&p->scx.dsq_node.priq);
+ p->scx.dsq_node.flags &= ~SCX_TASK_DSQ_ON_PRIQ;
+ }
+
+ list_del_init(&p->scx.dsq_node.list);
+}
+
+static bool task_linked_on_dsq(struct task_struct *p)
+{
+ return !list_empty(&p->scx.dsq_node.list);
+}
+
static void dispatch_dequeue(struct rq *rq, struct task_struct *p)
{
struct scx_dispatch_q *dsq = p->scx.dsq;
bool is_local = dsq == &rq->scx.local_dsq;
if (!dsq) {
- WARN_ON_ONCE(!list_empty(&p->scx.dsq_node));
+ WARN_ON_ONCE(task_linked_on_dsq(p));
/*
* When dispatching directly from the BPF scheduler to a local
* DSQ, the task isn't associated with any DSQ but
@@ -1444,8 +1524,8 @@ static void dispatch_dequeue(struct rq *rq, struct task_struct *p)
*/
if (p->scx.holding_cpu < 0) {
/* @p must still be on @dsq, dequeue */
- WARN_ON_ONCE(list_empty(&p->scx.dsq_node));
- list_del_init(&p->scx.dsq_node);
+ WARN_ON_ONCE(!task_linked_on_dsq(p));
+ task_unlink_from_dsq(p, dsq);
dsq_mod_nr(dsq, -1);
} else {
/*
@@ -1454,7 +1534,7 @@ static void dispatch_dequeue(struct rq *rq, struct task_struct *p)
* holding_cpu which tells dispatch_to_local_dsq() that it lost
* the race.
*/
- WARN_ON_ONCE(!list_empty(&p->scx.dsq_node));
+ WARN_ON_ONCE(task_linked_on_dsq(p));
p->scx.holding_cpu = -1;
}
p->scx.dsq = NULL;
@@ -1949,7 +2029,8 @@ static void consume_local_task(struct rq *rq, struct scx_dispatch_q *dsq,
/* @dsq is locked and @p is on this rq */
WARN_ON_ONCE(p->scx.holding_cpu >= 0);
- list_move_tail(&p->scx.dsq_node, &rq->scx.local_dsq.list);
+ task_unlink_from_dsq(p, dsq);
+ list_add_tail(&p->scx.dsq_node.list, &rq->scx.local_dsq.list);
dsq_mod_nr(dsq, -1);
dsq_mod_nr(&rq->scx.local_dsq, 1);
p->scx.dsq = &rq->scx.local_dsq;
@@ -1992,7 +2073,7 @@ static bool consume_remote_task(struct rq *rq, struct rq_flags *rf,
* move_task_to_local_dsq().
*/
WARN_ON_ONCE(p->scx.holding_cpu >= 0);
- list_del_init(&p->scx.dsq_node);
+ task_unlink_from_dsq(p, dsq);
dsq_mod_nr(dsq, -1);
p->scx.holding_cpu = raw_smp_processor_id();
raw_spin_unlock(&dsq->lock);
@@ -2024,7 +2105,7 @@ static bool consume_dispatch_q(struct rq *rq, struct rq_flags *rf,
raw_spin_lock(&dsq->lock);
- list_for_each_entry(p, &dsq->list, scx.dsq_node) {
+ list_for_each_entry(p, &dsq->list, scx.dsq_node.list) {
struct rq *task_rq = task_rq(p);
if (rq == task_rq) {
@@ -2543,7 +2624,7 @@ static void put_prev_task_scx(struct rq *rq, struct task_struct *p)
static struct task_struct *first_local_task(struct rq *rq)
{
return list_first_entry_or_null(&rq->scx.local_dsq.list,
- struct task_struct, scx.dsq_node);
+ struct task_struct, scx.dsq_node.list);
}
static struct task_struct *pick_next_task_scx(struct rq *rq)
@@ -3225,7 +3306,8 @@ void init_scx_entity(struct sched_ext_entity *scx)
*/
memset(scx, 0, offsetof(struct sched_ext_entity, tasks_node));
- INIT_LIST_HEAD(&scx->dsq_node);
+ INIT_LIST_HEAD(&scx->dsq_node.list);
+ RB_CLEAR_NODE(&scx->dsq_node.priq);
scx->sticky_cpu = -1;
scx->holding_cpu = -1;
INIT_LIST_HEAD(&scx->runnable_node);
@@ -4070,12 +4152,13 @@ static void scx_dump_task(struct seq_buf *s, struct scx_dump_ctx *dctx,
dump_line(s, " %c%c %s[%d] %+ldms",
marker, task_state_to_char(p), p->comm, p->pid,
jiffies_delta_msecs(p->scx.runnable_at, dctx->at_jiffies));
- dump_line(s, " scx_state/flags=%u/0x%x ops_state/qseq=%lu/%lu",
+ dump_line(s, " scx_state/flags=%u/0x%x dsq_flags=0x%x ops_state/qseq=%lu/%lu",
scx_get_task_state(p), p->scx.flags & ~SCX_TASK_STATE_MASK,
- ops_state & SCX_OPSS_STATE_MASK,
+ p->scx.dsq_node.flags, ops_state & SCX_OPSS_STATE_MASK,
ops_state >> SCX_OPSS_QSEQ_SHIFT);
- dump_line(s, " sticky/holding_cpu=%d/%d dsq_id=%s",
- p->scx.sticky_cpu, p->scx.holding_cpu, dsq_id_buf);
+ dump_line(s, " sticky/holding_cpu=%d/%d dsq_id=%s dsq_vtime=%llu",
+ p->scx.sticky_cpu, p->scx.holding_cpu, dsq_id_buf,
+ p->scx.dsq_vtime);
dump_line(s, " cpus=%*pb", cpumask_pr_args(p->cpus_ptr));
if (SCX_HAS_OP(dump_task)) {
@@ -4663,6 +4746,9 @@ static int bpf_scx_btf_struct_access(struct bpf_verifier_log *log,
if (off >= offsetof(struct task_struct, scx.slice) &&
off + size <= offsetofend(struct task_struct, scx.slice))
return SCALAR_VALUE;
+ if (off >= offsetof(struct task_struct, scx.dsq_vtime) &&
+ off + size <= offsetofend(struct task_struct, scx.dsq_vtime))
+ return SCALAR_VALUE;
if (off >= offsetof(struct task_struct, scx.disallow) &&
off + size <= offsetofend(struct task_struct, scx.disallow))
return SCALAR_VALUE;
@@ -5298,10 +5384,44 @@ __bpf_kfunc void scx_bpf_dispatch(struct task_struct *p, u64 dsq_id, u64 slice,
scx_dispatch_commit(p, dsq_id, enq_flags);
}
+/**
+ * scx_bpf_dispatch_vtime - Dispatch a task into the vtime priority queue of a DSQ
+ * @p: task_struct to dispatch
+ * @dsq_id: DSQ to dispatch to
+ * @slice: duration @p can run for in nsecs
+ * @vtime: @p's ordering inside the vtime-sorted queue of the target DSQ
+ * @enq_flags: SCX_ENQ_*
+ *
+ * Dispatch @p into the vtime priority queue of the DSQ identified by @dsq_id.
+ * Tasks queued into the priority queue are ordered by @vtime and always
+ * consumed after the tasks in the FIFO queue. All other aspects are identical
+ * to scx_bpf_dispatch().
+ *
+ * @vtime ordering is according to time_before64() which considers wrapping. A
+ * numerically larger vtime may indicate an earlier position in the ordering and
+ * vice-versa.
+ */
+__bpf_kfunc void scx_bpf_dispatch_vtime(struct task_struct *p, u64 dsq_id,
+ u64 slice, u64 vtime, u64 enq_flags)
+{
+ if (!scx_dispatch_preamble(p, enq_flags))
+ return;
+
+ if (slice)
+ p->scx.slice = slice;
+ else
+ p->scx.slice = p->scx.slice ?: 1;
+
+ p->scx.dsq_vtime = vtime;
+
+ scx_dispatch_commit(p, dsq_id, enq_flags | SCX_ENQ_DSQ_PRIQ);
+}
+
__bpf_kfunc_end_defs();
BTF_KFUNCS_START(scx_kfunc_ids_enqueue_dispatch)
BTF_ID_FLAGS(func, scx_bpf_dispatch, KF_RCU)
+BTF_ID_FLAGS(func, scx_bpf_dispatch_vtime, KF_RCU)
BTF_KFUNCS_END(scx_kfunc_ids_enqueue_dispatch)
static const struct btf_kfunc_id_set scx_kfunc_set_enqueue_dispatch = {
diff --git a/tools/sched_ext/include/scx/common.bpf.h b/tools/sched_ext/include/scx/common.bpf.h
index 8686f84497db..3fa87084cf17 100644
--- a/tools/sched_ext/include/scx/common.bpf.h
+++ b/tools/sched_ext/include/scx/common.bpf.h
@@ -31,6 +31,7 @@ static inline void ___vmlinux_h_sanity_check___(void)
s32 scx_bpf_create_dsq(u64 dsq_id, s32 node) __ksym;
s32 scx_bpf_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags, bool *is_idle) __ksym;
void scx_bpf_dispatch(struct task_struct *p, u64 dsq_id, u64 slice, u64 enq_flags) __ksym;
+void scx_bpf_dispatch_vtime(struct task_struct *p, u64 dsq_id, u64 slice, u64 vtime, u64 enq_flags) __ksym;
u32 scx_bpf_dispatch_nr_slots(void) __ksym;
void scx_bpf_dispatch_cancel(void) __ksym;
bool scx_bpf_consume(u64 dsq_id) __ksym;
diff --git a/tools/sched_ext/scx_simple.bpf.c b/tools/sched_ext/scx_simple.bpf.c
index 6bb13a3c801b..ed7e8d535fc5 100644
--- a/tools/sched_ext/scx_simple.bpf.c
+++ b/tools/sched_ext/scx_simple.bpf.c
@@ -2,11 +2,20 @@
/*
* A simple scheduler.
*
- * A simple global FIFO scheduler. It also demonstrates the following niceties.
+ * By default, it operates as a simple global weighted vtime scheduler and can
+ * be switched to FIFO scheduling. It also demonstrates the following niceties.
*
* - Statistics tracking how many tasks are queued to local and global dsq's.
* - Termination notification for userspace.
*
+ * While very simple, this scheduler should work reasonably well on CPUs with a
+ * uniform L3 cache topology. While preemption is not implemented, the fact that
+ * the scheduling queue is shared across all CPUs means that whatever is at the
+ * front of the queue is likely to be executed fairly quickly given enough
+ * number of CPUs. The FIFO scheduling mode may be beneficial to some workloads
+ * but comes with the usual problems with FIFO scheduling where saturating
+ * threads can easily drown out interactive ones.
+ *
* Copyright (c) 2022 Meta Platforms, Inc. and affiliates.
* Copyright (c) 2022 Tejun Heo <tj@kernel.org>
* Copyright (c) 2022 David Vernet <dvernet@meta.com>
@@ -15,8 +24,20 @@
char _license[] SEC("license") = "GPL";
+const volatile bool fifo_sched;
+
+static u64 vtime_now;
UEI_DEFINE(uei);
+/*
+ * Built-in DSQs such as SCX_DSQ_GLOBAL cannot be used as priority queues
+ * (meaning, cannot be dispatched to with scx_bpf_dispatch_vtime()). We
+ * therefore create a separate DSQ with ID 0 that we dispatch to and consume
+ * from. If scx_simple only supported global FIFO scheduling, then we could
+ * just use SCX_DSQ_GLOBAL.
+ */
+#define SHARED_DSQ 0
+
struct {
__uint(type, BPF_MAP_TYPE_PERCPU_ARRAY);
__uint(key_size, sizeof(u32));
@@ -31,6 +52,11 @@ static void stat_inc(u32 idx)
(*cnt_p)++;
}
+static inline bool vtime_before(u64 a, u64 b)
+{
+ return (s64)(a - b) < 0;
+}
+
s32 BPF_STRUCT_OPS(simple_select_cpu, struct task_struct *p, s32 prev_cpu, u64 wake_flags)
{
bool is_idle = false;
@@ -48,7 +74,69 @@ s32 BPF_STRUCT_OPS(simple_select_cpu, struct task_struct *p, s32 prev_cpu, u64 w
void BPF_STRUCT_OPS(simple_enqueue, struct task_struct *p, u64 enq_flags)
{
stat_inc(1); /* count global queueing */
- scx_bpf_dispatch(p, SCX_DSQ_GLOBAL, SCX_SLICE_DFL, enq_flags);
+
+ if (fifo_sched) {
+ scx_bpf_dispatch(p, SHARED_DSQ, SCX_SLICE_DFL, enq_flags);
+ } else {
+ u64 vtime = p->scx.dsq_vtime;
+
+ /*
+ * Limit the amount of budget that an idling task can accumulate
+ * to one slice.
+ */
+ if (vtime_before(vtime, vtime_now - SCX_SLICE_DFL))
+ vtime = vtime_now - SCX_SLICE_DFL;
+
+ scx_bpf_dispatch_vtime(p, SHARED_DSQ, SCX_SLICE_DFL, vtime,
+ enq_flags);
+ }
+}
+
+void BPF_STRUCT_OPS(simple_dispatch, s32 cpu, struct task_struct *prev)
+{
+ scx_bpf_consume(SHARED_DSQ);
+}
+
+void BPF_STRUCT_OPS(simple_running, struct task_struct *p)
+{
+ if (fifo_sched)
+ return;
+
+ /*
+ * Global vtime always progresses forward as tasks start executing. The
+ * test and update can be performed concurrently from multiple CPUs and
+ * thus racy. Any error should be contained and temporary. Let's just
+ * live with it.
+ */
+ if (vtime_before(vtime_now, p->scx.dsq_vtime))
+ vtime_now = p->scx.dsq_vtime;
+}
+
+void BPF_STRUCT_OPS(simple_stopping, struct task_struct *p, bool runnable)
+{
+ if (fifo_sched)
+ return;
+
+ /*
+ * Scale the execution time by the inverse of the weight and charge.
+ *
+ * Note that the default yield implementation yields by setting
+ * @p->scx.slice to zero and the following would treat the yielding task
+ * as if it has consumed all its slice. If this penalizes yielding tasks
+ * too much, determine the execution time by taking explicit timestamps
+ * instead of depending on @p->scx.slice.
+ */
+ p->scx.dsq_vtime += (SCX_SLICE_DFL - p->scx.slice) * 100 / p->scx.weight;
+}
+
+void BPF_STRUCT_OPS(simple_enable, struct task_struct *p)
+{
+ p->scx.dsq_vtime = vtime_now;
+}
+
+s32 BPF_STRUCT_OPS_SLEEPABLE(simple_init)
+{
+ return scx_bpf_create_dsq(SHARED_DSQ, -1);
}
void BPF_STRUCT_OPS(simple_exit, struct scx_exit_info *ei)
@@ -59,5 +147,10 @@ void BPF_STRUCT_OPS(simple_exit, struct scx_exit_info *ei)
SCX_OPS_DEFINE(simple_ops,
.select_cpu = (void *)simple_select_cpu,
.enqueue = (void *)simple_enqueue,
+ .dispatch = (void *)simple_dispatch,
+ .running = (void *)simple_running,
+ .stopping = (void *)simple_stopping,
+ .enable = (void *)simple_enable,
+ .init = (void *)simple_init,
.exit = (void *)simple_exit,
.name = "simple");
diff --git a/tools/sched_ext/scx_simple.c b/tools/sched_ext/scx_simple.c
index bead482e1383..76d83199545c 100644
--- a/tools/sched_ext/scx_simple.c
+++ b/tools/sched_ext/scx_simple.c
@@ -17,8 +17,9 @@ const char help_fmt[] =
"\n"
"See the top-level comment in .bpf.c for more details.\n"
"\n"
-"Usage: %s [-v]\n"
+"Usage: %s [-f] [-v]\n"
"\n"
+" -f Use FIFO scheduling instead of weighted vtime scheduling\n"
" -v Print libbpf debug messages\n"
" -h Display this help and exit\n";
@@ -70,8 +71,11 @@ int main(int argc, char **argv)
restart:
skel = SCX_OPS_OPEN(simple_ops, scx_simple);
- while ((opt = getopt(argc, argv, "vh")) != -1) {
+ while ((opt = getopt(argc, argv, "fvh")) != -1) {
switch (opt) {
+ case 'f':
+ skel->rodata->fifo_sched = true;
+ break;
case 'v':
verbose = true;
break;
--
2.45.2
Implementation Analysis
Overview
Prior to this patch, all DSQs were strict FIFOs. A task dispatched earlier was always consumed earlier. This is sufficient for many schedulers but precludes vtime-based scheduling policies (Weighted Fair Queueing, Earliest Deadline First, etc.) within a DSQ. This patch adds an optional priority queue mode to user-created DSQs, ordered by a per-task dsq_vtime field. The BPF scheduler uses the new scx_bpf_dispatch_vtime() kfunc to dispatch into the priority queue. Built-in DSQs (SCX_DSQ_LOCAL, SCX_DSQ_GLOBAL) remain FIFO-only.
scx_simple is updated to default to weighted vtime scheduling using this new capability.
Code Walkthrough
scx_dispatch_q struct extension (include/linux/sched/ext.h):
struct scx_dispatch_q {
raw_spinlock_t lock;
struct list_head list; /* tasks in dispatch order */
struct rb_root priq; /* used to order by p->scx.dsq_vtime */
u32 nr;
u64 id;
...
};
An rb_root is added alongside the existing list_head. In FIFO mode, the rbtree is empty. In PRIQ mode, tasks are inserted into both the rbtree (ordered by vtime) and the list (mirroring rbtree order).
struct scx_dsq_node replaces the old struct list_head dsq_node in sched_ext_entity:
struct scx_dsq_node {
struct list_head list; /* dispatch order */
struct rb_node priq; /* p->scx.dsq_vtime order */
u32 flags; /* SCX_TASK_DSQ_* flags */
};
/* scx_entity.dsq_flags */
enum scx_ent_dsq_flags {
SCX_TASK_DSQ_ON_PRIQ = 1 << 0, /* task is on priority queue */
};
The flags field is protected by the DSQ lock (unlike p->scx.flags which is rq-lock protected). This separation was explicitly noted in the v3 commit message: merging them into p->scx.flags caused flag corruption.
p->scx.dsq_vtime (include/linux/sched/ext.h):
u64 dsq_vtime;
The vtime key for priority queue insertion. Set by scx_bpf_dispatch_vtime(), or modifiable directly by BPF. The comment warns: "Modifying it while a task is queued on a dsq may mangle the ordering and is not recommended." The BPF verifier allows direct write access to this field (added to bpf_scx_btf_struct_access()).
scx_bpf_dispatch_vtime() — the new kfunc:
__bpf_kfunc void scx_bpf_dispatch_vtime(struct task_struct *p, u64 dsq_id,
u64 slice, u64 vtime, u64 enq_flags)
{
if (!scx_dispatch_preamble(p, enq_flags))
return;
if (slice)
p->scx.slice = slice;
else
p->scx.slice = p->scx.slice ?: 1;
p->scx.dsq_vtime = vtime;
scx_dispatch_commit(p, dsq_id, enq_flags | SCX_ENQ_DSQ_PRIQ);
}
Sets dsq_vtime and calls scx_dispatch_commit() with the internal SCX_ENQ_DSQ_PRIQ flag. The slice handling ensures a task with zero slice gets a minimal non-zero slice rather than immediately expiring.
dispatch_enqueue() changes — the PRIQ path in kernel/sched/ext.c:
if (enq_flags & SCX_ENQ_DSQ_PRIQ) {
/* check: built-in DSQs cannot use PRIQ */
if (RB_EMPTY_ROOT(&dsq->priq) && !list_empty(&dsq->list))
scx_ops_error("DSQ ... already had FIFO-enqueued tasks");
p->scx.dsq_node.flags |= SCX_TASK_DSQ_ON_PRIQ;
rb_add(&p->scx.dsq_node.priq, &dsq->priq, scx_dsq_priq_less);
/* mirror rbtree order onto the list */
rbp = rb_prev(&p->scx.dsq_node.priq);
if (rbp) {
struct task_struct *prev = container_of(rbp, ...);
list_add(&p->scx.dsq_node.list, &prev->scx.dsq_node.list);
} else {
list_add(&p->scx.dsq_node.list, &dsq->list);
}
} else {
/* FIFO path, check no PRIQ tasks exist */
if (!RB_EMPTY_ROOT(&dsq->priq))
scx_ops_error("DSQ ... already had PRIQ-enqueued tasks");
...
}
The key insight: tasks are always on dsq->list regardless of mode. In PRIQ mode, the list mirrors the rbtree order. This means all consumers (which iterate dsq->list) work identically for both modes — the PRIQ complexity is entirely in enqueue and dequeue.
scx_dsq_priq_less() — the rbtree comparator:
static bool scx_dsq_priq_less(struct rb_node *node_a, const struct rb_node *node_b)
{
...
return time_before64(a->scx.dsq_vtime, b->scx.dsq_vtime);
}
time_before64() handles u64 wraparound correctly. The rbtree is a min-heap by vtime: the leftmost node has the smallest (earliest) vtime and is at the head of the list.
task_unlink_from_dsq() — new helper for dequeue:
static void task_unlink_from_dsq(struct task_struct *p, struct scx_dispatch_q *dsq)
{
if (p->scx.dsq_node.flags & SCX_TASK_DSQ_ON_PRIQ) {
rb_erase(&p->scx.dsq_node.priq, &dsq->priq);
RB_CLEAR_NODE(&p->scx.dsq_node.priq);
p->scx.dsq_node.flags &= ~SCX_TASK_DSQ_ON_PRIQ;
}
list_del_init(&p->scx.dsq_node.list);
}
Centralizes the dual-structure dequeue. All callers of the old list_del_init(&p->scx.dsq_node) are updated to use this.
scx_simple vtime scheduler — the canonical example:
void BPF_STRUCT_OPS(simple_enqueue, struct task_struct *p, u64 enq_flags)
{
if (fifo_sched) {
scx_bpf_dispatch(p, SHARED_DSQ, SCX_SLICE_DFL, enq_flags);
} else {
u64 vtime = p->scx.dsq_vtime;
/* cap idle tasks: don't let them accumulate more than one slice */
if (vtime_before(vtime, vtime_now - SCX_SLICE_DFL))
vtime = vtime_now - SCX_SLICE_DFL;
scx_bpf_dispatch_vtime(p, SHARED_DSQ, SCX_SLICE_DFL, vtime, enq_flags);
}
}
void BPF_STRUCT_OPS(simple_stopping, struct task_struct *p, bool runnable)
{
/* charge vtime proportionally to weight */
p->scx.dsq_vtime += (SCX_SLICE_DFL - p->scx.slice) * 100 / p->scx.weight;
}
Tasks accumulate vtime proportional to (time_used / weight). Lower weight = more vtime per real time = later in the priority queue. The idle task budget cap (vtime_now - SCX_SLICE_DFL) prevents idling tasks from accumulating unbounded negative debt.
Key Concepts
scx_bpf_dispatch_vtime(p, dsq_id, slice, vtime, enq_flags): Dispatchpinto the priority queue of DSQdsq_idwith ordering keyvtime. Lower vtime = earlier position.- DSQ mode is per-DSQ and detected dynamically: A DSQ is in PRIQ mode as soon as a task is dispatched with
SCX_ENQ_DSQ_PRIQ. Mixing both modes in the same DSQ triggers an ops error. - Built-in DSQs are FIFO-only:
SCX_DSQ_LOCALandSCX_DSQ_GLOBALwill error if used withscx_bpf_dispatch_vtime(). dsq_vtimeis BPF-writable: BPF code can modifyp->scx.dsq_vtimedirectly (not just via the kfunc), but only when the task is not currently enqueued.- List always maintained: Consumer paths (
consume_dispatch_q,first_local_task, etc.) only walkdsq->list. PRIQ mode does not require any consumer changes. - vtime wrapping:
time_before64()handles 64-bit wrapping correctly. A scheduler that increments vtime by small amounts over billions of tasks will not break.
Locking and Concurrency Notes
dsq_node.flags(includingSCX_TASK_DSQ_ON_PRIQ) is protected by the DSQ'sraw_spinlock_t lock, not by rq lock. This was a bug fix from v3 — previously mixing these protection domains caused flag corruption.dispatch_enqueue()for non-local DSQs holdsdsq->lock. The PRIQ path's rbtree operations and list operations are both under this lock.p->scx.dsq_vtimeis not protected by any lock when accessed fromops.stopping()(where the BPF scheduler charges vtime). This is intentional — the task is running (not queued) at that point, so there is no DSQ lock contention. Access from multiple CPUs simultaneously would require explicit BPF synchronization.vtime_nowin scx_simple is a global u64 updated racily from multiple CPUs insimple_running(). The comment acknowledges this: "Any error should be contained and temporary."
Integration with Kernel Subsystems
The rbtree (struct rb_root / rb_add() / rb_erase()) is the standard kernel red-black tree implementation from include/linux/rbtree.h. No new data structures are introduced. The comparator scx_dsq_priq_less() uses time_before64() from include/linux/jiffies.h for correct wrap-aware 64-bit time comparison.
The change to init_task.c is a mechanical fix needed because dsq_node changed from a list_head to a struct scx_dsq_node — the initializer must now target the embedded list field.
What Maintainers Need to Know
- Never mix FIFO and PRIQ dispatch to the same DSQ. The mode check at enqueue time is an ops error (scheduler exits). Design your scheduler to use dedicated DSQs for each mode.
- Built-in DSQs are always FIFO. Use
scx_bpf_create_dsq()to create a user DSQ if you need vtime ordering. See scx_simple'sSHARED_DSQ = 0pattern. - vtime is not automatically managed. You must update
p->scx.dsq_vtimeinops.stopping()(or another appropriate callback) to track per-task virtual time. If you never update it, all tasks have vtime=0 and the DSQ degrades to LIFO (insertion order by rbtree position). - Budget capping for idle tasks: Tasks that were idle accumulate no vtime debt, which would make them unfairly high-priority. The scx_simple pattern of
if vtime < vtime_now - slice: vtime = vtime_now - slicecaps the budget to one slice worth of credit. Adapt this threshold to your policy. dsq_vtimeis writable from BPF via the BTF struct access mechanism. This allows advanced schedulers to implement out-of-band priority boosts by directly modifying the field. Be careful not to do this while the task is enqueued.- Debugging:
dsq_vtime=%lluis now printed in the per-task debug dump (scx_dump_task()), anddsq_flags=0x%xshows theSCX_TASK_DSQ_ON_PRIQflag.
Connection to Other Patches
- Patch 27/30 (core-sched):
touch_core_sched_dispatch()is called indirect_dispatch()andfinish_dispatch()— both of which eventually calldispatch_enqueue(). This is called before the PRIQ/FIFO logic, so core-sched timestamps are updated regardless of DSQ mode. - Patch 29/30 (documentation): The documentation explicitly describes
scx_bpf_dispatch_vtime(), the FIFO vs. priority queue distinction, and the restriction that built-in DSQs must usescx_bpf_dispatch(). - Patch 30/30 (selftests): The
ddsp_vtimelocal_failtest verifies that dispatching toSCX_DSQ_LOCALwith vtime correctly triggers an ops error. Theselect_cpu_vtimetest verifies correct vtime ordering behavior.
Documentation and Testing (Patches 29–30)
Overview
A feature that cannot be verified is a feature that cannot be maintained. Patches 29 and 30 close the loop on the sched_ext series by providing the official reference documentation and a kernel self-test suite. These are not peripheral additions — in the Linux kernel review process, documentation and selftests are expected for any significant feature that targets mainline, and their quality signals the maturity and maintainability of the feature itself.
Patch 29 adds Documentation/scheduler/sched-ext.rst, the canonical prose documentation for
the sched_ext framework. Patch 30 adds tools/testing/selftests/sched_ext/, a test suite that
exercises the correctness of the core dispatch mechanism, error handling, and DSQ lifecycle.
For a maintainer, these two patches are as important as the implementation patches. The documentation defines the contract that future patches must preserve. The selftests define the behavioral invariants that regressions must not break.
Why These Patches Are Needed
The Documentation Gap
By the time the implementation patches are in place (patches 08–12 and 20–28), a developer wanting to write a BPF scheduler must understand:
- What
sched_ext_opscallbacks are available and when each is called. - The dispatch queue concept and the relationship between global DSQ, local DSQs, and user DSQs.
- How tasks move through the scheduler lifecycle.
- What BPF helpers are available and what each one does.
- What happens when the BPF scheduler makes an error.
- How to load and unload a BPF scheduler safely.
This information is scattered across include/linux/sched/ext.h, kernel/sched/ext.c, and the
example schedulers. Without a unified reference document, every new sched_ext developer would
have to reverse-engineer these contracts from source code. Patch 29 provides the unified
reference.
The Testing Gap
The core sched_ext patches are large and complex. Manual testing with scx_simple and
scx_example_qmap verifies that the happy path works, but does not verify:
- What happens when
scx_bpf_dispatch()is called with an invalid DSQ ID? - What happens when
ops.enqueue()dispatches to a local DSQ of a different CPU? - What happens when the BPF scheduler tries to use vtime ordering on a FIFO DSQ?
- What happens when the scheduler exits cleanly vs. exits with an error?
Without automated tests covering these cases, a future change to the dispatch or DSQ code could break error handling silently — the system might panic, return incorrect results, or silently succeed when it should have triggered an error exit.
Patch 30 covers these cases systematically, providing regression protection for the behavioral invariants established by the core implementation.
Key Concepts
PATCH 29 — Documentation/scheduler/sched-ext.rst
The documentation is structured as a developer reference, not a tutorial. It covers:
Framework overview: What sched_ext is, what problem it solves, and how it relates to the
existing scheduler class hierarchy. This section explains the positioning of ext_sched_class
between fair_sched_class and idle_sched_class and what it means for task priorities.
Writing a BPF scheduler: A walkthrough of the minimal BPF scheduler (equivalent to
scx_simple) with annotations explaining each callback and helper. This section establishes
the conceptual model: the BPF program implements struct sched_ext_ops, and each operation
callback corresponds to a specific scheduling event.
The ops callback reference: For each sched_ext_ops member function:
- When it is called (the kernel event that triggers it).
- What the BPF program is expected to do (the contract).
- What happens if the BPF program does not implement it (the default behavior).
- What happens if the BPF program returns an error (the error exit conditions).
This reference is the normative specification for sched_ext behavior. Any future change to when a callback is called, or to the contract of what the BPF program must do, is a change to this specification and must update the documentation.
Dispatch queue concept: Explains the three DSQ types (global, local, user-defined), how
tasks flow between them, and the lifecycle of a user-defined DSQ (create in ops.init(),
destroy in ops.exit()). The documentation makes explicit that tasks must always end up in a
DSQ — there is no mechanism for a BPF program to "hold" a task without placing it in a queue.
BPF helper reference: For each scx_bpf_* function:
- Its signature and arguments.
- Pre/post conditions (what must be true before calling, what is guaranteed after).
- Which
sched_ext_opscallbacks it may be called from. - Thread safety guarantees.
Error handling and exit states: Explains the difference between:
SCX_EXIT_NONE(scheduler not loaded or shut down cleanly).SCX_EXIT_DONE(BPF program calledscx_bpf_exit()explicitly — clean shutdown).SCX_EXIT_UNREG(BPF program unregistered via bpftool/BPF link drop).SCX_EXIT_ERROR(kernel detected misbehavior — watchdog, invalid dispatch, etc.).SCX_EXIT_SYSRQ(user pressed Alt+SysRq+S).
Understanding exit states is critical for operators diagnosing why a BPF scheduler terminated.
Example usage: Shows how to load a BPF scheduler using bpf_skel__open(), set ops
callbacks, and load it into the kernel. This section is aimed at BPF scheduler developers who
need a quick start guide.
PATCH 30 — tools/testing/selftests/sched_ext/
The test suite in tools/testing/selftests/sched_ext/ consists of several test programs, each
testing a specific behavioral aspect of the sched_ext implementation. The tests are designed to
run without root (where possible) or with minimal privilege.
DSQ creation and destruction (test_create_dsq): Creates a user-defined DSQ, dispatches
tasks to it, and then destroys it. Verifies:
scx_bpf_create_dsq()succeeds with a valid ID.scx_bpf_create_dsq()fails withEEXISTif the ID is already taken.scx_bpf_destroy_dsq()correctly frees the DSQ.- Using a destroyed DSQ in
scx_bpf_dispatch()triggers an error exit.
Dispatch to local DSQ (test_local_dsq): Dispatches tasks to the calling CPU's local DSQ
and verifies that they run. Verifies:
- Tasks dispatched to
SCX_DSQ_LOCALrun on the dispatching CPU. - Tasks dispatched to
SCX_DSQ_LOCAL_ON(cpu)run on the specified CPU (cross-CPU dispatch).
Error conditions (test_bogus_dsq, test_vtime_misuse): Intentionally misbehaves to
verify that the kernel's error detection works:
test_bogus_dsq: Callsscx_bpf_dispatch(p, INVALID_DSQ_ID, ...). Verifies that the scheduler exits withSCX_EXIT_ERROR, not with a kernel panic.test_vtime_misuse: Mixes FIFO and vtime dispatches to the same DSQ. Verifies that this triggers an error exit with a meaningful reason string.
Local-on dispatch (test_local_on): Tests the SCX_DSQ_LOCAL_ON mechanism where a task
is dispatched to a specific CPU's local DSQ from a different CPU. Verifies that the task
actually runs on the target CPU (CPU affinity is respected in dispatch).
Enqueue flags (test_enqueue_flags): Tests the SCX_ENQ_* flags that control enqueue
behavior:
SCX_ENQ_WAKEUP: Task is waking up from sleep.SCX_ENQ_LAST: This is the last task being enqueued in a batch.SCX_ENQ_HEAD: Task should go to the head of its DSQ (priority boost).
Exit behavior (test_exit): Verifies the clean shutdown path:
- BPF scheduler calls
scx_bpf_exit(reason)explicitly. - The scheduler exits with
SCX_EXIT_DONE, notSCX_EXIT_ERROR. - The
reasonstring is available in the exit state debugfs files. - All tasks return to CFS after the scheduler exits.
prog_run test (test_prog_run): Uses BPF_PROG_RUN to invoke individual BPF callbacks
(without loading the full scheduler) and verifies their return values and side effects. This
allows unit testing of individual callbacks in isolation — without paying the overhead of
loading a full BPF scheduler for each test case.
The prog_run approach is particularly valuable for testing error conditions: you can inject
invalid arguments (e.g., a NULL task pointer, an out-of-range CPU ID) directly into a callback
and verify that the callback returns the expected error code without needing to reproduce the
exact kernel state that would naturally trigger that condition.
Test infrastructure: The test suite uses the kernel's kselftest framework:
- Tests are run by
make -C tools/testing/selftests/sched_ext run_tests. - Each test is a separate binary that forks a worker process, loads the BPF test scheduler, runs the test scenario, and checks the result.
- Tests that require
CAP_BPF(loading BPF programs) are automatically skipped in unprivileged environments. - Tests verify cleanup: after each test, the BPF scheduler is unloaded and all tasks return to CFS. A test that leaks a loaded BPF scheduler causes subsequent tests to fail, providing built-in leak detection.
Connections Between Patches
PATCH 29 (documentation)
└─→ Documents the contracts established by PATCHES 08-28
└─→ The ops callback reference is the normative specification that PATCH 30
tests verify against
PATCH 30 (selftests)
└─→ Tests the core dispatch mechanism from PATCH 09
└─→ Tests the error exit paths that PATCHES 11-12 implement
└─→ Tests DSQ vtime ordering from PATCH 28
└─→ Tests the lifecycle callbacks from PATCH 20 via prog_run
What to Focus On
For a maintainer, the critical lessons from this group:
-
Documentation as specification. The
sched-ext.rstdocument is the normative specification for sched_ext behavior. When a patch changes when a callback is called, or changes the contract of a BPF helper, the documentation must be updated in the same patch. Accepting a behavioral change without a documentation update creates a situation where the code and spec diverge, making it impossible for BPF scheduler developers to know which to trust. -
Selftest coverage as a merge gate. In the Linux kernel, selftests for a feature are expected to pass before the feature is merged. When reviewing future sched_ext patches that change behavioral aspects (e.g., new exit conditions, new DSQ ordering modes, new enqueue flags), verify that the selftest suite covers the new behavior. A patch that adds a new feature without a corresponding selftest case creates a gap that will likely be exploited by a future regression.
-
Error path testing. The
test_bogus_dsqandtest_vtime_misusetests are specifically testing error paths — they intentionally trigger misbehavior and verify the kernel's response. These tests are more valuable than happy-path tests from a maintenance perspective because they verify that the safety mechanisms work. When reviewing the selftests, be skeptical of any feature that has no error path tests. -
The prog_run approach for unit testing.
test_prog_rundemonstrates how to test individual BPF callbacks in isolation usingBPF_PROG_RUN. This technique should be used for any new BPF callback that has non-trivial logic. Unit testing callbacks in isolation is faster, more targeted, and easier to debug than full integration tests that require reproducing complex scheduler state. -
Test cleanup as a first-class concern. Each test verifies that cleanup is complete: BPF scheduler unloaded, all tasks on CFS, no leaked DSQs. This cleanup verification is not just housekeeping — it is a test of the
scx_ops_disable_workfn()path. A future change to the disable path that introduces a cleanup bug will be caught by the first test that runs after the broken test. Maintaining this cleanup discipline in new tests is essential. -
rst documentation format and kernel doc conventions.
sched-ext.rstfollows the kernel documentation conventions: reStructuredText format, cross-references to other kernel docs using:doc:roles, function documentation using.. c:function::directives. When adding new sections to this document or adding documentation for new BPF helpers, follow these conventions to ensure the document renders correctly inmake htmldocsand integrates with the kernel's documentation build system.
[PATCH 29/30] sched_ext: Documentation: scheduler: Document extensible scheduler class
View on Lore: https://lore.kernel.org/all/20240618212056.2833381-30-tj@kernel.org
Commit Message
Add Documentation/scheduler/sched-ext.rst which gives a high-level overview
and pointers to the examples.
v6: - Add paragraph explaining debug dump.
v5: - Updated to reflect /sys/kernel interface change. Kconfig options
added.
v4: - README improved, reformatted in markdown and renamed to README.md.
v3: - Added tools/sched_ext/README.
- Dropped _example prefix from scheduler names.
v2: - Apply minor edits suggested by Bagas. Caveats section dropped as all
of them are addressed.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: David Vernet <dvernet@meta.com>
Acked-by: Josh Don <joshdon@google.com>
Acked-by: Hao Luo <haoluo@google.com>
Acked-by: Barret Rhoden <brho@google.com>
Cc: Bagas Sanjaya <bagasdotme@gmail.com>
---
Documentation/scheduler/index.rst | 1 +
Documentation/scheduler/sched-ext.rst | 314 ++++++++++++++++++++++++++
include/linux/sched/ext.h | 2 +
kernel/Kconfig.preempt | 1 +
kernel/sched/ext.c | 2 +
kernel/sched/ext.h | 2 +
tools/sched_ext/README.md | 258 +++++++++++++++++++++
7 files changed, 580 insertions(+)
create mode 100644 Documentation/scheduler/sched-ext.rst
create mode 100644 tools/sched_ext/README.md
diff --git a/Documentation/scheduler/index.rst b/Documentation/scheduler/index.rst
index 43bd8a145b7a..0611dc3dda8e 100644
--- a/Documentation/scheduler/index.rst
+++ b/Documentation/scheduler/index.rst
@@ -20,6 +20,7 @@ Scheduler
sched-nice-design
sched-rt-group
sched-stats
+ sched-ext
sched-debug
text_files
diff --git a/Documentation/scheduler/sched-ext.rst b/Documentation/scheduler/sched-ext.rst
new file mode 100644
index 000000000000..497eeaa5ecbe
--- /dev/null
+++ b/Documentation/scheduler/sched-ext.rst
@@ -0,0 +1,314 @@
+==========================
+Extensible Scheduler Class
+==========================
+
+sched_ext is a scheduler class whose behavior can be defined by a set of BPF
+programs - the BPF scheduler.
+
+* sched_ext exports a full scheduling interface so that any scheduling
+ algorithm can be implemented on top.
+
+* The BPF scheduler can group CPUs however it sees fit and schedule them
+ together, as tasks aren't tied to specific CPUs at the time of wakeup.
+
+* The BPF scheduler can be turned on and off dynamically anytime.
+
+* The system integrity is maintained no matter what the BPF scheduler does.
+ The default scheduling behavior is restored anytime an error is detected,
+ a runnable task stalls, or on invoking the SysRq key sequence
+ :kbd:`SysRq-S`.
+
+* When the BPF scheduler triggers an error, debug information is dumped to
+ aid debugging. The debug dump is passed to and printed out by the
+ scheduler binary. The debug dump can also be accessed through the
+ `sched_ext_dump` tracepoint. The SysRq key sequence :kbd:`SysRq-D`
+ triggers a debug dump. This doesn't terminate the BPF scheduler and can
+ only be read through the tracepoint.
+
+Switching to and from sched_ext
+===============================
+
+``CONFIG_SCHED_CLASS_EXT`` is the config option to enable sched_ext and
+``tools/sched_ext`` contains the example schedulers. The following config
+options should be enabled to use sched_ext:
+
+.. code-block:: none
+
+ CONFIG_BPF=y
+ CONFIG_SCHED_CLASS_EXT=y
+ CONFIG_BPF_SYSCALL=y
+ CONFIG_BPF_JIT=y
+ CONFIG_DEBUG_INFO_BTF=y
+ CONFIG_BPF_JIT_ALWAYS_ON=y
+ CONFIG_BPF_JIT_DEFAULT_ON=y
+ CONFIG_PAHOLE_HAS_SPLIT_BTF=y
+ CONFIG_PAHOLE_HAS_BTF_TAG=y
+
+sched_ext is used only when the BPF scheduler is loaded and running.
+
+If a task explicitly sets its scheduling policy to ``SCHED_EXT``, it will be
+treated as ``SCHED_NORMAL`` and scheduled by CFS until the BPF scheduler is
+loaded. On load, such tasks will be switched to and scheduled by sched_ext.
+
+The BPF scheduler can choose to schedule all normal and lower class tasks by
+calling ``scx_bpf_switch_all()`` from its ``init()`` operation. In this
+case, all ``SCHED_NORMAL``, ``SCHED_BATCH``, ``SCHED_IDLE`` and
+``SCHED_EXT`` tasks are scheduled by sched_ext. In the example schedulers,
+this mode can be selected with the ``-a`` option.
+
+Terminating the sched_ext scheduler program, triggering :kbd:`SysRq-S`, or
+detection of any internal error including stalled runnable tasks aborts the
+BPF scheduler and reverts all tasks back to CFS.
+
+.. code-block:: none
+
+ # make -j16 -C tools/sched_ext
+ # tools/sched_ext/scx_simple
+ local=0 global=3
+ local=5 global=24
+ local=9 global=44
+ local=13 global=56
+ local=17 global=72
+ ^CEXIT: BPF scheduler unregistered
+
+The current status of the BPF scheduler can be determined as follows:
+
+.. code-block:: none
+
+ # cat /sys/kernel/sched_ext/state
+ enabled
+ # cat /sys/kernel/sched_ext/root/ops
+ simple
+
+``tools/sched_ext/scx_show_state.py`` is a drgn script which shows more
+detailed information:
+
+.. code-block:: none
+
+ # tools/sched_ext/scx_show_state.py
+ ops : simple
+ enabled : 1
+ switching_all : 1
+ switched_all : 1
+ enable_state : enabled (2)
+ bypass_depth : 0
+ nr_rejected : 0
+
+If ``CONFIG_SCHED_DEBUG`` is set, whether a given task is on sched_ext can
+be determined as follows:
+
+.. code-block:: none
+
+ # grep ext /proc/self/sched
+ ext.enabled : 1
+
+The Basics
+==========
+
+Userspace can implement an arbitrary BPF scheduler by loading a set of BPF
+programs that implement ``struct sched_ext_ops``. The only mandatory field
+is ``ops.name`` which must be a valid BPF object name. All operations are
+optional. The following modified excerpt is from
+``tools/sched/scx_simple.bpf.c`` showing a minimal global FIFO scheduler.
+
+.. code-block:: c
+
+ /*
+ * Decide which CPU a task should be migrated to before being
+ * enqueued (either at wakeup, fork time, or exec time). If an
+ * idle core is found by the default ops.select_cpu() implementation,
+ * then dispatch the task directly to SCX_DSQ_LOCAL and skip the
+ * ops.enqueue() callback.
+ *
+ * Note that this implementation has exactly the same behavior as the
+ * default ops.select_cpu implementation. The behavior of the scheduler
+ * would be exactly same if the implementation just didn't define the
+ * simple_select_cpu() struct_ops prog.
+ */
+ s32 BPF_STRUCT_OPS(simple_select_cpu, struct task_struct *p,
+ s32 prev_cpu, u64 wake_flags)
+ {
+ s32 cpu;
+ /* Need to initialize or the BPF verifier will reject the program */
+ bool direct = false;
+
+ cpu = scx_bpf_select_cpu_dfl(p, prev_cpu, wake_flags, &direct);
+
+ if (direct)
+ scx_bpf_dispatch(p, SCX_DSQ_LOCAL, SCX_SLICE_DFL, 0);
+
+ return cpu;
+ }
+
+ /*
+ * Do a direct dispatch of a task to the global DSQ. This ops.enqueue()
+ * callback will only be invoked if we failed to find a core to dispatch
+ * to in ops.select_cpu() above.
+ *
+ * Note that this implementation has exactly the same behavior as the
+ * default ops.enqueue implementation, which just dispatches the task
+ * to SCX_DSQ_GLOBAL. The behavior of the scheduler would be exactly same
+ * if the implementation just didn't define the simple_enqueue struct_ops
+ * prog.
+ */
+ void BPF_STRUCT_OPS(simple_enqueue, struct task_struct *p, u64 enq_flags)
+ {
+ scx_bpf_dispatch(p, SCX_DSQ_GLOBAL, SCX_SLICE_DFL, enq_flags);
+ }
+
+ s32 BPF_STRUCT_OPS(simple_init)
+ {
+ /*
+ * All SCHED_OTHER, SCHED_IDLE, and SCHED_BATCH tasks should
+ * use sched_ext.
+ */
+ scx_bpf_switch_all();
+ return 0;
+ }
+
+ void BPF_STRUCT_OPS(simple_exit, struct scx_exit_info *ei)
+ {
+ exit_type = ei->type;
+ }
+
+ SEC(".struct_ops")
+ struct sched_ext_ops simple_ops = {
+ .select_cpu = (void *)simple_select_cpu,
+ .enqueue = (void *)simple_enqueue,
+ .init = (void *)simple_init,
+ .exit = (void *)simple_exit,
+ .name = "simple",
+ };
+
+Dispatch Queues
+---------------
+
+To match the impedance between the scheduler core and the BPF scheduler,
+sched_ext uses DSQs (dispatch queues) which can operate as both a FIFO and a
+priority queue. By default, there is one global FIFO (``SCX_DSQ_GLOBAL``),
+and one local dsq per CPU (``SCX_DSQ_LOCAL``). The BPF scheduler can manage
+an arbitrary number of dsq's using ``scx_bpf_create_dsq()`` and
+``scx_bpf_destroy_dsq()``.
+
+A CPU always executes a task from its local DSQ. A task is "dispatched" to a
+DSQ. A non-local DSQ is "consumed" to transfer a task to the consuming CPU's
+local DSQ.
+
+When a CPU is looking for the next task to run, if the local DSQ is not
+empty, the first task is picked. Otherwise, the CPU tries to consume the
+global DSQ. If that doesn't yield a runnable task either, ``ops.dispatch()``
+is invoked.
+
+Scheduling Cycle
+----------------
+
+The following briefly shows how a waking task is scheduled and executed.
+
+1. When a task is waking up, ``ops.select_cpu()`` is the first operation
+ invoked. This serves two purposes. First, CPU selection optimization
+ hint. Second, waking up the selected CPU if idle.
+
+ The CPU selected by ``ops.select_cpu()`` is an optimization hint and not
+ binding. The actual decision is made at the last step of scheduling.
+ However, there is a small performance gain if the CPU
+ ``ops.select_cpu()`` returns matches the CPU the task eventually runs on.
+
+ A side-effect of selecting a CPU is waking it up from idle. While a BPF
+ scheduler can wake up any cpu using the ``scx_bpf_kick_cpu()`` helper,
+ using ``ops.select_cpu()`` judiciously can be simpler and more efficient.
+
+ A task can be immediately dispatched to a DSQ from ``ops.select_cpu()`` by
+ calling ``scx_bpf_dispatch()``. If the task is dispatched to
+ ``SCX_DSQ_LOCAL`` from ``ops.select_cpu()``, it will be dispatched to the
+ local DSQ of whichever CPU is returned from ``ops.select_cpu()``.
+ Additionally, dispatching directly from ``ops.select_cpu()`` will cause the
+ ``ops.enqueue()`` callback to be skipped.
+
+ Note that the scheduler core will ignore an invalid CPU selection, for
+ example, if it's outside the allowed cpumask of the task.
+
+2. Once the target CPU is selected, ``ops.enqueue()`` is invoked (unless the
+ task was dispatched directly from ``ops.select_cpu()``). ``ops.enqueue()``
+ can make one of the following decisions:
+
+ * Immediately dispatch the task to either the global or local DSQ by
+ calling ``scx_bpf_dispatch()`` with ``SCX_DSQ_GLOBAL`` or
+ ``SCX_DSQ_LOCAL``, respectively.
+
+ * Immediately dispatch the task to a custom DSQ by calling
+ ``scx_bpf_dispatch()`` with a DSQ ID which is smaller than 2^63.
+
+ * Queue the task on the BPF side.
+
+3. When a CPU is ready to schedule, it first looks at its local DSQ. If
+ empty, it then looks at the global DSQ. If there still isn't a task to
+ run, ``ops.dispatch()`` is invoked which can use the following two
+ functions to populate the local DSQ.
+
+ * ``scx_bpf_dispatch()`` dispatches a task to a DSQ. Any target DSQ can
+ be used - ``SCX_DSQ_LOCAL``, ``SCX_DSQ_LOCAL_ON | cpu``,
+ ``SCX_DSQ_GLOBAL`` or a custom DSQ. While ``scx_bpf_dispatch()``
+ currently can't be called with BPF locks held, this is being worked on
+ and will be supported. ``scx_bpf_dispatch()`` schedules dispatching
+ rather than performing them immediately. There can be up to
+ ``ops.dispatch_max_batch`` pending tasks.
+
+ * ``scx_bpf_consume()`` tranfers a task from the specified non-local DSQ
+ to the dispatching DSQ. This function cannot be called with any BPF
+ locks held. ``scx_bpf_consume()`` flushes the pending dispatched tasks
+ before trying to consume the specified DSQ.
+
+4. After ``ops.dispatch()`` returns, if there are tasks in the local DSQ,
+ the CPU runs the first one. If empty, the following steps are taken:
+
+ * Try to consume the global DSQ. If successful, run the task.
+
+ * If ``ops.dispatch()`` has dispatched any tasks, retry #3.
+
+ * If the previous task is an SCX task and still runnable, keep executing
+ it (see ``SCX_OPS_ENQ_LAST``).
+
+ * Go idle.
+
+Note that the BPF scheduler can always choose to dispatch tasks immediately
+in ``ops.enqueue()`` as illustrated in the above simple example. If only the
+built-in DSQs are used, there is no need to implement ``ops.dispatch()`` as
+a task is never queued on the BPF scheduler and both the local and global
+DSQs are consumed automatically.
+
+``scx_bpf_dispatch()`` queues the task on the FIFO of the target DSQ. Use
+``scx_bpf_dispatch_vtime()`` for the priority queue. Internal DSQs such as
+``SCX_DSQ_LOCAL`` and ``SCX_DSQ_GLOBAL`` do not support priority-queue
+dispatching, and must be dispatched to with ``scx_bpf_dispatch()``. See the
+function documentation and usage in ``tools/sched_ext/scx_simple.bpf.c`` for
+more information.
+
+Where to Look
+=============
+
+* ``include/linux/sched/ext.h`` defines the core data structures, ops table
+ and constants.
+
+* ``kernel/sched/ext.c`` contains sched_ext core implementation and helpers.
+ The functions prefixed with ``scx_bpf_`` can be called from the BPF
+ scheduler.
+
+* ``tools/sched_ext/`` hosts example BPF scheduler implementations.
+
+ * ``scx_simple[.bpf].c``: Minimal global FIFO scheduler example using a
+ custom DSQ.
+
+ * ``scx_qmap[.bpf].c``: A multi-level FIFO scheduler supporting five
+ levels of priority implemented with ``BPF_MAP_TYPE_QUEUE``.
+
+ABI Instability
+===============
+
+The APIs provided by sched_ext to BPF schedulers programs have no stability
+guarantees. This includes the ops table callbacks and constants defined in
+``include/linux/sched/ext.h``, as well as the ``scx_bpf_`` kfuncs defined in
+``kernel/sched/ext.c``.
+
+While we will attempt to provide a relatively stable API surface when
+possible, they are subject to change without warning between kernel
+versions.
diff --git a/include/linux/sched/ext.h b/include/linux/sched/ext.h
index 9cee193dab19..fe9a67ffe6b1 100644
--- a/include/linux/sched/ext.h
+++ b/include/linux/sched/ext.h
@@ -1,5 +1,7 @@
/* SPDX-License-Identifier: GPL-2.0 */
/*
+ * BPF extensible scheduler class: Documentation/scheduler/sched-ext.rst
+ *
* Copyright (c) 2022 Meta Platforms, Inc. and affiliates.
* Copyright (c) 2022 Tejun Heo <tj@kernel.org>
* Copyright (c) 2022 David Vernet <dvernet@meta.com>
diff --git a/kernel/Kconfig.preempt b/kernel/Kconfig.preempt
index 7dde5e424ac3..f035c87d02f1 100644
--- a/kernel/Kconfig.preempt
+++ b/kernel/Kconfig.preempt
@@ -156,4 +156,5 @@ config SCHED_CLASS_EXT
similar to struct sched_class.
For more information:
+ Documentation/scheduler/sched-ext.rst
https://github.com/sched-ext/scx
diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c
index f186c576e7d9..f814e84ceeb3 100644
--- a/kernel/sched/ext.c
+++ b/kernel/sched/ext.c
@@ -1,5 +1,7 @@
/* SPDX-License-Identifier: GPL-2.0 */
/*
+ * BPF extensible scheduler class: Documentation/scheduler/sched-ext.rst
+ *
* Copyright (c) 2022 Meta Platforms, Inc. and affiliates.
* Copyright (c) 2022 Tejun Heo <tj@kernel.org>
* Copyright (c) 2022 David Vernet <dvernet@meta.com>
diff --git a/kernel/sched/ext.h b/kernel/sched/ext.h
index 6555878c5da3..c41d742b5d62 100644
--- a/kernel/sched/ext.h
+++ b/kernel/sched/ext.h
@@ -1,5 +1,7 @@
/* SPDX-License-Identifier: GPL-2.0 */
/*
+ * BPF extensible scheduler class: Documentation/scheduler/sched-ext.rst
+ *
* Copyright (c) 2022 Meta Platforms, Inc. and affiliates.
* Copyright (c) 2022 Tejun Heo <tj@kernel.org>
* Copyright (c) 2022 David Vernet <dvernet@meta.com>
diff --git a/tools/sched_ext/README.md b/tools/sched_ext/README.md
new file mode 100644
index 000000000000..8efe70cc4363
--- /dev/null
+++ b/tools/sched_ext/README.md
@@ -0,0 +1,258 @@
+SCHED_EXT EXAMPLE SCHEDULERS
+============================
+
+# Introduction
+
+This directory contains a number of example sched_ext schedulers. These
+schedulers are meant to provide examples of different types of schedulers
+that can be built using sched_ext, and illustrate how various features of
+sched_ext can be used.
+
+Some of the examples are performant, production-ready schedulers. That is, for
+the correct workload and with the correct tuning, they may be deployed in a
+production environment with acceptable or possibly even improved performance.
+Others are just examples that in practice, would not provide acceptable
+performance (though they could be improved to get there).
+
+This README will describe these example schedulers, including describing the
+types of workloads or scenarios they're designed to accommodate, and whether or
+not they're production ready. For more details on any of these schedulers,
+please see the header comment in their .bpf.c file.
+
+
+# Compiling the examples
+
+There are a few toolchain dependencies for compiling the example schedulers.
+
+## Toolchain dependencies
+
+1. clang >= 16.0.0
+
+The schedulers are BPF programs, and therefore must be compiled with clang. gcc
+is actively working on adding a BPF backend compiler as well, but are still
+missing some features such as BTF type tags which are necessary for using
+kptrs.
+
+2. pahole >= 1.25
+
+You may need pahole in order to generate BTF from DWARF.
+
+3. rust >= 1.70.0
+
+Rust schedulers uses features present in the rust toolchain >= 1.70.0. You
+should be able to use the stable build from rustup, but if that doesn't
+work, try using the rustup nightly build.
+
+There are other requirements as well, such as make, but these are the main /
+non-trivial ones.
+
+## Compiling the kernel
+
+In order to run a sched_ext scheduler, you'll have to run a kernel compiled
+with the patches in this repository, and with a minimum set of necessary
+Kconfig options:
+
+```
+CONFIG_BPF=y
+CONFIG_SCHED_CLASS_EXT=y
+CONFIG_BPF_SYSCALL=y
+CONFIG_BPF_JIT=y
+CONFIG_DEBUG_INFO_BTF=y
+```
+
+It's also recommended that you also include the following Kconfig options:
+
+```
+CONFIG_BPF_JIT_ALWAYS_ON=y
+CONFIG_BPF_JIT_DEFAULT_ON=y
+CONFIG_PAHOLE_HAS_SPLIT_BTF=y
+CONFIG_PAHOLE_HAS_BTF_TAG=y
+```
+
+There is a `Kconfig` file in this directory whose contents you can append to
+your local `.config` file, as long as there are no conflicts with any existing
+options in the file.
+
+## Getting a vmlinux.h file
+
+You may notice that most of the example schedulers include a "vmlinux.h" file.
+This is a large, auto-generated header file that contains all of the types
+defined in some vmlinux binary that was compiled with
+[BTF](https://docs.kernel.org/bpf/btf.html) (i.e. with the BTF-related Kconfig
+options specified above).
+
+The header file is created using `bpftool`, by passing it a vmlinux binary
+compiled with BTF as follows:
+
+```bash
+$ bpftool btf dump file /path/to/vmlinux format c > vmlinux.h
+```
+
+`bpftool` analyzes all of the BTF encodings in the binary, and produces a
+header file that can be included by BPF programs to access those types. For
+example, using vmlinux.h allows a scheduler to access fields defined directly
+in vmlinux as follows:
+
+```c
+#include "vmlinux.h"
+// vmlinux.h is also implicitly included by scx_common.bpf.h.
+#include "scx_common.bpf.h"
+
+/*
+ * vmlinux.h provides definitions for struct task_struct and
+ * struct scx_enable_args.
+ */
+void BPF_STRUCT_OPS(example_enable, struct task_struct *p,
+ struct scx_enable_args *args)
+{
+ bpf_printk("Task %s enabled in example scheduler", p->comm);
+}
+
+// vmlinux.h provides the definition for struct sched_ext_ops.
+SEC(".struct_ops.link")
+struct sched_ext_ops example_ops {
+ .enable = (void *)example_enable,
+ .name = "example",
+}
+```
+
+The scheduler build system will generate this vmlinux.h file as part of the
+scheduler build pipeline. It looks for a vmlinux file in the following
+dependency order:
+
+1. If the O= environment variable is defined, at `$O/vmlinux`
+2. If the KBUILD_OUTPUT= environment variable is defined, at
+ `$KBUILD_OUTPUT/vmlinux`
+3. At `../../vmlinux` (i.e. at the root of the kernel tree where you're
+ compiling the schedulers)
+3. `/sys/kernel/btf/vmlinux`
+4. `/boot/vmlinux-$(uname -r)`
+
+In other words, if you have compiled a kernel in your local repo, its vmlinux
+file will be used to generate vmlinux.h. Otherwise, it will be the vmlinux of
+the kernel you're currently running on. This means that if you're running on a
+kernel with sched_ext support, you may not need to compile a local kernel at
+all.
+
+### Aside on CO-RE
+
+One of the cooler features of BPF is that it supports
+[CO-RE](https://nakryiko.com/posts/bpf-core-reference-guide/) (Compile Once Run
+Everywhere). This feature allows you to reference fields inside of structs with
+types defined internal to the kernel, and not have to recompile if you load the
+BPF program on a different kernel with the field at a different offset. In our
+example above, we print out a task name with `p->comm`. CO-RE would perform
+relocations for that access when the program is loaded to ensure that it's
+referencing the correct offset for the currently running kernel.
+
+## Compiling the schedulers
+
+Once you have your toolchain setup, and a vmlinux that can be used to generate
+a full vmlinux.h file, you can compile the schedulers using `make`:
+
+```bash
+$ make -j($nproc)
+```
+
+# Example schedulers
+
+This directory contains the following example schedulers. These schedulers are
+for testing and demonstrating different aspects of sched_ext. While some may be
+useful in limited scenarios, they are not intended to be practical.
+
+For more scheduler implementations, tools and documentation, visit
+https://github.com/sched-ext/scx.
+
+## scx_simple
+
+A simple scheduler that provides an example of a minimal sched_ext scheduler.
+scx_simple can be run in either global weighted vtime mode, or FIFO mode.
+
+Though very simple, in limited scenarios, this scheduler can perform reasonably
+well on single-socket systems with a unified L3 cache.
+
+## scx_qmap
+
+Another simple, yet slightly more complex scheduler that provides an example of
+a basic weighted FIFO queuing policy. It also provides examples of some common
+useful BPF features, such as sleepable per-task storage allocation in the
+`ops.prep_enable()` callback, and using the `BPF_MAP_TYPE_QUEUE` map type to
+enqueue tasks. It also illustrates how core-sched support could be implemented.
+
+## scx_central
+
+A "central" scheduler where scheduling decisions are made from a single CPU.
+This scheduler illustrates how scheduling decisions can be dispatched from a
+single CPU, allowing other cores to run with infinite slices, without timer
+ticks, and without having to incur the overhead of making scheduling decisions.
+
+The approach demonstrated by this scheduler may be useful for any workload that
+benefits from minimizing scheduling overhead and timer ticks. An example of
+where this could be particularly useful is running VMs, where running with
+infinite slices and no timer ticks allows the VM to avoid unnecessary expensive
+vmexits.
+
+
+# Troubleshooting
+
+There are a number of common issues that you may run into when building the
+schedulers. We'll go over some of the common ones here.
+
+## Build Failures
+
+### Old version of clang
+
+```
+error: static assertion failed due to requirement 'SCX_DSQ_FLAG_BUILTIN': bpftool generated vmlinux.h is missing high bits for 64bit enums, upgrade clang and pahole
+ _Static_assert(SCX_DSQ_FLAG_BUILTIN,
+ ^~~~~~~~~~~~~~~~~~~~
+1 error generated.
+```
+
+This means you built the kernel or the schedulers with an older version of
+clang than what's supported (i.e. older than 16.0.0). To remediate this:
+
+1. `which clang` to make sure you're using a sufficiently new version of clang.
+
+2. `make fullclean` in the root path of the repository, and rebuild the kernel
+ and schedulers.
+
+3. Rebuild the kernel, and then your example schedulers.
+
+The schedulers are also cleaned if you invoke `make mrproper` in the root
+directory of the tree.
+
+### Stale kernel build / incomplete vmlinux.h file
+
+As described above, you'll need a `vmlinux.h` file that was generated from a
+vmlinux built with BTF, and with sched_ext support enabled. If you don't,
+you'll see errors such as the following which indicate that a type being
+referenced in a scheduler is unknown:
+
+```
+/path/to/sched_ext/tools/sched_ext/user_exit_info.h:25:23: note: forward declaration of 'struct scx_exit_info'
+
+const struct scx_exit_info *ei)
+
+^
+```
+
+In order to resolve this, please follow the steps above in
+[Getting a vmlinux.h file](#getting-a-vmlinuxh-file) in order to ensure your
+schedulers are using a vmlinux.h file that includes the requisite types.
+
+## Misc
+
+### llvm: [OFF]
+
+You may see the following output when building the schedulers:
+
+```
+Auto-detecting system features:
+... clang-bpf-co-re: [ on ]
+... llvm: [ OFF ]
+... libcap: [ on ]
+... libbfd: [ on ]
+```
+
+Seeing `llvm: [ OFF ]` here is not an issue. You can safely ignore.
--
2.45.2
Diff
---
Documentation/scheduler/index.rst | 1 +
Documentation/scheduler/sched-ext.rst | 314 ++++++++++++++++++++++++++
include/linux/sched/ext.h | 2 +
kernel/Kconfig.preempt | 1 +
kernel/sched/ext.c | 2 +
kernel/sched/ext.h | 2 +
tools/sched_ext/README.md | 258 +++++++++++++++++++++
7 files changed, 580 insertions(+)
create mode 100644 Documentation/scheduler/sched-ext.rst
create mode 100644 tools/sched_ext/README.md
diff --git a/Documentation/scheduler/index.rst b/Documentation/scheduler/index.rst
index 43bd8a145b7a..0611dc3dda8e 100644
--- a/Documentation/scheduler/index.rst
+++ b/Documentation/scheduler/index.rst
@@ -20,6 +20,7 @@ Scheduler
sched-nice-design
sched-rt-group
sched-stats
+ sched-ext
sched-debug
text_files
diff --git a/Documentation/scheduler/sched-ext.rst b/Documentation/scheduler/sched-ext.rst
new file mode 100644
index 000000000000..497eeaa5ecbe
--- /dev/null
+++ b/Documentation/scheduler/sched-ext.rst
@@ -0,0 +1,314 @@
+==========================
+Extensible Scheduler Class
+==========================
+
+sched_ext is a scheduler class whose behavior can be defined by a set of BPF
+programs - the BPF scheduler.
+
+* sched_ext exports a full scheduling interface so that any scheduling
+ algorithm can be implemented on top.
+
+* The BPF scheduler can group CPUs however it sees fit and schedule them
+ together, as tasks aren't tied to specific CPUs at the time of wakeup.
+
+* The BPF scheduler can be turned on and off dynamically anytime.
+
+* The system integrity is maintained no matter what the BPF scheduler does.
+ The default scheduling behavior is restored anytime an error is detected,
+ a runnable task stalls, or on invoking the SysRq key sequence
+ :kbd:`SysRq-S`.
+
+* When the BPF scheduler triggers an error, debug information is dumped to
+ aid debugging. The debug dump is passed to and printed out by the
+ scheduler binary. The debug dump can also be accessed through the
+ `sched_ext_dump` tracepoint. The SysRq key sequence :kbd:`SysRq-D`
+ triggers a debug dump. This doesn't terminate the BPF scheduler and can
+ only be read through the tracepoint.
+
+Switching to and from sched_ext
+===============================
+
+``CONFIG_SCHED_CLASS_EXT`` is the config option to enable sched_ext and
+``tools/sched_ext`` contains the example schedulers. The following config
+options should be enabled to use sched_ext:
+
+.. code-block:: none
+
+ CONFIG_BPF=y
+ CONFIG_SCHED_CLASS_EXT=y
+ CONFIG_BPF_SYSCALL=y
+ CONFIG_BPF_JIT=y
+ CONFIG_DEBUG_INFO_BTF=y
+ CONFIG_BPF_JIT_ALWAYS_ON=y
+ CONFIG_BPF_JIT_DEFAULT_ON=y
+ CONFIG_PAHOLE_HAS_SPLIT_BTF=y
+ CONFIG_PAHOLE_HAS_BTF_TAG=y
+
+sched_ext is used only when the BPF scheduler is loaded and running.
+
+If a task explicitly sets its scheduling policy to ``SCHED_EXT``, it will be
+treated as ``SCHED_NORMAL`` and scheduled by CFS until the BPF scheduler is
+loaded. On load, such tasks will be switched to and scheduled by sched_ext.
+
+The BPF scheduler can choose to schedule all normal and lower class tasks by
+calling ``scx_bpf_switch_all()`` from its ``init()`` operation. In this
+case, all ``SCHED_NORMAL``, ``SCHED_BATCH``, ``SCHED_IDLE`` and
+``SCHED_EXT`` tasks are scheduled by sched_ext. In the example schedulers,
+this mode can be selected with the ``-a`` option.
+
+Terminating the sched_ext scheduler program, triggering :kbd:`SysRq-S`, or
+detection of any internal error including stalled runnable tasks aborts the
+BPF scheduler and reverts all tasks back to CFS.
+
+.. code-block:: none
+
+ # make -j16 -C tools/sched_ext
+ # tools/sched_ext/scx_simple
+ local=0 global=3
+ local=5 global=24
+ local=9 global=44
+ local=13 global=56
+ local=17 global=72
+ ^CEXIT: BPF scheduler unregistered
+
+The current status of the BPF scheduler can be determined as follows:
+
+.. code-block:: none
+
+ # cat /sys/kernel/sched_ext/state
+ enabled
+ # cat /sys/kernel/sched_ext/root/ops
+ simple
+
+``tools/sched_ext/scx_show_state.py`` is a drgn script which shows more
+detailed information:
+
+.. code-block:: none
+
+ # tools/sched_ext/scx_show_state.py
+ ops : simple
+ enabled : 1
+ switching_all : 1
+ switched_all : 1
+ enable_state : enabled (2)
+ bypass_depth : 0
+ nr_rejected : 0
+
+If ``CONFIG_SCHED_DEBUG`` is set, whether a given task is on sched_ext can
+be determined as follows:
+
+.. code-block:: none
+
+ # grep ext /proc/self/sched
+ ext.enabled : 1
+
+The Basics
+==========
+
+Userspace can implement an arbitrary BPF scheduler by loading a set of BPF
+programs that implement ``struct sched_ext_ops``. The only mandatory field
+is ``ops.name`` which must be a valid BPF object name. All operations are
+optional. The following modified excerpt is from
+``tools/sched/scx_simple.bpf.c`` showing a minimal global FIFO scheduler.
+
+.. code-block:: c
+
+ /*
+ * Decide which CPU a task should be migrated to before being
+ * enqueued (either at wakeup, fork time, or exec time). If an
+ * idle core is found by the default ops.select_cpu() implementation,
+ * then dispatch the task directly to SCX_DSQ_LOCAL and skip the
+ * ops.enqueue() callback.
+ *
+ * Note that this implementation has exactly the same behavior as the
+ * default ops.select_cpu implementation. The behavior of the scheduler
+ * would be exactly same if the implementation just didn't define the
+ * simple_select_cpu() struct_ops prog.
+ */
+ s32 BPF_STRUCT_OPS(simple_select_cpu, struct task_struct *p,
+ s32 prev_cpu, u64 wake_flags)
+ {
+ s32 cpu;
+ /* Need to initialize or the BPF verifier will reject the program */
+ bool direct = false;
+
+ cpu = scx_bpf_select_cpu_dfl(p, prev_cpu, wake_flags, &direct);
+
+ if (direct)
+ scx_bpf_dispatch(p, SCX_DSQ_LOCAL, SCX_SLICE_DFL, 0);
+
+ return cpu;
+ }
+
+ /*
+ * Do a direct dispatch of a task to the global DSQ. This ops.enqueue()
+ * callback will only be invoked if we failed to find a core to dispatch
+ * to in ops.select_cpu() above.
+ *
+ * Note that this implementation has exactly the same behavior as the
+ * default ops.enqueue implementation, which just dispatches the task
+ * to SCX_DSQ_GLOBAL. The behavior of the scheduler would be exactly same
+ * if the implementation just didn't define the simple_enqueue struct_ops
+ * prog.
+ */
+ void BPF_STRUCT_OPS(simple_enqueue, struct task_struct *p, u64 enq_flags)
+ {
+ scx_bpf_dispatch(p, SCX_DSQ_GLOBAL, SCX_SLICE_DFL, enq_flags);
+ }
+
+ s32 BPF_STRUCT_OPS(simple_init)
+ {
+ /*
+ * All SCHED_OTHER, SCHED_IDLE, and SCHED_BATCH tasks should
+ * use sched_ext.
+ */
+ scx_bpf_switch_all();
+ return 0;
+ }
+
+ void BPF_STRUCT_OPS(simple_exit, struct scx_exit_info *ei)
+ {
+ exit_type = ei->type;
+ }
+
+ SEC(".struct_ops")
+ struct sched_ext_ops simple_ops = {
+ .select_cpu = (void *)simple_select_cpu,
+ .enqueue = (void *)simple_enqueue,
+ .init = (void *)simple_init,
+ .exit = (void *)simple_exit,
+ .name = "simple",
+ };
+
+Dispatch Queues
+---------------
+
+To match the impedance between the scheduler core and the BPF scheduler,
+sched_ext uses DSQs (dispatch queues) which can operate as both a FIFO and a
+priority queue. By default, there is one global FIFO (``SCX_DSQ_GLOBAL``),
+and one local dsq per CPU (``SCX_DSQ_LOCAL``). The BPF scheduler can manage
+an arbitrary number of dsq's using ``scx_bpf_create_dsq()`` and
+``scx_bpf_destroy_dsq()``.
+
+A CPU always executes a task from its local DSQ. A task is "dispatched" to a
+DSQ. A non-local DSQ is "consumed" to transfer a task to the consuming CPU's
+local DSQ.
+
+When a CPU is looking for the next task to run, if the local DSQ is not
+empty, the first task is picked. Otherwise, the CPU tries to consume the
+global DSQ. If that doesn't yield a runnable task either, ``ops.dispatch()``
+is invoked.
+
+Scheduling Cycle
+----------------
+
+The following briefly shows how a waking task is scheduled and executed.
+
+1. When a task is waking up, ``ops.select_cpu()`` is the first operation
+ invoked. This serves two purposes. First, CPU selection optimization
+ hint. Second, waking up the selected CPU if idle.
+
+ The CPU selected by ``ops.select_cpu()`` is an optimization hint and not
+ binding. The actual decision is made at the last step of scheduling.
+ However, there is a small performance gain if the CPU
+ ``ops.select_cpu()`` returns matches the CPU the task eventually runs on.
+
+ A side-effect of selecting a CPU is waking it up from idle. While a BPF
+ scheduler can wake up any cpu using the ``scx_bpf_kick_cpu()`` helper,
+ using ``ops.select_cpu()`` judiciously can be simpler and more efficient.
+
+ A task can be immediately dispatched to a DSQ from ``ops.select_cpu()`` by
+ calling ``scx_bpf_dispatch()``. If the task is dispatched to
+ ``SCX_DSQ_LOCAL`` from ``ops.select_cpu()``, it will be dispatched to the
+ local DSQ of whichever CPU is returned from ``ops.select_cpu()``.
+ Additionally, dispatching directly from ``ops.select_cpu()`` will cause the
+ ``ops.enqueue()`` callback to be skipped.
+
+ Note that the scheduler core will ignore an invalid CPU selection, for
+ example, if it's outside the allowed cpumask of the task.
+
+2. Once the target CPU is selected, ``ops.enqueue()`` is invoked (unless the
+ task was dispatched directly from ``ops.select_cpu()``). ``ops.enqueue()``
+ can make one of the following decisions:
+
+ * Immediately dispatch the task to either the global or local DSQ by
+ calling ``scx_bpf_dispatch()`` with ``SCX_DSQ_GLOBAL`` or
+ ``SCX_DSQ_LOCAL``, respectively.
+
+ * Immediately dispatch the task to a custom DSQ by calling
+ ``scx_bpf_dispatch()`` with a DSQ ID which is smaller than 2^63.
+
+ * Queue the task on the BPF side.
+
+3. When a CPU is ready to schedule, it first looks at its local DSQ. If
+ empty, it then looks at the global DSQ. If there still isn't a task to
+ run, ``ops.dispatch()`` is invoked which can use the following two
+ functions to populate the local DSQ.
+
+ * ``scx_bpf_dispatch()`` dispatches a task to a DSQ. Any target DSQ can
+ be used - ``SCX_DSQ_LOCAL``, ``SCX_DSQ_LOCAL_ON | cpu``,
+ ``SCX_DSQ_GLOBAL`` or a custom DSQ. While ``scx_bpf_dispatch()``
+ currently can't be called with BPF locks held, this is being worked on
+ and will be supported. ``scx_bpf_dispatch()`` schedules dispatching
+ rather than performing them immediately. There can be up to
+ ``ops.dispatch_max_batch`` pending tasks.
+
+ * ``scx_bpf_consume()`` tranfers a task from the specified non-local DSQ
+ to the dispatching DSQ. This function cannot be called with any BPF
+ locks held. ``scx_bpf_consume()`` flushes the pending dispatched tasks
+ before trying to consume the specified DSQ.
+
+4. After ``ops.dispatch()`` returns, if there are tasks in the local DSQ,
+ the CPU runs the first one. If empty, the following steps are taken:
+
+ * Try to consume the global DSQ. If successful, run the task.
+
+ * If ``ops.dispatch()`` has dispatched any tasks, retry #3.
+
+ * If the previous task is an sched_ext task and still runnable, keep executing
+ it (see ``SCX_OPS_ENQ_LAST``).
+
+ * Go idle.
+
+Note that the BPF scheduler can always choose to dispatch tasks immediately
+in ``ops.enqueue()`` as illustrated in the above simple example. If only the
+built-in DSQs are used, there is no need to implement ``ops.dispatch()`` as
+a task is never queued on the BPF scheduler and both the local and global
+DSQs are consumed automatically.
+
+``scx_bpf_dispatch()`` queues the task on the FIFO of the target DSQ. Use
+``scx_bpf_dispatch_vtime()`` for the priority queue. Internal DSQs such as
+``SCX_DSQ_LOCAL`` and ``SCX_DSQ_GLOBAL`` do not support priority-queue
+dispatching, and must be dispatched to with ``scx_bpf_dispatch()``. See the
+function documentation and usage in ``tools/sched_ext/scx_simple.bpf.c`` for
+more information.
+
+Where to Look
+=============
+
+* ``include/linux/sched/ext.h`` defines the core data structures, ops table
+ and constants.
+
+* ``kernel/sched/ext.c`` contains sched_ext core implementation and helpers.
+ The functions prefixed with ``scx_bpf_`` can be called from the BPF
+ scheduler.
+
+* ``tools/sched_ext/`` hosts example BPF scheduler implementations.
+
+ * ``scx_simple[.bpf].c``: Minimal global FIFO scheduler example using a
+ custom DSQ.
+
+ * ``scx_qmap[.bpf].c``: A multi-level FIFO scheduler supporting five
+ levels of priority implemented with ``BPF_MAP_TYPE_QUEUE``.
+
+ABI Instability
+===============
+
+The APIs provided by sched_ext to BPF schedulers programs have no stability
+guarantees. This includes the ops table callbacks and constants defined in
+``include/linux/sched/ext.h``, as well as the ``scx_bpf_`` kfuncs defined in
+``kernel/sched/ext.c``.
+
+While we will attempt to provide a relatively stable API surface when
+possible, they are subject to change without warning between kernel
+versions.
diff --git a/include/linux/sched/ext.h b/include/linux/sched/ext.h
index 9cee193dab19..fe9a67ffe6b1 100644
--- a/include/linux/sched/ext.h
+++ b/include/linux/sched/ext.h
@@ -1,5 +1,7 @@
/* SPDX-License-Identifier: GPL-2.0 */
/*
+ * BPF extensible scheduler class: Documentation/scheduler/sched-ext.rst
+ *
* Copyright (c) 2022 Meta Platforms, Inc. and affiliates.
* Copyright (c) 2022 Tejun Heo <tj@kernel.org>
* Copyright (c) 2022 David Vernet <dvernet@meta.com>
diff --git a/kernel/Kconfig.preempt b/kernel/Kconfig.preempt
index 7dde5e424ac3..f035c87d02f1 100644
--- a/kernel/Kconfig.preempt
+++ b/kernel/Kconfig.preempt
@@ -156,4 +156,5 @@ config SCHED_CLASS_EXT
similar to struct sched_class.
For more information:
+ Documentation/scheduler/sched-ext.rst
https://github.com/sched-ext/scx
diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c
index f186c576e7d9..f814e84ceeb3 100644
--- a/kernel/sched/ext.c
+++ b/kernel/sched/ext.c
@@ -1,5 +1,7 @@
/* SPDX-License-Identifier: GPL-2.0 */
/*
+ * BPF extensible scheduler class: Documentation/scheduler/sched-ext.rst
+ *
* Copyright (c) 2022 Meta Platforms, Inc. and affiliates.
* Copyright (c) 2022 Tejun Heo <tj@kernel.org>
* Copyright (c) 2022 David Vernet <dvernet@meta.com>
diff --git a/kernel/sched/ext.h b/kernel/sched/ext.h
index 6555878c5da3..c41d742b5d62 100644
--- a/kernel/sched/ext.h
+++ b/kernel/sched/ext.h
@@ -1,5 +1,7 @@
/* SPDX-License-Identifier: GPL-2.0 */
/*
+ * BPF extensible scheduler class: Documentation/scheduler/sched-ext.rst
+ *
* Copyright (c) 2022 Meta Platforms, Inc. and affiliates.
* Copyright (c) 2022 Tejun Heo <tj@kernel.org>
* Copyright (c) 2022 David Vernet <dvernet@meta.com>
diff --git a/tools/sched_ext/README.md b/tools/sched_ext/README.md
new file mode 100644
index 000000000000..8efe70cc4363
--- /dev/null
+++ b/tools/sched_ext/README.md
@@ -0,0 +1,258 @@
+SCHED_EXT EXAMPLE SCHEDULERS
+============================
+
+# Introduction
+
+This directory contains a number of example sched_ext schedulers. These
+schedulers are meant to provide examples of different types of schedulers
+that can be built using sched_ext, and illustrate how various features of
+sched_ext can be used.
+
+Some of the examples are performant, production-ready schedulers. That is, for
+the correct workload and with the correct tuning, they may be deployed in a
+production environment with acceptable or possibly even improved performance.
+Others are just examples that in practice, would not provide acceptable
+performance (though they could be improved to get there).
+
+This README will describe these example schedulers, including describing the
+types of workloads or scenarios they're designed to accommodate, and whether or
+not they're production ready. For more details on any of these schedulers,
+please see the header comment in their .bpf.c file.
+
+
+# Compiling the examples
+
+There are a few toolchain dependencies for compiling the example schedulers.
+
+## Toolchain dependencies
+
+1. clang >= 16.0.0
+
+The schedulers are BPF programs, and therefore must be compiled with clang. gcc
+is actively working on adding a BPF backend compiler as well, but are still
+missing some features such as BTF type tags which are necessary for using
+kptrs.
+
+2. pahole >= 1.25
+
+You may need pahole in order to generate BTF from DWARF.
+
+3. rust >= 1.70.0
+
+Rust schedulers uses features present in the rust toolchain >= 1.70.0. You
+should be able to use the stable build from rustup, but if that doesn't
+work, try using the rustup nightly build.
+
+There are other requirements as well, such as make, but these are the main /
+non-trivial ones.
+
+## Compiling the kernel
+
+In order to run a sched_ext scheduler, you'll have to run a kernel compiled
+with the patches in this repository, and with a minimum set of necessary
+Kconfig options:
+
+```
+CONFIG_BPF=y
+CONFIG_SCHED_CLASS_EXT=y
+CONFIG_BPF_SYSCALL=y
+CONFIG_BPF_JIT=y
+CONFIG_DEBUG_INFO_BTF=y
+```
+
+It's also recommended that you also include the following Kconfig options:
+
+```
+CONFIG_BPF_JIT_ALWAYS_ON=y
+CONFIG_BPF_JIT_DEFAULT_ON=y
+CONFIG_PAHOLE_HAS_SPLIT_BTF=y
+CONFIG_PAHOLE_HAS_BTF_TAG=y
+```
+
+There is a `Kconfig` file in this directory whose contents you can append to
+your local `.config` file, as long as there are no conflicts with any existing
+options in the file.
+
+## Getting a vmlinux.h file
+
+You may notice that most of the example schedulers include a "vmlinux.h" file.
+This is a large, auto-generated header file that contains all of the types
+defined in some vmlinux binary that was compiled with
+[BTF](https://docs.kernel.org/bpf/btf.html) (i.e. with the BTF-related Kconfig
+options specified above).
+
+The header file is created using `bpftool`, by passing it a vmlinux binary
+compiled with BTF as follows:
+
+```bash
+$ bpftool btf dump file /path/to/vmlinux format c > vmlinux.h
+```
+
+`bpftool` analyzes all of the BTF encodings in the binary, and produces a
+header file that can be included by BPF programs to access those types. For
+example, using vmlinux.h allows a scheduler to access fields defined directly
+in vmlinux as follows:
+
+```c
+#include "vmlinux.h"
+// vmlinux.h is also implicitly included by scx_common.bpf.h.
+#include "scx_common.bpf.h"
+
+/*
+ * vmlinux.h provides definitions for struct task_struct and
+ * struct scx_enable_args.
+ */
+void BPF_STRUCT_OPS(example_enable, struct task_struct *p,
+ struct scx_enable_args *args)
+{
+ bpf_printk("Task %s enabled in example scheduler", p->comm);
+}
+
+// vmlinux.h provides the definition for struct sched_ext_ops.
+SEC(".struct_ops.link")
+struct sched_ext_ops example_ops {
+ .enable = (void *)example_enable,
+ .name = "example",
+}
+```
+
+The scheduler build system will generate this vmlinux.h file as part of the
+scheduler build pipeline. It looks for a vmlinux file in the following
+dependency order:
+
+1. If the O= environment variable is defined, at `$O/vmlinux`
+2. If the KBUILD_OUTPUT= environment variable is defined, at
+ `$KBUILD_OUTPUT/vmlinux`
+3. At `../../vmlinux` (i.e. at the root of the kernel tree where you're
+ compiling the schedulers)
+3. `/sys/kernel/btf/vmlinux`
+4. `/boot/vmlinux-$(uname -r)`
+
+In other words, if you have compiled a kernel in your local repo, its vmlinux
+file will be used to generate vmlinux.h. Otherwise, it will be the vmlinux of
+the kernel you're currently running on. This means that if you're running on a
+kernel with sched_ext support, you may not need to compile a local kernel at
+all.
+
+### Aside on CO-RE
+
+One of the cooler features of BPF is that it supports
+[CO-RE](https://nakryiko.com/posts/bpf-core-reference-guide/) (Compile Once Run
+Everywhere). This feature allows you to reference fields inside of structs with
+types defined internal to the kernel, and not have to recompile if you load the
+BPF program on a different kernel with the field at a different offset. In our
+example above, we print out a task name with `p->comm`. CO-RE would perform
+relocations for that access when the program is loaded to ensure that it's
+referencing the correct offset for the currently running kernel.
+
+## Compiling the schedulers
+
+Once you have your toolchain setup, and a vmlinux that can be used to generate
+a full vmlinux.h file, you can compile the schedulers using `make`:
+
+```bash
+$ make -j($nproc)
+```
+
+# Example schedulers
+
+This directory contains the following example schedulers. These schedulers are
+for testing and demonstrating different aspects of sched_ext. While some may be
+useful in limited scenarios, they are not intended to be practical.
+
+For more scheduler implementations, tools and documentation, visit
+https://github.com/sched-ext/scx.
+
+## scx_simple
+
+A simple scheduler that provides an example of a minimal sched_ext scheduler.
+scx_simple can be run in either global weighted vtime mode, or FIFO mode.
+
+Though very simple, in limited scenarios, this scheduler can perform reasonably
+well on single-socket systems with a unified L3 cache.
+
+## scx_qmap
+
+Another simple, yet slightly more complex scheduler that provides an example of
+a basic weighted FIFO queuing policy. It also provides examples of some common
+useful BPF features, such as sleepable per-task storage allocation in the
+`ops.prep_enable()` callback, and using the `BPF_MAP_TYPE_QUEUE` map type to
+enqueue tasks. It also illustrates how core-sched support could be implemented.
+
+## scx_central
+
+A "central" scheduler where scheduling decisions are made from a single CPU.
+This scheduler illustrates how scheduling decisions can be dispatched from a
+single CPU, allowing other cores to run with infinite slices, without timer
+ticks, and without having to incur the overhead of making scheduling decisions.
+
+The approach demonstrated by this scheduler may be useful for any workload that
+benefits from minimizing scheduling overhead and timer ticks. An example of
+where this could be particularly useful is running VMs, where running with
+infinite slices and no timer ticks allows the VM to avoid unnecessary expensive
+vmexits.
+
+
+# Troubleshooting
+
+There are a number of common issues that you may run into when building the
+schedulers. We'll go over some of the common ones here.
+
+## Build Failures
+
+### Old version of clang
+
+```
+error: static assertion failed due to requirement 'SCX_DSQ_FLAG_BUILTIN': bpftool generated vmlinux.h is missing high bits for 64bit enums, upgrade clang and pahole
+ _Static_assert(SCX_DSQ_FLAG_BUILTIN,
+ ^~~~~~~~~~~~~~~~~~~~
+1 error generated.
+```
+
+This means you built the kernel or the schedulers with an older version of
+clang than what's supported (i.e. older than 16.0.0). To remediate this:
+
+1. `which clang` to make sure you're using a sufficiently new version of clang.
+
+2. `make fullclean` in the root path of the repository, and rebuild the kernel
+ and schedulers.
+
+3. Rebuild the kernel, and then your example schedulers.
+
+The schedulers are also cleaned if you invoke `make mrproper` in the root
+directory of the tree.
+
+### Stale kernel build / incomplete vmlinux.h file
+
+As described above, you'll need a `vmlinux.h` file that was generated from a
+vmlinux built with BTF, and with sched_ext support enabled. If you don't,
+you'll see errors such as the following which indicate that a type being
+referenced in a scheduler is unknown:
+
+```
+/path/to/sched_ext/tools/sched_ext/user_exit_info.h:25:23: note: forward declaration of 'struct scx_exit_info'
+
+const struct scx_exit_info *ei)
+
+^
+```
+
+In order to resolve this, please follow the steps above in
+[Getting a vmlinux.h file](#getting-a-vmlinuxh-file) in order to ensure your
+schedulers are using a vmlinux.h file that includes the requisite types.
+
+## Misc
+
+### llvm: [OFF]
+
+You may see the following output when building the schedulers:
+
+```
+Auto-detecting system features:
+... clang-bpf-co-re: [ on ]
+... llvm: [ OFF ]
+... libcap: [ on ]
+... libbfd: [ on ]
+```
+
+Seeing `llvm: [ OFF ]` here is not an issue. You can safely ignore.
--
2.45.2
Implementation Analysis
Overview
This patch adds the official kernel documentation for sched_ext. It creates Documentation/scheduler/sched-ext.rst (314 lines) and tools/sched_ext/README.md (258 lines). The RST document is added to the scheduler documentation index. Source file headers in ext.c, ext.h, and include/linux/sched/ext.h are updated to reference the documentation. The Kconfig help text gains a link to the RST document.
This patch marks the official API contract for sched_ext — what it documents is what is supported.
Code Walkthrough
Documentation/scheduler/sched-ext.rst — the kernel documentation:
The document covers five main areas:
-
Feature overview: sched_ext exports a full scheduling interface; the BPF scheduler can group CPUs freely; it can be turned on/off dynamically; system integrity is always maintained (fallback to CFS on errors, stalls, or SysRq-S); debug dumps via tracepoint
sched_ext_dumpand SysRq-D. -
Switching to/from sched_ext: Required Kconfig options (
CONFIG_SCHED_CLASS_EXT,CONFIG_BPF_SYSCALL,CONFIG_BPF_JIT,CONFIG_DEBUG_INFO_BTF, etc.). Task migration semantics: tasks withSCHED_EXTpolicy run under CFS until a BPF scheduler is loaded.scx_bpf_switch_all()makes allSCHED_NORMAL/BATCH/IDLEtasks use sched_ext. -
The Basics: A complete minimal scheduler example (select_cpu → enqueue → init → exit). This is the canonical "getting started" code.
-
Dispatch Queues: The DSQ concept — FIFO and priority queue modes,
SCX_DSQ_GLOBALandSCX_DSQ_LOCAL,scx_bpf_create_dsq/destroy_dsq, the full 4-step scheduling cycle from wakeup to execution. -
Where to Look: Pointers to
include/linux/sched/ext.h(ops table, constants),kernel/sched/ext.c(implementation, kfuncs), andtools/sched_ext/(examples).
ABI Instability section: Explicitly documents that the sched_ext BPF API has no stability guarantees between kernel versions. The ops callbacks, constants, and kfuncs are all subject to change. This is a critical statement for BPF scheduler authors.
tools/sched_ext/README.md — the practitioner guide:
Covers toolchain requirements (clang >= 16.0.0 for BPF, pahole >= 1.25 for BTF, rust >= 1.70.0 for Rust schedulers), kernel Kconfig options, how to obtain and use vmlinux.h via bpftool btf dump, CO-RE (Compile Once Run Everywhere) explanation, and descriptions of the example schedulers (scx_simple, scx_qmap, scx_central). Includes a troubleshooting section for common build failures.
Cross-references added to source files: ext.c, ext.h, and include/linux/sched/ext.h each gain:
/* BPF extensible scheduler class: Documentation/scheduler/sched-ext.rst */
This makes it easy to find the documentation when reading kernel source.
Kconfig update (kernel/Kconfig.preempt):
For more information:
Documentation/scheduler/sched-ext.rst
https://github.com/sched-ext/scx
Key Concepts
The scheduling cycle (from the docs), which every BPF scheduler author must understand:
ops.select_cpu()— CPU selection hint + wake idle CPU. Can dispatch directly toSCX_DSQ_LOCALto skipops.enqueue().ops.enqueue()— dispatch to global, local, or custom DSQ, or hold in BPF queues.ops.dispatch()— called when local DSQ is empty. Usescx_bpf_dispatch()orscx_bpf_consume()to populate local DSQ.- Task execution: local DSQ → global DSQ →
ops.dispatch()retry →SCX_OPS_ENQ_LASTfallback → idle.
ABI instability: The BPF-facing API (struct sched_ext_ops, kfuncs, constants) has no kernel ABI stability guarantees. This is distinct from normal kernel ABI — BPF schedulers must be compiled for a specific kernel version or use CO-RE where applicable.
scx_bpf_dispatch_vtime() documentation in the Dispatch Queues section explicitly states:
- Use for PRIQ DSQs
- Internal DSQs (LOCAL, GLOBAL) must use
scx_bpf_dispatch()only
Locking and Concurrency Notes
The documentation itself does not introduce new locking. However, the scheduling cycle description implicitly conveys the locking context:
select_cpuandenqueueare called in the task-enqueue path (rq lock held for the target CPU)dispatchis called in the balance/pick path (rq lock held for the dispatching CPU)
The docs note: "While scx_bpf_dispatch() currently can't be called with BPF locks held, this is being worked on and will be supported." This is an accurate forward-looking statement as of this patch series.
Integration with Kernel Subsystems
The RST file is added to Documentation/scheduler/index.rst, making it part of the standard kernel documentation build. It will appear in the rendered kernel docs at the same level as sched-design-CFS.rst, sched-deadline.rst, etc.
The tools/sched_ext/README.md is a developer-facing document for users of the tools/sched_ext/ directory. It is not part of the kernel documentation build system.
What Maintainers Need to Know
- The ABI instability section is the most important part of the documentation for a maintainer. Future patches that change ops callbacks, kfunc signatures, or constants must update
Documentation/scheduler/sched-ext.rstaccordingly. - The scheduling cycle documentation is the API contract. Any change to when/how
select_cpu,enqueue,dispatch, or other callbacks fire must be reflected here. - The "Where to Look" section should be updated whenever new primary source files are added (e.g., if
kernel/sched/ext_helpers.cwere split out). tools/sched_ext/README.mdshould be updated when example schedulers are added or removed, or when build requirements change (new minimum clang version, etc.).- The cross-reference comments in
ext.c,ext.h, andsched/ext.hare a convention — new files related to sched_ext should follow the same pattern.
Connection to Other Patches
- All prior patches (1-28): This documentation describes the complete API assembled by those patches. The scheduling cycle, DSQ model, vtime dispatch, ops callbacks (including cpu_acquire/release, cpu_online/offline, core_sched_before) are all documented here.
- Patch 28/30 (vtime DSQs): The DSQ section explicitly describes
scx_bpf_dispatch_vtime()and its restrictions. The scx_simple example in the README references the vtime scheduler mode. - Patch 30/30 (selftests): The README in this patch and the Makefile in the selftests patch together form the developer onboarding experience.
[PATCH 30/30] sched_ext: Add selftests
View on Lore: https://lore.kernel.org/all/20240618212056.2833381-31-tj@kernel.org
Commit Message
From: David Vernet <dvernet@meta.com>
Add basic selftests.
Signed-off-by: David Vernet <dvernet@meta.com>
Acked-by: Tejun Heo <tj@kernel.org>
---
tools/testing/selftests/sched_ext/.gitignore | 6 +
tools/testing/selftests/sched_ext/Makefile | 218 ++++++++++++++++++
tools/testing/selftests/sched_ext/config | 9 +
.../selftests/sched_ext/create_dsq.bpf.c | 58 +++++
.../testing/selftests/sched_ext/create_dsq.c | 57 +++++
.../sched_ext/ddsp_bogus_dsq_fail.bpf.c | 42 ++++
.../selftests/sched_ext/ddsp_bogus_dsq_fail.c | 57 +++++
.../sched_ext/ddsp_vtimelocal_fail.bpf.c | 39 ++++
.../sched_ext/ddsp_vtimelocal_fail.c | 56 +++++
.../selftests/sched_ext/dsp_local_on.bpf.c | 65 ++++++
.../selftests/sched_ext/dsp_local_on.c | 58 +++++
.../sched_ext/enq_last_no_enq_fails.bpf.c | 21 ++
.../sched_ext/enq_last_no_enq_fails.c | 60 +++++
.../sched_ext/enq_select_cpu_fails.bpf.c | 43 ++++
.../sched_ext/enq_select_cpu_fails.c | 61 +++++
tools/testing/selftests/sched_ext/exit.bpf.c | 84 +++++++
tools/testing/selftests/sched_ext/exit.c | 55 +++++
tools/testing/selftests/sched_ext/exit_test.h | 20 ++
.../testing/selftests/sched_ext/hotplug.bpf.c | 61 +++++
tools/testing/selftests/sched_ext/hotplug.c | 168 ++++++++++++++
.../selftests/sched_ext/hotplug_test.h | 15 ++
.../sched_ext/init_enable_count.bpf.c | 53 +++++
.../selftests/sched_ext/init_enable_count.c | 166 +++++++++++++
.../testing/selftests/sched_ext/maximal.bpf.c | 132 +++++++++++
tools/testing/selftests/sched_ext/maximal.c | 51 ++++
.../selftests/sched_ext/maybe_null.bpf.c | 36 +++
.../testing/selftests/sched_ext/maybe_null.c | 49 ++++
.../sched_ext/maybe_null_fail_dsp.bpf.c | 25 ++
.../sched_ext/maybe_null_fail_yld.bpf.c | 28 +++
.../testing/selftests/sched_ext/minimal.bpf.c | 21 ++
tools/testing/selftests/sched_ext/minimal.c | 58 +++++
.../selftests/sched_ext/prog_run.bpf.c | 32 +++
tools/testing/selftests/sched_ext/prog_run.c | 78 +++++++
.../testing/selftests/sched_ext/reload_loop.c | 75 ++++++
tools/testing/selftests/sched_ext/runner.c | 201 ++++++++++++++++
tools/testing/selftests/sched_ext/scx_test.h | 131 +++++++++++
.../selftests/sched_ext/select_cpu_dfl.bpf.c | 40 ++++
.../selftests/sched_ext/select_cpu_dfl.c | 72 ++++++
.../sched_ext/select_cpu_dfl_nodispatch.bpf.c | 89 +++++++
.../sched_ext/select_cpu_dfl_nodispatch.c | 72 ++++++
.../sched_ext/select_cpu_dispatch.bpf.c | 41 ++++
.../selftests/sched_ext/select_cpu_dispatch.c | 70 ++++++
.../select_cpu_dispatch_bad_dsq.bpf.c | 37 +++
.../sched_ext/select_cpu_dispatch_bad_dsq.c | 56 +++++
.../select_cpu_dispatch_dbl_dsp.bpf.c | 38 +++
.../sched_ext/select_cpu_dispatch_dbl_dsp.c | 56 +++++
.../sched_ext/select_cpu_vtime.bpf.c | 92 ++++++++
.../selftests/sched_ext/select_cpu_vtime.c | 59 +++++
.../selftests/sched_ext/test_example.c | 49 ++++
tools/testing/selftests/sched_ext/util.c | 71 ++++++
tools/testing/selftests/sched_ext/util.h | 13 ++
51 files changed, 3244 insertions(+)
create mode 100644 tools/testing/selftests/sched_ext/.gitignore
create mode 100644 tools/testing/selftests/sched_ext/Makefile
create mode 100644 tools/testing/selftests/sched_ext/config
create mode 100644 tools/testing/selftests/sched_ext/create_dsq.bpf.c
create mode 100644 tools/testing/selftests/sched_ext/create_dsq.c
create mode 100644 tools/testing/selftests/sched_ext/ddsp_bogus_dsq_fail.bpf.c
create mode 100644 tools/testing/selftests/sched_ext/ddsp_bogus_dsq_fail.c
create mode 100644 tools/testing/selftests/sched_ext/ddsp_vtimelocal_fail.bpf.c
create mode 100644 tools/testing/selftests/sched_ext/ddsp_vtimelocal_fail.c
create mode 100644 tools/testing/selftests/sched_ext/dsp_local_on.bpf.c
create mode 100644 tools/testing/selftests/sched_ext/dsp_local_on.c
create mode 100644 tools/testing/selftests/sched_ext/enq_last_no_enq_fails.bpf.c
create mode 100644 tools/testing/selftests/sched_ext/enq_last_no_enq_fails.c
create mode 100644 tools/testing/selftests/sched_ext/enq_select_cpu_fails.bpf.c
create mode 100644 tools/testing/selftests/sched_ext/enq_select_cpu_fails.c
create mode 100644 tools/testing/selftests/sched_ext/exit.bpf.c
create mode 100644 tools/testing/selftests/sched_ext/exit.c
create mode 100644 tools/testing/selftests/sched_ext/exit_test.h
create mode 100644 tools/testing/selftests/sched_ext/hotplug.bpf.c
create mode 100644 tools/testing/selftests/sched_ext/hotplug.c
create mode 100644 tools/testing/selftests/sched_ext/hotplug_test.h
create mode 100644 tools/testing/selftests/sched_ext/init_enable_count.bpf.c
create mode 100644 tools/testing/selftests/sched_ext/init_enable_count.c
create mode 100644 tools/testing/selftests/sched_ext/maximal.bpf.c
create mode 100644 tools/testing/selftests/sched_ext/maximal.c
create mode 100644 tools/testing/selftests/sched_ext/maybe_null.bpf.c
create mode 100644 tools/testing/selftests/sched_ext/maybe_null.c
create mode 100644 tools/testing/selftests/sched_ext/maybe_null_fail_dsp.bpf.c
create mode 100644 tools/testing/selftests/sched_ext/maybe_null_fail_yld.bpf.c
create mode 100644 tools/testing/selftests/sched_ext/minimal.bpf.c
create mode 100644 tools/testing/selftests/sched_ext/minimal.c
create mode 100644 tools/testing/selftests/sched_ext/prog_run.bpf.c
create mode 100644 tools/testing/selftests/sched_ext/prog_run.c
create mode 100644 tools/testing/selftests/sched_ext/reload_loop.c
create mode 100644 tools/testing/selftests/sched_ext/runner.c
create mode 100644 tools/testing/selftests/sched_ext/scx_test.h
create mode 100644 tools/testing/selftests/sched_ext/select_cpu_dfl.bpf.c
create mode 100644 tools/testing/selftests/sched_ext/select_cpu_dfl.c
create mode 100644 tools/testing/selftests/sched_ext/select_cpu_dfl_nodispatch.bpf.c
create mode 100644 tools/testing/selftests/sched_ext/select_cpu_dfl_nodispatch.c
create mode 100644 tools/testing/selftests/sched_ext/select_cpu_dispatch.bpf.c
create mode 100644 tools/testing/selftests/sched_ext/select_cpu_dispatch.c
create mode 100644 tools/testing/selftests/sched_ext/select_cpu_dispatch_bad_dsq.bpf.c
create mode 100644 tools/testing/selftests/sched_ext/select_cpu_dispatch_bad_dsq.c
create mode 100644 tools/testing/selftests/sched_ext/select_cpu_dispatch_dbl_dsp.bpf.c
create mode 100644 tools/testing/selftests/sched_ext/select_cpu_dispatch_dbl_dsp.c
create mode 100644 tools/testing/selftests/sched_ext/select_cpu_vtime.bpf.c
create mode 100644 tools/testing/selftests/sched_ext/select_cpu_vtime.c
create mode 100644 tools/testing/selftests/sched_ext/test_example.c
create mode 100644 tools/testing/selftests/sched_ext/util.c
create mode 100644 tools/testing/selftests/sched_ext/util.h
diff --git a/tools/testing/selftests/sched_ext/.gitignore b/tools/testing/selftests/sched_ext/.gitignore
new file mode 100644
index 000000000000..ae5491a114c0
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/.gitignore
@@ -0,0 +1,6 @@
+*
+!*.c
+!*.h
+!Makefile
+!.gitignore
+!config
diff --git a/tools/testing/selftests/sched_ext/Makefile b/tools/testing/selftests/sched_ext/Makefile
new file mode 100644
index 000000000000..0754a2c110a1
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/Makefile
@@ -0,0 +1,218 @@
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2022 Meta Platforms, Inc. and affiliates.
+include ../../../build/Build.include
+include ../../../scripts/Makefile.arch
+include ../../../scripts/Makefile.include
+include ../lib.mk
+
+ifneq ($(LLVM),)
+ifneq ($(filter %/,$(LLVM)),)
+LLVM_PREFIX := $(LLVM)
+else ifneq ($(filter -%,$(LLVM)),)
+LLVM_SUFFIX := $(LLVM)
+endif
+
+CC := $(LLVM_PREFIX)clang$(LLVM_SUFFIX) $(CLANG_FLAGS) -fintegrated-as
+else
+CC := gcc
+endif # LLVM
+
+ifneq ($(CROSS_COMPILE),)
+$(error CROSS_COMPILE not supported for scx selftests)
+endif # CROSS_COMPILE
+
+CURDIR := $(abspath .)
+REPOROOT := $(abspath ../../../..)
+TOOLSDIR := $(REPOROOT)/tools
+LIBDIR := $(TOOLSDIR)/lib
+BPFDIR := $(LIBDIR)/bpf
+TOOLSINCDIR := $(TOOLSDIR)/include
+BPFTOOLDIR := $(TOOLSDIR)/bpf/bpftool
+APIDIR := $(TOOLSINCDIR)/uapi
+GENDIR := $(REPOROOT)/include/generated
+GENHDR := $(GENDIR)/autoconf.h
+SCXTOOLSDIR := $(TOOLSDIR)/sched_ext
+SCXTOOLSINCDIR := $(TOOLSDIR)/sched_ext/include
+
+OUTPUT_DIR := $(CURDIR)/build
+OBJ_DIR := $(OUTPUT_DIR)/obj
+INCLUDE_DIR := $(OUTPUT_DIR)/include
+BPFOBJ_DIR := $(OBJ_DIR)/libbpf
+SCXOBJ_DIR := $(OBJ_DIR)/sched_ext
+BPFOBJ := $(BPFOBJ_DIR)/libbpf.a
+LIBBPF_OUTPUT := $(OBJ_DIR)/libbpf/libbpf.a
+DEFAULT_BPFTOOL := $(OUTPUT_DIR)/sbin/bpftool
+HOST_BUILD_DIR := $(OBJ_DIR)
+HOST_OUTPUT_DIR := $(OUTPUT_DIR)
+
+VMLINUX_BTF_PATHS ?= ../../../../vmlinux \
+ /sys/kernel/btf/vmlinux \
+ /boot/vmlinux-$(shell uname -r)
+VMLINUX_BTF ?= $(abspath $(firstword $(wildcard $(VMLINUX_BTF_PATHS))))
+ifeq ($(VMLINUX_BTF),)
+$(error Cannot find a vmlinux for VMLINUX_BTF at any of "$(VMLINUX_BTF_PATHS)")
+endif
+
+BPFTOOL ?= $(DEFAULT_BPFTOOL)
+
+ifneq ($(wildcard $(GENHDR)),)
+ GENFLAGS := -DHAVE_GENHDR
+endif
+
+CFLAGS += -g -O2 -rdynamic -pthread -Wall -Werror $(GENFLAGS) \
+ -I$(INCLUDE_DIR) -I$(GENDIR) -I$(LIBDIR) \
+ -I$(TOOLSINCDIR) -I$(APIDIR) -I$(CURDIR)/include -I$(SCXTOOLSINCDIR)
+
+# Silence some warnings when compiled with clang
+ifneq ($(LLVM),)
+CFLAGS += -Wno-unused-command-line-argument
+endif
+
+LDFLAGS = -lelf -lz -lpthread -lzstd
+
+IS_LITTLE_ENDIAN = $(shell $(CC) -dM -E - </dev/null | \
+ grep 'define __BYTE_ORDER__ __ORDER_LITTLE_ENDIAN__')
+
+# Get Clang's default includes on this system, as opposed to those seen by
+# '-target bpf'. This fixes "missing" files on some architectures/distros,
+# such as asm/byteorder.h, asm/socket.h, asm/sockios.h, sys/cdefs.h etc.
+#
+# Use '-idirafter': Don't interfere with include mechanics except where the
+# build would have failed anyways.
+define get_sys_includes
+$(shell $(1) -v -E - </dev/null 2>&1 \
+ | sed -n '/<...> search starts here:/,/End of search list./{ s| \(/.*\)|-idirafter \1|p }') \
+$(shell $(1) -dM -E - </dev/null | grep '__riscv_xlen ' | awk '{printf("-D__riscv_xlen=%d -D__BITS_PER_LONG=%d", $$3, $$3)}')
+endef
+
+BPF_CFLAGS = -g -D__TARGET_ARCH_$(SRCARCH) \
+ $(if $(IS_LITTLE_ENDIAN),-mlittle-endian,-mbig-endian) \
+ -I$(CURDIR)/include -I$(CURDIR)/include/bpf-compat \
+ -I$(INCLUDE_DIR) -I$(APIDIR) -I$(SCXTOOLSINCDIR) \
+ -I$(REPOROOT)/include \
+ $(call get_sys_includes,$(CLANG)) \
+ -Wall -Wno-compare-distinct-pointer-types \
+ -Wno-incompatible-function-pointer-types \
+ -O2 -mcpu=v3
+
+# sort removes libbpf duplicates when not cross-building
+MAKE_DIRS := $(sort $(OBJ_DIR)/libbpf $(OBJ_DIR)/libbpf \
+ $(OBJ_DIR)/bpftool $(OBJ_DIR)/resolve_btfids \
+ $(INCLUDE_DIR) $(SCXOBJ_DIR))
+
+$(MAKE_DIRS):
+ $(call msg,MKDIR,,$@)
+ $(Q)mkdir -p $@
+
+$(BPFOBJ): $(wildcard $(BPFDIR)/*.[ch] $(BPFDIR)/Makefile) \
+ $(APIDIR)/linux/bpf.h \
+ | $(OBJ_DIR)/libbpf
+ $(Q)$(MAKE) $(submake_extras) -C $(BPFDIR) OUTPUT=$(OBJ_DIR)/libbpf/ \
+ EXTRA_CFLAGS='-g -O0 -fPIC' \
+ DESTDIR=$(OUTPUT_DIR) prefix= all install_headers
+
+$(DEFAULT_BPFTOOL): $(wildcard $(BPFTOOLDIR)/*.[ch] $(BPFTOOLDIR)/Makefile) \
+ $(LIBBPF_OUTPUT) | $(OBJ_DIR)/bpftool
+ $(Q)$(MAKE) $(submake_extras) -C $(BPFTOOLDIR) \
+ ARCH= CROSS_COMPILE= CC=$(HOSTCC) LD=$(HOSTLD) \
+ EXTRA_CFLAGS='-g -O0' \
+ OUTPUT=$(OBJ_DIR)/bpftool/ \
+ LIBBPF_OUTPUT=$(OBJ_DIR)/libbpf/ \
+ LIBBPF_DESTDIR=$(OUTPUT_DIR)/ \
+ prefix= DESTDIR=$(OUTPUT_DIR)/ install-bin
+
+$(INCLUDE_DIR)/vmlinux.h: $(VMLINUX_BTF) $(BPFTOOL) | $(INCLUDE_DIR)
+ifeq ($(VMLINUX_H),)
+ $(call msg,GEN,,$@)
+ $(Q)$(BPFTOOL) btf dump file $(VMLINUX_BTF) format c > $@
+else
+ $(call msg,CP,,$@)
+ $(Q)cp "$(VMLINUX_H)" $@
+endif
+
+$(SCXOBJ_DIR)/%.bpf.o: %.bpf.c $(INCLUDE_DIR)/vmlinux.h | $(BPFOBJ) $(SCXOBJ_DIR)
+ $(call msg,CLNG-BPF,,$(notdir $@))
+ $(Q)$(CLANG) $(BPF_CFLAGS) -target bpf -c $< -o $@
+
+$(INCLUDE_DIR)/%.bpf.skel.h: $(SCXOBJ_DIR)/%.bpf.o $(INCLUDE_DIR)/vmlinux.h $(BPFTOOL) | $(INCLUDE_DIR)
+ $(eval sched=$(notdir $@))
+ $(call msg,GEN-SKEL,,$(sched))
+ $(Q)$(BPFTOOL) gen object $(<:.o=.linked1.o) $<
+ $(Q)$(BPFTOOL) gen object $(<:.o=.linked2.o) $(<:.o=.linked1.o)
+ $(Q)$(BPFTOOL) gen object $(<:.o=.linked3.o) $(<:.o=.linked2.o)
+ $(Q)diff $(<:.o=.linked2.o) $(<:.o=.linked3.o)
+ $(Q)$(BPFTOOL) gen skeleton $(<:.o=.linked3.o) name $(subst .bpf.skel.h,,$(sched)) > $@
+ $(Q)$(BPFTOOL) gen subskeleton $(<:.o=.linked3.o) name $(subst .bpf.skel.h,,$(sched)) > $(@:.skel.h=.subskel.h)
+
+################
+# C schedulers #
+################
+
+override define CLEAN
+ rm -rf $(OUTPUT_DIR)
+ rm -f *.o *.bpf.o *.bpf.skel.h *.bpf.subskel.h
+ rm -f $(TEST_GEN_PROGS)
+ rm -f runner
+endef
+
+# Every testcase takes all of the BPF progs are dependencies by default. This
+# allows testcases to load any BPF scheduler, which is useful for testcases
+# that don't need their own prog to run their test.
+all_test_bpfprogs := $(foreach prog,$(wildcard *.bpf.c),$(INCLUDE_DIR)/$(patsubst %.c,%.skel.h,$(prog)))
+
+auto-test-targets := \
+ create_dsq \
+ enq_last_no_enq_fails \
+ enq_select_cpu_fails \
+ ddsp_bogus_dsq_fail \
+ ddsp_vtimelocal_fail \
+ dsp_local_on \
+ exit \
+ hotplug \
+ init_enable_count \
+ maximal \
+ maybe_null \
+ minimal \
+ prog_run \
+ reload_loop \
+ select_cpu_dfl \
+ select_cpu_dfl_nodispatch \
+ select_cpu_dispatch \
+ select_cpu_dispatch_bad_dsq \
+ select_cpu_dispatch_dbl_dsp \
+ select_cpu_vtime \
+ test_example \
+
+testcase-targets := $(addsuffix .o,$(addprefix $(SCXOBJ_DIR)/,$(auto-test-targets)))
+
+$(SCXOBJ_DIR)/runner.o: runner.c | $(SCXOBJ_DIR)
+ $(CC) $(CFLAGS) -c $< -o $@
+
+# Create all of the test targets object files, whose testcase objects will be
+# registered into the runner in ELF constructors.
+#
+# Note that we must do double expansion here in order to support conditionally
+# compiling BPF object files only if one is present, as the wildcard Make
+# function doesn't support using implicit rules otherwise.
+$(testcase-targets): $(SCXOBJ_DIR)/%.o: %.c $(SCXOBJ_DIR)/runner.o $(all_test_bpfprogs) | $(SCXOBJ_DIR)
+ $(eval test=$(patsubst %.o,%.c,$(notdir $@)))
+ $(CC) $(CFLAGS) -c $< -o $@ $(SCXOBJ_DIR)/runner.o
+
+$(SCXOBJ_DIR)/util.o: util.c | $(SCXOBJ_DIR)
+ $(CC) $(CFLAGS) -c $< -o $@
+
+runner: $(SCXOBJ_DIR)/runner.o $(SCXOBJ_DIR)/util.o $(BPFOBJ) $(testcase-targets)
+ @echo "$(testcase-targets)"
+ $(CC) $(CFLAGS) -o $@ $^ $(LDFLAGS)
+
+TEST_GEN_PROGS := runner
+
+all: runner
+
+.PHONY: all clean help
+
+.DEFAULT_GOAL := all
+
+.DELETE_ON_ERROR:
+
+.SECONDARY:
diff --git a/tools/testing/selftests/sched_ext/config b/tools/testing/selftests/sched_ext/config
new file mode 100644
index 000000000000..0de9b4ee249d
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/config
@@ -0,0 +1,9 @@
+CONFIG_SCHED_DEBUG=y
+CONFIG_SCHED_CLASS_EXT=y
+CONFIG_CGROUPS=y
+CONFIG_CGROUP_SCHED=y
+CONFIG_EXT_GROUP_SCHED=y
+CONFIG_BPF=y
+CONFIG_BPF_SYSCALL=y
+CONFIG_DEBUG_INFO=y
+CONFIG_DEBUG_INFO_BTF=y
diff --git a/tools/testing/selftests/sched_ext/create_dsq.bpf.c b/tools/testing/selftests/sched_ext/create_dsq.bpf.c
new file mode 100644
index 000000000000..23f79ed343f0
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/create_dsq.bpf.c
@@ -0,0 +1,58 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Create and destroy DSQs in a loop.
+ *
+ * Copyright (c) 2024 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2024 David Vernet <dvernet@meta.com>
+ */
+
+#include <scx/common.bpf.h>
+
+char _license[] SEC("license") = "GPL";
+
+void BPF_STRUCT_OPS(create_dsq_exit_task, struct task_struct *p,
+ struct scx_exit_task_args *args)
+{
+ scx_bpf_destroy_dsq(p->pid);
+}
+
+s32 BPF_STRUCT_OPS_SLEEPABLE(create_dsq_init_task, struct task_struct *p,
+ struct scx_init_task_args *args)
+{
+ s32 err;
+
+ err = scx_bpf_create_dsq(p->pid, -1);
+ if (err)
+ scx_bpf_error("Failed to create DSQ for %s[%d]",
+ p->comm, p->pid);
+
+ return err;
+}
+
+s32 BPF_STRUCT_OPS_SLEEPABLE(create_dsq_init)
+{
+ u32 i;
+ s32 err;
+
+ bpf_for(i, 0, 1024) {
+ err = scx_bpf_create_dsq(i, -1);
+ if (err) {
+ scx_bpf_error("Failed to create DSQ %d", i);
+ return 0;
+ }
+ }
+
+ bpf_for(i, 0, 1024) {
+ scx_bpf_destroy_dsq(i);
+ }
+
+ return 0;
+}
+
+SEC(".struct_ops.link")
+struct sched_ext_ops create_dsq_ops = {
+ .init_task = create_dsq_init_task,
+ .exit_task = create_dsq_exit_task,
+ .init = create_dsq_init,
+ .name = "create_dsq",
+};
diff --git a/tools/testing/selftests/sched_ext/create_dsq.c b/tools/testing/selftests/sched_ext/create_dsq.c
new file mode 100644
index 000000000000..fa946d9146d4
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/create_dsq.c
@@ -0,0 +1,57 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2024 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2024 David Vernet <dvernet@meta.com>
+ */
+#include <bpf/bpf.h>
+#include <scx/common.h>
+#include <sys/wait.h>
+#include <unistd.h>
+#include "create_dsq.bpf.skel.h"
+#include "scx_test.h"
+
+static enum scx_test_status setup(void **ctx)
+{
+ struct create_dsq *skel;
+
+ skel = create_dsq__open_and_load();
+ if (!skel) {
+ SCX_ERR("Failed to open and load skel");
+ return SCX_TEST_FAIL;
+ }
+ *ctx = skel;
+
+ return SCX_TEST_PASS;
+}
+
+static enum scx_test_status run(void *ctx)
+{
+ struct create_dsq *skel = ctx;
+ struct bpf_link *link;
+
+ link = bpf_map__attach_struct_ops(skel->maps.create_dsq_ops);
+ if (!link) {
+ SCX_ERR("Failed to attach scheduler");
+ return SCX_TEST_FAIL;
+ }
+
+ bpf_link__destroy(link);
+
+ return SCX_TEST_PASS;
+}
+
+static void cleanup(void *ctx)
+{
+ struct create_dsq *skel = ctx;
+
+ create_dsq__destroy(skel);
+}
+
+struct scx_test create_dsq = {
+ .name = "create_dsq",
+ .description = "Create and destroy a dsq in a loop",
+ .setup = setup,
+ .run = run,
+ .cleanup = cleanup,
+};
+REGISTER_SCX_TEST(&create_dsq)
diff --git a/tools/testing/selftests/sched_ext/ddsp_bogus_dsq_fail.bpf.c b/tools/testing/selftests/sched_ext/ddsp_bogus_dsq_fail.bpf.c
new file mode 100644
index 000000000000..e97ad41d354a
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/ddsp_bogus_dsq_fail.bpf.c
@@ -0,0 +1,42 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2024 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2024 David Vernet <dvernet@meta.com>
+ * Copyright (c) 2024 Tejun Heo <tj@kernel.org>
+ */
+#include <scx/common.bpf.h>
+
+char _license[] SEC("license") = "GPL";
+
+UEI_DEFINE(uei);
+
+s32 BPF_STRUCT_OPS(ddsp_bogus_dsq_fail_select_cpu, struct task_struct *p,
+ s32 prev_cpu, u64 wake_flags)
+{
+ s32 cpu = scx_bpf_pick_idle_cpu(p->cpus_ptr, 0);
+
+ if (cpu >= 0) {
+ /*
+ * If we dispatch to a bogus DSQ that will fall back to the
+ * builtin global DSQ, we fail gracefully.
+ */
+ scx_bpf_dispatch_vtime(p, 0xcafef00d, SCX_SLICE_DFL,
+ p->scx.dsq_vtime, 0);
+ return cpu;
+ }
+
+ return prev_cpu;
+}
+
+void BPF_STRUCT_OPS(ddsp_bogus_dsq_fail_exit, struct scx_exit_info *ei)
+{
+ UEI_RECORD(uei, ei);
+}
+
+SEC(".struct_ops.link")
+struct sched_ext_ops ddsp_bogus_dsq_fail_ops = {
+ .select_cpu = ddsp_bogus_dsq_fail_select_cpu,
+ .exit = ddsp_bogus_dsq_fail_exit,
+ .name = "ddsp_bogus_dsq_fail",
+ .timeout_ms = 1000U,
+};
diff --git a/tools/testing/selftests/sched_ext/ddsp_bogus_dsq_fail.c b/tools/testing/selftests/sched_ext/ddsp_bogus_dsq_fail.c
new file mode 100644
index 000000000000..e65d22f23f3b
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/ddsp_bogus_dsq_fail.c
@@ -0,0 +1,57 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2024 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2024 David Vernet <dvernet@meta.com>
+ * Copyright (c) 2024 Tejun Heo <tj@kernel.org>
+ */
+#include <bpf/bpf.h>
+#include <scx/common.h>
+#include <sys/wait.h>
+#include <unistd.h>
+#include "ddsp_bogus_dsq_fail.bpf.skel.h"
+#include "scx_test.h"
+
+static enum scx_test_status setup(void **ctx)
+{
+ struct ddsp_bogus_dsq_fail *skel;
+
+ skel = ddsp_bogus_dsq_fail__open_and_load();
+ SCX_FAIL_IF(!skel, "Failed to open and load skel");
+ *ctx = skel;
+
+ return SCX_TEST_PASS;
+}
+
+static enum scx_test_status run(void *ctx)
+{
+ struct ddsp_bogus_dsq_fail *skel = ctx;
+ struct bpf_link *link;
+
+ link = bpf_map__attach_struct_ops(skel->maps.ddsp_bogus_dsq_fail_ops);
+ SCX_FAIL_IF(!link, "Failed to attach struct_ops");
+
+ sleep(1);
+
+ SCX_EQ(skel->data->uei.kind, EXIT_KIND(SCX_EXIT_ERROR));
+ bpf_link__destroy(link);
+
+ return SCX_TEST_PASS;
+}
+
+static void cleanup(void *ctx)
+{
+ struct ddsp_bogus_dsq_fail *skel = ctx;
+
+ ddsp_bogus_dsq_fail__destroy(skel);
+}
+
+struct scx_test ddsp_bogus_dsq_fail = {
+ .name = "ddsp_bogus_dsq_fail",
+ .description = "Verify we gracefully fail, and fall back to using a "
+ "built-in DSQ, if we do a direct dispatch to an invalid"
+ " DSQ in ops.select_cpu()",
+ .setup = setup,
+ .run = run,
+ .cleanup = cleanup,
+};
+REGISTER_SCX_TEST(&ddsp_bogus_dsq_fail)
diff --git a/tools/testing/selftests/sched_ext/ddsp_vtimelocal_fail.bpf.c b/tools/testing/selftests/sched_ext/ddsp_vtimelocal_fail.bpf.c
new file mode 100644
index 000000000000..dde7e7dafbfb
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/ddsp_vtimelocal_fail.bpf.c
@@ -0,0 +1,39 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2024 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2024 David Vernet <dvernet@meta.com>
+ * Copyright (c) 2024 Tejun Heo <tj@kernel.org>
+ */
+#include <scx/common.bpf.h>
+
+char _license[] SEC("license") = "GPL";
+
+UEI_DEFINE(uei);
+
+s32 BPF_STRUCT_OPS(ddsp_vtimelocal_fail_select_cpu, struct task_struct *p,
+ s32 prev_cpu, u64 wake_flags)
+{
+ s32 cpu = scx_bpf_pick_idle_cpu(p->cpus_ptr, 0);
+
+ if (cpu >= 0) {
+ /* Shouldn't be allowed to vtime dispatch to a builtin DSQ. */
+ scx_bpf_dispatch_vtime(p, SCX_DSQ_LOCAL, SCX_SLICE_DFL,
+ p->scx.dsq_vtime, 0);
+ return cpu;
+ }
+
+ return prev_cpu;
+}
+
+void BPF_STRUCT_OPS(ddsp_vtimelocal_fail_exit, struct scx_exit_info *ei)
+{
+ UEI_RECORD(uei, ei);
+}
+
+SEC(".struct_ops.link")
+struct sched_ext_ops ddsp_vtimelocal_fail_ops = {
+ .select_cpu = ddsp_vtimelocal_fail_select_cpu,
+ .exit = ddsp_vtimelocal_fail_exit,
+ .name = "ddsp_vtimelocal_fail",
+ .timeout_ms = 1000U,
+};
diff --git a/tools/testing/selftests/sched_ext/ddsp_vtimelocal_fail.c b/tools/testing/selftests/sched_ext/ddsp_vtimelocal_fail.c
new file mode 100644
index 000000000000..abafee587cd6
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/ddsp_vtimelocal_fail.c
@@ -0,0 +1,56 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2024 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2024 David Vernet <dvernet@meta.com>
+ * Copyright (c) 2024 Tejun Heo <tj@kernel.org>
+ */
+#include <bpf/bpf.h>
+#include <scx/common.h>
+#include <unistd.h>
+#include "ddsp_vtimelocal_fail.bpf.skel.h"
+#include "scx_test.h"
+
+static enum scx_test_status setup(void **ctx)
+{
+ struct ddsp_vtimelocal_fail *skel;
+
+ skel = ddsp_vtimelocal_fail__open_and_load();
+ SCX_FAIL_IF(!skel, "Failed to open and load skel");
+ *ctx = skel;
+
+ return SCX_TEST_PASS;
+}
+
+static enum scx_test_status run(void *ctx)
+{
+ struct ddsp_vtimelocal_fail *skel = ctx;
+ struct bpf_link *link;
+
+ link = bpf_map__attach_struct_ops(skel->maps.ddsp_vtimelocal_fail_ops);
+ SCX_FAIL_IF(!link, "Failed to attach struct_ops");
+
+ sleep(1);
+
+ SCX_EQ(skel->data->uei.kind, EXIT_KIND(SCX_EXIT_ERROR));
+ bpf_link__destroy(link);
+
+ return SCX_TEST_PASS;
+}
+
+static void cleanup(void *ctx)
+{
+ struct ddsp_vtimelocal_fail *skel = ctx;
+
+ ddsp_vtimelocal_fail__destroy(skel);
+}
+
+struct scx_test ddsp_vtimelocal_fail = {
+ .name = "ddsp_vtimelocal_fail",
+ .description = "Verify we gracefully fail, and fall back to using a "
+ "built-in DSQ, if we do a direct vtime dispatch to a "
+ "built-in DSQ from DSQ in ops.select_cpu()",
+ .setup = setup,
+ .run = run,
+ .cleanup = cleanup,
+};
+REGISTER_SCX_TEST(&ddsp_vtimelocal_fail)
diff --git a/tools/testing/selftests/sched_ext/dsp_local_on.bpf.c b/tools/testing/selftests/sched_ext/dsp_local_on.bpf.c
new file mode 100644
index 000000000000..efb4672decb4
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/dsp_local_on.bpf.c
@@ -0,0 +1,65 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2024 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2024 David Vernet <dvernet@meta.com>
+ */
+#include <scx/common.bpf.h>
+
+char _license[] SEC("license") = "GPL";
+const volatile s32 nr_cpus;
+
+UEI_DEFINE(uei);
+
+struct {
+ __uint(type, BPF_MAP_TYPE_QUEUE);
+ __uint(max_entries, 8192);
+ __type(value, s32);
+} queue SEC(".maps");
+
+s32 BPF_STRUCT_OPS(dsp_local_on_select_cpu, struct task_struct *p,
+ s32 prev_cpu, u64 wake_flags)
+{
+ return prev_cpu;
+}
+
+void BPF_STRUCT_OPS(dsp_local_on_enqueue, struct task_struct *p,
+ u64 enq_flags)
+{
+ s32 pid = p->pid;
+
+ if (bpf_map_push_elem(&queue, &pid, 0))
+ scx_bpf_error("Failed to enqueue %s[%d]", p->comm, p->pid);
+}
+
+void BPF_STRUCT_OPS(dsp_local_on_dispatch, s32 cpu, struct task_struct *prev)
+{
+ s32 pid, target;
+ struct task_struct *p;
+
+ if (bpf_map_pop_elem(&queue, &pid))
+ return;
+
+ p = bpf_task_from_pid(pid);
+ if (!p)
+ return;
+
+ target = bpf_get_prandom_u32() % nr_cpus;
+
+ scx_bpf_dispatch(p, SCX_DSQ_LOCAL_ON | target, SCX_SLICE_DFL, 0);
+ bpf_task_release(p);
+}
+
+void BPF_STRUCT_OPS(dsp_local_on_exit, struct scx_exit_info *ei)
+{
+ UEI_RECORD(uei, ei);
+}
+
+SEC(".struct_ops.link")
+struct sched_ext_ops dsp_local_on_ops = {
+ .select_cpu = dsp_local_on_select_cpu,
+ .enqueue = dsp_local_on_enqueue,
+ .dispatch = dsp_local_on_dispatch,
+ .exit = dsp_local_on_exit,
+ .name = "dsp_local_on",
+ .timeout_ms = 1000U,
+};
diff --git a/tools/testing/selftests/sched_ext/dsp_local_on.c b/tools/testing/selftests/sched_ext/dsp_local_on.c
new file mode 100644
index 000000000000..472851b56854
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/dsp_local_on.c
@@ -0,0 +1,58 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2024 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2024 David Vernet <dvernet@meta.com>
+ */
+#include <bpf/bpf.h>
+#include <scx/common.h>
+#include <unistd.h>
+#include "dsp_local_on.bpf.skel.h"
+#include "scx_test.h"
+
+static enum scx_test_status setup(void **ctx)
+{
+ struct dsp_local_on *skel;
+
+ skel = dsp_local_on__open();
+ SCX_FAIL_IF(!skel, "Failed to open");
+
+ skel->rodata->nr_cpus = libbpf_num_possible_cpus();
+ SCX_FAIL_IF(dsp_local_on__load(skel), "Failed to load skel");
+ *ctx = skel;
+
+ return SCX_TEST_PASS;
+}
+
+static enum scx_test_status run(void *ctx)
+{
+ struct dsp_local_on *skel = ctx;
+ struct bpf_link *link;
+
+ link = bpf_map__attach_struct_ops(skel->maps.dsp_local_on_ops);
+ SCX_FAIL_IF(!link, "Failed to attach struct_ops");
+
+ /* Just sleeping is fine, plenty of scheduling events happening */
+ sleep(1);
+
+ SCX_EQ(skel->data->uei.kind, EXIT_KIND(SCX_EXIT_ERROR));
+ bpf_link__destroy(link);
+
+ return SCX_TEST_PASS;
+}
+
+static void cleanup(void *ctx)
+{
+ struct dsp_local_on *skel = ctx;
+
+ dsp_local_on__destroy(skel);
+}
+
+struct scx_test dsp_local_on = {
+ .name = "dsp_local_on",
+ .description = "Verify we can directly dispatch tasks to a local DSQs "
+ "from osp.dispatch()",
+ .setup = setup,
+ .run = run,
+ .cleanup = cleanup,
+};
+REGISTER_SCX_TEST(&dsp_local_on)
diff --git a/tools/testing/selftests/sched_ext/enq_last_no_enq_fails.bpf.c b/tools/testing/selftests/sched_ext/enq_last_no_enq_fails.bpf.c
new file mode 100644
index 000000000000..b0b99531d5d5
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/enq_last_no_enq_fails.bpf.c
@@ -0,0 +1,21 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * A scheduler that validates the behavior of direct dispatching with a default
+ * select_cpu implementation.
+ *
+ * Copyright (c) 2023 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2023 David Vernet <dvernet@meta.com>
+ * Copyright (c) 2023 Tejun Heo <tj@kernel.org>
+ */
+
+#include <scx/common.bpf.h>
+
+char _license[] SEC("license") = "GPL";
+
+SEC(".struct_ops.link")
+struct sched_ext_ops enq_last_no_enq_fails_ops = {
+ .name = "enq_last_no_enq_fails",
+ /* Need to define ops.enqueue() with SCX_OPS_ENQ_LAST */
+ .flags = SCX_OPS_ENQ_LAST,
+ .timeout_ms = 1000U,
+};
diff --git a/tools/testing/selftests/sched_ext/enq_last_no_enq_fails.c b/tools/testing/selftests/sched_ext/enq_last_no_enq_fails.c
new file mode 100644
index 000000000000..2a3eda5e2c0b
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/enq_last_no_enq_fails.c
@@ -0,0 +1,60 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2023 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2023 David Vernet <dvernet@meta.com>
+ * Copyright (c) 2023 Tejun Heo <tj@kernel.org>
+ */
+#include <bpf/bpf.h>
+#include <scx/common.h>
+#include <sys/wait.h>
+#include <unistd.h>
+#include "enq_last_no_enq_fails.bpf.skel.h"
+#include "scx_test.h"
+
+static enum scx_test_status setup(void **ctx)
+{
+ struct enq_last_no_enq_fails *skel;
+
+ skel = enq_last_no_enq_fails__open_and_load();
+ if (!skel) {
+ SCX_ERR("Failed to open and load skel");
+ return SCX_TEST_FAIL;
+ }
+ *ctx = skel;
+
+ return SCX_TEST_PASS;
+}
+
+static enum scx_test_status run(void *ctx)
+{
+ struct enq_last_no_enq_fails *skel = ctx;
+ struct bpf_link *link;
+
+ link = bpf_map__attach_struct_ops(skel->maps.enq_last_no_enq_fails_ops);
+ if (link) {
+ SCX_ERR("Incorrectly succeeded in to attaching scheduler");
+ return SCX_TEST_FAIL;
+ }
+
+ bpf_link__destroy(link);
+
+ return SCX_TEST_PASS;
+}
+
+static void cleanup(void *ctx)
+{
+ struct enq_last_no_enq_fails *skel = ctx;
+
+ enq_last_no_enq_fails__destroy(skel);
+}
+
+struct scx_test enq_last_no_enq_fails = {
+ .name = "enq_last_no_enq_fails",
+ .description = "Verify we fail to load a scheduler if we specify "
+ "the SCX_OPS_ENQ_LAST flag without defining "
+ "ops.enqueue()",
+ .setup = setup,
+ .run = run,
+ .cleanup = cleanup,
+};
+REGISTER_SCX_TEST(&enq_last_no_enq_fails)
diff --git a/tools/testing/selftests/sched_ext/enq_select_cpu_fails.bpf.c b/tools/testing/selftests/sched_ext/enq_select_cpu_fails.bpf.c
new file mode 100644
index 000000000000..b3dfc1033cd6
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/enq_select_cpu_fails.bpf.c
@@ -0,0 +1,43 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2023 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2023 David Vernet <dvernet@meta.com>
+ * Copyright (c) 2023 Tejun Heo <tj@kernel.org>
+ */
+
+#include <scx/common.bpf.h>
+
+char _license[] SEC("license") = "GPL";
+
+/* Manually specify the signature until the kfunc is added to the scx repo. */
+s32 scx_bpf_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags,
+ bool *found) __ksym;
+
+s32 BPF_STRUCT_OPS(enq_select_cpu_fails_select_cpu, struct task_struct *p,
+ s32 prev_cpu, u64 wake_flags)
+{
+ return prev_cpu;
+}
+
+void BPF_STRUCT_OPS(enq_select_cpu_fails_enqueue, struct task_struct *p,
+ u64 enq_flags)
+{
+ /*
+ * Need to initialize the variable or the verifier will fail to load.
+ * Improving these semantics is actively being worked on.
+ */
+ bool found = false;
+
+ /* Can only call from ops.select_cpu() */
+ scx_bpf_select_cpu_dfl(p, 0, 0, &found);
+
+ scx_bpf_dispatch(p, SCX_DSQ_GLOBAL, SCX_SLICE_DFL, enq_flags);
+}
+
+SEC(".struct_ops.link")
+struct sched_ext_ops enq_select_cpu_fails_ops = {
+ .select_cpu = enq_select_cpu_fails_select_cpu,
+ .enqueue = enq_select_cpu_fails_enqueue,
+ .name = "enq_select_cpu_fails",
+ .timeout_ms = 1000U,
+};
diff --git a/tools/testing/selftests/sched_ext/enq_select_cpu_fails.c b/tools/testing/selftests/sched_ext/enq_select_cpu_fails.c
new file mode 100644
index 000000000000..dd1350e5f002
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/enq_select_cpu_fails.c
@@ -0,0 +1,61 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2023 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2023 David Vernet <dvernet@meta.com>
+ * Copyright (c) 2023 Tejun Heo <tj@kernel.org>
+ */
+#include <bpf/bpf.h>
+#include <scx/common.h>
+#include <sys/wait.h>
+#include <unistd.h>
+#include "enq_select_cpu_fails.bpf.skel.h"
+#include "scx_test.h"
+
+static enum scx_test_status setup(void **ctx)
+{
+ struct enq_select_cpu_fails *skel;
+
+ skel = enq_select_cpu_fails__open_and_load();
+ if (!skel) {
+ SCX_ERR("Failed to open and load skel");
+ return SCX_TEST_FAIL;
+ }
+ *ctx = skel;
+
+ return SCX_TEST_PASS;
+}
+
+static enum scx_test_status run(void *ctx)
+{
+ struct enq_select_cpu_fails *skel = ctx;
+ struct bpf_link *link;
+
+ link = bpf_map__attach_struct_ops(skel->maps.enq_select_cpu_fails_ops);
+ if (!link) {
+ SCX_ERR("Failed to attach scheduler");
+ return SCX_TEST_FAIL;
+ }
+
+ sleep(1);
+
+ bpf_link__destroy(link);
+
+ return SCX_TEST_PASS;
+}
+
+static void cleanup(void *ctx)
+{
+ struct enq_select_cpu_fails *skel = ctx;
+
+ enq_select_cpu_fails__destroy(skel);
+}
+
+struct scx_test enq_select_cpu_fails = {
+ .name = "enq_select_cpu_fails",
+ .description = "Verify we fail to call scx_bpf_select_cpu_dfl() "
+ "from ops.enqueue()",
+ .setup = setup,
+ .run = run,
+ .cleanup = cleanup,
+};
+REGISTER_SCX_TEST(&enq_select_cpu_fails)
diff --git a/tools/testing/selftests/sched_ext/exit.bpf.c b/tools/testing/selftests/sched_ext/exit.bpf.c
new file mode 100644
index 000000000000..ae12ddaac921
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/exit.bpf.c
@@ -0,0 +1,84 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2024 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2024 David Vernet <dvernet@meta.com>
+ */
+
+#include <scx/common.bpf.h>
+
+char _license[] SEC("license") = "GPL";
+
+#include "exit_test.h"
+
+const volatile int exit_point;
+UEI_DEFINE(uei);
+
+#define EXIT_CLEANLY() scx_bpf_exit(exit_point, "%d", exit_point)
+
+s32 BPF_STRUCT_OPS(exit_select_cpu, struct task_struct *p,
+ s32 prev_cpu, u64 wake_flags)
+{
+ bool found;
+
+ if (exit_point == EXIT_SELECT_CPU)
+ EXIT_CLEANLY();
+
+ return scx_bpf_select_cpu_dfl(p, prev_cpu, wake_flags, &found);
+}
+
+void BPF_STRUCT_OPS(exit_enqueue, struct task_struct *p, u64 enq_flags)
+{
+ if (exit_point == EXIT_ENQUEUE)
+ EXIT_CLEANLY();
+
+ scx_bpf_dispatch(p, SCX_DSQ_GLOBAL, SCX_SLICE_DFL, enq_flags);
+}
+
+void BPF_STRUCT_OPS(exit_dispatch, s32 cpu, struct task_struct *p)
+{
+ if (exit_point == EXIT_DISPATCH)
+ EXIT_CLEANLY();
+
+ scx_bpf_consume(SCX_DSQ_GLOBAL);
+}
+
+void BPF_STRUCT_OPS(exit_enable, struct task_struct *p)
+{
+ if (exit_point == EXIT_ENABLE)
+ EXIT_CLEANLY();
+}
+
+s32 BPF_STRUCT_OPS(exit_init_task, struct task_struct *p,
+ struct scx_init_task_args *args)
+{
+ if (exit_point == EXIT_INIT_TASK)
+ EXIT_CLEANLY();
+
+ return 0;
+}
+
+void BPF_STRUCT_OPS(exit_exit, struct scx_exit_info *ei)
+{
+ UEI_RECORD(uei, ei);
+}
+
+s32 BPF_STRUCT_OPS_SLEEPABLE(exit_init)
+{
+ if (exit_point == EXIT_INIT)
+ EXIT_CLEANLY();
+
+ return 0;
+}
+
+SEC(".struct_ops.link")
+struct sched_ext_ops exit_ops = {
+ .select_cpu = exit_select_cpu,
+ .enqueue = exit_enqueue,
+ .dispatch = exit_dispatch,
+ .init_task = exit_init_task,
+ .enable = exit_enable,
+ .exit = exit_exit,
+ .init = exit_init,
+ .name = "exit",
+ .timeout_ms = 1000U,
+};
diff --git a/tools/testing/selftests/sched_ext/exit.c b/tools/testing/selftests/sched_ext/exit.c
new file mode 100644
index 000000000000..31bcd06e21cd
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/exit.c
@@ -0,0 +1,55 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2024 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2024 David Vernet <dvernet@meta.com>
+ */
+#include <bpf/bpf.h>
+#include <sched.h>
+#include <scx/common.h>
+#include <sys/wait.h>
+#include <unistd.h>
+#include "exit.bpf.skel.h"
+#include "scx_test.h"
+
+#include "exit_test.h"
+
+static enum scx_test_status run(void *ctx)
+{
+ enum exit_test_case tc;
+
+ for (tc = 0; tc < NUM_EXITS; tc++) {
+ struct exit *skel;
+ struct bpf_link *link;
+ char buf[16];
+
+ skel = exit__open();
+ skel->rodata->exit_point = tc;
+ exit__load(skel);
+ link = bpf_map__attach_struct_ops(skel->maps.exit_ops);
+ if (!link) {
+ SCX_ERR("Failed to attach scheduler");
+ exit__destroy(skel);
+ return SCX_TEST_FAIL;
+ }
+
+ /* Assumes uei.kind is written last */
+ while (skel->data->uei.kind == EXIT_KIND(SCX_EXIT_NONE))
+ sched_yield();
+
+ SCX_EQ(skel->data->uei.kind, EXIT_KIND(SCX_EXIT_UNREG_BPF));
+ SCX_EQ(skel->data->uei.exit_code, tc);
+ sprintf(buf, "%d", tc);
+ SCX_ASSERT(!strcmp(skel->data->uei.msg, buf));
+ bpf_link__destroy(link);
+ exit__destroy(skel);
+ }
+
+ return SCX_TEST_PASS;
+}
+
+struct scx_test exit_test = {
+ .name = "exit",
+ .description = "Verify we can cleanly exit a scheduler in multiple places",
+ .run = run,
+};
+REGISTER_SCX_TEST(&exit_test)
diff --git a/tools/testing/selftests/sched_ext/exit_test.h b/tools/testing/selftests/sched_ext/exit_test.h
new file mode 100644
index 000000000000..94f0268b9cb8
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/exit_test.h
@@ -0,0 +1,20 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2024 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2024 David Vernet <dvernet@meta.com>
+ */
+
+#ifndef __EXIT_TEST_H__
+#define __EXIT_TEST_H__
+
+enum exit_test_case {
+ EXIT_SELECT_CPU,
+ EXIT_ENQUEUE,
+ EXIT_DISPATCH,
+ EXIT_ENABLE,
+ EXIT_INIT_TASK,
+ EXIT_INIT,
+ NUM_EXITS,
+};
+
+#endif // # __EXIT_TEST_H__
diff --git a/tools/testing/selftests/sched_ext/hotplug.bpf.c b/tools/testing/selftests/sched_ext/hotplug.bpf.c
new file mode 100644
index 000000000000..8f2601db39f3
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/hotplug.bpf.c
@@ -0,0 +1,61 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2024 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2024 David Vernet <dvernet@meta.com>
+ */
+
+#include <scx/common.bpf.h>
+
+char _license[] SEC("license") = "GPL";
+
+#include "hotplug_test.h"
+
+UEI_DEFINE(uei);
+
+void BPF_STRUCT_OPS(hotplug_exit, struct scx_exit_info *ei)
+{
+ UEI_RECORD(uei, ei);
+}
+
+static void exit_from_hotplug(s32 cpu, bool onlining)
+{
+ /*
+ * Ignored, just used to verify that we can invoke blocking kfuncs
+ * from the hotplug path.
+ */
+ scx_bpf_create_dsq(0, -1);
+
+ s64 code = SCX_ECODE_ACT_RESTART | HOTPLUG_EXIT_RSN;
+
+ if (onlining)
+ code |= HOTPLUG_ONLINING;
+
+ scx_bpf_exit(code, "hotplug event detected (%d going %s)", cpu,
+ onlining ? "online" : "offline");
+}
+
+void BPF_STRUCT_OPS_SLEEPABLE(hotplug_cpu_online, s32 cpu)
+{
+ exit_from_hotplug(cpu, true);
+}
+
+void BPF_STRUCT_OPS_SLEEPABLE(hotplug_cpu_offline, s32 cpu)
+{
+ exit_from_hotplug(cpu, false);
+}
+
+SEC(".struct_ops.link")
+struct sched_ext_ops hotplug_cb_ops = {
+ .cpu_online = hotplug_cpu_online,
+ .cpu_offline = hotplug_cpu_offline,
+ .exit = hotplug_exit,
+ .name = "hotplug_cbs",
+ .timeout_ms = 1000U,
+};
+
+SEC(".struct_ops.link")
+struct sched_ext_ops hotplug_nocb_ops = {
+ .exit = hotplug_exit,
+ .name = "hotplug_nocbs",
+ .timeout_ms = 1000U,
+};
diff --git a/tools/testing/selftests/sched_ext/hotplug.c b/tools/testing/selftests/sched_ext/hotplug.c
new file mode 100644
index 000000000000..87bf220b1bce
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/hotplug.c
@@ -0,0 +1,168 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2024 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2024 David Vernet <dvernet@meta.com>
+ */
+#include <bpf/bpf.h>
+#include <sched.h>
+#include <scx/common.h>
+#include <sched.h>
+#include <sys/wait.h>
+#include <unistd.h>
+
+#include "hotplug_test.h"
+#include "hotplug.bpf.skel.h"
+#include "scx_test.h"
+#include "util.h"
+
+const char *online_path = "/sys/devices/system/cpu/cpu1/online";
+
+static bool is_cpu_online(void)
+{
+ return file_read_long(online_path) > 0;
+}
+
+static void toggle_online_status(bool online)
+{
+ long val = online ? 1 : 0;
+ int ret;
+
+ ret = file_write_long(online_path, val);
+ if (ret != 0)
+ fprintf(stderr, "Failed to bring CPU %s (%s)",
+ online ? "online" : "offline", strerror(errno));
+}
+
+static enum scx_test_status setup(void **ctx)
+{
+ if (!is_cpu_online())
+ return SCX_TEST_SKIP;
+
+ return SCX_TEST_PASS;
+}
+
+static enum scx_test_status test_hotplug(bool onlining, bool cbs_defined)
+{
+ struct hotplug *skel;
+ struct bpf_link *link;
+ long kind, code;
+
+ SCX_ASSERT(is_cpu_online());
+
+ skel = hotplug__open_and_load();
+ SCX_ASSERT(skel);
+
+ /* Testing the offline -> online path, so go offline before starting */
+ if (onlining)
+ toggle_online_status(0);
+
+ if (cbs_defined) {
+ kind = SCX_KIND_VAL(SCX_EXIT_UNREG_BPF);
+ code = SCX_ECODE_VAL(SCX_ECODE_ACT_RESTART) | HOTPLUG_EXIT_RSN;
+ if (onlining)
+ code |= HOTPLUG_ONLINING;
+ } else {
+ kind = SCX_KIND_VAL(SCX_EXIT_UNREG_KERN);
+ code = SCX_ECODE_VAL(SCX_ECODE_ACT_RESTART) |
+ SCX_ECODE_VAL(SCX_ECODE_RSN_HOTPLUG);
+ }
+
+ if (cbs_defined)
+ link = bpf_map__attach_struct_ops(skel->maps.hotplug_cb_ops);
+ else
+ link = bpf_map__attach_struct_ops(skel->maps.hotplug_nocb_ops);
+
+ if (!link) {
+ SCX_ERR("Failed to attach scheduler");
+ hotplug__destroy(skel);
+ return SCX_TEST_FAIL;
+ }
+
+ toggle_online_status(onlining ? 1 : 0);
+
+ while (!UEI_EXITED(skel, uei))
+ sched_yield();
+
+ SCX_EQ(skel->data->uei.kind, kind);
+ SCX_EQ(UEI_REPORT(skel, uei), code);
+
+ if (!onlining)
+ toggle_online_status(1);
+
+ bpf_link__destroy(link);
+ hotplug__destroy(skel);
+
+ return SCX_TEST_PASS;
+}
+
+static enum scx_test_status test_hotplug_attach(void)
+{
+ struct hotplug *skel;
+ struct bpf_link *link;
+ enum scx_test_status status = SCX_TEST_PASS;
+ long kind, code;
+
+ SCX_ASSERT(is_cpu_online());
+ SCX_ASSERT(scx_hotplug_seq() > 0);
+
+ skel = SCX_OPS_OPEN(hotplug_nocb_ops, hotplug);
+ SCX_ASSERT(skel);
+
+ SCX_OPS_LOAD(skel, hotplug_nocb_ops, hotplug, uei);
+
+ /*
+ * Take the CPU offline to increment the global hotplug seq, which
+ * should cause attach to fail due to us setting the hotplug seq above
+ */
+ toggle_online_status(0);
+ link = bpf_map__attach_struct_ops(skel->maps.hotplug_nocb_ops);
+
+ toggle_online_status(1);
+
+ SCX_ASSERT(link);
+ while (!UEI_EXITED(skel, uei))
+ sched_yield();
+
+ kind = SCX_KIND_VAL(SCX_EXIT_UNREG_KERN);
+ code = SCX_ECODE_VAL(SCX_ECODE_ACT_RESTART) |
+ SCX_ECODE_VAL(SCX_ECODE_RSN_HOTPLUG);
+ SCX_EQ(skel->data->uei.kind, kind);
+ SCX_EQ(UEI_REPORT(skel, uei), code);
+
+ bpf_link__destroy(link);
+ hotplug__destroy(skel);
+
+ return status;
+}
+
+static enum scx_test_status run(void *ctx)
+{
+
+#define HP_TEST(__onlining, __cbs_defined) ({ \
+ if (test_hotplug(__onlining, __cbs_defined) != SCX_TEST_PASS) \
+ return SCX_TEST_FAIL; \
+})
+
+ HP_TEST(true, true);
+ HP_TEST(false, true);
+ HP_TEST(true, false);
+ HP_TEST(false, false);
+
+#undef HP_TEST
+
+ return test_hotplug_attach();
+}
+
+static void cleanup(void *ctx)
+{
+ toggle_online_status(1);
+}
+
+struct scx_test hotplug_test = {
+ .name = "hotplug",
+ .description = "Verify hotplug behavior",
+ .setup = setup,
+ .run = run,
+ .cleanup = cleanup,
+};
+REGISTER_SCX_TEST(&hotplug_test)
diff --git a/tools/testing/selftests/sched_ext/hotplug_test.h b/tools/testing/selftests/sched_ext/hotplug_test.h
new file mode 100644
index 000000000000..73d236f90787
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/hotplug_test.h
@@ -0,0 +1,15 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2024 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2024 David Vernet <dvernet@meta.com>
+ */
+
+#ifndef __HOTPLUG_TEST_H__
+#define __HOTPLUG_TEST_H__
+
+enum hotplug_test_flags {
+ HOTPLUG_EXIT_RSN = 1LLU << 0,
+ HOTPLUG_ONLINING = 1LLU << 1,
+};
+
+#endif // # __HOTPLUG_TEST_H__
diff --git a/tools/testing/selftests/sched_ext/init_enable_count.bpf.c b/tools/testing/selftests/sched_ext/init_enable_count.bpf.c
new file mode 100644
index 000000000000..47ea89a626c3
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/init_enable_count.bpf.c
@@ -0,0 +1,53 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * A scheduler that verifies that we do proper counting of init, enable, etc
+ * callbacks.
+ *
+ * Copyright (c) 2023 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2023 David Vernet <dvernet@meta.com>
+ * Copyright (c) 2023 Tejun Heo <tj@kernel.org>
+ */
+
+#include <scx/common.bpf.h>
+
+char _license[] SEC("license") = "GPL";
+
+u64 init_task_cnt, exit_task_cnt, enable_cnt, disable_cnt;
+u64 init_fork_cnt, init_transition_cnt;
+
+s32 BPF_STRUCT_OPS_SLEEPABLE(cnt_init_task, struct task_struct *p,
+ struct scx_init_task_args *args)
+{
+ __sync_fetch_and_add(&init_task_cnt, 1);
+
+ if (args->fork)
+ __sync_fetch_and_add(&init_fork_cnt, 1);
+ else
+ __sync_fetch_and_add(&init_transition_cnt, 1);
+
+ return 0;
+}
+
+void BPF_STRUCT_OPS(cnt_exit_task, struct task_struct *p)
+{
+ __sync_fetch_and_add(&exit_task_cnt, 1);
+}
+
+void BPF_STRUCT_OPS(cnt_enable, struct task_struct *p)
+{
+ __sync_fetch_and_add(&enable_cnt, 1);
+}
+
+void BPF_STRUCT_OPS(cnt_disable, struct task_struct *p)
+{
+ __sync_fetch_and_add(&disable_cnt, 1);
+}
+
+SEC(".struct_ops.link")
+struct sched_ext_ops init_enable_count_ops = {
+ .init_task = cnt_init_task,
+ .exit_task = cnt_exit_task,
+ .enable = cnt_enable,
+ .disable = cnt_disable,
+ .name = "init_enable_count",
+};
diff --git a/tools/testing/selftests/sched_ext/init_enable_count.c b/tools/testing/selftests/sched_ext/init_enable_count.c
new file mode 100644
index 000000000000..97d45f1e5597
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/init_enable_count.c
@@ -0,0 +1,166 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2023 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2023 David Vernet <dvernet@meta.com>
+ * Copyright (c) 2023 Tejun Heo <tj@kernel.org>
+ */
+#include <stdio.h>
+#include <unistd.h>
+#include <sched.h>
+#include <bpf/bpf.h>
+#include <scx/common.h>
+#include <sys/wait.h>
+#include "scx_test.h"
+#include "init_enable_count.bpf.skel.h"
+
+#define SCHED_EXT 7
+
+static struct init_enable_count *
+open_load_prog(bool global)
+{
+ struct init_enable_count *skel;
+
+ skel = init_enable_count__open();
+ SCX_BUG_ON(!skel, "Failed to open skel");
+
+ if (!global)
+ skel->struct_ops.init_enable_count_ops->flags |= SCX_OPS_SWITCH_PARTIAL;
+
+ SCX_BUG_ON(init_enable_count__load(skel), "Failed to load skel");
+
+ return skel;
+}
+
+static enum scx_test_status run_test(bool global)
+{
+ struct init_enable_count *skel;
+ struct bpf_link *link;
+ const u32 num_children = 5, num_pre_forks = 1024;
+ int ret, i, status;
+ struct sched_param param = {};
+ pid_t pids[num_pre_forks];
+
+ skel = open_load_prog(global);
+
+ /*
+ * Fork a bunch of children before we attach the scheduler so that we
+ * ensure (at least in practical terms) that there are more tasks that
+ * transition from SCHED_OTHER -> SCHED_EXT than there are tasks that
+ * take the fork() path either below or in other processes.
+ */
+ for (i = 0; i < num_pre_forks; i++) {
+ pids[i] = fork();
+ SCX_FAIL_IF(pids[i] < 0, "Failed to fork child");
+ if (pids[i] == 0) {
+ sleep(1);
+ exit(0);
+ }
+ }
+
+ link = bpf_map__attach_struct_ops(skel->maps.init_enable_count_ops);
+ SCX_FAIL_IF(!link, "Failed to attach struct_ops");
+
+ for (i = 0; i < num_pre_forks; i++) {
+ SCX_FAIL_IF(waitpid(pids[i], &status, 0) != pids[i],
+ "Failed to wait for pre-forked child\n");
+
+ SCX_FAIL_IF(status != 0, "Pre-forked child %d exited with status %d\n", i,
+ status);
+ }
+
+ bpf_link__destroy(link);
+ SCX_GE(skel->bss->init_task_cnt, num_pre_forks);
+ SCX_GE(skel->bss->exit_task_cnt, num_pre_forks);
+
+ link = bpf_map__attach_struct_ops(skel->maps.init_enable_count_ops);
+ SCX_FAIL_IF(!link, "Failed to attach struct_ops");
+
+ /* SCHED_EXT children */
+ for (i = 0; i < num_children; i++) {
+ pids[i] = fork();
+ SCX_FAIL_IF(pids[i] < 0, "Failed to fork child");
+
+ if (pids[i] == 0) {
+ ret = sched_setscheduler(0, SCHED_EXT, ¶m);
+ SCX_BUG_ON(ret, "Failed to set sched to sched_ext");
+
+ /*
+ * Reset to SCHED_OTHER for half of them. Counts for
+ * everything should still be the same regardless, as
+ * ops.disable() is invoked even if a task is still on
+ * SCHED_EXT before it exits.
+ */
+ if (i % 2 == 0) {
+ ret = sched_setscheduler(0, SCHED_OTHER, ¶m);
+ SCX_BUG_ON(ret, "Failed to reset sched to normal");
+ }
+ exit(0);
+ }
+ }
+ for (i = 0; i < num_children; i++) {
+ SCX_FAIL_IF(waitpid(pids[i], &status, 0) != pids[i],
+ "Failed to wait for SCX child\n");
+
+ SCX_FAIL_IF(status != 0, "SCX child %d exited with status %d\n", i,
+ status);
+ }
+
+ /* SCHED_OTHER children */
+ for (i = 0; i < num_children; i++) {
+ pids[i] = fork();
+ if (pids[i] == 0)
+ exit(0);
+ }
+
+ for (i = 0; i < num_children; i++) {
+ SCX_FAIL_IF(waitpid(pids[i], &status, 0) != pids[i],
+ "Failed to wait for normal child\n");
+
+ SCX_FAIL_IF(status != 0, "Normal child %d exited with status %d\n", i,
+ status);
+ }
+
+ bpf_link__destroy(link);
+
+ SCX_GE(skel->bss->init_task_cnt, 2 * num_children);
+ SCX_GE(skel->bss->exit_task_cnt, 2 * num_children);
+
+ if (global) {
+ SCX_GE(skel->bss->enable_cnt, 2 * num_children);
+ SCX_GE(skel->bss->disable_cnt, 2 * num_children);
+ } else {
+ SCX_EQ(skel->bss->enable_cnt, num_children);
+ SCX_EQ(skel->bss->disable_cnt, num_children);
+ }
+ /*
+ * We forked a ton of tasks before we attached the scheduler above, so
+ * this should be fine. Technically it could be flaky if a ton of forks
+ * are happening at the same time in other processes, but that should
+ * be exceedingly unlikely.
+ */
+ SCX_GT(skel->bss->init_transition_cnt, skel->bss->init_fork_cnt);
+ SCX_GE(skel->bss->init_fork_cnt, 2 * num_children);
+
+ init_enable_count__destroy(skel);
+
+ return SCX_TEST_PASS;
+}
+
+static enum scx_test_status run(void *ctx)
+{
+ enum scx_test_status status;
+
+ status = run_test(true);
+ if (status != SCX_TEST_PASS)
+ return status;
+
+ return run_test(false);
+}
+
+struct scx_test init_enable_count = {
+ .name = "init_enable_count",
+ .description = "Verify we do the correct amount of counting of init, "
+ "enable, etc callbacks.",
+ .run = run,
+};
+REGISTER_SCX_TEST(&init_enable_count)
diff --git a/tools/testing/selftests/sched_ext/maximal.bpf.c b/tools/testing/selftests/sched_ext/maximal.bpf.c
new file mode 100644
index 000000000000..44612fdaf399
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/maximal.bpf.c
@@ -0,0 +1,132 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * A scheduler with every callback defined.
+ *
+ * This scheduler defines every callback.
+ *
+ * Copyright (c) 2024 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2024 David Vernet <dvernet@meta.com>
+ */
+
+#include <scx/common.bpf.h>
+
+char _license[] SEC("license") = "GPL";
+
+s32 BPF_STRUCT_OPS(maximal_select_cpu, struct task_struct *p, s32 prev_cpu,
+ u64 wake_flags)
+{
+ return prev_cpu;
+}
+
+void BPF_STRUCT_OPS(maximal_enqueue, struct task_struct *p, u64 enq_flags)
+{
+ scx_bpf_dispatch(p, SCX_DSQ_GLOBAL, SCX_SLICE_DFL, enq_flags);
+}
+
+void BPF_STRUCT_OPS(maximal_dequeue, struct task_struct *p, u64 deq_flags)
+{}
+
+void BPF_STRUCT_OPS(maximal_dispatch, s32 cpu, struct task_struct *prev)
+{
+ scx_bpf_consume(SCX_DSQ_GLOBAL);
+}
+
+void BPF_STRUCT_OPS(maximal_runnable, struct task_struct *p, u64 enq_flags)
+{}
+
+void BPF_STRUCT_OPS(maximal_running, struct task_struct *p)
+{}
+
+void BPF_STRUCT_OPS(maximal_stopping, struct task_struct *p, bool runnable)
+{}
+
+void BPF_STRUCT_OPS(maximal_quiescent, struct task_struct *p, u64 deq_flags)
+{}
+
+bool BPF_STRUCT_OPS(maximal_yield, struct task_struct *from,
+ struct task_struct *to)
+{
+ return false;
+}
+
+bool BPF_STRUCT_OPS(maximal_core_sched_before, struct task_struct *a,
+ struct task_struct *b)
+{
+ return false;
+}
+
+void BPF_STRUCT_OPS(maximal_set_weight, struct task_struct *p, u32 weight)
+{}
+
+void BPF_STRUCT_OPS(maximal_set_cpumask, struct task_struct *p,
+ const struct cpumask *cpumask)
+{}
+
+void BPF_STRUCT_OPS(maximal_update_idle, s32 cpu, bool idle)
+{}
+
+void BPF_STRUCT_OPS(maximal_cpu_acquire, s32 cpu,
+ struct scx_cpu_acquire_args *args)
+{}
+
+void BPF_STRUCT_OPS(maximal_cpu_release, s32 cpu,
+ struct scx_cpu_release_args *args)
+{}
+
+void BPF_STRUCT_OPS(maximal_cpu_online, s32 cpu)
+{}
+
+void BPF_STRUCT_OPS(maximal_cpu_offline, s32 cpu)
+{}
+
+s32 BPF_STRUCT_OPS(maximal_init_task, struct task_struct *p,
+ struct scx_init_task_args *args)
+{
+ return 0;
+}
+
+void BPF_STRUCT_OPS(maximal_enable, struct task_struct *p)
+{}
+
+void BPF_STRUCT_OPS(maximal_exit_task, struct task_struct *p,
+ struct scx_exit_task_args *args)
+{}
+
+void BPF_STRUCT_OPS(maximal_disable, struct task_struct *p)
+{}
+
+s32 BPF_STRUCT_OPS_SLEEPABLE(maximal_init)
+{
+ return 0;
+}
+
+void BPF_STRUCT_OPS(maximal_exit, struct scx_exit_info *info)
+{}
+
+SEC(".struct_ops.link")
+struct sched_ext_ops maximal_ops = {
+ .select_cpu = maximal_select_cpu,
+ .enqueue = maximal_enqueue,
+ .dequeue = maximal_dequeue,
+ .dispatch = maximal_dispatch,
+ .runnable = maximal_runnable,
+ .running = maximal_running,
+ .stopping = maximal_stopping,
+ .quiescent = maximal_quiescent,
+ .yield = maximal_yield,
+ .core_sched_before = maximal_core_sched_before,
+ .set_weight = maximal_set_weight,
+ .set_cpumask = maximal_set_cpumask,
+ .update_idle = maximal_update_idle,
+ .cpu_acquire = maximal_cpu_acquire,
+ .cpu_release = maximal_cpu_release,
+ .cpu_online = maximal_cpu_online,
+ .cpu_offline = maximal_cpu_offline,
+ .init_task = maximal_init_task,
+ .enable = maximal_enable,
+ .exit_task = maximal_exit_task,
+ .disable = maximal_disable,
+ .init = maximal_init,
+ .exit = maximal_exit,
+ .name = "maximal",
+};
diff --git a/tools/testing/selftests/sched_ext/maximal.c b/tools/testing/selftests/sched_ext/maximal.c
new file mode 100644
index 000000000000..f38fc973c380
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/maximal.c
@@ -0,0 +1,51 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2024 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2024 David Vernet <dvernet@meta.com>
+ */
+#include <bpf/bpf.h>
+#include <scx/common.h>
+#include <sys/wait.h>
+#include <unistd.h>
+#include "maximal.bpf.skel.h"
+#include "scx_test.h"
+
+static enum scx_test_status setup(void **ctx)
+{
+ struct maximal *skel;
+
+ skel = maximal__open_and_load();
+ SCX_FAIL_IF(!skel, "Failed to open and load skel");
+ *ctx = skel;
+
+ return SCX_TEST_PASS;
+}
+
+static enum scx_test_status run(void *ctx)
+{
+ struct maximal *skel = ctx;
+ struct bpf_link *link;
+
+ link = bpf_map__attach_struct_ops(skel->maps.maximal_ops);
+ SCX_FAIL_IF(!link, "Failed to attach scheduler");
+
+ bpf_link__destroy(link);
+
+ return SCX_TEST_PASS;
+}
+
+static void cleanup(void *ctx)
+{
+ struct maximal *skel = ctx;
+
+ maximal__destroy(skel);
+}
+
+struct scx_test maximal = {
+ .name = "maximal",
+ .description = "Verify we can load a scheduler with every callback defined",
+ .setup = setup,
+ .run = run,
+ .cleanup = cleanup,
+};
+REGISTER_SCX_TEST(&maximal)
diff --git a/tools/testing/selftests/sched_ext/maybe_null.bpf.c b/tools/testing/selftests/sched_ext/maybe_null.bpf.c
new file mode 100644
index 000000000000..27d0f386acfb
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/maybe_null.bpf.c
@@ -0,0 +1,36 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2024 Meta Platforms, Inc. and affiliates.
+ */
+
+#include <scx/common.bpf.h>
+
+char _license[] SEC("license") = "GPL";
+
+u64 vtime_test;
+
+void BPF_STRUCT_OPS(maybe_null_running, struct task_struct *p)
+{}
+
+void BPF_STRUCT_OPS(maybe_null_success_dispatch, s32 cpu, struct task_struct *p)
+{
+ if (p != NULL)
+ vtime_test = p->scx.dsq_vtime;
+}
+
+bool BPF_STRUCT_OPS(maybe_null_success_yield, struct task_struct *from,
+ struct task_struct *to)
+{
+ if (to)
+ bpf_printk("Yielding to %s[%d]", to->comm, to->pid);
+
+ return false;
+}
+
+SEC(".struct_ops.link")
+struct sched_ext_ops maybe_null_success = {
+ .dispatch = maybe_null_success_dispatch,
+ .yield = maybe_null_success_yield,
+ .enable = maybe_null_running,
+ .name = "minimal",
+};
diff --git a/tools/testing/selftests/sched_ext/maybe_null.c b/tools/testing/selftests/sched_ext/maybe_null.c
new file mode 100644
index 000000000000..31cfafb0cf65
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/maybe_null.c
@@ -0,0 +1,49 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2024 Meta Platforms, Inc. and affiliates.
+ */
+#include <bpf/bpf.h>
+#include <scx/common.h>
+#include <sys/wait.h>
+#include <unistd.h>
+#include "maybe_null.bpf.skel.h"
+#include "maybe_null_fail_dsp.bpf.skel.h"
+#include "maybe_null_fail_yld.bpf.skel.h"
+#include "scx_test.h"
+
+static enum scx_test_status run(void *ctx)
+{
+ struct maybe_null *skel;
+ struct maybe_null_fail_dsp *fail_dsp;
+ struct maybe_null_fail_yld *fail_yld;
+
+ skel = maybe_null__open_and_load();
+ if (!skel) {
+ SCX_ERR("Failed to open and load maybe_null skel");
+ return SCX_TEST_FAIL;
+ }
+ maybe_null__destroy(skel);
+
+ fail_dsp = maybe_null_fail_dsp__open_and_load();
+ if (fail_dsp) {
+ maybe_null_fail_dsp__destroy(fail_dsp);
+ SCX_ERR("Should failed to open and load maybe_null_fail_dsp skel");
+ return SCX_TEST_FAIL;
+ }
+
+ fail_yld = maybe_null_fail_yld__open_and_load();
+ if (fail_yld) {
+ maybe_null_fail_yld__destroy(fail_yld);
+ SCX_ERR("Should failed to open and load maybe_null_fail_yld skel");
+ return SCX_TEST_FAIL;
+ }
+
+ return SCX_TEST_PASS;
+}
+
+struct scx_test maybe_null = {
+ .name = "maybe_null",
+ .description = "Verify if PTR_MAYBE_NULL work for .dispatch",
+ .run = run,
+};
+REGISTER_SCX_TEST(&maybe_null)
diff --git a/tools/testing/selftests/sched_ext/maybe_null_fail_dsp.bpf.c b/tools/testing/selftests/sched_ext/maybe_null_fail_dsp.bpf.c
new file mode 100644
index 000000000000..c0641050271d
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/maybe_null_fail_dsp.bpf.c
@@ -0,0 +1,25 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2024 Meta Platforms, Inc. and affiliates.
+ */
+
+#include <scx/common.bpf.h>
+
+char _license[] SEC("license") = "GPL";
+
+u64 vtime_test;
+
+void BPF_STRUCT_OPS(maybe_null_running, struct task_struct *p)
+{}
+
+void BPF_STRUCT_OPS(maybe_null_fail_dispatch, s32 cpu, struct task_struct *p)
+{
+ vtime_test = p->scx.dsq_vtime;
+}
+
+SEC(".struct_ops.link")
+struct sched_ext_ops maybe_null_fail = {
+ .dispatch = maybe_null_fail_dispatch,
+ .enable = maybe_null_running,
+ .name = "maybe_null_fail_dispatch",
+};
diff --git a/tools/testing/selftests/sched_ext/maybe_null_fail_yld.bpf.c b/tools/testing/selftests/sched_ext/maybe_null_fail_yld.bpf.c
new file mode 100644
index 000000000000..3c1740028e3b
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/maybe_null_fail_yld.bpf.c
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2024 Meta Platforms, Inc. and affiliates.
+ */
+
+#include <scx/common.bpf.h>
+
+char _license[] SEC("license") = "GPL";
+
+u64 vtime_test;
+
+void BPF_STRUCT_OPS(maybe_null_running, struct task_struct *p)
+{}
+
+bool BPF_STRUCT_OPS(maybe_null_fail_yield, struct task_struct *from,
+ struct task_struct *to)
+{
+ bpf_printk("Yielding to %s[%d]", to->comm, to->pid);
+
+ return false;
+}
+
+SEC(".struct_ops.link")
+struct sched_ext_ops maybe_null_fail = {
+ .yield = maybe_null_fail_yield,
+ .enable = maybe_null_running,
+ .name = "maybe_null_fail_yield",
+};
diff --git a/tools/testing/selftests/sched_ext/minimal.bpf.c b/tools/testing/selftests/sched_ext/minimal.bpf.c
new file mode 100644
index 000000000000..6a7eccef0104
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/minimal.bpf.c
@@ -0,0 +1,21 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * A completely minimal scheduler.
+ *
+ * This scheduler defines the absolute minimal set of struct sched_ext_ops
+ * fields: its name. It should _not_ fail to be loaded, and can be used to
+ * exercise the default scheduling paths in ext.c.
+ *
+ * Copyright (c) 2023 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2023 David Vernet <dvernet@meta.com>
+ * Copyright (c) 2023 Tejun Heo <tj@kernel.org>
+ */
+
+#include <scx/common.bpf.h>
+
+char _license[] SEC("license") = "GPL";
+
+SEC(".struct_ops.link")
+struct sched_ext_ops minimal_ops = {
+ .name = "minimal",
+};
diff --git a/tools/testing/selftests/sched_ext/minimal.c b/tools/testing/selftests/sched_ext/minimal.c
new file mode 100644
index 000000000000..6c5db8ebbf8a
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/minimal.c
@@ -0,0 +1,58 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2023 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2023 David Vernet <dvernet@meta.com>
+ * Copyright (c) 2023 Tejun Heo <tj@kernel.org>
+ */
+#include <bpf/bpf.h>
+#include <scx/common.h>
+#include <sys/wait.h>
+#include <unistd.h>
+#include "minimal.bpf.skel.h"
+#include "scx_test.h"
+
+static enum scx_test_status setup(void **ctx)
+{
+ struct minimal *skel;
+
+ skel = minimal__open_and_load();
+ if (!skel) {
+ SCX_ERR("Failed to open and load skel");
+ return SCX_TEST_FAIL;
+ }
+ *ctx = skel;
+
+ return SCX_TEST_PASS;
+}
+
+static enum scx_test_status run(void *ctx)
+{
+ struct minimal *skel = ctx;
+ struct bpf_link *link;
+
+ link = bpf_map__attach_struct_ops(skel->maps.minimal_ops);
+ if (!link) {
+ SCX_ERR("Failed to attach scheduler");
+ return SCX_TEST_FAIL;
+ }
+
+ bpf_link__destroy(link);
+
+ return SCX_TEST_PASS;
+}
+
+static void cleanup(void *ctx)
+{
+ struct minimal *skel = ctx;
+
+ minimal__destroy(skel);
+}
+
+struct scx_test minimal = {
+ .name = "minimal",
+ .description = "Verify we can load a fully minimal scheduler",
+ .setup = setup,
+ .run = run,
+ .cleanup = cleanup,
+};
+REGISTER_SCX_TEST(&minimal)
diff --git a/tools/testing/selftests/sched_ext/prog_run.bpf.c b/tools/testing/selftests/sched_ext/prog_run.bpf.c
new file mode 100644
index 000000000000..fd2c8f12af16
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/prog_run.bpf.c
@@ -0,0 +1,32 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * A scheduler that validates that we can invoke sched_ext kfuncs in
+ * BPF_PROG_TYPE_SYSCALL programs.
+ *
+ * Copyright (c) 2024 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2024 David Vernet <dvernet@meta.com>
+ */
+
+#include <scx/common.bpf.h>
+
+UEI_DEFINE(uei);
+
+char _license[] SEC("license") = "GPL";
+
+SEC("syscall")
+int BPF_PROG(prog_run_syscall)
+{
+ scx_bpf_exit(0xdeadbeef, "Exited from PROG_RUN");
+ return 0;
+}
+
+void BPF_STRUCT_OPS(prog_run_exit, struct scx_exit_info *ei)
+{
+ UEI_RECORD(uei, ei);
+}
+
+SEC(".struct_ops.link")
+struct sched_ext_ops prog_run_ops = {
+ .exit = prog_run_exit,
+ .name = "prog_run",
+};
diff --git a/tools/testing/selftests/sched_ext/prog_run.c b/tools/testing/selftests/sched_ext/prog_run.c
new file mode 100644
index 000000000000..3cd57ef8daaa
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/prog_run.c
@@ -0,0 +1,78 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2024 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2024 David Vernet <dvernet@meta.com>
+ */
+#include <bpf/bpf.h>
+#include <sched.h>
+#include <scx/common.h>
+#include <sys/wait.h>
+#include <unistd.h>
+#include "prog_run.bpf.skel.h"
+#include "scx_test.h"
+
+static enum scx_test_status setup(void **ctx)
+{
+ struct prog_run *skel;
+
+ skel = prog_run__open_and_load();
+ if (!skel) {
+ SCX_ERR("Failed to open and load skel");
+ return SCX_TEST_FAIL;
+ }
+ *ctx = skel;
+
+ return SCX_TEST_PASS;
+}
+
+static enum scx_test_status run(void *ctx)
+{
+ struct prog_run *skel = ctx;
+ struct bpf_link *link;
+ int prog_fd, err = 0;
+
+ prog_fd = bpf_program__fd(skel->progs.prog_run_syscall);
+ if (prog_fd < 0) {
+ SCX_ERR("Failed to get BPF_PROG_RUN prog");
+ return SCX_TEST_FAIL;
+ }
+
+ LIBBPF_OPTS(bpf_test_run_opts, topts);
+
+ link = bpf_map__attach_struct_ops(skel->maps.prog_run_ops);
+ if (!link) {
+ SCX_ERR("Failed to attach scheduler");
+ close(prog_fd);
+ return SCX_TEST_FAIL;
+ }
+
+ err = bpf_prog_test_run_opts(prog_fd, &topts);
+ SCX_EQ(err, 0);
+
+ /* Assumes uei.kind is written last */
+ while (skel->data->uei.kind == EXIT_KIND(SCX_EXIT_NONE))
+ sched_yield();
+
+ SCX_EQ(skel->data->uei.kind, EXIT_KIND(SCX_EXIT_UNREG_BPF));
+ SCX_EQ(skel->data->uei.exit_code, 0xdeadbeef);
+ close(prog_fd);
+ bpf_link__destroy(link);
+
+ return SCX_TEST_PASS;
+}
+
+static void cleanup(void *ctx)
+{
+ struct prog_run *skel = ctx;
+
+ prog_run__destroy(skel);
+}
+
+struct scx_test prog_run = {
+ .name = "prog_run",
+ .description = "Verify we can call into a scheduler with BPF_PROG_RUN, and invoke kfuncs",
+ .setup = setup,
+ .run = run,
+ .cleanup = cleanup,
+};
+REGISTER_SCX_TEST(&prog_run)
diff --git a/tools/testing/selftests/sched_ext/reload_loop.c b/tools/testing/selftests/sched_ext/reload_loop.c
new file mode 100644
index 000000000000..5cfba2d6e056
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/reload_loop.c
@@ -0,0 +1,75 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2024 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2024 David Vernet <dvernet@meta.com>
+ */
+#include <bpf/bpf.h>
+#include <pthread.h>
+#include <scx/common.h>
+#include <sys/wait.h>
+#include <unistd.h>
+#include "maximal.bpf.skel.h"
+#include "scx_test.h"
+
+static struct maximal *skel;
+static pthread_t threads[2];
+
+bool force_exit = false;
+
+static enum scx_test_status setup(void **ctx)
+{
+ skel = maximal__open_and_load();
+ if (!skel) {
+ SCX_ERR("Failed to open and load skel");
+ return SCX_TEST_FAIL;
+ }
+
+ return SCX_TEST_PASS;
+}
+
+static void *do_reload_loop(void *arg)
+{
+ u32 i;
+
+ for (i = 0; i < 1024 && !force_exit; i++) {
+ struct bpf_link *link;
+
+ link = bpf_map__attach_struct_ops(skel->maps.maximal_ops);
+ if (link)
+ bpf_link__destroy(link);
+ }
+
+ return NULL;
+}
+
+static enum scx_test_status run(void *ctx)
+{
+ int err;
+ void *ret;
+
+ err = pthread_create(&threads[0], NULL, do_reload_loop, NULL);
+ SCX_FAIL_IF(err, "Failed to create thread 0");
+
+ err = pthread_create(&threads[1], NULL, do_reload_loop, NULL);
+ SCX_FAIL_IF(err, "Failed to create thread 1");
+
+ SCX_FAIL_IF(pthread_join(threads[0], &ret), "thread 0 failed");
+ SCX_FAIL_IF(pthread_join(threads[1], &ret), "thread 1 failed");
+
+ return SCX_TEST_PASS;
+}
+
+static void cleanup(void *ctx)
+{
+ force_exit = true;
+ maximal__destroy(skel);
+}
+
+struct scx_test reload_loop = {
+ .name = "reload_loop",
+ .description = "Stress test loading and unloading schedulers repeatedly in a tight loop",
+ .setup = setup,
+ .run = run,
+ .cleanup = cleanup,
+};
+REGISTER_SCX_TEST(&reload_loop)
diff --git a/tools/testing/selftests/sched_ext/runner.c b/tools/testing/selftests/sched_ext/runner.c
new file mode 100644
index 000000000000..eab48c7ff309
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/runner.c
@@ -0,0 +1,201 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2024 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2024 David Vernet <dvernet@meta.com>
+ * Copyright (c) 2024 Tejun Heo <tj@kernel.org>
+ */
+#include <stdio.h>
+#include <unistd.h>
+#include <signal.h>
+#include <libgen.h>
+#include <bpf/bpf.h>
+#include "scx_test.h"
+
+const char help_fmt[] =
+"The runner for sched_ext tests.\n"
+"\n"
+"The runner is statically linked against all testcases, and runs them all serially.\n"
+"It's required for the testcases to be serial, as only a single host-wide sched_ext\n"
+"scheduler may be loaded at any given time."
+"\n"
+"Usage: %s [-t TEST] [-h]\n"
+"\n"
+" -t TEST Only run tests whose name includes this string\n"
+" -s Include print output for skipped tests\n"
+" -q Don't print the test descriptions during run\n"
+" -h Display this help and exit\n";
+
+static volatile int exit_req;
+static bool quiet, print_skipped;
+
+#define MAX_SCX_TESTS 2048
+
+static struct scx_test __scx_tests[MAX_SCX_TESTS];
+static unsigned __scx_num_tests = 0;
+
+static void sigint_handler(int simple)
+{
+ exit_req = 1;
+}
+
+static void print_test_preamble(const struct scx_test *test, bool quiet)
+{
+ printf("===== START =====\n");
+ printf("TEST: %s\n", test->name);
+ if (!quiet)
+ printf("DESCRIPTION: %s\n", test->description);
+ printf("OUTPUT:\n");
+}
+
+static const char *status_to_result(enum scx_test_status status)
+{
+ switch (status) {
+ case SCX_TEST_PASS:
+ case SCX_TEST_SKIP:
+ return "ok";
+ case SCX_TEST_FAIL:
+ return "not ok";
+ default:
+ return "<UNKNOWN>";
+ }
+}
+
+static void print_test_result(const struct scx_test *test,
+ enum scx_test_status status,
+ unsigned int testnum)
+{
+ const char *result = status_to_result(status);
+ const char *directive = status == SCX_TEST_SKIP ? "SKIP " : "";
+
+ printf("%s %u %s # %s\n", result, testnum, test->name, directive);
+ printf("===== END =====\n");
+}
+
+static bool should_skip_test(const struct scx_test *test, const char * filter)
+{
+ return !strstr(test->name, filter);
+}
+
+static enum scx_test_status run_test(const struct scx_test *test)
+{
+ enum scx_test_status status;
+ void *context = NULL;
+
+ if (test->setup) {
+ status = test->setup(&context);
+ if (status != SCX_TEST_PASS)
+ return status;
+ }
+
+ status = test->run(context);
+
+ if (test->cleanup)
+ test->cleanup(context);
+
+ return status;
+}
+
+static bool test_valid(const struct scx_test *test)
+{
+ if (!test) {
+ fprintf(stderr, "NULL test detected\n");
+ return false;
+ }
+
+ if (!test->name) {
+ fprintf(stderr,
+ "Test with no name found. Must specify test name.\n");
+ return false;
+ }
+
+ if (!test->description) {
+ fprintf(stderr, "Test %s requires description.\n", test->name);
+ return false;
+ }
+
+ if (!test->run) {
+ fprintf(stderr, "Test %s has no run() callback\n", test->name);
+ return false;
+ }
+
+ return true;
+}
+
+int main(int argc, char **argv)
+{
+ const char *filter = NULL;
+ unsigned testnum = 0, i;
+ unsigned passed = 0, skipped = 0, failed = 0;
+ int opt;
+
+ signal(SIGINT, sigint_handler);
+ signal(SIGTERM, sigint_handler);
+
+ libbpf_set_strict_mode(LIBBPF_STRICT_ALL);
+
+ while ((opt = getopt(argc, argv, "qst:h")) != -1) {
+ switch (opt) {
+ case 'q':
+ quiet = true;
+ break;
+ case 's':
+ print_skipped = true;
+ break;
+ case 't':
+ filter = optarg;
+ break;
+ default:
+ fprintf(stderr, help_fmt, basename(argv[0]));
+ return opt != 'h';
+ }
+ }
+
+ for (i = 0; i < __scx_num_tests; i++) {
+ enum scx_test_status status;
+ struct scx_test *test = &__scx_tests[i];
+
+ if (filter && should_skip_test(test, filter)) {
+ /*
+ * Printing the skipped tests and their preambles can
+ * add a lot of noise to the runner output. Printing
+ * this is only really useful for CI, so let's skip it
+ * by default.
+ */
+ if (print_skipped) {
+ print_test_preamble(test, quiet);
+ print_test_result(test, SCX_TEST_SKIP, ++testnum);
+ }
+ continue;
+ }
+
+ print_test_preamble(test, quiet);
+ status = run_test(test);
+ print_test_result(test, status, ++testnum);
+ switch (status) {
+ case SCX_TEST_PASS:
+ passed++;
+ break;
+ case SCX_TEST_SKIP:
+ skipped++;
+ break;
+ case SCX_TEST_FAIL:
+ failed++;
+ break;
+ }
+ }
+ printf("\n\n=============================\n\n");
+ printf("RESULTS:\n\n");
+ printf("PASSED: %u\n", passed);
+ printf("SKIPPED: %u\n", skipped);
+ printf("FAILED: %u\n", failed);
+
+ return 0;
+}
+
+void scx_test_register(struct scx_test *test)
+{
+ SCX_BUG_ON(!test_valid(test), "Invalid test found");
+ SCX_BUG_ON(__scx_num_tests >= MAX_SCX_TESTS, "Maximum tests exceeded");
+
+ __scx_tests[__scx_num_tests++] = *test;
+}
diff --git a/tools/testing/selftests/sched_ext/scx_test.h b/tools/testing/selftests/sched_ext/scx_test.h
new file mode 100644
index 000000000000..90b8d6915bb7
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/scx_test.h
@@ -0,0 +1,131 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2023 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2023 Tejun Heo <tj@kernel.org>
+ * Copyright (c) 2023 David Vernet <dvernet@meta.com>
+ */
+
+#ifndef __SCX_TEST_H__
+#define __SCX_TEST_H__
+
+#include <errno.h>
+#include <scx/common.h>
+#include <scx/compat.h>
+
+enum scx_test_status {
+ SCX_TEST_PASS = 0,
+ SCX_TEST_SKIP,
+ SCX_TEST_FAIL,
+};
+
+#define EXIT_KIND(__ent) __COMPAT_ENUM_OR_ZERO("scx_exit_kind", #__ent)
+
+struct scx_test {
+ /**
+ * name - The name of the testcase.
+ */
+ const char *name;
+
+ /**
+ * description - A description of your testcase: what it tests and is
+ * meant to validate.
+ */
+ const char *description;
+
+ /*
+ * setup - Setup the test.
+ * @ctx: A pointer to a context object that will be passed to run and
+ * cleanup.
+ *
+ * An optional callback that allows a testcase to perform setup for its
+ * run. A test may return SCX_TEST_SKIP to skip the run.
+ */
+ enum scx_test_status (*setup)(void **ctx);
+
+ /*
+ * run - Run the test.
+ * @ctx: Context set in the setup() callback. If @ctx was not set in
+ * setup(), it is NULL.
+ *
+ * The main test. Callers should return one of:
+ *
+ * - SCX_TEST_PASS: Test passed
+ * - SCX_TEST_SKIP: Test should be skipped
+ * - SCX_TEST_FAIL: Test failed
+ *
+ * This callback must be defined.
+ */
+ enum scx_test_status (*run)(void *ctx);
+
+ /*
+ * cleanup - Perform cleanup following the test
+ * @ctx: Context set in the setup() callback. If @ctx was not set in
+ * setup(), it is NULL.
+ *
+ * An optional callback that allows a test to perform cleanup after
+ * being run. This callback is run even if the run() callback returns
+ * SCX_TEST_SKIP or SCX_TEST_FAIL. It is not run if setup() returns
+ * SCX_TEST_SKIP or SCX_TEST_FAIL.
+ */
+ void (*cleanup)(void *ctx);
+};
+
+void scx_test_register(struct scx_test *test);
+
+#define REGISTER_SCX_TEST(__test) \
+ __attribute__((constructor)) \
+ static void ___scxregister##__LINE__(void) \
+ { \
+ scx_test_register(__test); \
+ }
+
+#define SCX_ERR(__fmt, ...) \
+ do { \
+ fprintf(stderr, "ERR: %s:%d\n", __FILE__, __LINE__); \
+ fprintf(stderr, __fmt"\n", ##__VA_ARGS__); \
+ } while (0)
+
+#define SCX_FAIL(__fmt, ...) \
+ do { \
+ SCX_ERR(__fmt, ##__VA_ARGS__); \
+ return SCX_TEST_FAIL; \
+ } while (0)
+
+#define SCX_FAIL_IF(__cond, __fmt, ...) \
+ do { \
+ if (__cond) \
+ SCX_FAIL(__fmt, ##__VA_ARGS__); \
+ } while (0)
+
+#define SCX_GT(_x, _y) SCX_FAIL_IF((_x) <= (_y), "Expected %s > %s (%lu > %lu)", \
+ #_x, #_y, (u64)(_x), (u64)(_y))
+#define SCX_GE(_x, _y) SCX_FAIL_IF((_x) < (_y), "Expected %s >= %s (%lu >= %lu)", \
+ #_x, #_y, (u64)(_x), (u64)(_y))
+#define SCX_LT(_x, _y) SCX_FAIL_IF((_x) >= (_y), "Expected %s < %s (%lu < %lu)", \
+ #_x, #_y, (u64)(_x), (u64)(_y))
+#define SCX_LE(_x, _y) SCX_FAIL_IF((_x) > (_y), "Expected %s <= %s (%lu <= %lu)", \
+ #_x, #_y, (u64)(_x), (u64)(_y))
+#define SCX_EQ(_x, _y) SCX_FAIL_IF((_x) != (_y), "Expected %s == %s (%lu == %lu)", \
+ #_x, #_y, (u64)(_x), (u64)(_y))
+#define SCX_ASSERT(_x) SCX_FAIL_IF(!(_x), "Expected %s to be true (%lu)", \
+ #_x, (u64)(_x))
+
+#define SCX_ECODE_VAL(__ecode) ({ \
+ u64 __val = 0; \
+ bool __found = false; \
+ \
+ __found = __COMPAT_read_enum("scx_exit_code", #__ecode, &__val); \
+ SCX_ASSERT(__found); \
+ (s64)__val; \
+})
+
+#define SCX_KIND_VAL(__kind) ({ \
+ u64 __val = 0; \
+ bool __found = false; \
+ \
+ __found = __COMPAT_read_enum("scx_exit_kind", #__kind, &__val); \
+ SCX_ASSERT(__found); \
+ __val; \
+})
+
+#endif // # __SCX_TEST_H__
diff --git a/tools/testing/selftests/sched_ext/select_cpu_dfl.bpf.c b/tools/testing/selftests/sched_ext/select_cpu_dfl.bpf.c
new file mode 100644
index 000000000000..2ed2991afafe
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/select_cpu_dfl.bpf.c
@@ -0,0 +1,40 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * A scheduler that validates the behavior of direct dispatching with a default
+ * select_cpu implementation.
+ *
+ * Copyright (c) 2023 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2023 David Vernet <dvernet@meta.com>
+ * Copyright (c) 2023 Tejun Heo <tj@kernel.org>
+ */
+
+#include <scx/common.bpf.h>
+
+char _license[] SEC("license") = "GPL";
+
+bool saw_local = false;
+
+static bool task_is_test(const struct task_struct *p)
+{
+ return !bpf_strncmp(p->comm, 9, "select_cpu");
+}
+
+void BPF_STRUCT_OPS(select_cpu_dfl_enqueue, struct task_struct *p,
+ u64 enq_flags)
+{
+ const struct cpumask *idle_mask = scx_bpf_get_idle_cpumask();
+
+ if (task_is_test(p) &&
+ bpf_cpumask_test_cpu(scx_bpf_task_cpu(p), idle_mask)) {
+ saw_local = true;
+ }
+ scx_bpf_put_idle_cpumask(idle_mask);
+
+ scx_bpf_dispatch(p, SCX_DSQ_GLOBAL, SCX_SLICE_DFL, enq_flags);
+}
+
+SEC(".struct_ops.link")
+struct sched_ext_ops select_cpu_dfl_ops = {
+ .enqueue = select_cpu_dfl_enqueue,
+ .name = "select_cpu_dfl",
+};
diff --git a/tools/testing/selftests/sched_ext/select_cpu_dfl.c b/tools/testing/selftests/sched_ext/select_cpu_dfl.c
new file mode 100644
index 000000000000..a53a40c2d2f0
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/select_cpu_dfl.c
@@ -0,0 +1,72 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2023 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2023 David Vernet <dvernet@meta.com>
+ * Copyright (c) 2023 Tejun Heo <tj@kernel.org>
+ */
+#include <bpf/bpf.h>
+#include <scx/common.h>
+#include <sys/wait.h>
+#include <unistd.h>
+#include "select_cpu_dfl.bpf.skel.h"
+#include "scx_test.h"
+
+#define NUM_CHILDREN 1028
+
+static enum scx_test_status setup(void **ctx)
+{
+ struct select_cpu_dfl *skel;
+
+ skel = select_cpu_dfl__open_and_load();
+ SCX_FAIL_IF(!skel, "Failed to open and load skel");
+ *ctx = skel;
+
+ return SCX_TEST_PASS;
+}
+
+static enum scx_test_status run(void *ctx)
+{
+ struct select_cpu_dfl *skel = ctx;
+ struct bpf_link *link;
+ pid_t pids[NUM_CHILDREN];
+ int i, status;
+
+ link = bpf_map__attach_struct_ops(skel->maps.select_cpu_dfl_ops);
+ SCX_FAIL_IF(!link, "Failed to attach scheduler");
+
+ for (i = 0; i < NUM_CHILDREN; i++) {
+ pids[i] = fork();
+ if (pids[i] == 0) {
+ sleep(1);
+ exit(0);
+ }
+ }
+
+ for (i = 0; i < NUM_CHILDREN; i++) {
+ SCX_EQ(waitpid(pids[i], &status, 0), pids[i]);
+ SCX_EQ(status, 0);
+ }
+
+ SCX_ASSERT(!skel->bss->saw_local);
+
+ bpf_link__destroy(link);
+
+ return SCX_TEST_PASS;
+}
+
+static void cleanup(void *ctx)
+{
+ struct select_cpu_dfl *skel = ctx;
+
+ select_cpu_dfl__destroy(skel);
+}
+
+struct scx_test select_cpu_dfl = {
+ .name = "select_cpu_dfl",
+ .description = "Verify the default ops.select_cpu() dispatches tasks "
+ "when idles cores are found, and skips ops.enqueue()",
+ .setup = setup,
+ .run = run,
+ .cleanup = cleanup,
+};
+REGISTER_SCX_TEST(&select_cpu_dfl)
diff --git a/tools/testing/selftests/sched_ext/select_cpu_dfl_nodispatch.bpf.c b/tools/testing/selftests/sched_ext/select_cpu_dfl_nodispatch.bpf.c
new file mode 100644
index 000000000000..4bb5abb2d369
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/select_cpu_dfl_nodispatch.bpf.c
@@ -0,0 +1,89 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * A scheduler that validates the behavior of direct dispatching with a default
+ * select_cpu implementation, and with the SCX_OPS_ENQ_DFL_NO_DISPATCH ops flag
+ * specified.
+ *
+ * Copyright (c) 2023 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2023 David Vernet <dvernet@meta.com>
+ * Copyright (c) 2023 Tejun Heo <tj@kernel.org>
+ */
+
+#include <scx/common.bpf.h>
+
+char _license[] SEC("license") = "GPL";
+
+bool saw_local = false;
+
+/* Per-task scheduling context */
+struct task_ctx {
+ bool force_local; /* CPU changed by ops.select_cpu() */
+};
+
+struct {
+ __uint(type, BPF_MAP_TYPE_TASK_STORAGE);
+ __uint(map_flags, BPF_F_NO_PREALLOC);
+ __type(key, int);
+ __type(value, struct task_ctx);
+} task_ctx_stor SEC(".maps");
+
+/* Manually specify the signature until the kfunc is added to the scx repo. */
+s32 scx_bpf_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags,
+ bool *found) __ksym;
+
+s32 BPF_STRUCT_OPS(select_cpu_dfl_nodispatch_select_cpu, struct task_struct *p,
+ s32 prev_cpu, u64 wake_flags)
+{
+ struct task_ctx *tctx;
+ s32 cpu;
+
+ tctx = bpf_task_storage_get(&task_ctx_stor, p, 0, 0);
+ if (!tctx) {
+ scx_bpf_error("task_ctx lookup failed");
+ return -ESRCH;
+ }
+
+ cpu = scx_bpf_select_cpu_dfl(p, prev_cpu, wake_flags,
+ &tctx->force_local);
+
+ return cpu;
+}
+
+void BPF_STRUCT_OPS(select_cpu_dfl_nodispatch_enqueue, struct task_struct *p,
+ u64 enq_flags)
+{
+ u64 dsq_id = SCX_DSQ_GLOBAL;
+ struct task_ctx *tctx;
+
+ tctx = bpf_task_storage_get(&task_ctx_stor, p, 0, 0);
+ if (!tctx) {
+ scx_bpf_error("task_ctx lookup failed");
+ return;
+ }
+
+ if (tctx->force_local) {
+ dsq_id = SCX_DSQ_LOCAL;
+ tctx->force_local = false;
+ saw_local = true;
+ }
+
+ scx_bpf_dispatch(p, dsq_id, SCX_SLICE_DFL, enq_flags);
+}
+
+s32 BPF_STRUCT_OPS(select_cpu_dfl_nodispatch_init_task,
+ struct task_struct *p, struct scx_init_task_args *args)
+{
+ if (bpf_task_storage_get(&task_ctx_stor, p, 0,
+ BPF_LOCAL_STORAGE_GET_F_CREATE))
+ return 0;
+ else
+ return -ENOMEM;
+}
+
+SEC(".struct_ops.link")
+struct sched_ext_ops select_cpu_dfl_nodispatch_ops = {
+ .select_cpu = select_cpu_dfl_nodispatch_select_cpu,
+ .enqueue = select_cpu_dfl_nodispatch_enqueue,
+ .init_task = select_cpu_dfl_nodispatch_init_task,
+ .name = "select_cpu_dfl_nodispatch",
+};
diff --git a/tools/testing/selftests/sched_ext/select_cpu_dfl_nodispatch.c b/tools/testing/selftests/sched_ext/select_cpu_dfl_nodispatch.c
new file mode 100644
index 000000000000..1d85bf4bf3a3
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/select_cpu_dfl_nodispatch.c
@@ -0,0 +1,72 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2023 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2023 David Vernet <dvernet@meta.com>
+ * Copyright (c) 2023 Tejun Heo <tj@kernel.org>
+ */
+#include <bpf/bpf.h>
+#include <scx/common.h>
+#include <sys/wait.h>
+#include <unistd.h>
+#include "select_cpu_dfl_nodispatch.bpf.skel.h"
+#include "scx_test.h"
+
+#define NUM_CHILDREN 1028
+
+static enum scx_test_status setup(void **ctx)
+{
+ struct select_cpu_dfl_nodispatch *skel;
+
+ skel = select_cpu_dfl_nodispatch__open_and_load();
+ SCX_FAIL_IF(!skel, "Failed to open and load skel");
+ *ctx = skel;
+
+ return SCX_TEST_PASS;
+}
+
+static enum scx_test_status run(void *ctx)
+{
+ struct select_cpu_dfl_nodispatch *skel = ctx;
+ struct bpf_link *link;
+ pid_t pids[NUM_CHILDREN];
+ int i, status;
+
+ link = bpf_map__attach_struct_ops(skel->maps.select_cpu_dfl_nodispatch_ops);
+ SCX_FAIL_IF(!link, "Failed to attach scheduler");
+
+ for (i = 0; i < NUM_CHILDREN; i++) {
+ pids[i] = fork();
+ if (pids[i] == 0) {
+ sleep(1);
+ exit(0);
+ }
+ }
+
+ for (i = 0; i < NUM_CHILDREN; i++) {
+ SCX_EQ(waitpid(pids[i], &status, 0), pids[i]);
+ SCX_EQ(status, 0);
+ }
+
+ SCX_ASSERT(skel->bss->saw_local);
+
+ bpf_link__destroy(link);
+
+ return SCX_TEST_PASS;
+}
+
+static void cleanup(void *ctx)
+{
+ struct select_cpu_dfl_nodispatch *skel = ctx;
+
+ select_cpu_dfl_nodispatch__destroy(skel);
+}
+
+struct scx_test select_cpu_dfl_nodispatch = {
+ .name = "select_cpu_dfl_nodispatch",
+ .description = "Verify behavior of scx_bpf_select_cpu_dfl() in "
+ "ops.select_cpu()",
+ .setup = setup,
+ .run = run,
+ .cleanup = cleanup,
+};
+REGISTER_SCX_TEST(&select_cpu_dfl_nodispatch)
diff --git a/tools/testing/selftests/sched_ext/select_cpu_dispatch.bpf.c b/tools/testing/selftests/sched_ext/select_cpu_dispatch.bpf.c
new file mode 100644
index 000000000000..f0b96a4a04b2
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/select_cpu_dispatch.bpf.c
@@ -0,0 +1,41 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * A scheduler that validates the behavior of direct dispatching with a default
+ * select_cpu implementation.
+ *
+ * Copyright (c) 2023 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2023 David Vernet <dvernet@meta.com>
+ * Copyright (c) 2023 Tejun Heo <tj@kernel.org>
+ */
+
+#include <scx/common.bpf.h>
+
+char _license[] SEC("license") = "GPL";
+
+s32 BPF_STRUCT_OPS(select_cpu_dispatch_select_cpu, struct task_struct *p,
+ s32 prev_cpu, u64 wake_flags)
+{
+ u64 dsq_id = SCX_DSQ_LOCAL;
+ s32 cpu = prev_cpu;
+
+ if (scx_bpf_test_and_clear_cpu_idle(cpu))
+ goto dispatch;
+
+ cpu = scx_bpf_pick_idle_cpu(p->cpus_ptr, 0);
+ if (cpu >= 0)
+ goto dispatch;
+
+ dsq_id = SCX_DSQ_GLOBAL;
+ cpu = prev_cpu;
+
+dispatch:
+ scx_bpf_dispatch(p, dsq_id, SCX_SLICE_DFL, 0);
+ return cpu;
+}
+
+SEC(".struct_ops.link")
+struct sched_ext_ops select_cpu_dispatch_ops = {
+ .select_cpu = select_cpu_dispatch_select_cpu,
+ .name = "select_cpu_dispatch",
+ .timeout_ms = 1000U,
+};
diff --git a/tools/testing/selftests/sched_ext/select_cpu_dispatch.c b/tools/testing/selftests/sched_ext/select_cpu_dispatch.c
new file mode 100644
index 000000000000..0309ca8785b3
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/select_cpu_dispatch.c
@@ -0,0 +1,70 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2023 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2023 David Vernet <dvernet@meta.com>
+ * Copyright (c) 2023 Tejun Heo <tj@kernel.org>
+ */
+#include <bpf/bpf.h>
+#include <scx/common.h>
+#include <sys/wait.h>
+#include <unistd.h>
+#include "select_cpu_dispatch.bpf.skel.h"
+#include "scx_test.h"
+
+#define NUM_CHILDREN 1028
+
+static enum scx_test_status setup(void **ctx)
+{
+ struct select_cpu_dispatch *skel;
+
+ skel = select_cpu_dispatch__open_and_load();
+ SCX_FAIL_IF(!skel, "Failed to open and load skel");
+ *ctx = skel;
+
+ return SCX_TEST_PASS;
+}
+
+static enum scx_test_status run(void *ctx)
+{
+ struct select_cpu_dispatch *skel = ctx;
+ struct bpf_link *link;
+ pid_t pids[NUM_CHILDREN];
+ int i, status;
+
+ link = bpf_map__attach_struct_ops(skel->maps.select_cpu_dispatch_ops);
+ SCX_FAIL_IF(!link, "Failed to attach scheduler");
+
+ for (i = 0; i < NUM_CHILDREN; i++) {
+ pids[i] = fork();
+ if (pids[i] == 0) {
+ sleep(1);
+ exit(0);
+ }
+ }
+
+ for (i = 0; i < NUM_CHILDREN; i++) {
+ SCX_EQ(waitpid(pids[i], &status, 0), pids[i]);
+ SCX_EQ(status, 0);
+ }
+
+ bpf_link__destroy(link);
+
+ return SCX_TEST_PASS;
+}
+
+static void cleanup(void *ctx)
+{
+ struct select_cpu_dispatch *skel = ctx;
+
+ select_cpu_dispatch__destroy(skel);
+}
+
+struct scx_test select_cpu_dispatch = {
+ .name = "select_cpu_dispatch",
+ .description = "Test direct dispatching to built-in DSQs from "
+ "ops.select_cpu()",
+ .setup = setup,
+ .run = run,
+ .cleanup = cleanup,
+};
+REGISTER_SCX_TEST(&select_cpu_dispatch)
diff --git a/tools/testing/selftests/sched_ext/select_cpu_dispatch_bad_dsq.bpf.c b/tools/testing/selftests/sched_ext/select_cpu_dispatch_bad_dsq.bpf.c
new file mode 100644
index 000000000000..7b42ddce0f56
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/select_cpu_dispatch_bad_dsq.bpf.c
@@ -0,0 +1,37 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * A scheduler that validates the behavior of direct dispatching with a default
+ * select_cpu implementation.
+ *
+ * Copyright (c) 2023 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2023 David Vernet <dvernet@meta.com>
+ * Copyright (c) 2023 Tejun Heo <tj@kernel.org>
+ */
+
+#include <scx/common.bpf.h>
+
+char _license[] SEC("license") = "GPL";
+
+UEI_DEFINE(uei);
+
+s32 BPF_STRUCT_OPS(select_cpu_dispatch_bad_dsq_select_cpu, struct task_struct *p,
+ s32 prev_cpu, u64 wake_flags)
+{
+ /* Dispatching to a random DSQ should fail. */
+ scx_bpf_dispatch(p, 0xcafef00d, SCX_SLICE_DFL, 0);
+
+ return prev_cpu;
+}
+
+void BPF_STRUCT_OPS(select_cpu_dispatch_bad_dsq_exit, struct scx_exit_info *ei)
+{
+ UEI_RECORD(uei, ei);
+}
+
+SEC(".struct_ops.link")
+struct sched_ext_ops select_cpu_dispatch_bad_dsq_ops = {
+ .select_cpu = select_cpu_dispatch_bad_dsq_select_cpu,
+ .exit = select_cpu_dispatch_bad_dsq_exit,
+ .name = "select_cpu_dispatch_bad_dsq",
+ .timeout_ms = 1000U,
+};
diff --git a/tools/testing/selftests/sched_ext/select_cpu_dispatch_bad_dsq.c b/tools/testing/selftests/sched_ext/select_cpu_dispatch_bad_dsq.c
new file mode 100644
index 000000000000..47eb6ed7627d
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/select_cpu_dispatch_bad_dsq.c
@@ -0,0 +1,56 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2023 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2023 David Vernet <dvernet@meta.com>
+ * Copyright (c) 2023 Tejun Heo <tj@kernel.org>
+ */
+#include <bpf/bpf.h>
+#include <scx/common.h>
+#include <sys/wait.h>
+#include <unistd.h>
+#include "select_cpu_dispatch_bad_dsq.bpf.skel.h"
+#include "scx_test.h"
+
+static enum scx_test_status setup(void **ctx)
+{
+ struct select_cpu_dispatch_bad_dsq *skel;
+
+ skel = select_cpu_dispatch_bad_dsq__open_and_load();
+ SCX_FAIL_IF(!skel, "Failed to open and load skel");
+ *ctx = skel;
+
+ return SCX_TEST_PASS;
+}
+
+static enum scx_test_status run(void *ctx)
+{
+ struct select_cpu_dispatch_bad_dsq *skel = ctx;
+ struct bpf_link *link;
+
+ link = bpf_map__attach_struct_ops(skel->maps.select_cpu_dispatch_bad_dsq_ops);
+ SCX_FAIL_IF(!link, "Failed to attach scheduler");
+
+ sleep(1);
+
+ SCX_EQ(skel->data->uei.kind, EXIT_KIND(SCX_EXIT_ERROR));
+ bpf_link__destroy(link);
+
+ return SCX_TEST_PASS;
+}
+
+static void cleanup(void *ctx)
+{
+ struct select_cpu_dispatch_bad_dsq *skel = ctx;
+
+ select_cpu_dispatch_bad_dsq__destroy(skel);
+}
+
+struct scx_test select_cpu_dispatch_bad_dsq = {
+ .name = "select_cpu_dispatch_bad_dsq",
+ .description = "Verify graceful failure if we direct-dispatch to a "
+ "bogus DSQ in ops.select_cpu()",
+ .setup = setup,
+ .run = run,
+ .cleanup = cleanup,
+};
+REGISTER_SCX_TEST(&select_cpu_dispatch_bad_dsq)
diff --git a/tools/testing/selftests/sched_ext/select_cpu_dispatch_dbl_dsp.bpf.c b/tools/testing/selftests/sched_ext/select_cpu_dispatch_dbl_dsp.bpf.c
new file mode 100644
index 000000000000..653e3dc0b4dc
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/select_cpu_dispatch_dbl_dsp.bpf.c
@@ -0,0 +1,38 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * A scheduler that validates the behavior of direct dispatching with a default
+ * select_cpu implementation.
+ *
+ * Copyright (c) 2023 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2023 David Vernet <dvernet@meta.com>
+ * Copyright (c) 2023 Tejun Heo <tj@kernel.org>
+ */
+
+#include <scx/common.bpf.h>
+
+char _license[] SEC("license") = "GPL";
+
+UEI_DEFINE(uei);
+
+s32 BPF_STRUCT_OPS(select_cpu_dispatch_dbl_dsp_select_cpu, struct task_struct *p,
+ s32 prev_cpu, u64 wake_flags)
+{
+ /* Dispatching twice in a row is disallowed. */
+ scx_bpf_dispatch(p, SCX_DSQ_GLOBAL, SCX_SLICE_DFL, 0);
+ scx_bpf_dispatch(p, SCX_DSQ_GLOBAL, SCX_SLICE_DFL, 0);
+
+ return prev_cpu;
+}
+
+void BPF_STRUCT_OPS(select_cpu_dispatch_dbl_dsp_exit, struct scx_exit_info *ei)
+{
+ UEI_RECORD(uei, ei);
+}
+
+SEC(".struct_ops.link")
+struct sched_ext_ops select_cpu_dispatch_dbl_dsp_ops = {
+ .select_cpu = select_cpu_dispatch_dbl_dsp_select_cpu,
+ .exit = select_cpu_dispatch_dbl_dsp_exit,
+ .name = "select_cpu_dispatch_dbl_dsp",
+ .timeout_ms = 1000U,
+};
diff --git a/tools/testing/selftests/sched_ext/select_cpu_dispatch_dbl_dsp.c b/tools/testing/selftests/sched_ext/select_cpu_dispatch_dbl_dsp.c
new file mode 100644
index 000000000000..48ff028a3c46
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/select_cpu_dispatch_dbl_dsp.c
@@ -0,0 +1,56 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2023 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2023 David Vernet <dvernet@meta.com>
+ * Copyright (c) 2023 Tejun Heo <tj@kernel.org>
+ */
+#include <bpf/bpf.h>
+#include <scx/common.h>
+#include <sys/wait.h>
+#include <unistd.h>
+#include "select_cpu_dispatch_dbl_dsp.bpf.skel.h"
+#include "scx_test.h"
+
+static enum scx_test_status setup(void **ctx)
+{
+ struct select_cpu_dispatch_dbl_dsp *skel;
+
+ skel = select_cpu_dispatch_dbl_dsp__open_and_load();
+ SCX_FAIL_IF(!skel, "Failed to open and load skel");
+ *ctx = skel;
+
+ return SCX_TEST_PASS;
+}
+
+static enum scx_test_status run(void *ctx)
+{
+ struct select_cpu_dispatch_dbl_dsp *skel = ctx;
+ struct bpf_link *link;
+
+ link = bpf_map__attach_struct_ops(skel->maps.select_cpu_dispatch_dbl_dsp_ops);
+ SCX_FAIL_IF(!link, "Failed to attach scheduler");
+
+ sleep(1);
+
+ SCX_EQ(skel->data->uei.kind, EXIT_KIND(SCX_EXIT_ERROR));
+ bpf_link__destroy(link);
+
+ return SCX_TEST_PASS;
+}
+
+static void cleanup(void *ctx)
+{
+ struct select_cpu_dispatch_dbl_dsp *skel = ctx;
+
+ select_cpu_dispatch_dbl_dsp__destroy(skel);
+}
+
+struct scx_test select_cpu_dispatch_dbl_dsp = {
+ .name = "select_cpu_dispatch_dbl_dsp",
+ .description = "Verify graceful failure if we dispatch twice to a "
+ "DSQ in ops.select_cpu()",
+ .setup = setup,
+ .run = run,
+ .cleanup = cleanup,
+};
+REGISTER_SCX_TEST(&select_cpu_dispatch_dbl_dsp)
diff --git a/tools/testing/selftests/sched_ext/select_cpu_vtime.bpf.c b/tools/testing/selftests/sched_ext/select_cpu_vtime.bpf.c
new file mode 100644
index 000000000000..7f3ebf4fc2ea
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/select_cpu_vtime.bpf.c
@@ -0,0 +1,92 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * A scheduler that validates that enqueue flags are properly stored and
+ * applied at dispatch time when a task is directly dispatched from
+ * ops.select_cpu(). We validate this by using scx_bpf_dispatch_vtime(), and
+ * making the test a very basic vtime scheduler.
+ *
+ * Copyright (c) 2024 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2024 David Vernet <dvernet@meta.com>
+ * Copyright (c) 2024 Tejun Heo <tj@kernel.org>
+ */
+
+#include <scx/common.bpf.h>
+
+char _license[] SEC("license") = "GPL";
+
+volatile bool consumed;
+
+static u64 vtime_now;
+
+#define VTIME_DSQ 0
+
+static inline bool vtime_before(u64 a, u64 b)
+{
+ return (s64)(a - b) < 0;
+}
+
+static inline u64 task_vtime(const struct task_struct *p)
+{
+ u64 vtime = p->scx.dsq_vtime;
+
+ if (vtime_before(vtime, vtime_now - SCX_SLICE_DFL))
+ return vtime_now - SCX_SLICE_DFL;
+ else
+ return vtime;
+}
+
+s32 BPF_STRUCT_OPS(select_cpu_vtime_select_cpu, struct task_struct *p,
+ s32 prev_cpu, u64 wake_flags)
+{
+ s32 cpu;
+
+ cpu = scx_bpf_pick_idle_cpu(p->cpus_ptr, 0);
+ if (cpu >= 0)
+ goto ddsp;
+
+ cpu = prev_cpu;
+ scx_bpf_test_and_clear_cpu_idle(cpu);
+ddsp:
+ scx_bpf_dispatch_vtime(p, VTIME_DSQ, SCX_SLICE_DFL, task_vtime(p), 0);
+ return cpu;
+}
+
+void BPF_STRUCT_OPS(select_cpu_vtime_dispatch, s32 cpu, struct task_struct *p)
+{
+ if (scx_bpf_consume(VTIME_DSQ))
+ consumed = true;
+}
+
+void BPF_STRUCT_OPS(select_cpu_vtime_running, struct task_struct *p)
+{
+ if (vtime_before(vtime_now, p->scx.dsq_vtime))
+ vtime_now = p->scx.dsq_vtime;
+}
+
+void BPF_STRUCT_OPS(select_cpu_vtime_stopping, struct task_struct *p,
+ bool runnable)
+{
+ p->scx.dsq_vtime += (SCX_SLICE_DFL - p->scx.slice) * 100 / p->scx.weight;
+}
+
+void BPF_STRUCT_OPS(select_cpu_vtime_enable, struct task_struct *p)
+{
+ p->scx.dsq_vtime = vtime_now;
+}
+
+s32 BPF_STRUCT_OPS_SLEEPABLE(select_cpu_vtime_init)
+{
+ return scx_bpf_create_dsq(VTIME_DSQ, -1);
+}
+
+SEC(".struct_ops.link")
+struct sched_ext_ops select_cpu_vtime_ops = {
+ .select_cpu = select_cpu_vtime_select_cpu,
+ .dispatch = select_cpu_vtime_dispatch,
+ .running = select_cpu_vtime_running,
+ .stopping = select_cpu_vtime_stopping,
+ .enable = select_cpu_vtime_enable,
+ .init = select_cpu_vtime_init,
+ .name = "select_cpu_vtime",
+ .timeout_ms = 1000U,
+};
diff --git a/tools/testing/selftests/sched_ext/select_cpu_vtime.c b/tools/testing/selftests/sched_ext/select_cpu_vtime.c
new file mode 100644
index 000000000000..b4629c2364f5
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/select_cpu_vtime.c
@@ -0,0 +1,59 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2024 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2024 David Vernet <dvernet@meta.com>
+ * Copyright (c) 2024 Tejun Heo <tj@kernel.org>
+ */
+#include <bpf/bpf.h>
+#include <scx/common.h>
+#include <sys/wait.h>
+#include <unistd.h>
+#include "select_cpu_vtime.bpf.skel.h"
+#include "scx_test.h"
+
+static enum scx_test_status setup(void **ctx)
+{
+ struct select_cpu_vtime *skel;
+
+ skel = select_cpu_vtime__open_and_load();
+ SCX_FAIL_IF(!skel, "Failed to open and load skel");
+ *ctx = skel;
+
+ return SCX_TEST_PASS;
+}
+
+static enum scx_test_status run(void *ctx)
+{
+ struct select_cpu_vtime *skel = ctx;
+ struct bpf_link *link;
+
+ SCX_ASSERT(!skel->bss->consumed);
+
+ link = bpf_map__attach_struct_ops(skel->maps.select_cpu_vtime_ops);
+ SCX_FAIL_IF(!link, "Failed to attach scheduler");
+
+ sleep(1);
+
+ SCX_ASSERT(skel->bss->consumed);
+
+ bpf_link__destroy(link);
+
+ return SCX_TEST_PASS;
+}
+
+static void cleanup(void *ctx)
+{
+ struct select_cpu_vtime *skel = ctx;
+
+ select_cpu_vtime__destroy(skel);
+}
+
+struct scx_test select_cpu_vtime = {
+ .name = "select_cpu_vtime",
+ .description = "Test doing direct vtime-dispatching from "
+ "ops.select_cpu(), to a non-built-in DSQ",
+ .setup = setup,
+ .run = run,
+ .cleanup = cleanup,
+};
+REGISTER_SCX_TEST(&select_cpu_vtime)
diff --git a/tools/testing/selftests/sched_ext/test_example.c b/tools/testing/selftests/sched_ext/test_example.c
new file mode 100644
index 000000000000..ce36cdf03cdc
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/test_example.c
@@ -0,0 +1,49 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2024 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2024 Tejun Heo <tj@kernel.org>
+ * Copyright (c) 2024 David Vernet <dvernet@meta.com>
+ */
+#include <bpf/bpf.h>
+#include <scx/common.h>
+#include "scx_test.h"
+
+static bool setup_called = false;
+static bool run_called = false;
+static bool cleanup_called = false;
+
+static int context = 10;
+
+static enum scx_test_status setup(void **ctx)
+{
+ setup_called = true;
+ *ctx = &context;
+
+ return SCX_TEST_PASS;
+}
+
+static enum scx_test_status run(void *ctx)
+{
+ int *arg = ctx;
+
+ SCX_ASSERT(setup_called);
+ SCX_ASSERT(!run_called && !cleanup_called);
+ SCX_EQ(*arg, context);
+
+ run_called = true;
+ return SCX_TEST_PASS;
+}
+
+static void cleanup (void *ctx)
+{
+ SCX_BUG_ON(!run_called || cleanup_called, "Wrong callbacks invoked");
+}
+
+struct scx_test example = {
+ .name = "example",
+ .description = "Validate the basic function of the test suite itself",
+ .setup = setup,
+ .run = run,
+ .cleanup = cleanup,
+};
+REGISTER_SCX_TEST(&example)
diff --git a/tools/testing/selftests/sched_ext/util.c b/tools/testing/selftests/sched_ext/util.c
new file mode 100644
index 000000000000..e47769c91918
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/util.c
@@ -0,0 +1,71 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2024 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2024 David Vernet <dvernet@meta.com>
+ */
+#include <errno.h>
+#include <fcntl.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+
+/* Returns read len on success, or -errno on failure. */
+static ssize_t read_text(const char *path, char *buf, size_t max_len)
+{
+ ssize_t len;
+ int fd;
+
+ fd = open(path, O_RDONLY);
+ if (fd < 0)
+ return -errno;
+
+ len = read(fd, buf, max_len - 1);
+
+ if (len >= 0)
+ buf[len] = 0;
+
+ close(fd);
+ return len < 0 ? -errno : len;
+}
+
+/* Returns written len on success, or -errno on failure. */
+static ssize_t write_text(const char *path, char *buf, ssize_t len)
+{
+ int fd;
+ ssize_t written;
+
+ fd = open(path, O_WRONLY | O_APPEND);
+ if (fd < 0)
+ return -errno;
+
+ written = write(fd, buf, len);
+ close(fd);
+ return written < 0 ? -errno : written;
+}
+
+long file_read_long(const char *path)
+{
+ char buf[128];
+
+
+ if (read_text(path, buf, sizeof(buf)) <= 0)
+ return -1;
+
+ return atol(buf);
+}
+
+int file_write_long(const char *path, long val)
+{
+ char buf[64];
+ int ret;
+
+ ret = sprintf(buf, "%lu", val);
+ if (ret < 0)
+ return ret;
+
+ if (write_text(path, buf, sizeof(buf)) <= 0)
+ return -1;
+
+ return 0;
+}
diff --git a/tools/testing/selftests/sched_ext/util.h b/tools/testing/selftests/sched_ext/util.h
new file mode 100644
index 000000000000..bc13dfec1267
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/util.h
@@ -0,0 +1,13 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2024 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2024 David Vernet <void@manifault.com>
+ */
+
+#ifndef __SCX_TEST_UTIL_H__
+#define __SCX_TEST_UTIL_H__
+
+long file_read_long(const char *path);
+int file_write_long(const char *path, long val);
+
+#endif // __SCX_TEST_H__
--
2.45.2
Diff
---
tools/testing/selftests/sched_ext/.gitignore | 6 +
tools/testing/selftests/sched_ext/Makefile | 218 ++++++++++++++++++
tools/testing/selftests/sched_ext/config | 9 +
.../selftests/sched_ext/create_dsq.bpf.c | 58 +++++
.../testing/selftests/sched_ext/create_dsq.c | 57 +++++
.../sched_ext/ddsp_bogus_dsq_fail.bpf.c | 42 ++++
.../selftests/sched_ext/ddsp_bogus_dsq_fail.c | 57 +++++
.../sched_ext/ddsp_vtimelocal_fail.bpf.c | 39 ++++
.../sched_ext/ddsp_vtimelocal_fail.c | 56 +++++
.../selftests/sched_ext/dsp_local_on.bpf.c | 65 ++++++
.../selftests/sched_ext/dsp_local_on.c | 58 +++++
.../sched_ext/enq_last_no_enq_fails.bpf.c | 21 ++
.../sched_ext/enq_last_no_enq_fails.c | 60 +++++
.../sched_ext/enq_select_cpu_fails.bpf.c | 43 ++++
.../sched_ext/enq_select_cpu_fails.c | 61 +++++
tools/testing/selftests/sched_ext/exit.bpf.c | 84 +++++++
tools/testing/selftests/sched_ext/exit.c | 55 +++++
tools/testing/selftests/sched_ext/exit_test.h | 20 ++
.../testing/selftests/sched_ext/hotplug.bpf.c | 61 +++++
tools/testing/selftests/sched_ext/hotplug.c | 168 ++++++++++++++
.../selftests/sched_ext/hotplug_test.h | 15 ++
.../sched_ext/init_enable_count.bpf.c | 53 +++++
.../selftests/sched_ext/init_enable_count.c | 166 +++++++++++++
.../testing/selftests/sched_ext/maximal.bpf.c | 132 +++++++++++
tools/testing/selftests/sched_ext/maximal.c | 51 ++++
.../selftests/sched_ext/maybe_null.bpf.c | 36 +++
.../testing/selftests/sched_ext/maybe_null.c | 49 ++++
.../sched_ext/maybe_null_fail_dsp.bpf.c | 25 ++
.../sched_ext/maybe_null_fail_yld.bpf.c | 28 +++
.../testing/selftests/sched_ext/minimal.bpf.c | 21 ++
tools/testing/selftests/sched_ext/minimal.c | 58 +++++
.../selftests/sched_ext/prog_run.bpf.c | 32 +++
tools/testing/selftests/sched_ext/prog_run.c | 78 +++++++
.../testing/selftests/sched_ext/reload_loop.c | 75 ++++++
tools/testing/selftests/sched_ext/runner.c | 201 ++++++++++++++++
tools/testing/selftests/sched_ext/scx_test.h | 131 +++++++++++
.../selftests/sched_ext/select_cpu_dfl.bpf.c | 40 ++++
.../selftests/sched_ext/select_cpu_dfl.c | 72 ++++++
.../sched_ext/select_cpu_dfl_nodispatch.bpf.c | 89 +++++++
.../sched_ext/select_cpu_dfl_nodispatch.c | 72 ++++++
.../sched_ext/select_cpu_dispatch.bpf.c | 41 ++++
.../selftests/sched_ext/select_cpu_dispatch.c | 70 ++++++
.../select_cpu_dispatch_bad_dsq.bpf.c | 37 +++
.../sched_ext/select_cpu_dispatch_bad_dsq.c | 56 +++++
.../select_cpu_dispatch_dbl_dsp.bpf.c | 38 +++
.../sched_ext/select_cpu_dispatch_dbl_dsp.c | 56 +++++
.../sched_ext/select_cpu_vtime.bpf.c | 92 ++++++++
.../selftests/sched_ext/select_cpu_vtime.c | 59 +++++
.../selftests/sched_ext/test_example.c | 49 ++++
tools/testing/selftests/sched_ext/util.c | 71 ++++++
tools/testing/selftests/sched_ext/util.h | 13 ++
51 files changed, 3244 insertions(+)
create mode 100644 tools/testing/selftests/sched_ext/.gitignore
create mode 100644 tools/testing/selftests/sched_ext/Makefile
create mode 100644 tools/testing/selftests/sched_ext/config
create mode 100644 tools/testing/selftests/sched_ext/create_dsq.bpf.c
create mode 100644 tools/testing/selftests/sched_ext/create_dsq.c
create mode 100644 tools/testing/selftests/sched_ext/ddsp_bogus_dsq_fail.bpf.c
create mode 100644 tools/testing/selftests/sched_ext/ddsp_bogus_dsq_fail.c
create mode 100644 tools/testing/selftests/sched_ext/ddsp_vtimelocal_fail.bpf.c
create mode 100644 tools/testing/selftests/sched_ext/ddsp_vtimelocal_fail.c
create mode 100644 tools/testing/selftests/sched_ext/dsp_local_on.bpf.c
create mode 100644 tools/testing/selftests/sched_ext/dsp_local_on.c
create mode 100644 tools/testing/selftests/sched_ext/enq_last_no_enq_fails.bpf.c
create mode 100644 tools/testing/selftests/sched_ext/enq_last_no_enq_fails.c
create mode 100644 tools/testing/selftests/sched_ext/enq_select_cpu_fails.bpf.c
create mode 100644 tools/testing/selftests/sched_ext/enq_select_cpu_fails.c
create mode 100644 tools/testing/selftests/sched_ext/exit.bpf.c
create mode 100644 tools/testing/selftests/sched_ext/exit.c
create mode 100644 tools/testing/selftests/sched_ext/exit_test.h
create mode 100644 tools/testing/selftests/sched_ext/hotplug.bpf.c
create mode 100644 tools/testing/selftests/sched_ext/hotplug.c
create mode 100644 tools/testing/selftests/sched_ext/hotplug_test.h
create mode 100644 tools/testing/selftests/sched_ext/init_enable_count.bpf.c
create mode 100644 tools/testing/selftests/sched_ext/init_enable_count.c
create mode 100644 tools/testing/selftests/sched_ext/maximal.bpf.c
create mode 100644 tools/testing/selftests/sched_ext/maximal.c
create mode 100644 tools/testing/selftests/sched_ext/maybe_null.bpf.c
create mode 100644 tools/testing/selftests/sched_ext/maybe_null.c
create mode 100644 tools/testing/selftests/sched_ext/maybe_null_fail_dsp.bpf.c
create mode 100644 tools/testing/selftests/sched_ext/maybe_null_fail_yld.bpf.c
create mode 100644 tools/testing/selftests/sched_ext/minimal.bpf.c
create mode 100644 tools/testing/selftests/sched_ext/minimal.c
create mode 100644 tools/testing/selftests/sched_ext/prog_run.bpf.c
create mode 100644 tools/testing/selftests/sched_ext/prog_run.c
create mode 100644 tools/testing/selftests/sched_ext/reload_loop.c
create mode 100644 tools/testing/selftests/sched_ext/runner.c
create mode 100644 tools/testing/selftests/sched_ext/scx_test.h
create mode 100644 tools/testing/selftests/sched_ext/select_cpu_dfl.bpf.c
create mode 100644 tools/testing/selftests/sched_ext/select_cpu_dfl.c
create mode 100644 tools/testing/selftests/sched_ext/select_cpu_dfl_nodispatch.bpf.c
create mode 100644 tools/testing/selftests/sched_ext/select_cpu_dfl_nodispatch.c
create mode 100644 tools/testing/selftests/sched_ext/select_cpu_dispatch.bpf.c
create mode 100644 tools/testing/selftests/sched_ext/select_cpu_dispatch.c
create mode 100644 tools/testing/selftests/sched_ext/select_cpu_dispatch_bad_dsq.bpf.c
create mode 100644 tools/testing/selftests/sched_ext/select_cpu_dispatch_bad_dsq.c
create mode 100644 tools/testing/selftests/sched_ext/select_cpu_dispatch_dbl_dsp.bpf.c
create mode 100644 tools/testing/selftests/sched_ext/select_cpu_dispatch_dbl_dsp.c
create mode 100644 tools/testing/selftests/sched_ext/select_cpu_vtime.bpf.c
create mode 100644 tools/testing/selftests/sched_ext/select_cpu_vtime.c
create mode 100644 tools/testing/selftests/sched_ext/test_example.c
create mode 100644 tools/testing/selftests/sched_ext/util.c
create mode 100644 tools/testing/selftests/sched_ext/util.h
diff --git a/tools/testing/selftests/sched_ext/.gitignore b/tools/testing/selftests/sched_ext/.gitignore
new file mode 100644
index 000000000000..ae5491a114c0
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/.gitignore
@@ -0,0 +1,6 @@
+*
+!*.c
+!*.h
+!Makefile
+!.gitignore
+!config
diff --git a/tools/testing/selftests/sched_ext/Makefile b/tools/testing/selftests/sched_ext/Makefile
new file mode 100644
index 000000000000..0754a2c110a1
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/Makefile
@@ -0,0 +1,218 @@
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2022 Meta Platforms, Inc. and affiliates.
+include ../../../build/Build.include
+include ../../../scripts/Makefile.arch
+include ../../../scripts/Makefile.include
+include ../lib.mk
+
+ifneq ($(LLVM),)
+ifneq ($(filter %/,$(LLVM)),)
+LLVM_PREFIX := $(LLVM)
+else ifneq ($(filter -%,$(LLVM)),)
+LLVM_SUFFIX := $(LLVM)
+endif
+
+CC := $(LLVM_PREFIX)clang$(LLVM_SUFFIX) $(CLANG_FLAGS) -fintegrated-as
+else
+CC := gcc
+endif # LLVM
+
+ifneq ($(CROSS_COMPILE),)
+$(error CROSS_COMPILE not supported for scx selftests)
+endif # CROSS_COMPILE
+
+CURDIR := $(abspath .)
+REPOROOT := $(abspath ../../../..)
+TOOLSDIR := $(REPOROOT)/tools
+LIBDIR := $(TOOLSDIR)/lib
+BPFDIR := $(LIBDIR)/bpf
+TOOLSINCDIR := $(TOOLSDIR)/include
+BPFTOOLDIR := $(TOOLSDIR)/bpf/bpftool
+APIDIR := $(TOOLSINCDIR)/uapi
+GENDIR := $(REPOROOT)/include/generated
+GENHDR := $(GENDIR)/autoconf.h
+SCXTOOLSDIR := $(TOOLSDIR)/sched_ext
+SCXTOOLSINCDIR := $(TOOLSDIR)/sched_ext/include
+
+OUTPUT_DIR := $(CURDIR)/build
+OBJ_DIR := $(OUTPUT_DIR)/obj
+INCLUDE_DIR := $(OUTPUT_DIR)/include
+BPFOBJ_DIR := $(OBJ_DIR)/libbpf
+SCXOBJ_DIR := $(OBJ_DIR)/sched_ext
+BPFOBJ := $(BPFOBJ_DIR)/libbpf.a
+LIBBPF_OUTPUT := $(OBJ_DIR)/libbpf/libbpf.a
+DEFAULT_BPFTOOL := $(OUTPUT_DIR)/sbin/bpftool
+HOST_BUILD_DIR := $(OBJ_DIR)
+HOST_OUTPUT_DIR := $(OUTPUT_DIR)
+
+VMLINUX_BTF_PATHS ?= ../../../../vmlinux \
+ /sys/kernel/btf/vmlinux \
+ /boot/vmlinux-$(shell uname -r)
+VMLINUX_BTF ?= $(abspath $(firstword $(wildcard $(VMLINUX_BTF_PATHS))))
+ifeq ($(VMLINUX_BTF),)
+$(error Cannot find a vmlinux for VMLINUX_BTF at any of "$(VMLINUX_BTF_PATHS)")
+endif
+
+BPFTOOL ?= $(DEFAULT_BPFTOOL)
+
+ifneq ($(wildcard $(GENHDR)),)
+ GENFLAGS := -DHAVE_GENHDR
+endif
+
+CFLAGS += -g -O2 -rdynamic -pthread -Wall -Werror $(GENFLAGS) \
+ -I$(INCLUDE_DIR) -I$(GENDIR) -I$(LIBDIR) \
+ -I$(TOOLSINCDIR) -I$(APIDIR) -I$(CURDIR)/include -I$(SCXTOOLSINCDIR)
+
+# Silence some warnings when compiled with clang
+ifneq ($(LLVM),)
+CFLAGS += -Wno-unused-command-line-argument
+endif
+
+LDFLAGS = -lelf -lz -lpthread -lzstd
+
+IS_LITTLE_ENDIAN = $(shell $(CC) -dM -E - </dev/null | \
+ grep 'define __BYTE_ORDER__ __ORDER_LITTLE_ENDIAN__')
+
+# Get Clang's default includes on this system, as opposed to those seen by
+# '-target bpf'. This fixes "missing" files on some architectures/distros,
+# such as asm/byteorder.h, asm/socket.h, asm/sockios.h, sys/cdefs.h etc.
+#
+# Use '-idirafter': Don't interfere with include mechanics except where the
+# build would have failed anyways.
+define get_sys_includes
+$(shell $(1) -v -E - </dev/null 2>&1 \
+ | sed -n '/<...> search starts here:/,/End of search list./{ s| \(/.*\)|-idirafter \1|p }') \
+$(shell $(1) -dM -E - </dev/null | grep '__riscv_xlen ' | awk '{printf("-D__riscv_xlen=%d -D__BITS_PER_LONG=%d", $$3, $$3)}')
+endef
+
+BPF_CFLAGS = -g -D__TARGET_ARCH_$(SRCARCH) \
+ $(if $(IS_LITTLE_ENDIAN),-mlittle-endian,-mbig-endian) \
+ -I$(CURDIR)/include -I$(CURDIR)/include/bpf-compat \
+ -I$(INCLUDE_DIR) -I$(APIDIR) -I$(SCXTOOLSINCDIR) \
+ -I$(REPOROOT)/include \
+ $(call get_sys_includes,$(CLANG)) \
+ -Wall -Wno-compare-distinct-pointer-types \
+ -Wno-incompatible-function-pointer-types \
+ -O2 -mcpu=v3
+
+# sort removes libbpf duplicates when not cross-building
+MAKE_DIRS := $(sort $(OBJ_DIR)/libbpf $(OBJ_DIR)/libbpf \
+ $(OBJ_DIR)/bpftool $(OBJ_DIR)/resolve_btfids \
+ $(INCLUDE_DIR) $(SCXOBJ_DIR))
+
+$(MAKE_DIRS):
+ $(call msg,MKDIR,,$@)
+ $(Q)mkdir -p $@
+
+$(BPFOBJ): $(wildcard $(BPFDIR)/*.[ch] $(BPFDIR)/Makefile) \
+ $(APIDIR)/linux/bpf.h \
+ | $(OBJ_DIR)/libbpf
+ $(Q)$(MAKE) $(submake_extras) -C $(BPFDIR) OUTPUT=$(OBJ_DIR)/libbpf/ \
+ EXTRA_CFLAGS='-g -O0 -fPIC' \
+ DESTDIR=$(OUTPUT_DIR) prefix= all install_headers
+
+$(DEFAULT_BPFTOOL): $(wildcard $(BPFTOOLDIR)/*.[ch] $(BPFTOOLDIR)/Makefile) \
+ $(LIBBPF_OUTPUT) | $(OBJ_DIR)/bpftool
+ $(Q)$(MAKE) $(submake_extras) -C $(BPFTOOLDIR) \
+ ARCH= CROSS_COMPILE= CC=$(HOSTCC) LD=$(HOSTLD) \
+ EXTRA_CFLAGS='-g -O0' \
+ OUTPUT=$(OBJ_DIR)/bpftool/ \
+ LIBBPF_OUTPUT=$(OBJ_DIR)/libbpf/ \
+ LIBBPF_DESTDIR=$(OUTPUT_DIR)/ \
+ prefix= DESTDIR=$(OUTPUT_DIR)/ install-bin
+
+$(INCLUDE_DIR)/vmlinux.h: $(VMLINUX_BTF) $(BPFTOOL) | $(INCLUDE_DIR)
+ifeq ($(VMLINUX_H),)
+ $(call msg,GEN,,$@)
+ $(Q)$(BPFTOOL) btf dump file $(VMLINUX_BTF) format c > $@
+else
+ $(call msg,CP,,$@)
+ $(Q)cp "$(VMLINUX_H)" $@
+endif
+
+$(SCXOBJ_DIR)/%.bpf.o: %.bpf.c $(INCLUDE_DIR)/vmlinux.h | $(BPFOBJ) $(SCXOBJ_DIR)
+ $(call msg,CLNG-BPF,,$(notdir $@))
+ $(Q)$(CLANG) $(BPF_CFLAGS) -target bpf -c $< -o $@
+
+$(INCLUDE_DIR)/%.bpf.skel.h: $(SCXOBJ_DIR)/%.bpf.o $(INCLUDE_DIR)/vmlinux.h $(BPFTOOL) | $(INCLUDE_DIR)
+ $(eval sched=$(notdir $@))
+ $(call msg,GEN-SKEL,,$(sched))
+ $(Q)$(BPFTOOL) gen object $(<:.o=.linked1.o) $<
+ $(Q)$(BPFTOOL) gen object $(<:.o=.linked2.o) $(<:.o=.linked1.o)
+ $(Q)$(BPFTOOL) gen object $(<:.o=.linked3.o) $(<:.o=.linked2.o)
+ $(Q)diff $(<:.o=.linked2.o) $(<:.o=.linked3.o)
+ $(Q)$(BPFTOOL) gen skeleton $(<:.o=.linked3.o) name $(subst .bpf.skel.h,,$(sched)) > $@
+ $(Q)$(BPFTOOL) gen subskeleton $(<:.o=.linked3.o) name $(subst .bpf.skel.h,,$(sched)) > $(@:.skel.h=.subskel.h)
+
+################
+# C schedulers #
+################
+
+override define CLEAN
+ rm -rf $(OUTPUT_DIR)
+ rm -f *.o *.bpf.o *.bpf.skel.h *.bpf.subskel.h
+ rm -f $(TEST_GEN_PROGS)
+ rm -f runner
+endef
+
+# Every testcase takes all of the BPF progs are dependencies by default. This
+# allows testcases to load any BPF scheduler, which is useful for testcases
+# that don't need their own prog to run their test.
+all_test_bpfprogs := $(foreach prog,$(wildcard *.bpf.c),$(INCLUDE_DIR)/$(patsubst %.c,%.skel.h,$(prog)))
+
+auto-test-targets := \
+ create_dsq \
+ enq_last_no_enq_fails \
+ enq_select_cpu_fails \
+ ddsp_bogus_dsq_fail \
+ ddsp_vtimelocal_fail \
+ dsp_local_on \
+ exit \
+ hotplug \
+ init_enable_count \
+ maximal \
+ maybe_null \
+ minimal \
+ prog_run \
+ reload_loop \
+ select_cpu_dfl \
+ select_cpu_dfl_nodispatch \
+ select_cpu_dispatch \
+ select_cpu_dispatch_bad_dsq \
+ select_cpu_dispatch_dbl_dsp \
+ select_cpu_vtime \
+ test_example \
+
+testcase-targets := $(addsuffix .o,$(addprefix $(SCXOBJ_DIR)/,$(auto-test-targets)))
+
+$(SCXOBJ_DIR)/runner.o: runner.c | $(SCXOBJ_DIR)
+ $(CC) $(CFLAGS) -c $< -o $@
+
+# Create all of the test targets object files, whose testcase objects will be
+# registered into the runner in ELF constructors.
+#
+# Note that we must do double expansion here in order to support conditionally
+# compiling BPF object files only if one is present, as the wildcard Make
+# function doesn't support using implicit rules otherwise.
+$(testcase-targets): $(SCXOBJ_DIR)/%.o: %.c $(SCXOBJ_DIR)/runner.o $(all_test_bpfprogs) | $(SCXOBJ_DIR)
+ $(eval test=$(patsubst %.o,%.c,$(notdir $@)))
+ $(CC) $(CFLAGS) -c $< -o $@ $(SCXOBJ_DIR)/runner.o
+
+$(SCXOBJ_DIR)/util.o: util.c | $(SCXOBJ_DIR)
+ $(CC) $(CFLAGS) -c $< -o $@
+
+runner: $(SCXOBJ_DIR)/runner.o $(SCXOBJ_DIR)/util.o $(BPFOBJ) $(testcase-targets)
+ @echo "$(testcase-targets)"
+ $(CC) $(CFLAGS) -o $@ $^ $(LDFLAGS)
+
+TEST_GEN_PROGS := runner
+
+all: runner
+
+.PHONY: all clean help
+
+.DEFAULT_GOAL := all
+
+.DELETE_ON_ERROR:
+
+.SECONDARY:
diff --git a/tools/testing/selftests/sched_ext/config b/tools/testing/selftests/sched_ext/config
new file mode 100644
index 000000000000..0de9b4ee249d
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/config
@@ -0,0 +1,9 @@
+CONFIG_SCHED_DEBUG=y
+CONFIG_SCHED_CLASS_EXT=y
+CONFIG_CGROUPS=y
+CONFIG_CGROUP_SCHED=y
+CONFIG_EXT_GROUP_SCHED=y
+CONFIG_BPF=y
+CONFIG_BPF_SYSCALL=y
+CONFIG_DEBUG_INFO=y
+CONFIG_DEBUG_INFO_BTF=y
diff --git a/tools/testing/selftests/sched_ext/create_dsq.bpf.c b/tools/testing/selftests/sched_ext/create_dsq.bpf.c
new file mode 100644
index 000000000000..23f79ed343f0
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/create_dsq.bpf.c
@@ -0,0 +1,58 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Create and destroy DSQs in a loop.
+ *
+ * Copyright (c) 2024 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2024 David Vernet <dvernet@meta.com>
+ */
+
+#include <scx/common.bpf.h>
+
+char _license[] SEC("license") = "GPL";
+
+void BPF_STRUCT_OPS(create_dsq_exit_task, struct task_struct *p,
+ struct scx_exit_task_args *args)
+{
+ scx_bpf_destroy_dsq(p->pid);
+}
+
+s32 BPF_STRUCT_OPS_SLEEPABLE(create_dsq_init_task, struct task_struct *p,
+ struct scx_init_task_args *args)
+{
+ s32 err;
+
+ err = scx_bpf_create_dsq(p->pid, -1);
+ if (err)
+ scx_bpf_error("Failed to create DSQ for %s[%d]",
+ p->comm, p->pid);
+
+ return err;
+}
+
+s32 BPF_STRUCT_OPS_SLEEPABLE(create_dsq_init)
+{
+ u32 i;
+ s32 err;
+
+ bpf_for(i, 0, 1024) {
+ err = scx_bpf_create_dsq(i, -1);
+ if (err) {
+ scx_bpf_error("Failed to create DSQ %d", i);
+ return 0;
+ }
+ }
+
+ bpf_for(i, 0, 1024) {
+ scx_bpf_destroy_dsq(i);
+ }
+
+ return 0;
+}
+
+SEC(".struct_ops.link")
+struct sched_ext_ops create_dsq_ops = {
+ .init_task = create_dsq_init_task,
+ .exit_task = create_dsq_exit_task,
+ .init = create_dsq_init,
+ .name = "create_dsq",
+};
diff --git a/tools/testing/selftests/sched_ext/create_dsq.c b/tools/testing/selftests/sched_ext/create_dsq.c
new file mode 100644
index 000000000000..fa946d9146d4
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/create_dsq.c
@@ -0,0 +1,57 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2024 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2024 David Vernet <dvernet@meta.com>
+ */
+#include <bpf/bpf.h>
+#include <scx/common.h>
+#include <sys/wait.h>
+#include <unistd.h>
+#include "create_dsq.bpf.skel.h"
+#include "scx_test.h"
+
+static enum scx_test_status setup(void **ctx)
+{
+ struct create_dsq *skel;
+
+ skel = create_dsq__open_and_load();
+ if (!skel) {
+ SCX_ERR("Failed to open and load skel");
+ return SCX_TEST_FAIL;
+ }
+ *ctx = skel;
+
+ return SCX_TEST_PASS;
+}
+
+static enum scx_test_status run(void *ctx)
+{
+ struct create_dsq *skel = ctx;
+ struct bpf_link *link;
+
+ link = bpf_map__attach_struct_ops(skel->maps.create_dsq_ops);
+ if (!link) {
+ SCX_ERR("Failed to attach scheduler");
+ return SCX_TEST_FAIL;
+ }
+
+ bpf_link__destroy(link);
+
+ return SCX_TEST_PASS;
+}
+
+static void cleanup(void *ctx)
+{
+ struct create_dsq *skel = ctx;
+
+ create_dsq__destroy(skel);
+}
+
+struct scx_test create_dsq = {
+ .name = "create_dsq",
+ .description = "Create and destroy a dsq in a loop",
+ .setup = setup,
+ .run = run,
+ .cleanup = cleanup,
+};
+REGISTER_SCX_TEST(&create_dsq)
diff --git a/tools/testing/selftests/sched_ext/ddsp_bogus_dsq_fail.bpf.c b/tools/testing/selftests/sched_ext/ddsp_bogus_dsq_fail.bpf.c
new file mode 100644
index 000000000000..e97ad41d354a
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/ddsp_bogus_dsq_fail.bpf.c
@@ -0,0 +1,42 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2024 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2024 David Vernet <dvernet@meta.com>
+ * Copyright (c) 2024 Tejun Heo <tj@kernel.org>
+ */
+#include <scx/common.bpf.h>
+
+char _license[] SEC("license") = "GPL";
+
+UEI_DEFINE(uei);
+
+s32 BPF_STRUCT_OPS(ddsp_bogus_dsq_fail_select_cpu, struct task_struct *p,
+ s32 prev_cpu, u64 wake_flags)
+{
+ s32 cpu = scx_bpf_pick_idle_cpu(p->cpus_ptr, 0);
+
+ if (cpu >= 0) {
+ /*
+ * If we dispatch to a bogus DSQ that will fall back to the
+ * builtin global DSQ, we fail gracefully.
+ */
+ scx_bpf_dispatch_vtime(p, 0xcafef00d, SCX_SLICE_DFL,
+ p->scx.dsq_vtime, 0);
+ return cpu;
+ }
+
+ return prev_cpu;
+}
+
+void BPF_STRUCT_OPS(ddsp_bogus_dsq_fail_exit, struct scx_exit_info *ei)
+{
+ UEI_RECORD(uei, ei);
+}
+
+SEC(".struct_ops.link")
+struct sched_ext_ops ddsp_bogus_dsq_fail_ops = {
+ .select_cpu = ddsp_bogus_dsq_fail_select_cpu,
+ .exit = ddsp_bogus_dsq_fail_exit,
+ .name = "ddsp_bogus_dsq_fail",
+ .timeout_ms = 1000U,
+};
diff --git a/tools/testing/selftests/sched_ext/ddsp_bogus_dsq_fail.c b/tools/testing/selftests/sched_ext/ddsp_bogus_dsq_fail.c
new file mode 100644
index 000000000000..e65d22f23f3b
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/ddsp_bogus_dsq_fail.c
@@ -0,0 +1,57 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2024 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2024 David Vernet <dvernet@meta.com>
+ * Copyright (c) 2024 Tejun Heo <tj@kernel.org>
+ */
+#include <bpf/bpf.h>
+#include <scx/common.h>
+#include <sys/wait.h>
+#include <unistd.h>
+#include "ddsp_bogus_dsq_fail.bpf.skel.h"
+#include "scx_test.h"
+
+static enum scx_test_status setup(void **ctx)
+{
+ struct ddsp_bogus_dsq_fail *skel;
+
+ skel = ddsp_bogus_dsq_fail__open_and_load();
+ SCX_FAIL_IF(!skel, "Failed to open and load skel");
+ *ctx = skel;
+
+ return SCX_TEST_PASS;
+}
+
+static enum scx_test_status run(void *ctx)
+{
+ struct ddsp_bogus_dsq_fail *skel = ctx;
+ struct bpf_link *link;
+
+ link = bpf_map__attach_struct_ops(skel->maps.ddsp_bogus_dsq_fail_ops);
+ SCX_FAIL_IF(!link, "Failed to attach struct_ops");
+
+ sleep(1);
+
+ SCX_EQ(skel->data->uei.kind, EXIT_KIND(SCX_EXIT_ERROR));
+ bpf_link__destroy(link);
+
+ return SCX_TEST_PASS;
+}
+
+static void cleanup(void *ctx)
+{
+ struct ddsp_bogus_dsq_fail *skel = ctx;
+
+ ddsp_bogus_dsq_fail__destroy(skel);
+}
+
+struct scx_test ddsp_bogus_dsq_fail = {
+ .name = "ddsp_bogus_dsq_fail",
+ .description = "Verify we gracefully fail, and fall back to using a "
+ "built-in DSQ, if we do a direct dispatch to an invalid"
+ " DSQ in ops.select_cpu()",
+ .setup = setup,
+ .run = run,
+ .cleanup = cleanup,
+};
+REGISTER_SCX_TEST(&ddsp_bogus_dsq_fail)
diff --git a/tools/testing/selftests/sched_ext/ddsp_vtimelocal_fail.bpf.c b/tools/testing/selftests/sched_ext/ddsp_vtimelocal_fail.bpf.c
new file mode 100644
index 000000000000..dde7e7dafbfb
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/ddsp_vtimelocal_fail.bpf.c
@@ -0,0 +1,39 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2024 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2024 David Vernet <dvernet@meta.com>
+ * Copyright (c) 2024 Tejun Heo <tj@kernel.org>
+ */
+#include <scx/common.bpf.h>
+
+char _license[] SEC("license") = "GPL";
+
+UEI_DEFINE(uei);
+
+s32 BPF_STRUCT_OPS(ddsp_vtimelocal_fail_select_cpu, struct task_struct *p,
+ s32 prev_cpu, u64 wake_flags)
+{
+ s32 cpu = scx_bpf_pick_idle_cpu(p->cpus_ptr, 0);
+
+ if (cpu >= 0) {
+ /* Shouldn't be allowed to vtime dispatch to a builtin DSQ. */
+ scx_bpf_dispatch_vtime(p, SCX_DSQ_LOCAL, SCX_SLICE_DFL,
+ p->scx.dsq_vtime, 0);
+ return cpu;
+ }
+
+ return prev_cpu;
+}
+
+void BPF_STRUCT_OPS(ddsp_vtimelocal_fail_exit, struct scx_exit_info *ei)
+{
+ UEI_RECORD(uei, ei);
+}
+
+SEC(".struct_ops.link")
+struct sched_ext_ops ddsp_vtimelocal_fail_ops = {
+ .select_cpu = ddsp_vtimelocal_fail_select_cpu,
+ .exit = ddsp_vtimelocal_fail_exit,
+ .name = "ddsp_vtimelocal_fail",
+ .timeout_ms = 1000U,
+};
diff --git a/tools/testing/selftests/sched_ext/ddsp_vtimelocal_fail.c b/tools/testing/selftests/sched_ext/ddsp_vtimelocal_fail.c
new file mode 100644
index 000000000000..abafee587cd6
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/ddsp_vtimelocal_fail.c
@@ -0,0 +1,56 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2024 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2024 David Vernet <dvernet@meta.com>
+ * Copyright (c) 2024 Tejun Heo <tj@kernel.org>
+ */
+#include <bpf/bpf.h>
+#include <scx/common.h>
+#include <unistd.h>
+#include "ddsp_vtimelocal_fail.bpf.skel.h"
+#include "scx_test.h"
+
+static enum scx_test_status setup(void **ctx)
+{
+ struct ddsp_vtimelocal_fail *skel;
+
+ skel = ddsp_vtimelocal_fail__open_and_load();
+ SCX_FAIL_IF(!skel, "Failed to open and load skel");
+ *ctx = skel;
+
+ return SCX_TEST_PASS;
+}
+
+static enum scx_test_status run(void *ctx)
+{
+ struct ddsp_vtimelocal_fail *skel = ctx;
+ struct bpf_link *link;
+
+ link = bpf_map__attach_struct_ops(skel->maps.ddsp_vtimelocal_fail_ops);
+ SCX_FAIL_IF(!link, "Failed to attach struct_ops");
+
+ sleep(1);
+
+ SCX_EQ(skel->data->uei.kind, EXIT_KIND(SCX_EXIT_ERROR));
+ bpf_link__destroy(link);
+
+ return SCX_TEST_PASS;
+}
+
+static void cleanup(void *ctx)
+{
+ struct ddsp_vtimelocal_fail *skel = ctx;
+
+ ddsp_vtimelocal_fail__destroy(skel);
+}
+
+struct scx_test ddsp_vtimelocal_fail = {
+ .name = "ddsp_vtimelocal_fail",
+ .description = "Verify we gracefully fail, and fall back to using a "
+ "built-in DSQ, if we do a direct vtime dispatch to a "
+ "built-in DSQ from DSQ in ops.select_cpu()",
+ .setup = setup,
+ .run = run,
+ .cleanup = cleanup,
+};
+REGISTER_SCX_TEST(&ddsp_vtimelocal_fail)
diff --git a/tools/testing/selftests/sched_ext/dsp_local_on.bpf.c b/tools/testing/selftests/sched_ext/dsp_local_on.bpf.c
new file mode 100644
index 000000000000..efb4672decb4
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/dsp_local_on.bpf.c
@@ -0,0 +1,65 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2024 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2024 David Vernet <dvernet@meta.com>
+ */
+#include <scx/common.bpf.h>
+
+char _license[] SEC("license") = "GPL";
+const volatile s32 nr_cpus;
+
+UEI_DEFINE(uei);
+
+struct {
+ __uint(type, BPF_MAP_TYPE_QUEUE);
+ __uint(max_entries, 8192);
+ __type(value, s32);
+} queue SEC(".maps");
+
+s32 BPF_STRUCT_OPS(dsp_local_on_select_cpu, struct task_struct *p,
+ s32 prev_cpu, u64 wake_flags)
+{
+ return prev_cpu;
+}
+
+void BPF_STRUCT_OPS(dsp_local_on_enqueue, struct task_struct *p,
+ u64 enq_flags)
+{
+ s32 pid = p->pid;
+
+ if (bpf_map_push_elem(&queue, &pid, 0))
+ scx_bpf_error("Failed to enqueue %s[%d]", p->comm, p->pid);
+}
+
+void BPF_STRUCT_OPS(dsp_local_on_dispatch, s32 cpu, struct task_struct *prev)
+{
+ s32 pid, target;
+ struct task_struct *p;
+
+ if (bpf_map_pop_elem(&queue, &pid))
+ return;
+
+ p = bpf_task_from_pid(pid);
+ if (!p)
+ return;
+
+ target = bpf_get_prandom_u32() % nr_cpus;
+
+ scx_bpf_dispatch(p, SCX_DSQ_LOCAL_ON | target, SCX_SLICE_DFL, 0);
+ bpf_task_release(p);
+}
+
+void BPF_STRUCT_OPS(dsp_local_on_exit, struct scx_exit_info *ei)
+{
+ UEI_RECORD(uei, ei);
+}
+
+SEC(".struct_ops.link")
+struct sched_ext_ops dsp_local_on_ops = {
+ .select_cpu = dsp_local_on_select_cpu,
+ .enqueue = dsp_local_on_enqueue,
+ .dispatch = dsp_local_on_dispatch,
+ .exit = dsp_local_on_exit,
+ .name = "dsp_local_on",
+ .timeout_ms = 1000U,
+};
diff --git a/tools/testing/selftests/sched_ext/dsp_local_on.c b/tools/testing/selftests/sched_ext/dsp_local_on.c
new file mode 100644
index 000000000000..472851b56854
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/dsp_local_on.c
@@ -0,0 +1,58 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2024 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2024 David Vernet <dvernet@meta.com>
+ */
+#include <bpf/bpf.h>
+#include <scx/common.h>
+#include <unistd.h>
+#include "dsp_local_on.bpf.skel.h"
+#include "scx_test.h"
+
+static enum scx_test_status setup(void **ctx)
+{
+ struct dsp_local_on *skel;
+
+ skel = dsp_local_on__open();
+ SCX_FAIL_IF(!skel, "Failed to open");
+
+ skel->rodata->nr_cpus = libbpf_num_possible_cpus();
+ SCX_FAIL_IF(dsp_local_on__load(skel), "Failed to load skel");
+ *ctx = skel;
+
+ return SCX_TEST_PASS;
+}
+
+static enum scx_test_status run(void *ctx)
+{
+ struct dsp_local_on *skel = ctx;
+ struct bpf_link *link;
+
+ link = bpf_map__attach_struct_ops(skel->maps.dsp_local_on_ops);
+ SCX_FAIL_IF(!link, "Failed to attach struct_ops");
+
+ /* Just sleeping is fine, plenty of scheduling events happening */
+ sleep(1);
+
+ SCX_EQ(skel->data->uei.kind, EXIT_KIND(SCX_EXIT_ERROR));
+ bpf_link__destroy(link);
+
+ return SCX_TEST_PASS;
+}
+
+static void cleanup(void *ctx)
+{
+ struct dsp_local_on *skel = ctx;
+
+ dsp_local_on__destroy(skel);
+}
+
+struct scx_test dsp_local_on = {
+ .name = "dsp_local_on",
+ .description = "Verify we can directly dispatch tasks to a local DSQs "
+ "from osp.dispatch()",
+ .setup = setup,
+ .run = run,
+ .cleanup = cleanup,
+};
+REGISTER_SCX_TEST(&dsp_local_on)
diff --git a/tools/testing/selftests/sched_ext/enq_last_no_enq_fails.bpf.c b/tools/testing/selftests/sched_ext/enq_last_no_enq_fails.bpf.c
new file mode 100644
index 000000000000..b0b99531d5d5
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/enq_last_no_enq_fails.bpf.c
@@ -0,0 +1,21 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * A scheduler that validates the behavior of direct dispatching with a default
+ * select_cpu implementation.
+ *
+ * Copyright (c) 2023 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2023 David Vernet <dvernet@meta.com>
+ * Copyright (c) 2023 Tejun Heo <tj@kernel.org>
+ */
+
+#include <scx/common.bpf.h>
+
+char _license[] SEC("license") = "GPL";
+
+SEC(".struct_ops.link")
+struct sched_ext_ops enq_last_no_enq_fails_ops = {
+ .name = "enq_last_no_enq_fails",
+ /* Need to define ops.enqueue() with SCX_OPS_ENQ_LAST */
+ .flags = SCX_OPS_ENQ_LAST,
+ .timeout_ms = 1000U,
+};
diff --git a/tools/testing/selftests/sched_ext/enq_last_no_enq_fails.c b/tools/testing/selftests/sched_ext/enq_last_no_enq_fails.c
new file mode 100644
index 000000000000..2a3eda5e2c0b
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/enq_last_no_enq_fails.c
@@ -0,0 +1,60 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2023 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2023 David Vernet <dvernet@meta.com>
+ * Copyright (c) 2023 Tejun Heo <tj@kernel.org>
+ */
+#include <bpf/bpf.h>
+#include <scx/common.h>
+#include <sys/wait.h>
+#include <unistd.h>
+#include "enq_last_no_enq_fails.bpf.skel.h"
+#include "scx_test.h"
+
+static enum scx_test_status setup(void **ctx)
+{
+ struct enq_last_no_enq_fails *skel;
+
+ skel = enq_last_no_enq_fails__open_and_load();
+ if (!skel) {
+ SCX_ERR("Failed to open and load skel");
+ return SCX_TEST_FAIL;
+ }
+ *ctx = skel;
+
+ return SCX_TEST_PASS;
+}
+
+static enum scx_test_status run(void *ctx)
+{
+ struct enq_last_no_enq_fails *skel = ctx;
+ struct bpf_link *link;
+
+ link = bpf_map__attach_struct_ops(skel->maps.enq_last_no_enq_fails_ops);
+ if (link) {
+ SCX_ERR("Incorrectly succeeded in to attaching scheduler");
+ return SCX_TEST_FAIL;
+ }
+
+ bpf_link__destroy(link);
+
+ return SCX_TEST_PASS;
+}
+
+static void cleanup(void *ctx)
+{
+ struct enq_last_no_enq_fails *skel = ctx;
+
+ enq_last_no_enq_fails__destroy(skel);
+}
+
+struct scx_test enq_last_no_enq_fails = {
+ .name = "enq_last_no_enq_fails",
+ .description = "Verify we fail to load a scheduler if we specify "
+ "the SCX_OPS_ENQ_LAST flag without defining "
+ "ops.enqueue()",
+ .setup = setup,
+ .run = run,
+ .cleanup = cleanup,
+};
+REGISTER_SCX_TEST(&enq_last_no_enq_fails)
diff --git a/tools/testing/selftests/sched_ext/enq_select_cpu_fails.bpf.c b/tools/testing/selftests/sched_ext/enq_select_cpu_fails.bpf.c
new file mode 100644
index 000000000000..b3dfc1033cd6
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/enq_select_cpu_fails.bpf.c
@@ -0,0 +1,43 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2023 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2023 David Vernet <dvernet@meta.com>
+ * Copyright (c) 2023 Tejun Heo <tj@kernel.org>
+ */
+
+#include <scx/common.bpf.h>
+
+char _license[] SEC("license") = "GPL";
+
+/* Manually specify the signature until the kfunc is added to the scx repo. */
+s32 scx_bpf_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags,
+ bool *found) __ksym;
+
+s32 BPF_STRUCT_OPS(enq_select_cpu_fails_select_cpu, struct task_struct *p,
+ s32 prev_cpu, u64 wake_flags)
+{
+ return prev_cpu;
+}
+
+void BPF_STRUCT_OPS(enq_select_cpu_fails_enqueue, struct task_struct *p,
+ u64 enq_flags)
+{
+ /*
+ * Need to initialize the variable or the verifier will fail to load.
+ * Improving these semantics is actively being worked on.
+ */
+ bool found = false;
+
+ /* Can only call from ops.select_cpu() */
+ scx_bpf_select_cpu_dfl(p, 0, 0, &found);
+
+ scx_bpf_dispatch(p, SCX_DSQ_GLOBAL, SCX_SLICE_DFL, enq_flags);
+}
+
+SEC(".struct_ops.link")
+struct sched_ext_ops enq_select_cpu_fails_ops = {
+ .select_cpu = enq_select_cpu_fails_select_cpu,
+ .enqueue = enq_select_cpu_fails_enqueue,
+ .name = "enq_select_cpu_fails",
+ .timeout_ms = 1000U,
+};
diff --git a/tools/testing/selftests/sched_ext/enq_select_cpu_fails.c b/tools/testing/selftests/sched_ext/enq_select_cpu_fails.c
new file mode 100644
index 000000000000..dd1350e5f002
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/enq_select_cpu_fails.c
@@ -0,0 +1,61 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2023 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2023 David Vernet <dvernet@meta.com>
+ * Copyright (c) 2023 Tejun Heo <tj@kernel.org>
+ */
+#include <bpf/bpf.h>
+#include <scx/common.h>
+#include <sys/wait.h>
+#include <unistd.h>
+#include "enq_select_cpu_fails.bpf.skel.h"
+#include "scx_test.h"
+
+static enum scx_test_status setup(void **ctx)
+{
+ struct enq_select_cpu_fails *skel;
+
+ skel = enq_select_cpu_fails__open_and_load();
+ if (!skel) {
+ SCX_ERR("Failed to open and load skel");
+ return SCX_TEST_FAIL;
+ }
+ *ctx = skel;
+
+ return SCX_TEST_PASS;
+}
+
+static enum scx_test_status run(void *ctx)
+{
+ struct enq_select_cpu_fails *skel = ctx;
+ struct bpf_link *link;
+
+ link = bpf_map__attach_struct_ops(skel->maps.enq_select_cpu_fails_ops);
+ if (!link) {
+ SCX_ERR("Failed to attach scheduler");
+ return SCX_TEST_FAIL;
+ }
+
+ sleep(1);
+
+ bpf_link__destroy(link);
+
+ return SCX_TEST_PASS;
+}
+
+static void cleanup(void *ctx)
+{
+ struct enq_select_cpu_fails *skel = ctx;
+
+ enq_select_cpu_fails__destroy(skel);
+}
+
+struct scx_test enq_select_cpu_fails = {
+ .name = "enq_select_cpu_fails",
+ .description = "Verify we fail to call scx_bpf_select_cpu_dfl() "
+ "from ops.enqueue()",
+ .setup = setup,
+ .run = run,
+ .cleanup = cleanup,
+};
+REGISTER_SCX_TEST(&enq_select_cpu_fails)
diff --git a/tools/testing/selftests/sched_ext/exit.bpf.c b/tools/testing/selftests/sched_ext/exit.bpf.c
new file mode 100644
index 000000000000..ae12ddaac921
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/exit.bpf.c
@@ -0,0 +1,84 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2024 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2024 David Vernet <dvernet@meta.com>
+ */
+
+#include <scx/common.bpf.h>
+
+char _license[] SEC("license") = "GPL";
+
+#include "exit_test.h"
+
+const volatile int exit_point;
+UEI_DEFINE(uei);
+
+#define EXIT_CLEANLY() scx_bpf_exit(exit_point, "%d", exit_point)
+
+s32 BPF_STRUCT_OPS(exit_select_cpu, struct task_struct *p,
+ s32 prev_cpu, u64 wake_flags)
+{
+ bool found;
+
+ if (exit_point == EXIT_SELECT_CPU)
+ EXIT_CLEANLY();
+
+ return scx_bpf_select_cpu_dfl(p, prev_cpu, wake_flags, &found);
+}
+
+void BPF_STRUCT_OPS(exit_enqueue, struct task_struct *p, u64 enq_flags)
+{
+ if (exit_point == EXIT_ENQUEUE)
+ EXIT_CLEANLY();
+
+ scx_bpf_dispatch(p, SCX_DSQ_GLOBAL, SCX_SLICE_DFL, enq_flags);
+}
+
+void BPF_STRUCT_OPS(exit_dispatch, s32 cpu, struct task_struct *p)
+{
+ if (exit_point == EXIT_DISPATCH)
+ EXIT_CLEANLY();
+
+ scx_bpf_consume(SCX_DSQ_GLOBAL);
+}
+
+void BPF_STRUCT_OPS(exit_enable, struct task_struct *p)
+{
+ if (exit_point == EXIT_ENABLE)
+ EXIT_CLEANLY();
+}
+
+s32 BPF_STRUCT_OPS(exit_init_task, struct task_struct *p,
+ struct scx_init_task_args *args)
+{
+ if (exit_point == EXIT_INIT_TASK)
+ EXIT_CLEANLY();
+
+ return 0;
+}
+
+void BPF_STRUCT_OPS(exit_exit, struct scx_exit_info *ei)
+{
+ UEI_RECORD(uei, ei);
+}
+
+s32 BPF_STRUCT_OPS_SLEEPABLE(exit_init)
+{
+ if (exit_point == EXIT_INIT)
+ EXIT_CLEANLY();
+
+ return 0;
+}
+
+SEC(".struct_ops.link")
+struct sched_ext_ops exit_ops = {
+ .select_cpu = exit_select_cpu,
+ .enqueue = exit_enqueue,
+ .dispatch = exit_dispatch,
+ .init_task = exit_init_task,
+ .enable = exit_enable,
+ .exit = exit_exit,
+ .init = exit_init,
+ .name = "exit",
+ .timeout_ms = 1000U,
+};
diff --git a/tools/testing/selftests/sched_ext/exit.c b/tools/testing/selftests/sched_ext/exit.c
new file mode 100644
index 000000000000..31bcd06e21cd
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/exit.c
@@ -0,0 +1,55 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2024 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2024 David Vernet <dvernet@meta.com>
+ */
+#include <bpf/bpf.h>
+#include <sched.h>
+#include <scx/common.h>
+#include <sys/wait.h>
+#include <unistd.h>
+#include "exit.bpf.skel.h"
+#include "scx_test.h"
+
+#include "exit_test.h"
+
+static enum scx_test_status run(void *ctx)
+{
+ enum exit_test_case tc;
+
+ for (tc = 0; tc < NUM_EXITS; tc++) {
+ struct exit *skel;
+ struct bpf_link *link;
+ char buf[16];
+
+ skel = exit__open();
+ skel->rodata->exit_point = tc;
+ exit__load(skel);
+ link = bpf_map__attach_struct_ops(skel->maps.exit_ops);
+ if (!link) {
+ SCX_ERR("Failed to attach scheduler");
+ exit__destroy(skel);
+ return SCX_TEST_FAIL;
+ }
+
+ /* Assumes uei.kind is written last */
+ while (skel->data->uei.kind == EXIT_KIND(SCX_EXIT_NONE))
+ sched_yield();
+
+ SCX_EQ(skel->data->uei.kind, EXIT_KIND(SCX_EXIT_UNREG_BPF));
+ SCX_EQ(skel->data->uei.exit_code, tc);
+ sprintf(buf, "%d", tc);
+ SCX_ASSERT(!strcmp(skel->data->uei.msg, buf));
+ bpf_link__destroy(link);
+ exit__destroy(skel);
+ }
+
+ return SCX_TEST_PASS;
+}
+
+struct scx_test exit_test = {
+ .name = "exit",
+ .description = "Verify we can cleanly exit a scheduler in multiple places",
+ .run = run,
+};
+REGISTER_SCX_TEST(&exit_test)
diff --git a/tools/testing/selftests/sched_ext/exit_test.h b/tools/testing/selftests/sched_ext/exit_test.h
new file mode 100644
index 000000000000..94f0268b9cb8
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/exit_test.h
@@ -0,0 +1,20 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2024 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2024 David Vernet <dvernet@meta.com>
+ */
+
+#ifndef __EXIT_TEST_H__
+#define __EXIT_TEST_H__
+
+enum exit_test_case {
+ EXIT_SELECT_CPU,
+ EXIT_ENQUEUE,
+ EXIT_DISPATCH,
+ EXIT_ENABLE,
+ EXIT_INIT_TASK,
+ EXIT_INIT,
+ NUM_EXITS,
+};
+
+#endif // # __EXIT_TEST_H__
diff --git a/tools/testing/selftests/sched_ext/hotplug.bpf.c b/tools/testing/selftests/sched_ext/hotplug.bpf.c
new file mode 100644
index 000000000000..8f2601db39f3
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/hotplug.bpf.c
@@ -0,0 +1,61 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2024 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2024 David Vernet <dvernet@meta.com>
+ */
+
+#include <scx/common.bpf.h>
+
+char _license[] SEC("license") = "GPL";
+
+#include "hotplug_test.h"
+
+UEI_DEFINE(uei);
+
+void BPF_STRUCT_OPS(hotplug_exit, struct scx_exit_info *ei)
+{
+ UEI_RECORD(uei, ei);
+}
+
+static void exit_from_hotplug(s32 cpu, bool onlining)
+{
+ /*
+ * Ignored, just used to verify that we can invoke blocking kfuncs
+ * from the hotplug path.
+ */
+ scx_bpf_create_dsq(0, -1);
+
+ s64 code = SCX_ECODE_ACT_RESTART | HOTPLUG_EXIT_RSN;
+
+ if (onlining)
+ code |= HOTPLUG_ONLINING;
+
+ scx_bpf_exit(code, "hotplug event detected (%d going %s)", cpu,
+ onlining ? "online" : "offline");
+}
+
+void BPF_STRUCT_OPS_SLEEPABLE(hotplug_cpu_online, s32 cpu)
+{
+ exit_from_hotplug(cpu, true);
+}
+
+void BPF_STRUCT_OPS_SLEEPABLE(hotplug_cpu_offline, s32 cpu)
+{
+ exit_from_hotplug(cpu, false);
+}
+
+SEC(".struct_ops.link")
+struct sched_ext_ops hotplug_cb_ops = {
+ .cpu_online = hotplug_cpu_online,
+ .cpu_offline = hotplug_cpu_offline,
+ .exit = hotplug_exit,
+ .name = "hotplug_cbs",
+ .timeout_ms = 1000U,
+};
+
+SEC(".struct_ops.link")
+struct sched_ext_ops hotplug_nocb_ops = {
+ .exit = hotplug_exit,
+ .name = "hotplug_nocbs",
+ .timeout_ms = 1000U,
+};
diff --git a/tools/testing/selftests/sched_ext/hotplug.c b/tools/testing/selftests/sched_ext/hotplug.c
new file mode 100644
index 000000000000..87bf220b1bce
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/hotplug.c
@@ -0,0 +1,168 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2024 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2024 David Vernet <dvernet@meta.com>
+ */
+#include <bpf/bpf.h>
+#include <sched.h>
+#include <scx/common.h>
+#include <sched.h>
+#include <sys/wait.h>
+#include <unistd.h>
+
+#include "hotplug_test.h"
+#include "hotplug.bpf.skel.h"
+#include "scx_test.h"
+#include "util.h"
+
+const char *online_path = "/sys/devices/system/cpu/cpu1/online";
+
+static bool is_cpu_online(void)
+{
+ return file_read_long(online_path) > 0;
+}
+
+static void toggle_online_status(bool online)
+{
+ long val = online ? 1 : 0;
+ int ret;
+
+ ret = file_write_long(online_path, val);
+ if (ret != 0)
+ fprintf(stderr, "Failed to bring CPU %s (%s)",
+ online ? "online" : "offline", strerror(errno));
+}
+
+static enum scx_test_status setup(void **ctx)
+{
+ if (!is_cpu_online())
+ return SCX_TEST_SKIP;
+
+ return SCX_TEST_PASS;
+}
+
+static enum scx_test_status test_hotplug(bool onlining, bool cbs_defined)
+{
+ struct hotplug *skel;
+ struct bpf_link *link;
+ long kind, code;
+
+ SCX_ASSERT(is_cpu_online());
+
+ skel = hotplug__open_and_load();
+ SCX_ASSERT(skel);
+
+ /* Testing the offline -> online path, so go offline before starting */
+ if (onlining)
+ toggle_online_status(0);
+
+ if (cbs_defined) {
+ kind = SCX_KIND_VAL(SCX_EXIT_UNREG_BPF);
+ code = SCX_ECODE_VAL(SCX_ECODE_ACT_RESTART) | HOTPLUG_EXIT_RSN;
+ if (onlining)
+ code |= HOTPLUG_ONLINING;
+ } else {
+ kind = SCX_KIND_VAL(SCX_EXIT_UNREG_KERN);
+ code = SCX_ECODE_VAL(SCX_ECODE_ACT_RESTART) |
+ SCX_ECODE_VAL(SCX_ECODE_RSN_HOTPLUG);
+ }
+
+ if (cbs_defined)
+ link = bpf_map__attach_struct_ops(skel->maps.hotplug_cb_ops);
+ else
+ link = bpf_map__attach_struct_ops(skel->maps.hotplug_nocb_ops);
+
+ if (!link) {
+ SCX_ERR("Failed to attach scheduler");
+ hotplug__destroy(skel);
+ return SCX_TEST_FAIL;
+ }
+
+ toggle_online_status(onlining ? 1 : 0);
+
+ while (!UEI_EXITED(skel, uei))
+ sched_yield();
+
+ SCX_EQ(skel->data->uei.kind, kind);
+ SCX_EQ(UEI_REPORT(skel, uei), code);
+
+ if (!onlining)
+ toggle_online_status(1);
+
+ bpf_link__destroy(link);
+ hotplug__destroy(skel);
+
+ return SCX_TEST_PASS;
+}
+
+static enum scx_test_status test_hotplug_attach(void)
+{
+ struct hotplug *skel;
+ struct bpf_link *link;
+ enum scx_test_status status = SCX_TEST_PASS;
+ long kind, code;
+
+ SCX_ASSERT(is_cpu_online());
+ SCX_ASSERT(scx_hotplug_seq() > 0);
+
+ skel = SCX_OPS_OPEN(hotplug_nocb_ops, hotplug);
+ SCX_ASSERT(skel);
+
+ SCX_OPS_LOAD(skel, hotplug_nocb_ops, hotplug, uei);
+
+ /*
+ * Take the CPU offline to increment the global hotplug seq, which
+ * should cause attach to fail due to us setting the hotplug seq above
+ */
+ toggle_online_status(0);
+ link = bpf_map__attach_struct_ops(skel->maps.hotplug_nocb_ops);
+
+ toggle_online_status(1);
+
+ SCX_ASSERT(link);
+ while (!UEI_EXITED(skel, uei))
+ sched_yield();
+
+ kind = SCX_KIND_VAL(SCX_EXIT_UNREG_KERN);
+ code = SCX_ECODE_VAL(SCX_ECODE_ACT_RESTART) |
+ SCX_ECODE_VAL(SCX_ECODE_RSN_HOTPLUG);
+ SCX_EQ(skel->data->uei.kind, kind);
+ SCX_EQ(UEI_REPORT(skel, uei), code);
+
+ bpf_link__destroy(link);
+ hotplug__destroy(skel);
+
+ return status;
+}
+
+static enum scx_test_status run(void *ctx)
+{
+
+#define HP_TEST(__onlining, __cbs_defined) ({ \
+ if (test_hotplug(__onlining, __cbs_defined) != SCX_TEST_PASS) \
+ return SCX_TEST_FAIL; \
+})
+
+ HP_TEST(true, true);
+ HP_TEST(false, true);
+ HP_TEST(true, false);
+ HP_TEST(false, false);
+
+#undef HP_TEST
+
+ return test_hotplug_attach();
+}
+
+static void cleanup(void *ctx)
+{
+ toggle_online_status(1);
+}
+
+struct scx_test hotplug_test = {
+ .name = "hotplug",
+ .description = "Verify hotplug behavior",
+ .setup = setup,
+ .run = run,
+ .cleanup = cleanup,
+};
+REGISTER_SCX_TEST(&hotplug_test)
diff --git a/tools/testing/selftests/sched_ext/hotplug_test.h b/tools/testing/selftests/sched_ext/hotplug_test.h
new file mode 100644
index 000000000000..73d236f90787
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/hotplug_test.h
@@ -0,0 +1,15 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2024 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2024 David Vernet <dvernet@meta.com>
+ */
+
+#ifndef __HOTPLUG_TEST_H__
+#define __HOTPLUG_TEST_H__
+
+enum hotplug_test_flags {
+ HOTPLUG_EXIT_RSN = 1LLU << 0,
+ HOTPLUG_ONLINING = 1LLU << 1,
+};
+
+#endif // # __HOTPLUG_TEST_H__
diff --git a/tools/testing/selftests/sched_ext/init_enable_count.bpf.c b/tools/testing/selftests/sched_ext/init_enable_count.bpf.c
new file mode 100644
index 000000000000..47ea89a626c3
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/init_enable_count.bpf.c
@@ -0,0 +1,53 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * A scheduler that verifies that we do proper counting of init, enable, etc
+ * callbacks.
+ *
+ * Copyright (c) 2023 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2023 David Vernet <dvernet@meta.com>
+ * Copyright (c) 2023 Tejun Heo <tj@kernel.org>
+ */
+
+#include <scx/common.bpf.h>
+
+char _license[] SEC("license") = "GPL";
+
+u64 init_task_cnt, exit_task_cnt, enable_cnt, disable_cnt;
+u64 init_fork_cnt, init_transition_cnt;
+
+s32 BPF_STRUCT_OPS_SLEEPABLE(cnt_init_task, struct task_struct *p,
+ struct scx_init_task_args *args)
+{
+ __sync_fetch_and_add(&init_task_cnt, 1);
+
+ if (args->fork)
+ __sync_fetch_and_add(&init_fork_cnt, 1);
+ else
+ __sync_fetch_and_add(&init_transition_cnt, 1);
+
+ return 0;
+}
+
+void BPF_STRUCT_OPS(cnt_exit_task, struct task_struct *p)
+{
+ __sync_fetch_and_add(&exit_task_cnt, 1);
+}
+
+void BPF_STRUCT_OPS(cnt_enable, struct task_struct *p)
+{
+ __sync_fetch_and_add(&enable_cnt, 1);
+}
+
+void BPF_STRUCT_OPS(cnt_disable, struct task_struct *p)
+{
+ __sync_fetch_and_add(&disable_cnt, 1);
+}
+
+SEC(".struct_ops.link")
+struct sched_ext_ops init_enable_count_ops = {
+ .init_task = cnt_init_task,
+ .exit_task = cnt_exit_task,
+ .enable = cnt_enable,
+ .disable = cnt_disable,
+ .name = "init_enable_count",
+};
diff --git a/tools/testing/selftests/sched_ext/init_enable_count.c b/tools/testing/selftests/sched_ext/init_enable_count.c
new file mode 100644
index 000000000000..97d45f1e5597
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/init_enable_count.c
@@ -0,0 +1,166 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2023 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2023 David Vernet <dvernet@meta.com>
+ * Copyright (c) 2023 Tejun Heo <tj@kernel.org>
+ */
+#include <stdio.h>
+#include <unistd.h>
+#include <sched.h>
+#include <bpf/bpf.h>
+#include <scx/common.h>
+#include <sys/wait.h>
+#include "scx_test.h"
+#include "init_enable_count.bpf.skel.h"
+
+#define SCHED_EXT 7
+
+static struct init_enable_count *
+open_load_prog(bool global)
+{
+ struct init_enable_count *skel;
+
+ skel = init_enable_count__open();
+ SCX_BUG_ON(!skel, "Failed to open skel");
+
+ if (!global)
+ skel->struct_ops.init_enable_count_ops->flags |= SCX_OPS_SWITCH_PARTIAL;
+
+ SCX_BUG_ON(init_enable_count__load(skel), "Failed to load skel");
+
+ return skel;
+}
+
+static enum scx_test_status run_test(bool global)
+{
+ struct init_enable_count *skel;
+ struct bpf_link *link;
+ const u32 num_children = 5, num_pre_forks = 1024;
+ int ret, i, status;
+ struct sched_param param = {};
+ pid_t pids[num_pre_forks];
+
+ skel = open_load_prog(global);
+
+ /*
+ * Fork a bunch of children before we attach the scheduler so that we
+ * ensure (at least in practical terms) that there are more tasks that
+ * transition from SCHED_OTHER -> SCHED_EXT than there are tasks that
+ * take the fork() path either below or in other processes.
+ */
+ for (i = 0; i < num_pre_forks; i++) {
+ pids[i] = fork();
+ SCX_FAIL_IF(pids[i] < 0, "Failed to fork child");
+ if (pids[i] == 0) {
+ sleep(1);
+ exit(0);
+ }
+ }
+
+ link = bpf_map__attach_struct_ops(skel->maps.init_enable_count_ops);
+ SCX_FAIL_IF(!link, "Failed to attach struct_ops");
+
+ for (i = 0; i < num_pre_forks; i++) {
+ SCX_FAIL_IF(waitpid(pids[i], &status, 0) != pids[i],
+ "Failed to wait for pre-forked child\n");
+
+ SCX_FAIL_IF(status != 0, "Pre-forked child %d exited with status %d\n", i,
+ status);
+ }
+
+ bpf_link__destroy(link);
+ SCX_GE(skel->bss->init_task_cnt, num_pre_forks);
+ SCX_GE(skel->bss->exit_task_cnt, num_pre_forks);
+
+ link = bpf_map__attach_struct_ops(skel->maps.init_enable_count_ops);
+ SCX_FAIL_IF(!link, "Failed to attach struct_ops");
+
+ /* SCHED_EXT children */
+ for (i = 0; i < num_children; i++) {
+ pids[i] = fork();
+ SCX_FAIL_IF(pids[i] < 0, "Failed to fork child");
+
+ if (pids[i] == 0) {
+ ret = sched_setscheduler(0, SCHED_EXT, ¶m);
+ SCX_BUG_ON(ret, "Failed to set sched to sched_ext");
+
+ /*
+ * Reset to SCHED_OTHER for half of them. Counts for
+ * everything should still be the same regardless, as
+ * ops.disable() is invoked even if a task is still on
+ * SCHED_EXT before it exits.
+ */
+ if (i % 2 == 0) {
+ ret = sched_setscheduler(0, SCHED_OTHER, ¶m);
+ SCX_BUG_ON(ret, "Failed to reset sched to normal");
+ }
+ exit(0);
+ }
+ }
+ for (i = 0; i < num_children; i++) {
+ SCX_FAIL_IF(waitpid(pids[i], &status, 0) != pids[i],
+ "Failed to wait for sched_ext child\n");
+
+ SCX_FAIL_IF(status != 0, "sched_ext child %d exited with status %d\n", i,
+ status);
+ }
+
+ /* SCHED_OTHER children */
+ for (i = 0; i < num_children; i++) {
+ pids[i] = fork();
+ if (pids[i] == 0)
+ exit(0);
+ }
+
+ for (i = 0; i < num_children; i++) {
+ SCX_FAIL_IF(waitpid(pids[i], &status, 0) != pids[i],
+ "Failed to wait for normal child\n");
+
+ SCX_FAIL_IF(status != 0, "Normal child %d exited with status %d\n", i,
+ status);
+ }
+
+ bpf_link__destroy(link);
+
+ SCX_GE(skel->bss->init_task_cnt, 2 * num_children);
+ SCX_GE(skel->bss->exit_task_cnt, 2 * num_children);
+
+ if (global) {
+ SCX_GE(skel->bss->enable_cnt, 2 * num_children);
+ SCX_GE(skel->bss->disable_cnt, 2 * num_children);
+ } else {
+ SCX_EQ(skel->bss->enable_cnt, num_children);
+ SCX_EQ(skel->bss->disable_cnt, num_children);
+ }
+ /*
+ * We forked a ton of tasks before we attached the scheduler above, so
+ * this should be fine. Technically it could be flaky if a ton of forks
+ * are happening at the same time in other processes, but that should
+ * be exceedingly unlikely.
+ */
+ SCX_GT(skel->bss->init_transition_cnt, skel->bss->init_fork_cnt);
+ SCX_GE(skel->bss->init_fork_cnt, 2 * num_children);
+
+ init_enable_count__destroy(skel);
+
+ return SCX_TEST_PASS;
+}
+
+static enum scx_test_status run(void *ctx)
+{
+ enum scx_test_status status;
+
+ status = run_test(true);
+ if (status != SCX_TEST_PASS)
+ return status;
+
+ return run_test(false);
+}
+
+struct scx_test init_enable_count = {
+ .name = "init_enable_count",
+ .description = "Verify we do the correct amount of counting of init, "
+ "enable, etc callbacks.",
+ .run = run,
+};
+REGISTER_SCX_TEST(&init_enable_count)
diff --git a/tools/testing/selftests/sched_ext/maximal.bpf.c b/tools/testing/selftests/sched_ext/maximal.bpf.c
new file mode 100644
index 000000000000..44612fdaf399
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/maximal.bpf.c
@@ -0,0 +1,132 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * A scheduler with every callback defined.
+ *
+ * This scheduler defines every callback.
+ *
+ * Copyright (c) 2024 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2024 David Vernet <dvernet@meta.com>
+ */
+
+#include <scx/common.bpf.h>
+
+char _license[] SEC("license") = "GPL";
+
+s32 BPF_STRUCT_OPS(maximal_select_cpu, struct task_struct *p, s32 prev_cpu,
+ u64 wake_flags)
+{
+ return prev_cpu;
+}
+
+void BPF_STRUCT_OPS(maximal_enqueue, struct task_struct *p, u64 enq_flags)
+{
+ scx_bpf_dispatch(p, SCX_DSQ_GLOBAL, SCX_SLICE_DFL, enq_flags);
+}
+
+void BPF_STRUCT_OPS(maximal_dequeue, struct task_struct *p, u64 deq_flags)
+{}
+
+void BPF_STRUCT_OPS(maximal_dispatch, s32 cpu, struct task_struct *prev)
+{
+ scx_bpf_consume(SCX_DSQ_GLOBAL);
+}
+
+void BPF_STRUCT_OPS(maximal_runnable, struct task_struct *p, u64 enq_flags)
+{}
+
+void BPF_STRUCT_OPS(maximal_running, struct task_struct *p)
+{}
+
+void BPF_STRUCT_OPS(maximal_stopping, struct task_struct *p, bool runnable)
+{}
+
+void BPF_STRUCT_OPS(maximal_quiescent, struct task_struct *p, u64 deq_flags)
+{}
+
+bool BPF_STRUCT_OPS(maximal_yield, struct task_struct *from,
+ struct task_struct *to)
+{
+ return false;
+}
+
+bool BPF_STRUCT_OPS(maximal_core_sched_before, struct task_struct *a,
+ struct task_struct *b)
+{
+ return false;
+}
+
+void BPF_STRUCT_OPS(maximal_set_weight, struct task_struct *p, u32 weight)
+{}
+
+void BPF_STRUCT_OPS(maximal_set_cpumask, struct task_struct *p,
+ const struct cpumask *cpumask)
+{}
+
+void BPF_STRUCT_OPS(maximal_update_idle, s32 cpu, bool idle)
+{}
+
+void BPF_STRUCT_OPS(maximal_cpu_acquire, s32 cpu,
+ struct scx_cpu_acquire_args *args)
+{}
+
+void BPF_STRUCT_OPS(maximal_cpu_release, s32 cpu,
+ struct scx_cpu_release_args *args)
+{}
+
+void BPF_STRUCT_OPS(maximal_cpu_online, s32 cpu)
+{}
+
+void BPF_STRUCT_OPS(maximal_cpu_offline, s32 cpu)
+{}
+
+s32 BPF_STRUCT_OPS(maximal_init_task, struct task_struct *p,
+ struct scx_init_task_args *args)
+{
+ return 0;
+}
+
+void BPF_STRUCT_OPS(maximal_enable, struct task_struct *p)
+{}
+
+void BPF_STRUCT_OPS(maximal_exit_task, struct task_struct *p,
+ struct scx_exit_task_args *args)
+{}
+
+void BPF_STRUCT_OPS(maximal_disable, struct task_struct *p)
+{}
+
+s32 BPF_STRUCT_OPS_SLEEPABLE(maximal_init)
+{
+ return 0;
+}
+
+void BPF_STRUCT_OPS(maximal_exit, struct scx_exit_info *info)
+{}
+
+SEC(".struct_ops.link")
+struct sched_ext_ops maximal_ops = {
+ .select_cpu = maximal_select_cpu,
+ .enqueue = maximal_enqueue,
+ .dequeue = maximal_dequeue,
+ .dispatch = maximal_dispatch,
+ .runnable = maximal_runnable,
+ .running = maximal_running,
+ .stopping = maximal_stopping,
+ .quiescent = maximal_quiescent,
+ .yield = maximal_yield,
+ .core_sched_before = maximal_core_sched_before,
+ .set_weight = maximal_set_weight,
+ .set_cpumask = maximal_set_cpumask,
+ .update_idle = maximal_update_idle,
+ .cpu_acquire = maximal_cpu_acquire,
+ .cpu_release = maximal_cpu_release,
+ .cpu_online = maximal_cpu_online,
+ .cpu_offline = maximal_cpu_offline,
+ .init_task = maximal_init_task,
+ .enable = maximal_enable,
+ .exit_task = maximal_exit_task,
+ .disable = maximal_disable,
+ .init = maximal_init,
+ .exit = maximal_exit,
+ .name = "maximal",
+};
diff --git a/tools/testing/selftests/sched_ext/maximal.c b/tools/testing/selftests/sched_ext/maximal.c
new file mode 100644
index 000000000000..f38fc973c380
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/maximal.c
@@ -0,0 +1,51 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2024 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2024 David Vernet <dvernet@meta.com>
+ */
+#include <bpf/bpf.h>
+#include <scx/common.h>
+#include <sys/wait.h>
+#include <unistd.h>
+#include "maximal.bpf.skel.h"
+#include "scx_test.h"
+
+static enum scx_test_status setup(void **ctx)
+{
+ struct maximal *skel;
+
+ skel = maximal__open_and_load();
+ SCX_FAIL_IF(!skel, "Failed to open and load skel");
+ *ctx = skel;
+
+ return SCX_TEST_PASS;
+}
+
+static enum scx_test_status run(void *ctx)
+{
+ struct maximal *skel = ctx;
+ struct bpf_link *link;
+
+ link = bpf_map__attach_struct_ops(skel->maps.maximal_ops);
+ SCX_FAIL_IF(!link, "Failed to attach scheduler");
+
+ bpf_link__destroy(link);
+
+ return SCX_TEST_PASS;
+}
+
+static void cleanup(void *ctx)
+{
+ struct maximal *skel = ctx;
+
+ maximal__destroy(skel);
+}
+
+struct scx_test maximal = {
+ .name = "maximal",
+ .description = "Verify we can load a scheduler with every callback defined",
+ .setup = setup,
+ .run = run,
+ .cleanup = cleanup,
+};
+REGISTER_SCX_TEST(&maximal)
diff --git a/tools/testing/selftests/sched_ext/maybe_null.bpf.c b/tools/testing/selftests/sched_ext/maybe_null.bpf.c
new file mode 100644
index 000000000000..27d0f386acfb
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/maybe_null.bpf.c
@@ -0,0 +1,36 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2024 Meta Platforms, Inc. and affiliates.
+ */
+
+#include <scx/common.bpf.h>
+
+char _license[] SEC("license") = "GPL";
+
+u64 vtime_test;
+
+void BPF_STRUCT_OPS(maybe_null_running, struct task_struct *p)
+{}
+
+void BPF_STRUCT_OPS(maybe_null_success_dispatch, s32 cpu, struct task_struct *p)
+{
+ if (p != NULL)
+ vtime_test = p->scx.dsq_vtime;
+}
+
+bool BPF_STRUCT_OPS(maybe_null_success_yield, struct task_struct *from,
+ struct task_struct *to)
+{
+ if (to)
+ bpf_printk("Yielding to %s[%d]", to->comm, to->pid);
+
+ return false;
+}
+
+SEC(".struct_ops.link")
+struct sched_ext_ops maybe_null_success = {
+ .dispatch = maybe_null_success_dispatch,
+ .yield = maybe_null_success_yield,
+ .enable = maybe_null_running,
+ .name = "minimal",
+};
diff --git a/tools/testing/selftests/sched_ext/maybe_null.c b/tools/testing/selftests/sched_ext/maybe_null.c
new file mode 100644
index 000000000000..31cfafb0cf65
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/maybe_null.c
@@ -0,0 +1,49 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2024 Meta Platforms, Inc. and affiliates.
+ */
+#include <bpf/bpf.h>
+#include <scx/common.h>
+#include <sys/wait.h>
+#include <unistd.h>
+#include "maybe_null.bpf.skel.h"
+#include "maybe_null_fail_dsp.bpf.skel.h"
+#include "maybe_null_fail_yld.bpf.skel.h"
+#include "scx_test.h"
+
+static enum scx_test_status run(void *ctx)
+{
+ struct maybe_null *skel;
+ struct maybe_null_fail_dsp *fail_dsp;
+ struct maybe_null_fail_yld *fail_yld;
+
+ skel = maybe_null__open_and_load();
+ if (!skel) {
+ SCX_ERR("Failed to open and load maybe_null skel");
+ return SCX_TEST_FAIL;
+ }
+ maybe_null__destroy(skel);
+
+ fail_dsp = maybe_null_fail_dsp__open_and_load();
+ if (fail_dsp) {
+ maybe_null_fail_dsp__destroy(fail_dsp);
+ SCX_ERR("Should failed to open and load maybe_null_fail_dsp skel");
+ return SCX_TEST_FAIL;
+ }
+
+ fail_yld = maybe_null_fail_yld__open_and_load();
+ if (fail_yld) {
+ maybe_null_fail_yld__destroy(fail_yld);
+ SCX_ERR("Should failed to open and load maybe_null_fail_yld skel");
+ return SCX_TEST_FAIL;
+ }
+
+ return SCX_TEST_PASS;
+}
+
+struct scx_test maybe_null = {
+ .name = "maybe_null",
+ .description = "Verify if PTR_MAYBE_NULL work for .dispatch",
+ .run = run,
+};
+REGISTER_SCX_TEST(&maybe_null)
diff --git a/tools/testing/selftests/sched_ext/maybe_null_fail_dsp.bpf.c b/tools/testing/selftests/sched_ext/maybe_null_fail_dsp.bpf.c
new file mode 100644
index 000000000000..c0641050271d
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/maybe_null_fail_dsp.bpf.c
@@ -0,0 +1,25 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2024 Meta Platforms, Inc. and affiliates.
+ */
+
+#include <scx/common.bpf.h>
+
+char _license[] SEC("license") = "GPL";
+
+u64 vtime_test;
+
+void BPF_STRUCT_OPS(maybe_null_running, struct task_struct *p)
+{}
+
+void BPF_STRUCT_OPS(maybe_null_fail_dispatch, s32 cpu, struct task_struct *p)
+{
+ vtime_test = p->scx.dsq_vtime;
+}
+
+SEC(".struct_ops.link")
+struct sched_ext_ops maybe_null_fail = {
+ .dispatch = maybe_null_fail_dispatch,
+ .enable = maybe_null_running,
+ .name = "maybe_null_fail_dispatch",
+};
diff --git a/tools/testing/selftests/sched_ext/maybe_null_fail_yld.bpf.c b/tools/testing/selftests/sched_ext/maybe_null_fail_yld.bpf.c
new file mode 100644
index 000000000000..3c1740028e3b
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/maybe_null_fail_yld.bpf.c
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2024 Meta Platforms, Inc. and affiliates.
+ */
+
+#include <scx/common.bpf.h>
+
+char _license[] SEC("license") = "GPL";
+
+u64 vtime_test;
+
+void BPF_STRUCT_OPS(maybe_null_running, struct task_struct *p)
+{}
+
+bool BPF_STRUCT_OPS(maybe_null_fail_yield, struct task_struct *from,
+ struct task_struct *to)
+{
+ bpf_printk("Yielding to %s[%d]", to->comm, to->pid);
+
+ return false;
+}
+
+SEC(".struct_ops.link")
+struct sched_ext_ops maybe_null_fail = {
+ .yield = maybe_null_fail_yield,
+ .enable = maybe_null_running,
+ .name = "maybe_null_fail_yield",
+};
diff --git a/tools/testing/selftests/sched_ext/minimal.bpf.c b/tools/testing/selftests/sched_ext/minimal.bpf.c
new file mode 100644
index 000000000000..6a7eccef0104
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/minimal.bpf.c
@@ -0,0 +1,21 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * A completely minimal scheduler.
+ *
+ * This scheduler defines the absolute minimal set of struct sched_ext_ops
+ * fields: its name. It should _not_ fail to be loaded, and can be used to
+ * exercise the default scheduling paths in ext.c.
+ *
+ * Copyright (c) 2023 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2023 David Vernet <dvernet@meta.com>
+ * Copyright (c) 2023 Tejun Heo <tj@kernel.org>
+ */
+
+#include <scx/common.bpf.h>
+
+char _license[] SEC("license") = "GPL";
+
+SEC(".struct_ops.link")
+struct sched_ext_ops minimal_ops = {
+ .name = "minimal",
+};
diff --git a/tools/testing/selftests/sched_ext/minimal.c b/tools/testing/selftests/sched_ext/minimal.c
new file mode 100644
index 000000000000..6c5db8ebbf8a
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/minimal.c
@@ -0,0 +1,58 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2023 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2023 David Vernet <dvernet@meta.com>
+ * Copyright (c) 2023 Tejun Heo <tj@kernel.org>
+ */
+#include <bpf/bpf.h>
+#include <scx/common.h>
+#include <sys/wait.h>
+#include <unistd.h>
+#include "minimal.bpf.skel.h"
+#include "scx_test.h"
+
+static enum scx_test_status setup(void **ctx)
+{
+ struct minimal *skel;
+
+ skel = minimal__open_and_load();
+ if (!skel) {
+ SCX_ERR("Failed to open and load skel");
+ return SCX_TEST_FAIL;
+ }
+ *ctx = skel;
+
+ return SCX_TEST_PASS;
+}
+
+static enum scx_test_status run(void *ctx)
+{
+ struct minimal *skel = ctx;
+ struct bpf_link *link;
+
+ link = bpf_map__attach_struct_ops(skel->maps.minimal_ops);
+ if (!link) {
+ SCX_ERR("Failed to attach scheduler");
+ return SCX_TEST_FAIL;
+ }
+
+ bpf_link__destroy(link);
+
+ return SCX_TEST_PASS;
+}
+
+static void cleanup(void *ctx)
+{
+ struct minimal *skel = ctx;
+
+ minimal__destroy(skel);
+}
+
+struct scx_test minimal = {
+ .name = "minimal",
+ .description = "Verify we can load a fully minimal scheduler",
+ .setup = setup,
+ .run = run,
+ .cleanup = cleanup,
+};
+REGISTER_SCX_TEST(&minimal)
diff --git a/tools/testing/selftests/sched_ext/prog_run.bpf.c b/tools/testing/selftests/sched_ext/prog_run.bpf.c
new file mode 100644
index 000000000000..fd2c8f12af16
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/prog_run.bpf.c
@@ -0,0 +1,32 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * A scheduler that validates that we can invoke sched_ext kfuncs in
+ * BPF_PROG_TYPE_SYSCALL programs.
+ *
+ * Copyright (c) 2024 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2024 David Vernet <dvernet@meta.com>
+ */
+
+#include <scx/common.bpf.h>
+
+UEI_DEFINE(uei);
+
+char _license[] SEC("license") = "GPL";
+
+SEC("syscall")
+int BPF_PROG(prog_run_syscall)
+{
+ scx_bpf_exit(0xdeadbeef, "Exited from PROG_RUN");
+ return 0;
+}
+
+void BPF_STRUCT_OPS(prog_run_exit, struct scx_exit_info *ei)
+{
+ UEI_RECORD(uei, ei);
+}
+
+SEC(".struct_ops.link")
+struct sched_ext_ops prog_run_ops = {
+ .exit = prog_run_exit,
+ .name = "prog_run",
+};
diff --git a/tools/testing/selftests/sched_ext/prog_run.c b/tools/testing/selftests/sched_ext/prog_run.c
new file mode 100644
index 000000000000..3cd57ef8daaa
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/prog_run.c
@@ -0,0 +1,78 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2024 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2024 David Vernet <dvernet@meta.com>
+ */
+#include <bpf/bpf.h>
+#include <sched.h>
+#include <scx/common.h>
+#include <sys/wait.h>
+#include <unistd.h>
+#include "prog_run.bpf.skel.h"
+#include "scx_test.h"
+
+static enum scx_test_status setup(void **ctx)
+{
+ struct prog_run *skel;
+
+ skel = prog_run__open_and_load();
+ if (!skel) {
+ SCX_ERR("Failed to open and load skel");
+ return SCX_TEST_FAIL;
+ }
+ *ctx = skel;
+
+ return SCX_TEST_PASS;
+}
+
+static enum scx_test_status run(void *ctx)
+{
+ struct prog_run *skel = ctx;
+ struct bpf_link *link;
+ int prog_fd, err = 0;
+
+ prog_fd = bpf_program__fd(skel->progs.prog_run_syscall);
+ if (prog_fd < 0) {
+ SCX_ERR("Failed to get BPF_PROG_RUN prog");
+ return SCX_TEST_FAIL;
+ }
+
+ LIBBPF_OPTS(bpf_test_run_opts, topts);
+
+ link = bpf_map__attach_struct_ops(skel->maps.prog_run_ops);
+ if (!link) {
+ SCX_ERR("Failed to attach scheduler");
+ close(prog_fd);
+ return SCX_TEST_FAIL;
+ }
+
+ err = bpf_prog_test_run_opts(prog_fd, &topts);
+ SCX_EQ(err, 0);
+
+ /* Assumes uei.kind is written last */
+ while (skel->data->uei.kind == EXIT_KIND(SCX_EXIT_NONE))
+ sched_yield();
+
+ SCX_EQ(skel->data->uei.kind, EXIT_KIND(SCX_EXIT_UNREG_BPF));
+ SCX_EQ(skel->data->uei.exit_code, 0xdeadbeef);
+ close(prog_fd);
+ bpf_link__destroy(link);
+
+ return SCX_TEST_PASS;
+}
+
+static void cleanup(void *ctx)
+{
+ struct prog_run *skel = ctx;
+
+ prog_run__destroy(skel);
+}
+
+struct scx_test prog_run = {
+ .name = "prog_run",
+ .description = "Verify we can call into a scheduler with BPF_PROG_RUN, and invoke kfuncs",
+ .setup = setup,
+ .run = run,
+ .cleanup = cleanup,
+};
+REGISTER_SCX_TEST(&prog_run)
diff --git a/tools/testing/selftests/sched_ext/reload_loop.c b/tools/testing/selftests/sched_ext/reload_loop.c
new file mode 100644
index 000000000000..5cfba2d6e056
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/reload_loop.c
@@ -0,0 +1,75 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2024 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2024 David Vernet <dvernet@meta.com>
+ */
+#include <bpf/bpf.h>
+#include <pthread.h>
+#include <scx/common.h>
+#include <sys/wait.h>
+#include <unistd.h>
+#include "maximal.bpf.skel.h"
+#include "scx_test.h"
+
+static struct maximal *skel;
+static pthread_t threads[2];
+
+bool force_exit = false;
+
+static enum scx_test_status setup(void **ctx)
+{
+ skel = maximal__open_and_load();
+ if (!skel) {
+ SCX_ERR("Failed to open and load skel");
+ return SCX_TEST_FAIL;
+ }
+
+ return SCX_TEST_PASS;
+}
+
+static void *do_reload_loop(void *arg)
+{
+ u32 i;
+
+ for (i = 0; i < 1024 && !force_exit; i++) {
+ struct bpf_link *link;
+
+ link = bpf_map__attach_struct_ops(skel->maps.maximal_ops);
+ if (link)
+ bpf_link__destroy(link);
+ }
+
+ return NULL;
+}
+
+static enum scx_test_status run(void *ctx)
+{
+ int err;
+ void *ret;
+
+ err = pthread_create(&threads[0], NULL, do_reload_loop, NULL);
+ SCX_FAIL_IF(err, "Failed to create thread 0");
+
+ err = pthread_create(&threads[1], NULL, do_reload_loop, NULL);
+ SCX_FAIL_IF(err, "Failed to create thread 1");
+
+ SCX_FAIL_IF(pthread_join(threads[0], &ret), "thread 0 failed");
+ SCX_FAIL_IF(pthread_join(threads[1], &ret), "thread 1 failed");
+
+ return SCX_TEST_PASS;
+}
+
+static void cleanup(void *ctx)
+{
+ force_exit = true;
+ maximal__destroy(skel);
+}
+
+struct scx_test reload_loop = {
+ .name = "reload_loop",
+ .description = "Stress test loading and unloading schedulers repeatedly in a tight loop",
+ .setup = setup,
+ .run = run,
+ .cleanup = cleanup,
+};
+REGISTER_SCX_TEST(&reload_loop)
diff --git a/tools/testing/selftests/sched_ext/runner.c b/tools/testing/selftests/sched_ext/runner.c
new file mode 100644
index 000000000000..eab48c7ff309
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/runner.c
@@ -0,0 +1,201 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2024 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2024 David Vernet <dvernet@meta.com>
+ * Copyright (c) 2024 Tejun Heo <tj@kernel.org>
+ */
+#include <stdio.h>
+#include <unistd.h>
+#include <signal.h>
+#include <libgen.h>
+#include <bpf/bpf.h>
+#include "scx_test.h"
+
+const char help_fmt[] =
+"The runner for sched_ext tests.\n"
+"\n"
+"The runner is statically linked against all testcases, and runs them all serially.\n"
+"It's required for the testcases to be serial, as only a single host-wide sched_ext\n"
+"scheduler may be loaded at any given time."
+"\n"
+"Usage: %s [-t TEST] [-h]\n"
+"\n"
+" -t TEST Only run tests whose name includes this string\n"
+" -s Include print output for skipped tests\n"
+" -q Don't print the test descriptions during run\n"
+" -h Display this help and exit\n";
+
+static volatile int exit_req;
+static bool quiet, print_skipped;
+
+#define MAX_SCX_TESTS 2048
+
+static struct scx_test __scx_tests[MAX_SCX_TESTS];
+static unsigned __scx_num_tests = 0;
+
+static void sigint_handler(int simple)
+{
+ exit_req = 1;
+}
+
+static void print_test_preamble(const struct scx_test *test, bool quiet)
+{
+ printf("===== START =====\n");
+ printf("TEST: %s\n", test->name);
+ if (!quiet)
+ printf("DESCRIPTION: %s\n", test->description);
+ printf("OUTPUT:\n");
+}
+
+static const char *status_to_result(enum scx_test_status status)
+{
+ switch (status) {
+ case SCX_TEST_PASS:
+ case SCX_TEST_SKIP:
+ return "ok";
+ case SCX_TEST_FAIL:
+ return "not ok";
+ default:
+ return "<UNKNOWN>";
+ }
+}
+
+static void print_test_result(const struct scx_test *test,
+ enum scx_test_status status,
+ unsigned int testnum)
+{
+ const char *result = status_to_result(status);
+ const char *directive = status == SCX_TEST_SKIP ? "SKIP " : "";
+
+ printf("%s %u %s # %s\n", result, testnum, test->name, directive);
+ printf("===== END =====\n");
+}
+
+static bool should_skip_test(const struct scx_test *test, const char * filter)
+{
+ return !strstr(test->name, filter);
+}
+
+static enum scx_test_status run_test(const struct scx_test *test)
+{
+ enum scx_test_status status;
+ void *context = NULL;
+
+ if (test->setup) {
+ status = test->setup(&context);
+ if (status != SCX_TEST_PASS)
+ return status;
+ }
+
+ status = test->run(context);
+
+ if (test->cleanup)
+ test->cleanup(context);
+
+ return status;
+}
+
+static bool test_valid(const struct scx_test *test)
+{
+ if (!test) {
+ fprintf(stderr, "NULL test detected\n");
+ return false;
+ }
+
+ if (!test->name) {
+ fprintf(stderr,
+ "Test with no name found. Must specify test name.\n");
+ return false;
+ }
+
+ if (!test->description) {
+ fprintf(stderr, "Test %s requires description.\n", test->name);
+ return false;
+ }
+
+ if (!test->run) {
+ fprintf(stderr, "Test %s has no run() callback\n", test->name);
+ return false;
+ }
+
+ return true;
+}
+
+int main(int argc, char **argv)
+{
+ const char *filter = NULL;
+ unsigned testnum = 0, i;
+ unsigned passed = 0, skipped = 0, failed = 0;
+ int opt;
+
+ signal(SIGINT, sigint_handler);
+ signal(SIGTERM, sigint_handler);
+
+ libbpf_set_strict_mode(LIBBPF_STRICT_ALL);
+
+ while ((opt = getopt(argc, argv, "qst:h")) != -1) {
+ switch (opt) {
+ case 'q':
+ quiet = true;
+ break;
+ case 's':
+ print_skipped = true;
+ break;
+ case 't':
+ filter = optarg;
+ break;
+ default:
+ fprintf(stderr, help_fmt, basename(argv[0]));
+ return opt != 'h';
+ }
+ }
+
+ for (i = 0; i < __scx_num_tests; i++) {
+ enum scx_test_status status;
+ struct scx_test *test = &__scx_tests[i];
+
+ if (filter && should_skip_test(test, filter)) {
+ /*
+ * Printing the skipped tests and their preambles can
+ * add a lot of noise to the runner output. Printing
+ * this is only really useful for CI, so let's skip it
+ * by default.
+ */
+ if (print_skipped) {
+ print_test_preamble(test, quiet);
+ print_test_result(test, SCX_TEST_SKIP, ++testnum);
+ }
+ continue;
+ }
+
+ print_test_preamble(test, quiet);
+ status = run_test(test);
+ print_test_result(test, status, ++testnum);
+ switch (status) {
+ case SCX_TEST_PASS:
+ passed++;
+ break;
+ case SCX_TEST_SKIP:
+ skipped++;
+ break;
+ case SCX_TEST_FAIL:
+ failed++;
+ break;
+ }
+ }
+ printf("\n\n=============================\n\n");
+ printf("RESULTS:\n\n");
+ printf("PASSED: %u\n", passed);
+ printf("SKIPPED: %u\n", skipped);
+ printf("FAILED: %u\n", failed);
+
+ return 0;
+}
+
+void scx_test_register(struct scx_test *test)
+{
+ SCX_BUG_ON(!test_valid(test), "Invalid test found");
+ SCX_BUG_ON(__scx_num_tests >= MAX_SCX_TESTS, "Maximum tests exceeded");
+
+ __scx_tests[__scx_num_tests++] = *test;
+}
diff --git a/tools/testing/selftests/sched_ext/scx_test.h b/tools/testing/selftests/sched_ext/scx_test.h
new file mode 100644
index 000000000000..90b8d6915bb7
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/scx_test.h
@@ -0,0 +1,131 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2023 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2023 Tejun Heo <tj@kernel.org>
+ * Copyright (c) 2023 David Vernet <dvernet@meta.com>
+ */
+
+#ifndef __SCX_TEST_H__
+#define __SCX_TEST_H__
+
+#include <errno.h>
+#include <scx/common.h>
+#include <scx/compat.h>
+
+enum scx_test_status {
+ SCX_TEST_PASS = 0,
+ SCX_TEST_SKIP,
+ SCX_TEST_FAIL,
+};
+
+#define EXIT_KIND(__ent) __COMPAT_ENUM_OR_ZERO("scx_exit_kind", #__ent)
+
+struct scx_test {
+ /**
+ * name - The name of the testcase.
+ */
+ const char *name;
+
+ /**
+ * description - A description of your testcase: what it tests and is
+ * meant to validate.
+ */
+ const char *description;
+
+ /*
+ * setup - Setup the test.
+ * @ctx: A pointer to a context object that will be passed to run and
+ * cleanup.
+ *
+ * An optional callback that allows a testcase to perform setup for its
+ * run. A test may return SCX_TEST_SKIP to skip the run.
+ */
+ enum scx_test_status (*setup)(void **ctx);
+
+ /*
+ * run - Run the test.
+ * @ctx: Context set in the setup() callback. If @ctx was not set in
+ * setup(), it is NULL.
+ *
+ * The main test. Callers should return one of:
+ *
+ * - SCX_TEST_PASS: Test passed
+ * - SCX_TEST_SKIP: Test should be skipped
+ * - SCX_TEST_FAIL: Test failed
+ *
+ * This callback must be defined.
+ */
+ enum scx_test_status (*run)(void *ctx);
+
+ /*
+ * cleanup - Perform cleanup following the test
+ * @ctx: Context set in the setup() callback. If @ctx was not set in
+ * setup(), it is NULL.
+ *
+ * An optional callback that allows a test to perform cleanup after
+ * being run. This callback is run even if the run() callback returns
+ * SCX_TEST_SKIP or SCX_TEST_FAIL. It is not run if setup() returns
+ * SCX_TEST_SKIP or SCX_TEST_FAIL.
+ */
+ void (*cleanup)(void *ctx);
+};
+
+void scx_test_register(struct scx_test *test);
+
+#define REGISTER_SCX_TEST(__test) \
+ __attribute__((constructor)) \
+ static void ___scxregister##__LINE__(void) \
+ { \
+ scx_test_register(__test); \
+ }
+
+#define SCX_ERR(__fmt, ...) \
+ do { \
+ fprintf(stderr, "ERR: %s:%d\n", __FILE__, __LINE__); \
+ fprintf(stderr, __fmt"\n", ##__VA_ARGS__); \
+ } while (0)
+
+#define SCX_FAIL(__fmt, ...) \
+ do { \
+ SCX_ERR(__fmt, ##__VA_ARGS__); \
+ return SCX_TEST_FAIL; \
+ } while (0)
+
+#define SCX_FAIL_IF(__cond, __fmt, ...) \
+ do { \
+ if (__cond) \
+ SCX_FAIL(__fmt, ##__VA_ARGS__); \
+ } while (0)
+
+#define SCX_GT(_x, _y) SCX_FAIL_IF((_x) <= (_y), "Expected %s > %s (%lu > %lu)", \
+ #_x, #_y, (u64)(_x), (u64)(_y))
+#define SCX_GE(_x, _y) SCX_FAIL_IF((_x) < (_y), "Expected %s >= %s (%lu >= %lu)", \
+ #_x, #_y, (u64)(_x), (u64)(_y))
+#define SCX_LT(_x, _y) SCX_FAIL_IF((_x) >= (_y), "Expected %s < %s (%lu < %lu)", \
+ #_x, #_y, (u64)(_x), (u64)(_y))
+#define SCX_LE(_x, _y) SCX_FAIL_IF((_x) > (_y), "Expected %s <= %s (%lu <= %lu)", \
+ #_x, #_y, (u64)(_x), (u64)(_y))
+#define SCX_EQ(_x, _y) SCX_FAIL_IF((_x) != (_y), "Expected %s == %s (%lu == %lu)", \
+ #_x, #_y, (u64)(_x), (u64)(_y))
+#define SCX_ASSERT(_x) SCX_FAIL_IF(!(_x), "Expected %s to be true (%lu)", \
+ #_x, (u64)(_x))
+
+#define SCX_ECODE_VAL(__ecode) ({ \
+ u64 __val = 0; \
+ bool __found = false; \
+ \
+ __found = __COMPAT_read_enum("scx_exit_code", #__ecode, &__val); \
+ SCX_ASSERT(__found); \
+ (s64)__val; \
+})
+
+#define SCX_KIND_VAL(__kind) ({ \
+ u64 __val = 0; \
+ bool __found = false; \
+ \
+ __found = __COMPAT_read_enum("scx_exit_kind", #__kind, &__val); \
+ SCX_ASSERT(__found); \
+ __val; \
+})
+
+#endif // # __SCX_TEST_H__
diff --git a/tools/testing/selftests/sched_ext/select_cpu_dfl.bpf.c b/tools/testing/selftests/sched_ext/select_cpu_dfl.bpf.c
new file mode 100644
index 000000000000..2ed2991afafe
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/select_cpu_dfl.bpf.c
@@ -0,0 +1,40 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * A scheduler that validates the behavior of direct dispatching with a default
+ * select_cpu implementation.
+ *
+ * Copyright (c) 2023 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2023 David Vernet <dvernet@meta.com>
+ * Copyright (c) 2023 Tejun Heo <tj@kernel.org>
+ */
+
+#include <scx/common.bpf.h>
+
+char _license[] SEC("license") = "GPL";
+
+bool saw_local = false;
+
+static bool task_is_test(const struct task_struct *p)
+{
+ return !bpf_strncmp(p->comm, 9, "select_cpu");
+}
+
+void BPF_STRUCT_OPS(select_cpu_dfl_enqueue, struct task_struct *p,
+ u64 enq_flags)
+{
+ const struct cpumask *idle_mask = scx_bpf_get_idle_cpumask();
+
+ if (task_is_test(p) &&
+ bpf_cpumask_test_cpu(scx_bpf_task_cpu(p), idle_mask)) {
+ saw_local = true;
+ }
+ scx_bpf_put_idle_cpumask(idle_mask);
+
+ scx_bpf_dispatch(p, SCX_DSQ_GLOBAL, SCX_SLICE_DFL, enq_flags);
+}
+
+SEC(".struct_ops.link")
+struct sched_ext_ops select_cpu_dfl_ops = {
+ .enqueue = select_cpu_dfl_enqueue,
+ .name = "select_cpu_dfl",
+};
diff --git a/tools/testing/selftests/sched_ext/select_cpu_dfl.c b/tools/testing/selftests/sched_ext/select_cpu_dfl.c
new file mode 100644
index 000000000000..a53a40c2d2f0
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/select_cpu_dfl.c
@@ -0,0 +1,72 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2023 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2023 David Vernet <dvernet@meta.com>
+ * Copyright (c) 2023 Tejun Heo <tj@kernel.org>
+ */
+#include <bpf/bpf.h>
+#include <scx/common.h>
+#include <sys/wait.h>
+#include <unistd.h>
+#include "select_cpu_dfl.bpf.skel.h"
+#include "scx_test.h"
+
+#define NUM_CHILDREN 1028
+
+static enum scx_test_status setup(void **ctx)
+{
+ struct select_cpu_dfl *skel;
+
+ skel = select_cpu_dfl__open_and_load();
+ SCX_FAIL_IF(!skel, "Failed to open and load skel");
+ *ctx = skel;
+
+ return SCX_TEST_PASS;
+}
+
+static enum scx_test_status run(void *ctx)
+{
+ struct select_cpu_dfl *skel = ctx;
+ struct bpf_link *link;
+ pid_t pids[NUM_CHILDREN];
+ int i, status;
+
+ link = bpf_map__attach_struct_ops(skel->maps.select_cpu_dfl_ops);
+ SCX_FAIL_IF(!link, "Failed to attach scheduler");
+
+ for (i = 0; i < NUM_CHILDREN; i++) {
+ pids[i] = fork();
+ if (pids[i] == 0) {
+ sleep(1);
+ exit(0);
+ }
+ }
+
+ for (i = 0; i < NUM_CHILDREN; i++) {
+ SCX_EQ(waitpid(pids[i], &status, 0), pids[i]);
+ SCX_EQ(status, 0);
+ }
+
+ SCX_ASSERT(!skel->bss->saw_local);
+
+ bpf_link__destroy(link);
+
+ return SCX_TEST_PASS;
+}
+
+static void cleanup(void *ctx)
+{
+ struct select_cpu_dfl *skel = ctx;
+
+ select_cpu_dfl__destroy(skel);
+}
+
+struct scx_test select_cpu_dfl = {
+ .name = "select_cpu_dfl",
+ .description = "Verify the default ops.select_cpu() dispatches tasks "
+ "when idles cores are found, and skips ops.enqueue()",
+ .setup = setup,
+ .run = run,
+ .cleanup = cleanup,
+};
+REGISTER_SCX_TEST(&select_cpu_dfl)
diff --git a/tools/testing/selftests/sched_ext/select_cpu_dfl_nodispatch.bpf.c b/tools/testing/selftests/sched_ext/select_cpu_dfl_nodispatch.bpf.c
new file mode 100644
index 000000000000..4bb5abb2d369
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/select_cpu_dfl_nodispatch.bpf.c
@@ -0,0 +1,89 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * A scheduler that validates the behavior of direct dispatching with a default
+ * select_cpu implementation, and with the SCX_OPS_ENQ_DFL_NO_DISPATCH ops flag
+ * specified.
+ *
+ * Copyright (c) 2023 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2023 David Vernet <dvernet@meta.com>
+ * Copyright (c) 2023 Tejun Heo <tj@kernel.org>
+ */
+
+#include <scx/common.bpf.h>
+
+char _license[] SEC("license") = "GPL";
+
+bool saw_local = false;
+
+/* Per-task scheduling context */
+struct task_ctx {
+ bool force_local; /* CPU changed by ops.select_cpu() */
+};
+
+struct {
+ __uint(type, BPF_MAP_TYPE_TASK_STORAGE);
+ __uint(map_flags, BPF_F_NO_PREALLOC);
+ __type(key, int);
+ __type(value, struct task_ctx);
+} task_ctx_stor SEC(".maps");
+
+/* Manually specify the signature until the kfunc is added to the scx repo. */
+s32 scx_bpf_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags,
+ bool *found) __ksym;
+
+s32 BPF_STRUCT_OPS(select_cpu_dfl_nodispatch_select_cpu, struct task_struct *p,
+ s32 prev_cpu, u64 wake_flags)
+{
+ struct task_ctx *tctx;
+ s32 cpu;
+
+ tctx = bpf_task_storage_get(&task_ctx_stor, p, 0, 0);
+ if (!tctx) {
+ scx_bpf_error("task_ctx lookup failed");
+ return -ESRCH;
+ }
+
+ cpu = scx_bpf_select_cpu_dfl(p, prev_cpu, wake_flags,
+ &tctx->force_local);
+
+ return cpu;
+}
+
+void BPF_STRUCT_OPS(select_cpu_dfl_nodispatch_enqueue, struct task_struct *p,
+ u64 enq_flags)
+{
+ u64 dsq_id = SCX_DSQ_GLOBAL;
+ struct task_ctx *tctx;
+
+ tctx = bpf_task_storage_get(&task_ctx_stor, p, 0, 0);
+ if (!tctx) {
+ scx_bpf_error("task_ctx lookup failed");
+ return;
+ }
+
+ if (tctx->force_local) {
+ dsq_id = SCX_DSQ_LOCAL;
+ tctx->force_local = false;
+ saw_local = true;
+ }
+
+ scx_bpf_dispatch(p, dsq_id, SCX_SLICE_DFL, enq_flags);
+}
+
+s32 BPF_STRUCT_OPS(select_cpu_dfl_nodispatch_init_task,
+ struct task_struct *p, struct scx_init_task_args *args)
+{
+ if (bpf_task_storage_get(&task_ctx_stor, p, 0,
+ BPF_LOCAL_STORAGE_GET_F_CREATE))
+ return 0;
+ else
+ return -ENOMEM;
+}
+
+SEC(".struct_ops.link")
+struct sched_ext_ops select_cpu_dfl_nodispatch_ops = {
+ .select_cpu = select_cpu_dfl_nodispatch_select_cpu,
+ .enqueue = select_cpu_dfl_nodispatch_enqueue,
+ .init_task = select_cpu_dfl_nodispatch_init_task,
+ .name = "select_cpu_dfl_nodispatch",
+};
diff --git a/tools/testing/selftests/sched_ext/select_cpu_dfl_nodispatch.c b/tools/testing/selftests/sched_ext/select_cpu_dfl_nodispatch.c
new file mode 100644
index 000000000000..1d85bf4bf3a3
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/select_cpu_dfl_nodispatch.c
@@ -0,0 +1,72 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2023 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2023 David Vernet <dvernet@meta.com>
+ * Copyright (c) 2023 Tejun Heo <tj@kernel.org>
+ */
+#include <bpf/bpf.h>
+#include <scx/common.h>
+#include <sys/wait.h>
+#include <unistd.h>
+#include "select_cpu_dfl_nodispatch.bpf.skel.h"
+#include "scx_test.h"
+
+#define NUM_CHILDREN 1028
+
+static enum scx_test_status setup(void **ctx)
+{
+ struct select_cpu_dfl_nodispatch *skel;
+
+ skel = select_cpu_dfl_nodispatch__open_and_load();
+ SCX_FAIL_IF(!skel, "Failed to open and load skel");
+ *ctx = skel;
+
+ return SCX_TEST_PASS;
+}
+
+static enum scx_test_status run(void *ctx)
+{
+ struct select_cpu_dfl_nodispatch *skel = ctx;
+ struct bpf_link *link;
+ pid_t pids[NUM_CHILDREN];
+ int i, status;
+
+ link = bpf_map__attach_struct_ops(skel->maps.select_cpu_dfl_nodispatch_ops);
+ SCX_FAIL_IF(!link, "Failed to attach scheduler");
+
+ for (i = 0; i < NUM_CHILDREN; i++) {
+ pids[i] = fork();
+ if (pids[i] == 0) {
+ sleep(1);
+ exit(0);
+ }
+ }
+
+ for (i = 0; i < NUM_CHILDREN; i++) {
+ SCX_EQ(waitpid(pids[i], &status, 0), pids[i]);
+ SCX_EQ(status, 0);
+ }
+
+ SCX_ASSERT(skel->bss->saw_local);
+
+ bpf_link__destroy(link);
+
+ return SCX_TEST_PASS;
+}
+
+static void cleanup(void *ctx)
+{
+ struct select_cpu_dfl_nodispatch *skel = ctx;
+
+ select_cpu_dfl_nodispatch__destroy(skel);
+}
+
+struct scx_test select_cpu_dfl_nodispatch = {
+ .name = "select_cpu_dfl_nodispatch",
+ .description = "Verify behavior of scx_bpf_select_cpu_dfl() in "
+ "ops.select_cpu()",
+ .setup = setup,
+ .run = run,
+ .cleanup = cleanup,
+};
+REGISTER_SCX_TEST(&select_cpu_dfl_nodispatch)
diff --git a/tools/testing/selftests/sched_ext/select_cpu_dispatch.bpf.c b/tools/testing/selftests/sched_ext/select_cpu_dispatch.bpf.c
new file mode 100644
index 000000000000..f0b96a4a04b2
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/select_cpu_dispatch.bpf.c
@@ -0,0 +1,41 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * A scheduler that validates the behavior of direct dispatching with a default
+ * select_cpu implementation.
+ *
+ * Copyright (c) 2023 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2023 David Vernet <dvernet@meta.com>
+ * Copyright (c) 2023 Tejun Heo <tj@kernel.org>
+ */
+
+#include <scx/common.bpf.h>
+
+char _license[] SEC("license") = "GPL";
+
+s32 BPF_STRUCT_OPS(select_cpu_dispatch_select_cpu, struct task_struct *p,
+ s32 prev_cpu, u64 wake_flags)
+{
+ u64 dsq_id = SCX_DSQ_LOCAL;
+ s32 cpu = prev_cpu;
+
+ if (scx_bpf_test_and_clear_cpu_idle(cpu))
+ goto dispatch;
+
+ cpu = scx_bpf_pick_idle_cpu(p->cpus_ptr, 0);
+ if (cpu >= 0)
+ goto dispatch;
+
+ dsq_id = SCX_DSQ_GLOBAL;
+ cpu = prev_cpu;
+
+dispatch:
+ scx_bpf_dispatch(p, dsq_id, SCX_SLICE_DFL, 0);
+ return cpu;
+}
+
+SEC(".struct_ops.link")
+struct sched_ext_ops select_cpu_dispatch_ops = {
+ .select_cpu = select_cpu_dispatch_select_cpu,
+ .name = "select_cpu_dispatch",
+ .timeout_ms = 1000U,
+};
diff --git a/tools/testing/selftests/sched_ext/select_cpu_dispatch.c b/tools/testing/selftests/sched_ext/select_cpu_dispatch.c
new file mode 100644
index 000000000000..0309ca8785b3
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/select_cpu_dispatch.c
@@ -0,0 +1,70 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2023 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2023 David Vernet <dvernet@meta.com>
+ * Copyright (c) 2023 Tejun Heo <tj@kernel.org>
+ */
+#include <bpf/bpf.h>
+#include <scx/common.h>
+#include <sys/wait.h>
+#include <unistd.h>
+#include "select_cpu_dispatch.bpf.skel.h"
+#include "scx_test.h"
+
+#define NUM_CHILDREN 1028
+
+static enum scx_test_status setup(void **ctx)
+{
+ struct select_cpu_dispatch *skel;
+
+ skel = select_cpu_dispatch__open_and_load();
+ SCX_FAIL_IF(!skel, "Failed to open and load skel");
+ *ctx = skel;
+
+ return SCX_TEST_PASS;
+}
+
+static enum scx_test_status run(void *ctx)
+{
+ struct select_cpu_dispatch *skel = ctx;
+ struct bpf_link *link;
+ pid_t pids[NUM_CHILDREN];
+ int i, status;
+
+ link = bpf_map__attach_struct_ops(skel->maps.select_cpu_dispatch_ops);
+ SCX_FAIL_IF(!link, "Failed to attach scheduler");
+
+ for (i = 0; i < NUM_CHILDREN; i++) {
+ pids[i] = fork();
+ if (pids[i] == 0) {
+ sleep(1);
+ exit(0);
+ }
+ }
+
+ for (i = 0; i < NUM_CHILDREN; i++) {
+ SCX_EQ(waitpid(pids[i], &status, 0), pids[i]);
+ SCX_EQ(status, 0);
+ }
+
+ bpf_link__destroy(link);
+
+ return SCX_TEST_PASS;
+}
+
+static void cleanup(void *ctx)
+{
+ struct select_cpu_dispatch *skel = ctx;
+
+ select_cpu_dispatch__destroy(skel);
+}
+
+struct scx_test select_cpu_dispatch = {
+ .name = "select_cpu_dispatch",
+ .description = "Test direct dispatching to built-in DSQs from "
+ "ops.select_cpu()",
+ .setup = setup,
+ .run = run,
+ .cleanup = cleanup,
+};
+REGISTER_SCX_TEST(&select_cpu_dispatch)
diff --git a/tools/testing/selftests/sched_ext/select_cpu_dispatch_bad_dsq.bpf.c b/tools/testing/selftests/sched_ext/select_cpu_dispatch_bad_dsq.bpf.c
new file mode 100644
index 000000000000..7b42ddce0f56
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/select_cpu_dispatch_bad_dsq.bpf.c
@@ -0,0 +1,37 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * A scheduler that validates the behavior of direct dispatching with a default
+ * select_cpu implementation.
+ *
+ * Copyright (c) 2023 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2023 David Vernet <dvernet@meta.com>
+ * Copyright (c) 2023 Tejun Heo <tj@kernel.org>
+ */
+
+#include <scx/common.bpf.h>
+
+char _license[] SEC("license") = "GPL";
+
+UEI_DEFINE(uei);
+
+s32 BPF_STRUCT_OPS(select_cpu_dispatch_bad_dsq_select_cpu, struct task_struct *p,
+ s32 prev_cpu, u64 wake_flags)
+{
+ /* Dispatching to a random DSQ should fail. */
+ scx_bpf_dispatch(p, 0xcafef00d, SCX_SLICE_DFL, 0);
+
+ return prev_cpu;
+}
+
+void BPF_STRUCT_OPS(select_cpu_dispatch_bad_dsq_exit, struct scx_exit_info *ei)
+{
+ UEI_RECORD(uei, ei);
+}
+
+SEC(".struct_ops.link")
+struct sched_ext_ops select_cpu_dispatch_bad_dsq_ops = {
+ .select_cpu = select_cpu_dispatch_bad_dsq_select_cpu,
+ .exit = select_cpu_dispatch_bad_dsq_exit,
+ .name = "select_cpu_dispatch_bad_dsq",
+ .timeout_ms = 1000U,
+};
diff --git a/tools/testing/selftests/sched_ext/select_cpu_dispatch_bad_dsq.c b/tools/testing/selftests/sched_ext/select_cpu_dispatch_bad_dsq.c
new file mode 100644
index 000000000000..47eb6ed7627d
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/select_cpu_dispatch_bad_dsq.c
@@ -0,0 +1,56 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2023 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2023 David Vernet <dvernet@meta.com>
+ * Copyright (c) 2023 Tejun Heo <tj@kernel.org>
+ */
+#include <bpf/bpf.h>
+#include <scx/common.h>
+#include <sys/wait.h>
+#include <unistd.h>
+#include "select_cpu_dispatch_bad_dsq.bpf.skel.h"
+#include "scx_test.h"
+
+static enum scx_test_status setup(void **ctx)
+{
+ struct select_cpu_dispatch_bad_dsq *skel;
+
+ skel = select_cpu_dispatch_bad_dsq__open_and_load();
+ SCX_FAIL_IF(!skel, "Failed to open and load skel");
+ *ctx = skel;
+
+ return SCX_TEST_PASS;
+}
+
+static enum scx_test_status run(void *ctx)
+{
+ struct select_cpu_dispatch_bad_dsq *skel = ctx;
+ struct bpf_link *link;
+
+ link = bpf_map__attach_struct_ops(skel->maps.select_cpu_dispatch_bad_dsq_ops);
+ SCX_FAIL_IF(!link, "Failed to attach scheduler");
+
+ sleep(1);
+
+ SCX_EQ(skel->data->uei.kind, EXIT_KIND(SCX_EXIT_ERROR));
+ bpf_link__destroy(link);
+
+ return SCX_TEST_PASS;
+}
+
+static void cleanup(void *ctx)
+{
+ struct select_cpu_dispatch_bad_dsq *skel = ctx;
+
+ select_cpu_dispatch_bad_dsq__destroy(skel);
+}
+
+struct scx_test select_cpu_dispatch_bad_dsq = {
+ .name = "select_cpu_dispatch_bad_dsq",
+ .description = "Verify graceful failure if we direct-dispatch to a "
+ "bogus DSQ in ops.select_cpu()",
+ .setup = setup,
+ .run = run,
+ .cleanup = cleanup,
+};
+REGISTER_SCX_TEST(&select_cpu_dispatch_bad_dsq)
diff --git a/tools/testing/selftests/sched_ext/select_cpu_dispatch_dbl_dsp.bpf.c b/tools/testing/selftests/sched_ext/select_cpu_dispatch_dbl_dsp.bpf.c
new file mode 100644
index 000000000000..653e3dc0b4dc
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/select_cpu_dispatch_dbl_dsp.bpf.c
@@ -0,0 +1,38 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * A scheduler that validates the behavior of direct dispatching with a default
+ * select_cpu implementation.
+ *
+ * Copyright (c) 2023 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2023 David Vernet <dvernet@meta.com>
+ * Copyright (c) 2023 Tejun Heo <tj@kernel.org>
+ */
+
+#include <scx/common.bpf.h>
+
+char _license[] SEC("license") = "GPL";
+
+UEI_DEFINE(uei);
+
+s32 BPF_STRUCT_OPS(select_cpu_dispatch_dbl_dsp_select_cpu, struct task_struct *p,
+ s32 prev_cpu, u64 wake_flags)
+{
+ /* Dispatching twice in a row is disallowed. */
+ scx_bpf_dispatch(p, SCX_DSQ_GLOBAL, SCX_SLICE_DFL, 0);
+ scx_bpf_dispatch(p, SCX_DSQ_GLOBAL, SCX_SLICE_DFL, 0);
+
+ return prev_cpu;
+}
+
+void BPF_STRUCT_OPS(select_cpu_dispatch_dbl_dsp_exit, struct scx_exit_info *ei)
+{
+ UEI_RECORD(uei, ei);
+}
+
+SEC(".struct_ops.link")
+struct sched_ext_ops select_cpu_dispatch_dbl_dsp_ops = {
+ .select_cpu = select_cpu_dispatch_dbl_dsp_select_cpu,
+ .exit = select_cpu_dispatch_dbl_dsp_exit,
+ .name = "select_cpu_dispatch_dbl_dsp",
+ .timeout_ms = 1000U,
+};
diff --git a/tools/testing/selftests/sched_ext/select_cpu_dispatch_dbl_dsp.c b/tools/testing/selftests/sched_ext/select_cpu_dispatch_dbl_dsp.c
new file mode 100644
index 000000000000..48ff028a3c46
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/select_cpu_dispatch_dbl_dsp.c
@@ -0,0 +1,56 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2023 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2023 David Vernet <dvernet@meta.com>
+ * Copyright (c) 2023 Tejun Heo <tj@kernel.org>
+ */
+#include <bpf/bpf.h>
+#include <scx/common.h>
+#include <sys/wait.h>
+#include <unistd.h>
+#include "select_cpu_dispatch_dbl_dsp.bpf.skel.h"
+#include "scx_test.h"
+
+static enum scx_test_status setup(void **ctx)
+{
+ struct select_cpu_dispatch_dbl_dsp *skel;
+
+ skel = select_cpu_dispatch_dbl_dsp__open_and_load();
+ SCX_FAIL_IF(!skel, "Failed to open and load skel");
+ *ctx = skel;
+
+ return SCX_TEST_PASS;
+}
+
+static enum scx_test_status run(void *ctx)
+{
+ struct select_cpu_dispatch_dbl_dsp *skel = ctx;
+ struct bpf_link *link;
+
+ link = bpf_map__attach_struct_ops(skel->maps.select_cpu_dispatch_dbl_dsp_ops);
+ SCX_FAIL_IF(!link, "Failed to attach scheduler");
+
+ sleep(1);
+
+ SCX_EQ(skel->data->uei.kind, EXIT_KIND(SCX_EXIT_ERROR));
+ bpf_link__destroy(link);
+
+ return SCX_TEST_PASS;
+}
+
+static void cleanup(void *ctx)
+{
+ struct select_cpu_dispatch_dbl_dsp *skel = ctx;
+
+ select_cpu_dispatch_dbl_dsp__destroy(skel);
+}
+
+struct scx_test select_cpu_dispatch_dbl_dsp = {
+ .name = "select_cpu_dispatch_dbl_dsp",
+ .description = "Verify graceful failure if we dispatch twice to a "
+ "DSQ in ops.select_cpu()",
+ .setup = setup,
+ .run = run,
+ .cleanup = cleanup,
+};
+REGISTER_SCX_TEST(&select_cpu_dispatch_dbl_dsp)
diff --git a/tools/testing/selftests/sched_ext/select_cpu_vtime.bpf.c b/tools/testing/selftests/sched_ext/select_cpu_vtime.bpf.c
new file mode 100644
index 000000000000..7f3ebf4fc2ea
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/select_cpu_vtime.bpf.c
@@ -0,0 +1,92 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * A scheduler that validates that enqueue flags are properly stored and
+ * applied at dispatch time when a task is directly dispatched from
+ * ops.select_cpu(). We validate this by using scx_bpf_dispatch_vtime(), and
+ * making the test a very basic vtime scheduler.
+ *
+ * Copyright (c) 2024 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2024 David Vernet <dvernet@meta.com>
+ * Copyright (c) 2024 Tejun Heo <tj@kernel.org>
+ */
+
+#include <scx/common.bpf.h>
+
+char _license[] SEC("license") = "GPL";
+
+volatile bool consumed;
+
+static u64 vtime_now;
+
+#define VTIME_DSQ 0
+
+static inline bool vtime_before(u64 a, u64 b)
+{
+ return (s64)(a - b) < 0;
+}
+
+static inline u64 task_vtime(const struct task_struct *p)
+{
+ u64 vtime = p->scx.dsq_vtime;
+
+ if (vtime_before(vtime, vtime_now - SCX_SLICE_DFL))
+ return vtime_now - SCX_SLICE_DFL;
+ else
+ return vtime;
+}
+
+s32 BPF_STRUCT_OPS(select_cpu_vtime_select_cpu, struct task_struct *p,
+ s32 prev_cpu, u64 wake_flags)
+{
+ s32 cpu;
+
+ cpu = scx_bpf_pick_idle_cpu(p->cpus_ptr, 0);
+ if (cpu >= 0)
+ goto ddsp;
+
+ cpu = prev_cpu;
+ scx_bpf_test_and_clear_cpu_idle(cpu);
+ddsp:
+ scx_bpf_dispatch_vtime(p, VTIME_DSQ, SCX_SLICE_DFL, task_vtime(p), 0);
+ return cpu;
+}
+
+void BPF_STRUCT_OPS(select_cpu_vtime_dispatch, s32 cpu, struct task_struct *p)
+{
+ if (scx_bpf_consume(VTIME_DSQ))
+ consumed = true;
+}
+
+void BPF_STRUCT_OPS(select_cpu_vtime_running, struct task_struct *p)
+{
+ if (vtime_before(vtime_now, p->scx.dsq_vtime))
+ vtime_now = p->scx.dsq_vtime;
+}
+
+void BPF_STRUCT_OPS(select_cpu_vtime_stopping, struct task_struct *p,
+ bool runnable)
+{
+ p->scx.dsq_vtime += (SCX_SLICE_DFL - p->scx.slice) * 100 / p->scx.weight;
+}
+
+void BPF_STRUCT_OPS(select_cpu_vtime_enable, struct task_struct *p)
+{
+ p->scx.dsq_vtime = vtime_now;
+}
+
+s32 BPF_STRUCT_OPS_SLEEPABLE(select_cpu_vtime_init)
+{
+ return scx_bpf_create_dsq(VTIME_DSQ, -1);
+}
+
+SEC(".struct_ops.link")
+struct sched_ext_ops select_cpu_vtime_ops = {
+ .select_cpu = select_cpu_vtime_select_cpu,
+ .dispatch = select_cpu_vtime_dispatch,
+ .running = select_cpu_vtime_running,
+ .stopping = select_cpu_vtime_stopping,
+ .enable = select_cpu_vtime_enable,
+ .init = select_cpu_vtime_init,
+ .name = "select_cpu_vtime",
+ .timeout_ms = 1000U,
+};
diff --git a/tools/testing/selftests/sched_ext/select_cpu_vtime.c b/tools/testing/selftests/sched_ext/select_cpu_vtime.c
new file mode 100644
index 000000000000..b4629c2364f5
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/select_cpu_vtime.c
@@ -0,0 +1,59 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2024 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2024 David Vernet <dvernet@meta.com>
+ * Copyright (c) 2024 Tejun Heo <tj@kernel.org>
+ */
+#include <bpf/bpf.h>
+#include <scx/common.h>
+#include <sys/wait.h>
+#include <unistd.h>
+#include "select_cpu_vtime.bpf.skel.h"
+#include "scx_test.h"
+
+static enum scx_test_status setup(void **ctx)
+{
+ struct select_cpu_vtime *skel;
+
+ skel = select_cpu_vtime__open_and_load();
+ SCX_FAIL_IF(!skel, "Failed to open and load skel");
+ *ctx = skel;
+
+ return SCX_TEST_PASS;
+}
+
+static enum scx_test_status run(void *ctx)
+{
+ struct select_cpu_vtime *skel = ctx;
+ struct bpf_link *link;
+
+ SCX_ASSERT(!skel->bss->consumed);
+
+ link = bpf_map__attach_struct_ops(skel->maps.select_cpu_vtime_ops);
+ SCX_FAIL_IF(!link, "Failed to attach scheduler");
+
+ sleep(1);
+
+ SCX_ASSERT(skel->bss->consumed);
+
+ bpf_link__destroy(link);
+
+ return SCX_TEST_PASS;
+}
+
+static void cleanup(void *ctx)
+{
+ struct select_cpu_vtime *skel = ctx;
+
+ select_cpu_vtime__destroy(skel);
+}
+
+struct scx_test select_cpu_vtime = {
+ .name = "select_cpu_vtime",
+ .description = "Test doing direct vtime-dispatching from "
+ "ops.select_cpu(), to a non-built-in DSQ",
+ .setup = setup,
+ .run = run,
+ .cleanup = cleanup,
+};
+REGISTER_SCX_TEST(&select_cpu_vtime)
diff --git a/tools/testing/selftests/sched_ext/test_example.c b/tools/testing/selftests/sched_ext/test_example.c
new file mode 100644
index 000000000000..ce36cdf03cdc
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/test_example.c
@@ -0,0 +1,49 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2024 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2024 Tejun Heo <tj@kernel.org>
+ * Copyright (c) 2024 David Vernet <dvernet@meta.com>
+ */
+#include <bpf/bpf.h>
+#include <scx/common.h>
+#include "scx_test.h"
+
+static bool setup_called = false;
+static bool run_called = false;
+static bool cleanup_called = false;
+
+static int context = 10;
+
+static enum scx_test_status setup(void **ctx)
+{
+ setup_called = true;
+ *ctx = &context;
+
+ return SCX_TEST_PASS;
+}
+
+static enum scx_test_status run(void *ctx)
+{
+ int *arg = ctx;
+
+ SCX_ASSERT(setup_called);
+ SCX_ASSERT(!run_called && !cleanup_called);
+ SCX_EQ(*arg, context);
+
+ run_called = true;
+ return SCX_TEST_PASS;
+}
+
+static void cleanup (void *ctx)
+{
+ SCX_BUG_ON(!run_called || cleanup_called, "Wrong callbacks invoked");
+}
+
+struct scx_test example = {
+ .name = "example",
+ .description = "Validate the basic function of the test suite itself",
+ .setup = setup,
+ .run = run,
+ .cleanup = cleanup,
+};
+REGISTER_SCX_TEST(&example)
diff --git a/tools/testing/selftests/sched_ext/util.c b/tools/testing/selftests/sched_ext/util.c
new file mode 100644
index 000000000000..e47769c91918
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/util.c
@@ -0,0 +1,71 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2024 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2024 David Vernet <dvernet@meta.com>
+ */
+#include <errno.h>
+#include <fcntl.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+
+/* Returns read len on success, or -errno on failure. */
+static ssize_t read_text(const char *path, char *buf, size_t max_len)
+{
+ ssize_t len;
+ int fd;
+
+ fd = open(path, O_RDONLY);
+ if (fd < 0)
+ return -errno;
+
+ len = read(fd, buf, max_len - 1);
+
+ if (len >= 0)
+ buf[len] = 0;
+
+ close(fd);
+ return len < 0 ? -errno : len;
+}
+
+/* Returns written len on success, or -errno on failure. */
+static ssize_t write_text(const char *path, char *buf, ssize_t len)
+{
+ int fd;
+ ssize_t written;
+
+ fd = open(path, O_WRONLY | O_APPEND);
+ if (fd < 0)
+ return -errno;
+
+ written = write(fd, buf, len);
+ close(fd);
+ return written < 0 ? -errno : written;
+}
+
+long file_read_long(const char *path)
+{
+ char buf[128];
+
+
+ if (read_text(path, buf, sizeof(buf)) <= 0)
+ return -1;
+
+ return atol(buf);
+}
+
+int file_write_long(const char *path, long val)
+{
+ char buf[64];
+ int ret;
+
+ ret = sprintf(buf, "%lu", val);
+ if (ret < 0)
+ return ret;
+
+ if (write_text(path, buf, sizeof(buf)) <= 0)
+ return -1;
+
+ return 0;
+}
diff --git a/tools/testing/selftests/sched_ext/util.h b/tools/testing/selftests/sched_ext/util.h
new file mode 100644
index 000000000000..bc13dfec1267
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/util.h
@@ -0,0 +1,13 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2024 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2024 David Vernet <void@manifault.com>
+ */
+
+#ifndef __SCX_TEST_UTIL_H__
+#define __SCX_TEST_UTIL_H__
+
+long file_read_long(const char *path);
+int file_write_long(const char *path, long val);
+
+#endif // __SCX_TEST_H__
--
2.45.2
Implementation Analysis
Overview
This patch (authored by David Vernet) establishes the tools/testing/selftests/sched_ext/ directory and provides a comprehensive selftest suite for sched_ext. The selftests are built with libbpf and BPF skeletons — each test is a minimal BPF scheduler (.bpf.c) paired with a C userspace driver (.c). They exercise specific features and error conditions rather than general scheduling correctness. 51 files are added, comprising over 3200 lines of test infrastructure and test cases.
Code Walkthrough
Test infrastructure — the shared framework files:
runner.c/scx_test.h: The test runner harness. DefinesSCX_TEST_RUN_ALL()and helpers for loading/attaching BPF skeletons, checking exit codes, and reporting PASS/FAIL. Tests register themselves via a linked list ofstruct scx_testentries.util.c/util.h: Utility functions shared across tests.Makefile(218 lines): Builds all.bpf.cfiles with clang, generates BPF skeletons, links each test binary. Follows the standardtools/testing/selftestsMakefile conventions. Includes../../../build/Build.include.config: Kernel Kconfig fragment specifying minimum requirements:CONFIG_SCHED_DEBUG=y CONFIG_SCHED_CLASS_EXT=y CONFIG_CGROUPS=y CONFIG_CGROUP_SCHED=y CONFIG_NO_HZ_FULL=y CONFIG_SCHED_CORE=y
Test cases and what they verify:
-
minimal: The simplest possible BPF scheduler — justops.initandops.exit. Verifies that a scheduler with no other ops loads, runs briefly, and exits cleanly. -
create_dsq: Creates a custom DSQ withscx_bpf_create_dsq(), dispatches tasks to it viaops.enqueue(), and destroys it inops.exit(). Verifies the DSQ create/consume/destroy lifecycle. -
dsp_local_on: Usesscx_bpf_dispatch_local_on(p, cpu, ...)to dispatch tasks directly to a specific CPU's local DSQ. Verifies that targeted local-DSQ dispatch works correctly and tasks actually run on the intended CPU. -
ddsp_bogus_dsq_fail: Attempts to dispatch a task to a DSQ ID that does not exist. Verifies that the BPF scheduler exits with an ops error (not a kernel panic or silent failure). -
ddsp_vtimelocal_fail: Attempts to callscx_bpf_dispatch_vtime()targetingSCX_DSQ_LOCAL. Verifies that this correctly triggers an ops error (built-in DSQs cannot be PRIQ). -
enq_last_no_enq_fails: Implementsops.enqueue()withSCX_OPS_ENQ_LASTset in flags, but the enqueue callback deliberately does not dispatch the task whenSCX_ENQ_LASTis received. Verifies that the watchdog/error detection catches this stall. -
enq_select_cpu_fails: Theops.select_cpu()callback returns an invalid CPU (e.g., out of range or not in the task's allowed cpumask). Verifies that the kernel catches this invalid selection and handles it gracefully. -
exit: Tests various exit scenarios —scx_bpf_exit()called with different exit codes, verifying that the correct exit kind and code are reported back to userspace viaops.exit(). -
prog_run: A basic smoke test that loads a complete BPF scheduler (with select_cpu, enqueue, dispatch), lets it run for a brief period with actual tasks, and verifies it ran without errors. -
hotplug: Tests CPU hotplug interaction. Exercises thecpu_online/offlinecallbacks, the auto-restart path when callbacks are absent, and thehotplug_seqrace detection mechanism. -
init_enable_count: Verifies thatops.init_task()andops.enable()are called the correct number of times as tasks transition in and out of sched_ext scheduling. -
maximal: Implements every optional op callback to ensure that the full ops table can be populated without crashes. This is a compilation and runtime coverage test. -
maybe_null/maybe_null_fail_dsp/maybe_null_fail_yld: Tests for nullable pointer handling in ops callbacks. Verifies that the BPF verifier correctly rejects programs that dereference potentially-null pointers without checks, and accepts programs that do null-check correctly. -
reload_loop: Repeatedly loads and unloads a BPF scheduler in a tight loop, verifying that the load/unload cycle is safe and leak-free. -
select_cpu_dfl/select_cpu_dfl_nodispatch/select_cpu_dispatch/select_cpu_dispatch_bad_dsq/select_cpu_dispatch_dbl_dsp: A suite of tests for theselect_cpupath. Tests include: using the default CPU selection, dispatching vs. not dispatching fromselect_cpu, dispatching to an invalid DSQ fromselect_cpu, and double-dispatch (dispatching twice for the same task fromselect_cpu). -
select_cpu_vtime: Tests vtime dispatch fromselect_cpu. Verifies thatscx_bpf_dispatch_vtime()works correctly when called from theselect_cpucontext. -
test_example: A meta-test that exercises the test framework itself.
Key Concepts
Test structure: Each test consists of:
- A
.bpf.cfile: the BPF scheduler program, compiled with clang and loaded via libbpf skeletons. - A
.cfile: the userspace driver that loads the skeleton, attaches it, lets it run briefly, then checks the exit condition.
Error detection tests (the *_fail tests): These tests intentionally trigger ops errors and verify that sched_ext catches them and exits with the expected error kind. This exercises the error handling and watchdog infrastructure.
scx_test.h conventions: Tests declare a struct scx_test with .name, .description, and .run function pointer. The runner collects all registered tests and executes them, reporting results.
Kernel config requirements: CONFIG_SCHED_CORE=y is listed in config, reflecting that this patch series requires core-sched to be buildable alongside sched_ext (from patch 27/30).
Locking and Concurrency Notes
The selftests themselves are userspace programs — locking is not a direct concern. However, the tests exercise kernel-side locking indirectly:
- Tests that use
SCX_OPS_ENQ_LASTexercise the path where rq lock is held with no runnable tasks. - The
reload_looptest exercises concurrent load/unload which exercises thescx_cgroup_rwsem/scx_fork_rwsemlocking inscx_ops_enable/disable. - The
hotplugtest exercises the interaction between hotplug locking (cpus_read_lock) and the sched_ext enable path.
Integration with Kernel Subsystems
The selftests integrate with the Linux kernel selftest framework (tools/testing/selftests/). The config file is picked up by kselftest infrastructure to set kernel build options when running selftests in CI. The Makefile follows the standard pattern for BPF-based selftests (similar to tools/testing/selftests/bpf/).
Required build dependencies: clang (for BPF compilation), libbpf, bpf skeletons (via bpftool). The CONFIG_SCHED_CLASS_EXT=y kernel option is required at runtime.
What Maintainers Need to Know
- Run selftests before submitting patches. The suite catches a wide range of error conditions, verifier rejections, and basic correctness issues. Run with:
make -C tools/testing/selftests/sched_ext && ./runner. - Add a selftest for every new error condition. The
ddsp_bogus_dsq_failandddsp_vtimelocal_failpatterns demonstrate the standard approach: write a BPF scheduler that intentionally triggers the error, verify the exit code matches the expected ops error. - Add a selftest for every new ops callback.
maximal.bpf.cshould be updated to include any new callback added tostruct sched_ext_ops, ensuring it compiles and runs without error. - The
configfile must be updated when new Kconfig requirements are added to sched_ext. TheCONFIG_SCHED_CORE=yentry was added in this patch precisely because patch 27/30 removed the!SCHED_COREdependency. maybe_nulltests are important for verifier correctness. If you add a new ops callback that takes a potentially-null pointer (e.g.,struct task_struct *arguments that may be null in some invocations), add correspondingmaybe_nulltests.- Test isolation: Each test loads a fresh BPF scheduler skeleton, runs briefly, and exits. Tests must not assume any scheduler state from previous tests. The
reload_looptest specifically validates that the kernel cleans up state correctly between loads.
Connection to Other Patches
- All patches 1-29: The selftests exercise features from every patch in the series.
create_dsqandselect_cpu_vtimetest patch 28 (vtime DSQs).hotplugtests patch 25 (cpu_online/offline).enq_last_no_enq_failstests theSCX_ENQ_LASTflag from earlier patches.ddsp_vtimelocal_failtests the built-in DSQ PRIQ restriction from patch 28. - Patch 29/30 (documentation): The
tools/sched_ext/README.mdfrom that patch and thetools/testing/selftests/sched_ext/directory from this patch together form the complete developer and QA tooling for sched_ext.
Community Follow-Up (Patches 32–42)
Overview
The story of sched_ext does not end with the submission of the v7 patchset. After a patchset of this scale is sent to LKML (Linux Kernel Mailing List), it enters a period of community review and post-merge stabilization. Patches 32–42 capture this post-submission phase: the review threads, the bugs discovered by the community, the fixes applied, and the discussion threads that shaped the final merged form of specific patches.
For a kernel maintainer, understanding this phase is as important as understanding the implementation itself. The post-submission review is where the kernel community's collected knowledge is applied to a new feature, and the issues raised reveal both the quality of the original implementation and the standards the community holds for scheduler code. The bug reports (particularly the NVIDIA-reported Makefile issue) demonstrate the kinds of integration problems that only surface when a large, diverse user base exercises the code.
This group divides into three subgroups:
- Patches 32–35: Community review of specific patches (documentation and
switching_to()). - Patches 36–38: NVIDIA-reported Makefile bug and its fix.
- Patches 39–42: Community review of the core implementation (patch 09/30).
Why This Phase Matters
Most study of kernel patches focuses on the diff — the code that was added or changed. But the review discussion attached to each patch often contains more information than the diff itself:
- Why alternatives were rejected. The reviewer asks "why not X?" and the author explains the tradeoff. This reasoning is not in the code.
- Edge cases the original author missed. A reviewer points out a scenario the author didn't consider, leading to a fix or a documentation clarification.
- Implicit assumptions made explicit. The community's questions reveal that something the author considered obvious needs to be stated explicitly in comments or documentation.
- Integration bugs. A third party (like NVIDIA) discovers a bug in a tooling file that the scheduler developers didn't test because they use different build workflows.
For a maintainer of sched_ext, the community follow-up phase provides the historical context needed to understand why the code is the way it is, and why certain changes that might appear to be improvements are actually inadvisable.
Key Concepts
PATCHES 32–33 — Documentation Review
The review of Documentation/scheduler/sched-ext.rst (patch 29) generated several comment
threads. Common categories of documentation review feedback in the kernel community:
Precision of callback timing descriptions. Reviewers asked for clarification on exactly
when callbacks like ops.runnable() vs ops.enqueue() are called, and whether the
distinction matters for BPF programs that only implement one but not the other. The resulting
discussion led to tightened language in the documentation about the ordering guarantees.
Error message accuracy. Some error condition descriptions were ambiguous — for example, "dispatch to invalid DSQ" could mean the DSQ ID doesn't exist, or the DSQ was destroyed, or the BPF program doesn't have permission. Reviewers pushed for precise enumeration of each error condition and the specific exit reason string each produces.
Example code correctness. The example code snippets in the documentation were reviewed for correctness against the actual API. Small discrepancies (wrong argument order, missing flags) were caught and corrected. This is particularly important because documentation code examples are often copied verbatim by users, amplifying any errors.
Coverage of race conditions. Reviewers asked whether the documentation adequately covered
the race conditions that BPF scheduler authors must handle — particularly around task migration
and the in_op_task serialization. The review led to a section explicitly documenting these
races and how in_op_task protects against them.
For a maintainer, documentation review comments are a direct window into what the community finds confusing or underspecified. Recurring questions about the same topic signal that the documentation needs restructuring, not just additional words.
PATCH 34 — Review of switching_to()
The switching_to() callback (patch 04) received particular scrutiny during post-submission
review because it is a change to the core scheduler infrastructure that affects all scheduler
classes, not just sched_ext. The concerns raised:
Ordering guarantees under concurrent migration. The review asked: what happens if a task
is being migrated to a new CPU at the same time as switching_to() is called? The original
implementation held the runqueue lock during switching_to(), which prevents the migration
from completing until switching_to() returns. Reviewers verified this was intentional and
that the lock ordering was correct.
Performance impact on non-SCX paths. switching_to() is called for all class transitions,
not just transitions into the ext class. Reviewers checked whether the new hook added any
overhead on the CFS-to-RT or RT-to-CFS paths (common class transitions that happen on every
sched_setscheduler() call on an RT system). The implementation correctly no-ops these cases
with a single branch that checks whether the new class is ext_sched_class.
Interaction with check_class_changing(). Several reviewers noted that the new
switching_to() callback, combined with the existing check_class_changing() and
check_class_changed() callbacks, created three separate notifications for a single class
transition. The discussion resulted in added comments clarifying the distinct purpose of each:
check_class_changing() is called with the old class for validation, switching_to() is
called on the new class for initialization, and check_class_changed() is called after the
transition completes for post-transition actions.
PATCH 35 — Further switching_to() Discussion
The switching_to() discussion continued across multiple LKML threads. A significant thread
examined the semantic distinction between switching_to() and switched_to():
switching_to(p): Called on the new class, with the task still in the old class. The new class initializes but cannot yet schedule the task.switched_to(p): Called on the new class, with the task already in the new class and potentially enqueued. The new class can now schedule the task.
The subtlety that reviewers focused on: there is a window between switching_to() and
switched_to() where the task is in transition. If another CPU tries to migrate the task
during this window, it must wait for the transition to complete. The review verified that the
runqueue lock covers this window correctly and that no scheduler-class-specific code accesses
the task's class state without holding the lock.
This discussion is a good example of the kernel community's attention to concurrent correctness: even when the code appears correct, reviewers ask for explicit verification of every concurrent access path.
PATCHES 36–38 — NVIDIA Makefile Bug
These three patches document a bug reported by NVIDIA engineers: make mrproper (the kernel's
"clean everything" target) was incorrectly deleting files that should not have been cleaned,
affecting the build workflow for systems with NVIDIA drivers installed alongside a sched_ext
kernel.
The bug: The sched_ext BPF skeleton headers (generated from BPF object files in
tools/sched_ext/) were placed in a path that make mrproper treated as generated output and
deleted. However, some of these files were also referenced by the NVIDIA module build system
(which generates its own headers in a separate build step that runs after the kernel build).
When mrproper deleted the sched_ext headers, the subsequent NVIDIA module build failed with
confusing errors about missing headers.
Why this matters for a maintainer: Build system bugs are particularly insidious because
they are environment-dependent — they only appear when specific third-party toolchains or build
workflows are used. The NVIDIA report revealed that make mrproper had inconsistent semantics:
it was supposed to clean only kernel-generated files, but some sched_ext files straddled the
boundary between "generated" and "source".
The fix (patches 37–38): The fix reclassified the affected sched_ext files as source files
(not generated output) and updated the .gitignore and Makefile clean/mrproper rules
accordingly. Patch 38 also adds a regression test: a CI check that verifies make mrproper
does not delete files that git status reports as untracked (source files that should not be
generated).
Lessons for maintainers: When adding tooling files (scripts, skeleton generators, BPF objects) alongside a kernel feature, the distinction between "source file" and "generated file" must be explicit in the Makefile. Files that are committed to the repository are source files. Files that are generated during the build are generated files. Mixing them in the same directory without explicit rules about which is which creates the class of bug NVIDIA encountered.
Additionally, the fact that a third-party (NVIDIA), not a kernel developer, discovered this bug demonstrates that out-of-tree module builds exercise kernel build infrastructure in ways that in-tree testing does not. Features that ship with kernel tooling (like sched_ext's BPF examples) must be validated against the full range of build configurations, including external module builds.
PATCHES 39–42 — Core Implementation Review
The review of the core implementation (patch 09, the ~4000-line main sched_ext patch) generated the most extensive discussion threads. Key themes:
DSQ lock ordering. The core implementation uses several locks: the runqueue lock, per-DSQ
locks, and the scx_tasks_lock that protects the global task list. Reviewers spent significant
effort verifying the lock ordering is consistent (no cycles, correct nesting). Patch 39 captures
the review thread that identified a potential lock ordering issue in the DSQ destruction path,
where scx_bpf_destroy_dsq() was acquiring locks in a different order than scx_ops_disable()
when both ran concurrently.
The scx_ops_bypass() invariant. The bypass mechanism (patch 26) was reviewed in the
context of the core implementation to verify it was invoked early enough. Reviewers constructed
scenarios where a task could receive a BPF callback during an ongoing scx_ops_enable() call
(before bypass mode was cleared), potentially calling BPF helpers on a partially initialized
scheduler state. Patch 40 captures the fix: ensuring bypass mode is cleared as the very last
step of scx_ops_enable(), after all data structures are fully initialized.
Interaction with cgroups. The scx_cgroup_* functions in the core implementation
manage the relationship between sched_ext tasks and cgroup hierarchy changes. Reviewers from
the cgroups maintainer community examined these functions for correct handling of the
cgroup css_task_iter locking rules. Patch 41 captures the resulting discussion and a fix
to ensure scx_cgroup_can_attach() releases the cgroup lock before calling into BPF code
(which may sleep).
BPF verifier bypass paths. A security-focused reviewer examined whether there were any
code paths that allowed BPF programs to bypass the verifier's safety checks — for example,
by calling scx_bpf_* helpers from a context the verifier had not authorized. The review
found that the bpf_prog_type check in the helper registration was correct and that all
scx_bpf_* helpers were only accessible from BPF_PROG_TYPE_STRUCT_OPS programs. Patch 42
captures this review and adds a comment in the code explaining why the prog_type restriction
is a security boundary, not just a validation convenience.
The Post-Submission Process as a Learning Resource
For a maintainer, the community follow-up patches are valuable for reasons beyond the specific bugs and discussions they contain:
Pattern recognition for review. The types of issues raised — lock ordering, bypass invariants, cgroup locking, build system semantics — are the same issues that arise in any large scheduler patch. Reading through these threads trains a reviewer to ask the right questions about future patches.
The cost of imprecision in documentation. The documentation review (patches 32–33) shows that ambiguous language in API documentation leads to user mistakes and follow-up questions. The investment in precise documentation at merge time pays off in reduced support burden.
Third-party integration as a test vector. The NVIDIA build bug (patches 36–38) shows that
user-space tooling shipped with a kernel feature must be tested against third-party build
workflows, not just in-tree builds. Any kernel feature that ships tooling (BPF skeletons,
test programs, Python scripts) should have CI coverage for mrproper, make clean, and
out-of-tree module builds.
Security review depth. The BPF verifier bypass review (patch 42) shows that the kernel community expects explicit documentation of security boundaries in code comments. "This check is correct because we only allow X prog_type" is not obvious from the code; stating it as a comment prevents future refactoring from accidentally removing the check while thinking it is redundant.
What to Focus On
For a maintainer, the critical lessons from this group:
-
Every lock acquire has a story. The lock ordering bug found in patches 39–40 demonstrates that even experienced kernel developers can introduce lock ordering issues in complex code. When reviewing sched_ext patches that touch the DSQ lock, runqueue lock, or
scx_tasks_lock, draw the lock ordering graph explicitly and verify it is acyclic. -
Bypass mode is an invariant, not a hint. The bypass mode bug (patch 40) — where BPF callbacks could be called before bypass mode was properly cleared — shows that the enable/disable sequencing of bypass mode is a hard invariant. When reviewing changes to
scx_ops_enable()orscx_ops_disable_workfn(), verify that bypass mode transitions happen at the exact correct point in the sequence. -
Cgroup locking rules are strict. The cgroup locking fix (patch 41) is a reminder that the cgroup subsystem has its own locking rules that are independent of the scheduler's locking rules. Any sched_ext code that calls into cgroup code must follow the cgroup locking documentation precisely. Calling BPF code (which may sleep via
bpf_spin_lock) while holding a cgroup lock is one specific pattern to watch for. -
Makefile semantics for feature tooling. The NVIDIA bug (patches 36–38) established the rule: files committed to the repository are source files and must not be deleted by
make mrproper. When reviewing patches that add tooling files (BPF skeletons, test scripts, Python tools), verify that the Makefile correctly classifies them as source vs. generated and thatmrproperbehavior is tested. -
Security boundary documentation is mandatory. The BPF verifier boundary review (patch 42) established that access control checks must be documented in comments explaining why they are security boundaries. When reviewing new
scx_bpf_*helpers or new BPF prog_type allowlists, require explicit comments stating the security intent of each access control check. -
Review threads as institutional memory. The discussions in patches 32–42 will never be in the source tree — they live only in LKML archives. For a maintainer, subscribing to the
linux-kernel@vger.kernel.organdlinux-sched@vger.kernel.orglists and archiving the sched_ext review threads is essential for understanding the why behind the what in the code. When a future contributor proposes reverting one of the decisions made during this review, the archived threads provide the evidence for why that decision was made.
Re: [PATCH 04/30] sched: Add sched_class->switching_to() and expose check_class_changing/changed()
View on Lore: https://lore.kernel.org/all/20240621165327.GA51310@lorien.usersys.redhat.com
Commit Message
On Tue, Jun 18, 2024 at 11:17:19AM -1000 Tejun Heo wrote:
> When a task switches to a new sched_class, the prev and new classes are
> notified through ->switched_from() and ->switched_to(), respectively, after
> the switching is done.
>
> A new BPF extensible sched_class will have callbacks that allow the BPF
> scheduler to keep track of relevant task states (like priority and cpumask).
> Those callbacks aren't called while a task is on a different sched_class.
> When a task comes back, we wanna tell the BPF progs the up-to-date state
"wanna" ? How about "want to"?
That makes me wanna stop reading right there... :)
> before the task gets enqueued, so we need a hook which is called before the
> switching is committed.
>
> This patch adds ->switching_to() which is called during sched_class switch
> through check_class_changing() before the task is restored. Also, this patch
> exposes check_class_changing/changed() in kernel/sched/sched.h. They will be
> used by the new BPF extensible sched_class to implement implicit sched_class
> switching which is used e.g. when falling back to CFS when the BPF scheduler
> fails or unloads.
>
> This is a prep patch and doesn't cause any behavior changes. The new
> operation and exposed functions aren't used yet.
>
> v3: Refreshed on top of tip:sched/core.
>
> v2: Improve patch description w/ details on planned use.
>
> Signed-off-by: Tejun Heo <tj@kernel.org>
> Reviewed-by: David Vernet <dvernet@meta.com>
> Acked-by: Josh Don <joshdon@google.com>
> Acked-by: Hao Luo <haoluo@google.com>
> Acked-by: Barret Rhoden <brho@google.com>
> ---
> kernel/sched/core.c | 12 ++++++++++++
> kernel/sched/sched.h | 3 +++
> kernel/sched/syscalls.c | 1 +
> 3 files changed, 16 insertions(+)
>
> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> index 48f9d00d0666..b088fbeaf26d 100644
> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
> @@ -2035,6 +2035,17 @@ inline int task_curr(const struct task_struct *p)
> return cpu_curr(task_cpu(p)) == p;
> }
>
> +/*
> + * ->switching_to() is called with the pi_lock and rq_lock held and must not
> + * mess with locking.
> + */
> +void check_class_changing(struct rq *rq, struct task_struct *p,
> + const struct sched_class *prev_class)
> +{
> + if (prev_class != p->sched_class && p->sched_class->switching_to)
> + p->sched_class->switching_to(rq, p);
> +}
Does this really need wrapper? The compiler may help but it doesn't seem to
but you're doing a function call and passing in prev_class just to do a
simple check. I guess it's not really a fast path. Just seemed like overkill.
I guess I did read past the commit message ...
Cheers,
Phil
> +
> /*
> * switched_from, switched_to and prio_changed must _NOT_ drop rq->lock,
> * use the balance_callback list if you want balancing.
> @@ -7021,6 +7032,7 @@ void rt_mutex_setprio(struct task_struct *p, struct task_struct *pi_task)
> }
>
> __setscheduler_prio(p, prio);
> + check_class_changing(rq, p, prev_class);
>
> if (queued)
> enqueue_task(rq, p, queue_flag);
> diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
> index a2399ccf259a..0ed4271cedf5 100644
> --- a/kernel/sched/sched.h
> +++ b/kernel/sched/sched.h
> @@ -2322,6 +2322,7 @@ struct sched_class {
> * cannot assume the switched_from/switched_to pair is serialized by
> * rq->lock. They are however serialized by p->pi_lock.
> */
> + void (*switching_to) (struct rq *this_rq, struct task_struct *task);
> void (*switched_from)(struct rq *this_rq, struct task_struct *task);
> void (*switched_to) (struct rq *this_rq, struct task_struct *task);
> void (*reweight_task)(struct rq *this_rq, struct task_struct *task,
> @@ -3608,6 +3609,8 @@ extern void set_load_weight(struct task_struct *p, bool update_load);
> extern void enqueue_task(struct rq *rq, struct task_struct *p, int flags);
> extern void dequeue_task(struct rq *rq, struct task_struct *p, int flags);
>
> +extern void check_class_changing(struct rq *rq, struct task_struct *p,
> + const struct sched_class *prev_class);
> extern void check_class_changed(struct rq *rq, struct task_struct *p,
> const struct sched_class *prev_class,
> int oldprio);
> diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
> index ae1b42775ef9..cf189bc3dd18 100644
> --- a/kernel/sched/syscalls.c
> +++ b/kernel/sched/syscalls.c
> @@ -797,6 +797,7 @@ int __sched_setscheduler(struct task_struct *p,
> __setscheduler_prio(p, newprio);
> }
> __setscheduler_uclamp(p, attr);
> + check_class_changing(rq, p, prev_class);
>
> if (queued) {
> /*
> --
> 2.45.2
>
>
--
Diff
No diff found.
Implementation Analysis
What This Email Addresses
This is Phil Auld's initial review of PATCH 04/30, which adds sched_class->switching_to() and exposes check_class_changing/changed(). Phil raises two separate issues in one email:
-
A style nit in the commit message: The word "wanna" in Tejun's commit message description. Phil flags it with some humor ("That makes me wanna stop reading right there...") but the underlying point is serious: kernel commit messages are permanent record and informal contractions are out of place.
-
A technical question about the
check_class_changing()wrapper: Phil questions whether the wrapper function is necessary. The implementation is only four lines: check if the class changed, then callswitching_toif it exists. Phil wonders if the compiler overhead of the function call and theprev_classparameter passing is justified for what amounts to a trivial guard.
The Technical Question in Depth
The check_class_changing() function Phil questions:
void check_class_changing(struct rq *rq, struct task_struct *p,
const struct sched_class *prev_class)
{
if (prev_class != p->sched_class && p->sched_class->switching_to)
p->sched_class->switching_to(rq, p);
}
Phil's concern: this function just wraps a pointer comparison and an optional function call. Why not inline it at both call sites (rt_mutex_setprio and __sched_setscheduler) rather than creating a dedicated wrapper?
This is a legitimate question about code organization vs. performance. The scheduler's hot paths are sensitive to unnecessary function calls. However, Phil himself hedges: "I guess it's not really a fast path."
Why This Matters for sched_ext Design
The switching_to() callback is critical to sched_ext's correctness. When a task switches back to SCX from another class, the BPF scheduler needs to be told the task's current state (weight, cpumask, etc.) before the task gets enqueued. The existing switched_to() callback fires after the switch is committed — too late for BPF to set up per-task state that influences the first scheduling decision.
The wrapper exists to mirror the existing check_class_changed() pattern, keeping the class-switching notification code symmetric and readable.
What the Community Decided
Tejun acknowledged the "wanna" typo and promised to fix it. On the wrapper question, Tejun's response (in patch-34.md) appeals to symmetry with check_class_changed() — a design consistency argument rather than a performance one.
Design Insights Revealed
This thread illustrates two kinds of review feedback that maintainers will regularly receive:
- Style feedback (easy to fix, just do it)
- "Is this necessary?" structural feedback (requires justification, not just correction)
The symmetry argument Tejun uses is a common and valid justification in the kernel: keeping analogous operations structured identically reduces cognitive overhead when reading the code, even if each individual instance is slightly over-engineered.
What Maintainers Should Know
When adding new lifecycle hooks to sched_class, follow the existing pattern for how those hooks are called. The kernel uses wrapper functions (check_class_changing, check_class_changed) rather than open-coding the "did the class change?" guard at every call site. This centralizes the logic and ensures future changes (e.g., adding tracing or a new condition) only need to happen in one place.
Re: [PATCH 04/30] sched: Add sched_class->switching_to() and expose check_class_changing/changed()
View on Lore: https://lore.kernel.org/all/ZnXSFrn6wNqk21GS@slm.duckdns.org
Commit Message
Hello, Phil.
On Fri, Jun 21, 2024 at 12:53:27PM -0400, Phil Auld wrote:
> > A new BPF extensible sched_class will have callbacks that allow the BPF
> > scheduler to keep track of relevant task states (like priority and cpumask).
> > Those callbacks aren't called while a task is on a different sched_class.
> > When a task comes back, we wanna tell the BPF progs the up-to-date state
>
> "wanna" ? How about "want to"?
>
> That makes me wanna stop reading right there... :)
Sorry about that. Have been watching for it recently but this log was
written a while ago, so...
> > +/*
> > + * ->switching_to() is called with the pi_lock and rq_lock held and must not
> > + * mess with locking.
> > + */
> > +void check_class_changing(struct rq *rq, struct task_struct *p,
> > + const struct sched_class *prev_class)
> > +{
> > + if (prev_class != p->sched_class && p->sched_class->switching_to)
> > + p->sched_class->switching_to(rq, p);
> > +}
>
> Does this really need wrapper? The compiler may help but it doesn't seem to
> but you're doing a function call and passing in prev_class just to do a
> simple check. I guess it's not really a fast path. Just seemed like overkill.
This doesn't really matter either way but wouldn't it look weird if it's not
symmetric with check_class_changed()?
Thanks.
--
tejun
Diff
No diff found.
Implementation Analysis
What This Email Addresses
This is Tejun's reply to Phil Auld's review of PATCH 04/30. Tejun responds to both points Phil raised:
-
On "wanna": Tejun simply apologizes and acknowledges the fix is needed. The commit message was written earlier and wasn't caught before sending. This is common in long-running patch series where the commit messages are drafted well before the final submission.
-
On the
check_class_changing()wrapper: Tejun's justification is symmetry withcheck_class_changed(). The question "wouldn't it look weird if it's not symmetric?" is rhetorical — the answer is yes. In kernel code, consistency between analogous mechanisms is a strong convention.
The Symmetry Argument Explained
The existing code has:
void check_class_changed(struct rq *rq, struct task_struct *p,
const struct sched_class *prev_class, int oldprio)
This fires after the class switch completes and calls switched_from() on the old class and switched_to() on the new class. It has been in the kernel for years.
The new check_class_changing() mirrors it structurally:
- Same call convention (rq, task, prev_class)
- Same placement at class-change call sites
- Only difference: fires before the switch commits and calls
switching_to()on the new class
Tejun's point is that a reader familiar with check_class_changed() will immediately understand check_class_changing() without needing to think about it. Asymmetry — e.g., open-coding one but wrapping the other — would require the reader to ask "why is this different?" without a good answer.
What the Community Decided
Phil accepted the symmetry argument in his follow-up (patch-35.md): "Fair enough. It was just a thought." The wrapper stays. The "wanna" typo gets fixed in the next revision.
Design Insights Revealed
This exchange shows how Tejun approaches design review: he doesn't dismiss the question but provides the actual reason for the choice. The reason here is not performance or correctness but readability and consistency — both of which are first-class concerns in scheduler code that will be read and maintained for years.
What Maintainers Should Know
When reviewing sched_ext patches that add new sched_class callbacks, look for whether the invocation pattern is consistent with existing callbacks. New hooks that are called differently from established patterns (e.g., some wrapped, some inlined) introduce cognitive overhead and invite "why?" questions that slow down review. Consistent patterns reduce review friction.
Re: [PATCH 04/30] sched: Add sched_class->switching_to() and expose check_class_changing/changed()
View on Lore: https://lore.kernel.org/all/20240621193223.GB51310@lorien.usersys.redhat.com
Commit Message
On Fri, Jun 21, 2024 at 09:18:46AM -1000 Tejun Heo wrote:
> Hello, Phil.
>
> On Fri, Jun 21, 2024 at 12:53:27PM -0400, Phil Auld wrote:
> > > A new BPF extensible sched_class will have callbacks that allow the BPF
> > > scheduler to keep track of relevant task states (like priority and cpumask).
> > > Those callbacks aren't called while a task is on a different sched_class.
> > > When a task comes back, we wanna tell the BPF progs the up-to-date state
> >
> > "wanna" ? How about "want to"?
> >
> > That makes me wanna stop reading right there... :)
>
> Sorry about that. Have been watching for it recently but this log was
> written a while ago, so...
>
> > > +/*
> > > + * ->switching_to() is called with the pi_lock and rq_lock held and must not
> > > + * mess with locking.
> > > + */
> > > +void check_class_changing(struct rq *rq, struct task_struct *p,
> > > + const struct sched_class *prev_class)
> > > +{
> > > + if (prev_class != p->sched_class && p->sched_class->switching_to)
> > > + p->sched_class->switching_to(rq, p);
> > > +}
> >
> > Does this really need wrapper? The compiler may help but it doesn't seem to
> > but you're doing a function call and passing in prev_class just to do a
> > simple check. I guess it's not really a fast path. Just seemed like overkill.
>
> This doesn't really matter either way but wouldn't it look weird if it's not
> symmetric with check_class_changed()?
Fair enough. It was just a thought.
Cheers,
Phil
>
> Thanks.
>
> --
> tejun
>
--
Diff
No diff found.
Implementation Analysis
What This Email Addresses
This is Phil Auld's closing reply in the three-message review thread about PATCH 04/30's switching_to() hook and the check_class_changing() wrapper. Phil accepts Tejun's symmetry argument with "Fair enough. It was just a thought."
This closes the technical discussion. The wrapper stays, the "wanna" gets fixed in the next revision, and the patch is effectively approved.
Why This Closing Email Matters
In kernel development, a reviewer withdrawing an objection is a meaningful signal. Phil raised a legitimate question (is the wrapper necessary?) and Tejun gave a concrete reason (symmetry). Phil's acceptance means:
- No
Requested-bychange is needed — the code stays as written - Phil's review can be credited as an approval rather than a blocking concern
- The thread is cleanly resolved, which matters when the patch maintainer assembles the final series for submission
A thread that ends without explicit resolution can come back during merge window review, where a maintainer might see the unresolved discussion and ask questions again.
The Full Thread in Context
Reading patches 33, 34, and 35 together shows the complete lifecycle of a minor review concern:
- patch-33: Phil raises the style nit and the structural question
- patch-34: Tejun acknowledges the typo and defends the structure with the symmetry argument
- patch-35: Phil accepts, thread closes cleanly
This is the normal, healthy pattern for kernel code review. Not every question requires a code change — sometimes the right outcome is the reviewer understanding the rationale and being satisfied with it.
Design Insights Revealed
The switching_to() callback itself, which this entire thread is about, exists because sched_ext needs a "pre-switch" notification that no prior scheduler class ever needed. All existing classes know their tasks' state at all times. SCX's BPF scheduler doesn't — its callbacks aren't called while a task is on a different class. So when a task returns to SCX, the BPF program needs to be told the current state before the task is placed on a DSQ.
This is a fundamental consequence of sched_ext's architecture: the BPF scheduler is an observer of kernel state, not a first-class participant in it. Keeping it synchronized requires explicit notifications at lifecycle transitions.
What Maintainers Should Know
When a reviewer raises a question and the author provides a rationale, the reviewer's explicit acknowledgment ("fair enough") is the expected closing for that thread. If a reviewer raises an issue and never follows up after the author responds, the concern is typically considered resolved by the community — but it's better practice to explicitly close it. As a maintainer, you can help by summarizing thread resolutions in your merge or tag messages.
Re: [PATCH 10/30] sched_ext: Add scx_simple and scx_example_qmap example schedulers
View on Lore: https://lore.kernel.org/all/ac065f1f-8754-4626-95db-2c9fcf02567b@nvidia.com
Commit Message
Hi Tejun,
On 18/06/2024 22:17, Tejun Heo wrote:
> Add two simple example BPF schedulers - simple and qmap.
>
> * simple: In terms of scheduling, it behaves identical to not having any
> operation implemented at all. The two operations it implements are only to
> improve visibility and exit handling. On certain homogeneous
> configurations, this actually can perform pretty well.
>
> * qmap: A fixed five level priority scheduler to demonstrate queueing PIDs
> on BPF maps for scheduling. While not very practical, this is useful as a
> simple example and will be used to demonstrate different features.
>
> v7: - Compat helpers stripped out in prepartion of upstreaming as the
> upstreamed patchset will be the baselinfe. Utility macros that can be
> used to implement compat features are kept.
>
> - Explicitly disable map autoattach on struct_ops to avoid trying to
> attach twice while maintaining compatbility with older libbpf.
>
> v6: - Common header files reorganized and cleaned up. Compat helpers are
> added to demonstrate how schedulers can maintain backward
> compatibility with older kernels while making use of newly added
> features.
>
> - simple_select_cpu() added to keep track of the number of local
> dispatches. This is needed because the default ops.select_cpu()
> implementation is updated to dispatch directly and won't call
> ops.enqueue().
>
> - Updated to reflect the sched_ext API changes. Switching all tasks is
> the default behavior now and scx_qmap supports partial switching when
> `-p` is specified.
>
> - tools/sched_ext/Kconfig dropped. This will be included in the doc
> instead.
>
> v5: - Improve Makefile. Build artifects are now collected into a separate
> dir which change be changed. Install and help targets are added and
> clean actually cleans everything.
>
> - MEMBER_VPTR() improved to improve access to structs. ARRAY_ELEM_PTR()
> and RESIZEABLE_ARRAY() are added to support resizable arrays in .bss.
>
> - Add scx_common.h which provides common utilities to user code such as
> SCX_BUG[_ON]() and RESIZE_ARRAY().
>
> - Use SCX_BUG[_ON]() to simplify error handling.
>
> v4: - Dropped _example prefix from scheduler names.
>
> v3: - Rename scx_example_dummy to scx_example_simple and restructure a bit
> to ease later additions. Comment updates.
>
> - Added declarations for BPF inline iterators. In the future, hopefully,
> these will be consolidated into a generic BPF header so that they
> don't need to be replicated here.
>
> v2: - Updated with the generic BPF cpumask helpers.
>
> Signed-off-by: Tejun Heo <tj@kernel.org>
> Reviewed-by: David Vernet <dvernet@meta.com>
> Acked-by: Josh Don <joshdon@google.com>
> Acked-by: Hao Luo <haoluo@google.com>
> Acked-by: Barret Rhoden <brho@google.com>
Our farm builders are currently failing to build -next and I am seeing the following error ...
f76698bd9a8c (HEAD -> refs/heads/buildbrain-branch, refs/remotes/m/master) Add linux-next specific files for 20240621
build-linux.sh: kernel_build - make mrproper
Makefile:83: *** Cannot find a vmlinux for VMLINUX_BTF at any of " ../../vmlinux /sys/kernel/btf/vmlinux /boot/vmlinux-4.15.0-136-generic". Stop.
Makefile:192: recipe for target 'sched_ext_clean' failed
make[2]: *** [sched_ext_clean] Error 2
Makefile:1361: recipe for target 'sched_ext' failed
make[1]: *** [sched_ext] Error 2
Makefile:240: recipe for target '__sub-make' failed
make: *** [__sub-make] Error 2
Reverting this change fixes the build. Any thoughts on what is happening here?
Thanks!
Jon
--
nvpublic
Diff
No diff found.
Implementation Analysis
What This Email Addresses
Jon Hunter (NVIDIA) reports that make mrproper is failing on linux-next builds that don't have a BTF-enabled kernel image available. The error message is:
Makefile:83: *** Cannot find a vmlinux for VMLINUX_BTF at any of
"../../vmlinux /sys/kernel/btf/vmlinux /boot/vmlinux-4.15.0-136-generic". Stop.
Makefile:192: recipe for target 'sched_ext_clean' failed
Jon bisected it to commit 2a52ca7c9896 ("sched_ext: Add scx_simple and scx_example_qmap example schedulers"), which added a tools_clean target to the top-level Makefile. Jon confirms that reverting that commit fixes the build.
Why This Bug Exists
The sched_ext BPF example schedulers require BTF (BPF Type Format) information extracted from a running kernel's vmlinux image. When the examples were added to tools/sched_ext/, their Makefile included logic to find a vmlinux for VMLINUX_BTF — this is needed to build the BPF skeletons.
The bug is that this BTF-finding logic runs even during sched_ext_clean, which is invoked by mrproper. Clean targets should never have build requirements — they exist to delete artifacts, not build them. A make mrproper on a fresh checkout (or on a machine without any kernel installed) should always succeed.
The root cause is a Makefile design error: the sched_ext_clean target inherited the full tools/sched_ext/Makefile dependency resolution, including the VMLINUX_BTF search, even though clean targets don't need it.
Why This Matters for sched_ext
This failure mode is particularly visible to people doing automated kernel builds and CI — exactly the audience that tests linux-next. NVIDIA's build farm hit it immediately. The failure is also confusing because mrproper is supposed to be the most reliable "start from scratch" target, and failing at that level suggests broken infrastructure.
More broadly, adding tools/ subtree targets to the top-level mrproper without verifying that those targets are unconditionally safe is a category of Makefile mistake that can affect any developer whose machine doesn't meet the build environment assumptions.
What the Community Decided
Tejun responded quickly with a fix (patch-37.md): drop the tools_clean target from the top-level Makefile entirely, removing sched_ext from mrproper's dependency chain. Jon confirmed the fix works (patch-38.md).
Design Insights Revealed
This bug reveals a tension in sched_ext's placement: the BPF scheduler tools live under tools/sched_ext/ (user-space build), but the kernel build system (Makefile) tried to integrate them into the standard mrproper target. That integration was premature — the tools have their own build requirements (BTF-enabled kernel) that the kernel build system cannot guarantee are met.
The fix defers the question of how to properly integrate tools-side cleaning with the kernel's top-level Makefile. Tejun notes in the fix: "The offending Makefile line is shared across BPF targets under tools/. Let's revisit them later."
What Maintainers Should Know
When adding tools/ subdirectory targets to the top-level Linux Makefile (especially to mrproper), verify that those targets are unconditionally safe: they must not require any external dependencies (no vmlinux, no running kernel, no installed headers). Clean targets that fail on fresh checkouts are a CI-breaking change and will be reported quickly by build farm operators. If a tools/ clean target has requirements, it should be in the tools' own Makefile, invocable explicitly, but not wired into the kernel's top-level clean targets.
[PATCH sched_ext/for-6.11] sched_ext: Drop tools_clean target from the top-level Makefile
View on Lore: https://lore.kernel.org/all/ZnokS4YL71S61g71@slm.duckdns.org
Commit Message
2a52ca7c9896 ("sched_ext: Add scx_simple and scx_example_qmap example
schedulers") added the tools_clean target which is triggered by mrproper.
The tools_clean target triggers the sched_ext_clean target in tools/. This
unfortunately makes mrproper fail when no BTF enabled kernel image is found:
Makefile:83: *** Cannot find a vmlinux for VMLINUX_BTF at any of " ../../vmlinux /sys/kernel/btf/vmlinux/boot/vmlinux-4.15.0-136-generic". Stop.
Makefile:192: recipe for target 'sched_ext_clean' failed
make[2]: *** [sched_ext_clean] Error 2
Makefile:1361: recipe for target 'sched_ext' failed
make[1]: *** [sched_ext] Error 2
Makefile:240: recipe for target '__sub-make' failed
make: *** [__sub-make] Error 2
Clean targets shouldn't fail like this but also it's really odd for mrproper
to single out and trigger the sched_ext_clean target when no other clean
targets under tools/ are triggered.
Fix builds by dropping the tools_clean target from the top-level Makefile.
The offending Makefile line is shared across BPF targets under tools/. Let's
revisit them later.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reported-by: Jon Hunter <jonathanh@nvidia.com>
Link: http://lkml.kernel.org/r/ac065f1f-8754-4626-95db-2c9fcf02567b@nvidia.com
Fixes: 2a52ca7c9896 ("sched_ext: Add scx_simple and scx_example_qmap example schedulers")
Cc: David Vernet <void@manifault.com>
---
Jon, this should fix it. I'll route this through sched_ext/for-6.11.
Thanks.
Makefile | 8 +-------
1 file changed, 1 insertion(+), 7 deletions(-)
--- a/Makefile
+++ b/Makefile
@@ -1355,12 +1355,6 @@ ifneq ($(wildcard $(resolve_btfids_O)),)
$(Q)$(MAKE) -sC $(srctree)/tools/bpf/resolve_btfids O=$(resolve_btfids_O) clean
endif
-tools-clean-targets := sched_ext
-PHONY += $(tools-clean-targets)
-$(tools-clean-targets):
- $(Q)$(MAKE) -sC tools $@_clean
-tools_clean: $(tools-clean-targets)
-
# Clear a bunch of variables before executing the submake
ifeq ($(quiet),silent_)
tools_silent=s
@@ -1533,7 +1527,7 @@ PHONY += $(mrproper-dirs) mrproper
$(mrproper-dirs):
$(Q)$(MAKE) $(clean)=$(patsubst _mrproper_%,%,$@)
-mrproper: clean $(mrproper-dirs) tools_clean
+mrproper: clean $(mrproper-dirs)
$(call cmd,rmfiles)
@find . $(RCS_FIND_IGNORE) \
\( -name '*.rmeta' \) \
Diff
---
Jon, this should fix it. I'll route this through sched_ext/for-6.11.
Thanks.
Makefile | 8 +-------
1 file changed, 1 insertion(+), 7 deletions(-)
--- a/Makefile
+++ b/Makefile
@@ -1355,12 +1355,6 @@ ifneq ($(wildcard $(resolve_btfids_O)),)
$(Q)$(MAKE) -sC $(srctree)/tools/bpf/resolve_btfids O=$(resolve_btfids_O) clean
endif
-tools-clean-targets := sched_ext
-PHONY += $(tools-clean-targets)
-$(tools-clean-targets):
- $(Q)$(MAKE) -sC tools $@_clean
-tools_clean: $(tools-clean-targets)
-
# Clear a bunch of variables before executing the submake
ifeq ($(quiet),silent_)
tools_silent=s
@@ -1533,7 +1527,7 @@ PHONY += $(mrproper-dirs) mrproper
$(mrproper-dirs):
$(Q)$(MAKE) $(clean)=$(patsubst _mrproper_%,%,$@)
-mrproper: clean $(mrproper-dirs) tools_clean
+mrproper: clean $(mrproper-dirs)
$(call cmd,rmfiles)
@find . $(RCS_FIND_IGNORE) \
\( -name '*.rmeta' \) \
Implementation Analysis
What This Patch Fixes
This is a targeted bug fix for the mrproper failure reported by Jon Hunter (NVIDIA) in patch-36.md. The fix is a deletion-only change to the top-level Makefile: it removes the tools_clean target and the tools-clean-targets variable that wired sched_ext_clean into mrproper.
Code Analysis
The patch removes two sections from the top-level Makefile:
Section 1 — The tools_clean mechanism:
-tools-clean-targets := sched_ext
-PHONY += $(tools-clean-targets)
-$(tools-clean-targets):
- $(Q)$(MAKE) -sC tools $@_clean
-tools_clean: $(tools-clean-targets)
This mechanism defined a variable tools-clean-targets containing sched_ext, declared it as a PHONY target, and created a rule that would descend into tools/ and invoke sched_ext_clean. The tools_clean target aggregated these.
Section 2 — The mrproper dependency:
-mrproper: clean $(mrproper-dirs) tools_clean
+mrproper: clean $(mrproper-dirs)
This single line change removes tools_clean from mrproper's prerequisite list. Without this, mrproper would invoke tools_clean, which invokes sched_ext_clean, which invokes the full tools/sched_ext/Makefile, which requires finding a vmlinux for VMLINUX_BTF — and fails if none exists.
Why Deletion Is the Right Fix
The correct fix is not to make sched_ext_clean tolerate missing BTF (though that would also work) — it's to not call it from mrproper at all. The kernel's top-level mrproper target is a baseline operation that must work unconditionally. Tying it to tools that have runtime build requirements violates this guarantee.
Tejun's commit message makes an important observation: mrproper was singling out the sched_ext clean target when "no other clean targets under tools/ are triggered." This inconsistency was itself a red flag — the integration was incomplete and asymmetric compared to other tools.
The comment "The offending Makefile line is shared across BPF targets under tools/" signals that there is a broader issue to address later: other BPF-related tools under tools/ may have similar Makefile patterns, and a proper integration with the kernel's clean targets will need to be designed more carefully.
Routing and Urgency
The commit message shows Tejun routing this fix through sched_ext/for-6.11, the staging branch for the 6.11 release cycle. This is the correct channel for a fix to a patch that's already in linux-next — it goes through the sched_ext tree rather than waiting for the next merge window. The Fixes: tag pointing to the introducing commit (2a52ca7c9896) and the Reported-by: crediting Jon Hunter follow standard kernel fix conventions.
What Maintainers Should Know
When a patch from your tree breaks linux-next (especially something as fundamental as mrproper), the expected response is:
- Acknowledge the report quickly
- Produce a minimal fix — do not refactor unrelated code
- Route through the appropriate -next branch with a
Fixes:tag andReported-by:attribution - Confirm the fix with the reporter before or shortly after sending
This patch is a model example of that pattern. The fix is surgical (8 lines removed, 1 changed), correctly attributed, and addresses exactly the reported failure mode without scope creep.
When adding new tools/ subdirectory targets to the top-level mrproper, verify that those targets are unconditionally safe: they must not require any external dependencies (no vmlinux, no running kernel, no installed headers).
Re: [PATCH sched_ext/for-6.11] sched_ext: Drop tools_clean target from the top-level Makefile
View on Lore: https://lore.kernel.org/all/ef376c13-2ca8-4f25-9cbd-fdca37351190@nvidia.com
Commit Message
On 25/06/2024 02:58, Tejun Heo wrote:
> 2a52ca7c9896 ("sched_ext: Add scx_simple and scx_example_qmap example
> schedulers") added the tools_clean target which is triggered by mrproper.
> The tools_clean target triggers the sched_ext_clean target in tools/. This
> unfortunately makes mrproper fail when no BTF enabled kernel image is found:
>
> Makefile:83: *** Cannot find a vmlinux for VMLINUX_BTF at any of " ../../vmlinux /sys/kernel/btf/vmlinux/boot/vmlinux-4.15.0-136-generic". Stop.
> Makefile:192: recipe for target 'sched_ext_clean' failed
> make[2]: *** [sched_ext_clean] Error 2
> Makefile:1361: recipe for target 'sched_ext' failed
> make[1]: *** [sched_ext] Error 2
> Makefile:240: recipe for target '__sub-make' failed
> make: *** [__sub-make] Error 2
>
> Clean targets shouldn't fail like this but also it's really odd for mrproper
> to single out and trigger the sched_ext_clean target when no other clean
> targets under tools/ are triggered.
>
> Fix builds by dropping the tools_clean target from the top-level Makefile.
> The offending Makefile line is shared across BPF targets under tools/. Let's
> revisit them later.
>
> Signed-off-by: Tejun Heo <tj@kernel.org>
> Reported-by: Jon Hunter <jonathanh@nvidia.com>
> Link: http://lkml.kernel.org/r/ac065f1f-8754-4626-95db-2c9fcf02567b@nvidia.com
> Fixes: 2a52ca7c9896 ("sched_ext: Add scx_simple and scx_example_qmap example schedulers")
> Cc: David Vernet <void@manifault.com>
> ---
> Jon, this should fix it. I'll route this through sched_ext/for-6.11.
>
> Thanks.
>
> Makefile | 8 +-------
> 1 file changed, 1 insertion(+), 7 deletions(-)
>
> --- a/Makefile
> +++ b/Makefile
> @@ -1355,12 +1355,6 @@ ifneq ($(wildcard $(resolve_btfids_O)),)
> $(Q)$(MAKE) -sC $(srctree)/tools/bpf/resolve_btfids O=$(resolve_btfids_O) clean
> endif
>
> -tools-clean-targets := sched_ext
> -PHONY += $(tools-clean-targets)
> -$(tools-clean-targets):
> - $(Q)$(MAKE) -sC tools $@_clean
> -tools_clean: $(tools-clean-targets)
> -
> # Clear a bunch of variables before executing the submake
> ifeq ($(quiet),silent_)
> tools_silent=s
> @@ -1533,7 +1527,7 @@ PHONY += $(mrproper-dirs) mrproper
> $(mrproper-dirs):
> $(Q)$(MAKE) $(clean)=$(patsubst _mrproper_%,%,$@)
>
> -mrproper: clean $(mrproper-dirs) tools_clean
> +mrproper: clean $(mrproper-dirs)
> $(call cmd,rmfiles)
> @find . $(RCS_FIND_IGNORE) \
> \( -name '*.rmeta' \) \
Fix it for me!
Tested-by: Jon Hunter <jonathanh@nvidia.com>
Thanks!
Jon
--
nvpublic
Diff
No diff found.
Implementation Analysis
What This Email Addresses
Jon Hunter (NVIDIA) confirms that Tejun's fix (patch-37.md) resolves the mrproper build failure he reported in patch-36.md. The reply is short: "Fix it for me!" followed by a Tested-by: tag.
The "Fix it for me!" line is friendly acknowledgment that the fix does exactly what was needed. The Tested-by: tag is the formal record.
The Tested-by Tag
A Tested-by: tag in a kernel patch has specific meaning: the credited person actually ran the code (or in this case, verified the build) and confirms it behaves correctly. It is distinct from Reviewed-by (code reviewed, logic sound) and Acked-by (I agree with this approach). For a build fix, Tested-by is the most directly meaningful tag: it says "I hit the failure, I applied the fix, the failure is gone."
This is particularly valuable here because Jon's build environment — NVIDIA's farm builders running linux-next on Ubuntu 4.15.0 kernels — was the one that exposed the bug. His confirmation that the fix works in that specific environment closes the loop.
The Full Bug-Fix Cycle in Context
Reading patches 36, 37, and 38 together shows the complete lifecycle of a build regression fix in the kernel:
- patch-36: Jon reports the
mrproperfailure with a precise error message and bisection result (reverts the introducing commit) - patch-37: Tejun produces a minimal fix, routes it through the sched_ext branch, addresses Jon directly
- patch-38: Jon confirms the fix works, provides
Tested-by
The turnaround is fast (reported Jun 21, fix sent Jun 25), which is important for a bug that breaks linux-next for any developer without a BTF-enabled kernel on their machine.
Design Insights Revealed
This thread demonstrates the value of linux-next as an integration testing surface. The bug was introduced in the main patch series and might not have been caught until after the 6.11 merge window if not for automated build farms like NVIDIA's testing linux-next continuously. For subsystems like sched_ext that integrate with the build system (tools Makefiles, top-level mrproper), having external testers with diverse build environments catches assumptions that the primary developers don't notice because their machines always have the right setup.
What Maintainers Should Know
When you receive a Tested-by: on a fix from the person who reported the original bug, include both the Reported-by: and Tested-by: tags in the final committed patch. The commit history then shows the complete picture: who found the bug, who fixed it, and who verified the fix. Tejun's fix already includes Reported-by: Jon Hunter — the Tested-by from this email would be added when the commit is finalized.
Re: [PATCH 09/30] sched_ext: Implement BPF extensible scheduler class
View on Lore: https://lore.kernel.org/all/Zn0joEebAdwjiTyT@gpd
Commit Message
On Tue, Jun 18, 2024 at 11:17:24AM -1000, Tejun Heo wrote:
...
> + /**
> + * set_weight - Set task weight
> + * @p: task to set weight for
> + * @weight: new eight [1..10000]
Small nit: eight -> weight
> + *
> + * Update @p's weight to @weight.
> + */
> + void (*set_weight)(struct task_struct *p, u32 weight);
-Andrea
Diff
No diff found.
Implementation Analysis
What This Email Addresses
Andrea Righi spots a typo in the sched_ext_ops kernel-doc comment for the set_weight callback:
/**
* set_weight - Set task weight
* @p: task to set weight for
* @weight: new eight [1..10000] ← "eight" should be "weight"
*
* Update @p's weight to @weight.
*/
void (*set_weight)(struct task_struct *p, u32 weight);
The word "eight" in the @weight parameter description should be "weight". This is a documentation typo that does not affect compilation or behavior.
Why set_weight Matters for sched_ext
The set_weight callback was added in v3 of the patch series (noted in the commit message: "ops.set_weight() added to allow BPF schedulers to track weight changes without polling p->scx.weight"). Without this callback, a BPF scheduler that wants to implement weight-aware scheduling has to either poll p->scx.weight on every scheduling decision (expensive) or miss updates entirely.
The callback's weight range [1..10000] maps to the Linux scheduler's nice-value-derived weight range. Weight 1 is the minimum (nice +19 equivalent), weight 10000 is an approximation of the nice 0 base weight (1024 in the kernel's SCHED_NORMAL scheme), and higher values correspond to negative nice values. BPF schedulers that implement proportional-share or weighted fair queueing need this value.
What the Community Decided
This is a trivial typo fix — it will be corrected in the next revision of the patch. No design changes are implied.
Design Insights Revealed
The set_weight callback is an example of a pattern repeated throughout sched_ext_ops: rather than having BPF schedulers poll task fields directly, sched_ext provides callbacks that notify the BPF scheduler when something changes. This push model is important because:
- BPF programs cannot safely dereference arbitrary task_struct fields without explicit whitelisting
- Polling from
ops.dispatch()orops.enqueue()would add overhead on every scheduling event - State changes (weight, cpumask, priority) happen infrequently; push notifications are efficient
The full set of "state notification" callbacks in sched_ext_ops includes set_weight, set_cpumask, and update_idle. Understanding that these exist as a group — and why they exist — is essential for writing correct BPF schedulers.
What Maintainers Should Know
Kernel-doc typos in sched_ext_ops callback descriptions are worth catching: those comments are the primary documentation BPF scheduler authors read when implementing a callback. Unlike internal kernel comments, these are part of the public BPF interface documentation and will appear in generated API docs. Reviewers like Andrea Righi who read the kernel-doc carefully are providing genuine value by catching these.
Re: [PATCH 09/30] sched_ext: Implement BPF extensible scheduler class
View on Lore: https://lore.kernel.org/all/20240807191004.GB47824@pauld.westford.csb
Commit Message
Hi Tejun,
On Tue, Jun 18, 2024 at 11:17:24AM -1000 Tejun Heo wrote:
> Implement a new scheduler class sched_ext (SCX), which allows scheduling
> policies to be implemented as BPF programs to achieve the following:
>
I looks like this is slated for v6.12 now? That would be good. My initial
experimentation with scx has been positive.
I just picked one email, not completely randomly.
> - Both enable and disable paths are a bit complicated. The enable path
> switches all tasks without blocking to avoid issues which can arise from
> partially switched states (e.g. the switching task itself being starved).
> The disable path can't trust the BPF scheduler at all, so it also has to
> guarantee forward progress without blocking. See scx_ops_enable() and
> scx_ops_disable_workfn().
I think, from a supportability point of view, there needs to be a pr_info, at least,
in each of these places, enable and disable, with the name of the scx scheduler. It
looks like there is at least a pr_error for when one gets ejected due to misbehavior.
But there needs to be a record of when such is loaded and unloaded.
Thoughts?
Cheers,
Phil
Diff
No diff found.
Implementation Analysis
What This Email Addresses
Phil Auld reviews PATCH 09/30 — the core sched_ext implementation — with two observations:
-
Timeline confirmation: Phil notes the patch appears slated for v6.12 and mentions his initial experimentation with SCX has been positive. This is significant community feedback: a Red Hat engineer who has tested the scheduler in practice and finds it works is meaningful evidence of real-world readiness.
-
Supportability concern: Phil raises a substantive operational issue about the
scx_ops_enable()andscx_ops_disable_workfn()paths. He notes that while there is error logging when a BPF scheduler is forcibly ejected due to misbehavior, there is nopr_info(informational log message) when a scheduler is successfully loaded or unloaded. His recommendation: at minimum, log the scheduler's name when it loads and when it unloads.
Why This Matters Operationally
Phil's concern is about supportability in production environments. In a typical Linux production deployment:
- An operator might enable a BPF scheduler as part of a performance optimization
- If the system later experiences scheduling anomalies, the first question is "which BPF scheduler was loaded, and when?"
- Without load/unload logging, this information is lost unless the operator was watching
dmesgat exactly the right moment
The sched_ext disable path already logs errors when a BPF scheduler misbehaves (scx_ops_error() generates a pr_err). But the normal load and unload paths were silent. This asymmetry means you can find out why a scheduler crashed, but not when it was running.
The Broader "Observability" Principle
Phil's feedback touches on a general principle in systems software: subsystem lifecycle events should be logged at an appropriate level even when they succeed. For a subsystem as significant as the scheduler, pr_info at load/unload is the minimum bar. This allows:
- Post-incident analysis correlating scheduling changes with system behavior changes
- Audit trails for environments with compliance requirements
- Debugging of unexpected failovers from SCX back to CFS
What the Community Decided
Tejun accepted the feedback immediately: "Sure, that's not difficult. Will do so soon." (patch-41.md). Phil offered to write the patch himself but deferred to Tejun once Tejun committed to doing it.
Design Insights Revealed
The scx_ops_enable() function in the initial patch series focused on correctness — switching all tasks atomically, setting up data structures, enabling static branches. Operational observability (logging) was a lower priority during development. Phil's review catching this before merge is the right time: adding pr_info calls to lifecycle paths is a non-controversial change that's easy to get right at merge time rather than as a follow-up patch in 6.13.
What Maintainers Should Know
When reviewing new subsystems, check that lifecycle transitions (init, load, unload, error) are all observable through the kernel log at appropriate severity levels:
pr_info: normal load/unload — "sched_ext: BPF scheduler 'scx_simple' enabled"pr_warn: unusual but non-fatal conditionspr_err: forced disable due to error
The sched_ext framework is unique in that it can be loaded and unloaded at runtime, which makes this particularly important — the scheduler is not a static kernel configuration choice.
Re: [PATCH 09/30] sched_ext: Implement BPF extensible scheduler class
View on Lore: https://lore.kernel.org/all/ZrPKZMvrl6kGFzo-@slm.duckdns.org
Commit Message
Hello, Phil.
On Wed, Aug 07, 2024 at 03:11:08PM -0400, Phil Auld wrote:
> On Tue, Jun 18, 2024 at 11:17:24AM -1000 Tejun Heo wrote:
> > Implement a new scheduler class sched_ext (SCX), which allows scheduling
> > policies to be implemented as BPF programs to achieve the following:
> >
>
> I looks like this is slated for v6.12 now? That would be good. My initial
> experimentation with scx has been positive.
Yeap and great to hear.
> I just picked one email, not completely randomly.
>
> > - Both enable and disable paths are a bit complicated. The enable path
> > switches all tasks without blocking to avoid issues which can arise from
> > partially switched states (e.g. the switching task itself being starved).
> > The disable path can't trust the BPF scheduler at all, so it also has to
> > guarantee forward progress without blocking. See scx_ops_enable() and
> > scx_ops_disable_workfn().
>
> I think, from a supportability point of view, there needs to be a pr_info, at least,
> in each of these places, enable and disable, with the name of the scx scheduler. It
> looks like there is at least a pr_error for when one gets ejected due to misbehavior.
> But there needs to be a record of when such is loaded and unloaded.
Sure, that's not difficult. Will do so soon.
Thanks.
--
tejun
Diff
No diff found.
Implementation Analysis
What This Email Addresses
This is Tejun's reply to Phil Auld's review of PATCH 09/30 (patch-40.md). Tejun responds to both points Phil raised:
-
v6.12 timeline: Tejun confirms ("Yeap") and acknowledges Phil's positive experimentation.
-
pr_info for load/unload: Tejun accepts the feedback without qualification — "Sure, that's not difficult. Will do so soon." This means the pr_info additions will be included in the revision submitted for 6.12, not deferred.
Significance of Immediate Acceptance
The speed and completeness of Tejun's acceptance matters. He doesn't argue for deferring this to a follow-up patch, doesn't ask for clarification, and doesn't propose a narrower version of the change. He simply agrees to do it. This is the appropriate response when a reviewer identifies a missing operational feature that:
- Is clearly correct (logging lifecycle events is unambiguously right)
- Has no design controversy (nobody argues against knowing when things load/unload)
- Is low-risk to implement (pr_info calls don't affect scheduling behavior)
For a patch series that has been through 7 iterations and is at the finish line for 6.12 merging, committing to this change rather than merging without it is a sign of good maintainer judgment — better to get it right now than ship without it.
The Timeline Context
This email is dated August 7, 2024 — nearly two months after the patch series was originally posted (June 18, 2024). The 6.12 merge window is approaching, and Phil is reviewing the patch series relatively late. Tejun's "will do so soon" reflects awareness that there's limited time but the change is simple enough to fit in.
What the Community Decided
Tejun committed to adding the pr_info calls. Phil explicitly offered to write the patch himself but was happy to let Tejun handle it (patch-42.md). The change was to be made before the 6.12 merge.
Design Insights Revealed
The exchange highlights a principle about when to include observability in a patch series versus adding it as follow-up work:
- During development: Focus on correctness, correctness, correctness. Logging can wait.
- Before merge: Add the minimum observability needed for production use. Load/unload logging for a runtime-loadable subsystem meets this bar.
- After merge: Add richer observability, tracing, and debugging hooks as the community identifies specific needs.
The pr_info additions Tejun committed to fit squarely in the "before merge" category — they are necessary for the feature to be usable in production without surprise.
What Maintainers Should Know
When you commit to a review feedback item with "will do so soon," follow through before the merge window closes. Reviewers who see their feedback addressed build trust in the maintainer process; reviewers who see accepted feedback silently dropped lose confidence. In this case, adding the pr_info calls is also a factual promise (not just a policy commitment), and the kernel community will remember if a maintainer's "will do so soon" doesn't appear in the final commit.
Re: [PATCH 09/30] sched_ext: Implement BPF extensible scheduler class
View on Lore: https://lore.kernel.org/all/20240807210431.GB80631@pauld.westford.csb
Commit Message
Hi Tejun,
On Wed, Aug 07, 2024 at 09:26:28AM -1000 Tejun Heo wrote:
> Hello, Phil.
>
> On Wed, Aug 07, 2024 at 03:11:08PM -0400, Phil Auld wrote:
> > On Tue, Jun 18, 2024 at 11:17:24AM -1000 Tejun Heo wrote:
> > > Implement a new scheduler class sched_ext (SCX), which allows scheduling
> > > policies to be implemented as BPF programs to achieve the following:
> > >
> >
> > I looks like this is slated for v6.12 now? That would be good. My initial
> > experimentation with scx has been positive.
>
> Yeap and great to hear.
>
> > I just picked one email, not completely randomly.
> >
> > > - Both enable and disable paths are a bit complicated. The enable path
> > > switches all tasks without blocking to avoid issues which can arise from
> > > partially switched states (e.g. the switching task itself being starved).
> > > The disable path can't trust the BPF scheduler at all, so it also has to
> > > guarantee forward progress without blocking. See scx_ops_enable() and
> > > scx_ops_disable_workfn().
> >
> > I think, from a supportability point of view, there needs to be a pr_info, at least,
> > in each of these places, enable and disable, with the name of the scx scheduler. It
> > looks like there is at least a pr_error for when one gets ejected due to misbehavior.
> > But there needs to be a record of when such is loaded and unloaded.
>
> Sure, that's not difficult. Will do so soon.
Thanks! That would be helpful. I was going to offer a patch but wanted to ask first
in case there was history.
But if you are willing to do it that's even better :)
Cheers,
Phil
>
> Thanks.
>
> --
> tejun
>
--
Diff
No diff found.
Implementation Analysis
What This Email Addresses
This is Phil Auld's closing reply in the PATCH 09/30 review thread about load/unload logging. Phil thanks Tejun for accepting the feedback, mentions he was ready to offer a patch himself if needed, and closes the thread with "if you are willing to do it that's even better :)".
This closes the three-message thread on PATCH 09/30 (patches 40, 41, 42). The feedback is accepted, the change is committed to, and no further action is needed from Phil.
The Full Thread in Context
Reading patches 40, 41, and 42 together shows a complete review cycle for a substantive (but non-controversial) feedback item:
- patch-40: Phil reviews PATCH 09/30, raises the supportability concern about missing pr_info at load/unload
- patch-41: Tejun accepts, commits to adding it
- patch-42: Phil acknowledges, closes the thread
The offer to write the patch himself ("I was going to offer a patch but wanted to ask first in case there was history") shows good kernel community etiquette: before sending unsolicited patches to a subsystem, it's polite to ask whether the maintainer has reasons for the current behavior. There might be history — maybe the logging was intentionally omitted, or maybe a larger observability redesign was planned. In this case, there was no such history, and Tejun simply hadn't gotten around to it.
Why Phil's Offer Matters
Phil's offer to write the patch, even though Tejun ultimately declined it, is meaningful for two reasons:
-
It accelerates the work: If Tejun had been slow to follow up, Phil had the option to send a patch directly and Tejun could review and apply it. This is the standard "if you're busy I can help" offer in open source communities.
-
It demonstrates engagement: Phil isn't just pointing out a problem and moving on — he's willing to invest time in fixing it. This kind of engaged review is more valuable to maintainers than pure criticism.
Design Insights Revealed
The PATCH 09/30 review thread (patches 40-42) is notable for what it doesn't discuss: the core design of sched_ext itself. By the time of this review, the fundamental architecture — DSQs, sched_ext_ops, the enable/disable paths, the task ownership model — was settled and apparently sound enough that reviewers were looking at operational concerns (logging) rather than architectural ones. This is a sign that the patch series had matured significantly across its 7 revisions.
The fact that Phil's initial experimentation with SCX was "positive" is also meaningful signal: it suggests the implementation is not only theoretically correct but practically usable, which is a high bar for a new scheduler class.
What Maintainers Should Know
Threads that end with the reporter offering to help are among the healthiest in open source kernel development — they signal that the community is engaged and wants the feature to succeed. As a maintainer, when someone offers to write a patch for missing functionality they identified, it's good practice to either accept the offer (with guidance on style and approach) or commit to a timeline for doing it yourself. Leaving such offers hanging without a response discourages future contributions.
Re: [PATCH 29/30] sched_ext: Documentation: scheduler: Document extensible scheduler class
View on Lore: https://lore.kernel.org/all/ZnLLAWbryU0-aqX1@archie.me
Commit Message
On Tue, Jun 18, 2024 at 11:17:44AM -1000, Tejun Heo wrote:
> Add Documentation/scheduler/sched-ext.rst which gives a high-level overview
> and pointers to the examples.
>
LGTM, thanks!
Reviewed-by: Bagas Sanjaya <bagasdotme@gmail.com>
--
An old man doll... just what I always wanted! - Clara
Diff
-----BEGIN PGP SIGNATURE-----
iHUEABYKAB0WIQSSYQ6Cy7oyFNCHrUH2uYlJVVFOowUCZnLK/AAKCRD2uYlJVVFO
o/m3AQDvlkShSHUwxaqKX5DV/HcD2PbL5R+9f+zPNpLRV5IVSwEAqHcCISspS8dU
UWAGJ9pJNmfVZ9sLaGeXH4uN2TvS0wU=
=ciJC
-----END PGP SIGNATURE-----
--QTBRT25tDLXH91dy--
Implementation Analysis
What This Email Addresses
This is a review acknowledgment from Bagas Sanjaya for PATCH 29/30, which adds Documentation/scheduler/sched-ext.rst — the official in-kernel documentation for the sched_ext framework.
The email is minimal: Bagas confirms the documentation patch looks good ("LGTM") and provides a Reviewed-by tag. The PGP signature at the bottom is standard practice for kernel contributors asserting the review's authenticity.
Why Documentation Matters for a New Scheduler Class
Adding a new scheduler class to the Linux kernel is a significant ABI and API commitment. Kernel documentation in Documentation/scheduler/ sets expectations for downstream users, distro packagers, and future contributors about:
- What sched_ext is and why it exists
- How to write a BPF scheduler
- Where to find the example schedulers
- The lifecycle of a BPF scheduler (load, run, unload, error handling)
A Reviewed-by from a documentation maintainer (Bagas Sanjaya maintains Linux kernel documentation) carries specific weight: it signals that the documentation is technically clear, well-formatted, and follows kernel documentation conventions (reStructuredText, appropriate cross-references, etc.).
What the Community Decided
The documentation patch was accepted as-is. No changes were requested. This is the expected outcome for documentation that accompanied a working implementation with multiple prior review rounds — by v7 of the patch series, the design was stable and the docs reflected it accurately.
Design Insights Revealed
The existence of this dedicated documentation patch — rather than inline comments alone — reflects the sched_ext team's philosophy that BPF schedulers are meant to be written by people who are not kernel scheduler experts. The documentation provides the high-level mental model needed to write a scheduler without needing to read the kernel source directly.
What Maintainers Should Know
A Reviewed-by on a documentation patch from a doc maintainer is part of the normal merge checklist for new subsystems. When reviewing future sched_ext patches that add new ops or change BPF interfaces, check whether Documentation/scheduler/sched-ext.rst has been updated accordingly. Interface changes without documentation updates are a common review failure mode.