Introduction to Harbinger

Related Pages

Related topics: Overall System Architecture, Deployment and Configuration

Relevant source files

Introduction to Harbinger

Harbinger is a comprehensive command and control (C2) framework designed for adversary emulation and red teaming operations. It provides capabilities for managing implants, proxies, tasks, and file operations, integrating various components to facilitate automated and structured engagements. The system leverages gRPC for robust agent-to-server communication, a FastAPI backend for its user interface and API, and Temporal for orchestrating complex workflows and asynchronous tasks. Harbinger aims to streamline the collection and processing of operational data, enabling in-depth analysis and automation of red team activities.

Core Architecture

Harbinger's architecture is composed of several interconnected services that work together to manage C2 operations, process data, and provide a user interface. The primary components include a FastAPI web application, a gRPC server, Temporal workers, and various Go-based C2 agents/connectors.

The overall system architecture can be visualized as follows:

graph TD
    User -->|Web UI/API| FastAPI_App[FastAPI App]
    FastAPI_App -->|DB Access| PostgreSQL[PostgreSQL DB]
    FastAPI_App -->|Graph DB Access| Neo4j[Neo4j Graph DB]
    FastAPI_App -->|Workflow Initiation| Temporal_Client[Temporal Client]
    Temporal_Client --> Temporal_Server[Temporal Server]
    Temporal_Server --> Temporal_Worker[Temporal Worker]
    Temporal_Worker -->|DB Access| PostgreSQL
    Temporal_Worker -->|Graph DB Access| Neo4j
    Temporal_Worker -->|File Storage| S3_Compatible[S3-Compatible Storage]
    C2_Agent[Go C2 Agent] -->|gRPC| gRPC_Server[gRPC Server]
    gRPC_Server -->|Queue Data| Temporal_Worker
    gRPC_Server -->|File Upload| S3_Compatible

Sources: harbinger/src/harbinger/config/app.py:33-59, harbinger/src/harbinger/rpc/server.py:44-53, Taskfile.yml:18-36, harbinger/src/harbinger/connectors/base.py:40-42

FastAPI Backend

The FastAPI application serves as the main entry point for user interaction and API access. It handles user authentication, serves the web interface, and exposes various CRUD (Create, Read, Update, Delete) endpoints for managing operational data such as domains, hosts, and files. It also acts as a client to the Temporal system, initiating workflows for complex operations. The application uses middleware to manage database sessions per request. Sources: harbinger/src/harbinger/config/app.py:33-59, harbinger/src/harbinger/database/router.py:43-46

Key routers include:

/: Main database CRUD operations.
/files: File management.
/templates: Job template management.
/auth, /users: User authentication and management.
/graph: Graph database interactions. Sources: harbinger/src/harbinger/config/app.py:61-75

# harbinger/src/harbinger/config/app.py
@app.middleware("http")
async def db_session_middleware(request: Request, call_next):
    response = Response("Internal server error", status_code=500)
    try:
        request.state.db = database.SessionLocal()
        response = await call_next(request)
    finally:
        await request.state.db.close()  # type: ignore
    return response

Sources: harbinger/src/harbinger/config/app.py:33-41

gRPC Server

The gRPC server acts as the communication hub for C2 implants and agents. It exposes a Harbinger service with methods for agents to send various types of operational data, including implant check-ins, proxy details, file metadata, task statuses, and task outputs. The server queues incoming data for processing by Temporal workers. Sources: harbinger/src/harbinger/rpc/server.py:44-53, go/proto/v1/messages_grpc.pb.go:34-45

The Harbinger service defines several full method names for client-server communication:

Method Name	Description
`Ping`	Checks server connectivity.
`SaveImplant`	Saves implant details.
`SaveProxy`	Saves proxy information.
`SaveFile`	Saves file metadata.
`C2TaskStatus`	Updates C2 task status.
`GetSettings`	Retrieves C2 server settings.
`SaveTask`	Saves task details.
`SaveTaskOutput`	Saves task command output.
`CheckFileExists`	Checks if a file exists on the server by hash.
`UploadFile`	Uploads file data (client streaming).
`DownloadFile`	Downloads file data (server streaming).
`SetC2ServerStatus`	Sets the status of a C2 server.
Sources: go/proto/v1/messages_grpc.pb.go:34-45

A typical interaction for saving task output involves the C2 agent sending a TaskOutputRequest to the gRPC server, which then processes and queues this data for a Temporal worker.

sequenceDiagram
    participant C2Agent as Go C2 Agent
    participant gRPCServer as gRPC Server
    participant TemporalWorker as Temporal Worker

    C2Agent->>gRPCServer: SaveTaskOutput(TaskOutputRequest)
    activate gRPCServer
    gRPCServer->>gRPCServer: Add TaskOutput to Queue
    gRPCServer-->>C2Agent: TaskOutputResponse
    deactivate gRPCServer
    TemporalWorker->>TemporalWorker: Process TaskOutput from Queue
    TemporalWorker->>PostgreSQL: Save C2OutputCreate

Sources: harbinger/src/harbinger/rpc/server.py:51, harbinger/src/harbinger/connectors/base.py:73-74, go/proto/v1/messages.pb.go:1982-2266

Temporal Workers

Temporal workers are responsible for executing long-running, fault-tolerant workflows and activities. They process data received from the gRPC server, interact with the database, perform file operations (upload/download), and initiate AI-driven analysis. This offloads complex and potentially time-consuming operations from the main API and gRPC servers, ensuring responsiveness and reliability. Sources: harbinger/src/harbinger/worker/activities.py:34-36, harbinger/src/harbinger/rpc/server.py:60-61

Examples of workflows initiated by the FastAPI backend or processed by workers include:

RunPlaybook: Executes a sequence of C2 jobs.
RunC2Job: Executes a single C2 command.
ParseFile: Processes uploaded files, potentially extracting metadata or parsing content.
CreateTimeline, CreateSummaries, CreateChecklist: Generate analytical outputs.
CreateC2ImplantSuggestion, CreateDomainSuggestion, CreateFileSuggestion, PrivEscSuggestions: AI-driven suggestions. Sources: harbinger/src/harbinger/database/router.py:30-40

Activities within these workflows handle specific atomic operations, such as saving implant data, task outputs, or fetching playbook details from the database. Sources: harbinger/src/harbinger/worker/activities.py:53-60, harbinger/src/harbinger/worker/activities.py:65-71

# harbinger/src/harbinger/rpc/server.py
class Harbinger(messages_pb2_grpc.HarbingerServicer):
    def __init__(self, client: Client):
        self.running = True
        self.client = client
        self.implant_queue: asyncio.Queue[schemas.C2ImplantCreate] = asyncio.Queue()
        self.task_queue: asyncio.Queue[schemas.C2TaskCreate] = asyncio.Queue()
        self.task_output_queue: asyncio.Queue[schemas.C2OutputCreate] = asyncio.Queue()
        self.proxy_queue: asyncio.Queue[schemas.ProxyCreate] = asyncio.Queue()
        self.task_status_queue: asyncio.Queue[schemas.C2TaskStatus] = asyncio.Queue()
        self.file_queue: asyncio.Queue[schemas.FileCreate] = asyncio.Queue()

    async def worker_loop(self):
        while self.running:
            while not self.implant_queue.empty():
                c2_implant = self.implant_queue.get_nowait()
                log.debug(c2_implant)
                try:
                    await activities.save_implant(c2_implant)

Sources: harbinger/src/harbinger/rpc/server.py:44-64

Data Models and Schemas

Harbinger defines its data structures using Protocol Buffers (for gRPC communication) and Pydantic schemas (for the Python backend and API validation). These models ensure data consistency and facilitate communication between different services and components.

Protocol Buffer Messages

The core communication between Go C2 agents and the Python gRPC server is defined using Protocol Buffers. These definitions are compiled into Go (.pb.go) and Python (_pb2.py) files.

For instance, the TaskOutputRequest message is used by C2 agents to send command output and associated data (processes, file lists) back to the server:

// go/proto/v1/messages.pb.go (or v1/messages.proto)
message TaskOutputRequest {
  string internal_id = 1;
  string c2_server_id = 2;
  string response_text = 3;
  string output_type = 4;
  string timestamp = 5;
  string internal_task_id = 6;
  string bucket = 7;
  string path = 8;
  repeated Process processes = 9;
  FileList file_list = 11;
}

Sources: harbinger/src/harbinger/proto/v1/messages_pb2.py:2266, go/proto/v1/messages.pb.go:1982-2266

The ShareFile message defines attributes for files shared or discovered on hosts:

// go/proto/v1/messages.pb.go (or v1/messages.proto)
message ShareFile {
  string type = 1;
  int64 size = 2;
  string last_accessed = 3;
  string last_modified = 4;
  string created = 5;
  string unc_path = 6;
  string name = 7;
}

Sources: harbinger/src/harbinger/proto/v1/messages_pb2.py:1773, go/proto/v1/messages.pb.go:1639-1773

Pydantic Schemas

On the Python backend, Pydantic models are used for data validation, serialization, and deserialization, particularly for API requests and database interactions. The Arguments schema defines common parameters for C2 jobs:

# harbinger/src/harbinger/job_templates/schemas.py
class Arguments(BaseModel):
    command: str = ""
    folder: str = ""
    path: str = ""
    sleep: int = 0
    jitter: int = 0
    remotename: str = ""
    host: str = ""
    arguments_str: str = ""
    source: str = ""
    destination: str = ""
    action: str = ""
    port: int = 0
    filename: str = ""
    powershell: str = ""
    tcp: bool = False
    udp: bool = False
    ipv4: bool = False
    ipv6: bool = False
    listening: bool = False
    pid: int = 0
    force: bool = False
    reghive: str = ""
    regkey: str = ""
    recurse: bool = False
    value: str = ""
    data: str = ""
    task_id: str = ""
    url: str = ""

Sources: harbinger/src/harbinger/job_templates/schemas.py:27-56

This Arguments schema maps closely to the HarbingerArguments struct used in Go workers:

// go/pkg/base_worker/structs.go
type HarbingerArguments struct {
	Sleep       *int   `json:"sleep,omitempty"`
	Jitter      *int   `json:"jitter,omitempty"`
	File        string `json:"file,omitempty"`
	Remotename  string `json:"remotename,omitempty"`
	Path        string `json:"path,omitempty"`
	Host        string `json:"host,omitempty"`
	Arguments   string `json:"arguments_str,omitempty"`
	Safe        bool   `json:"safe,omitempty"`
	Source      string `json:"source,omitempty"`
	Dest        string `json:"dest,omitempty"`
	Port        int    `json:"port,omitempty"`
	Action      string `json:"action,omitempty"`
	Command     string `json:"command,omitempty"`
	Folder      string `json:"folder,omitempty"`
	Destination string `json:"destination,omitempty"`
	Filename    string `json:"filename,omitempty"`
	Cmdline     string `json:"cmdline,omitempty"`
	Hwbp        bool   `json:"hwbp,omitempty"`
}

Sources: go/pkg/base_worker/structs.go:60-80

The ShareFile interface in the frontend TypeScript models also reflects the data structure for file information:

// harbinger/interface/src/models.ts
export interface ShareFile {
  type: string;
  file_id: string;
  parent_id: string;
  share_id: string;
  size: number;
  last_accessed: string;
  last_modified: string;
  created: string;
  unc_path: string;
  depth: number;
  name: string;
  downloaded: boolean;
  indexed: boolean;
  id: string;
  time_created: string;
  extension: string;
}

Sources: harbinger/interface/src/models.ts:36-53

Job and Task Execution

Harbinger facilitates the execution of C2 jobs and playbooks. Playbooks are sequences of C2 jobs that can be run on implants. The system uses Temporal workflows (RunPlaybook, RunC2Job) to manage the lifecycle of these operations, ensuring resilience and visibility.

Job Templates

Job templates define the structure and arguments for various C2 commands. These templates are categorized by C2 type (e.g., c2, proxy) and can be retrieved via the FastAPI endpoint /templates/{c2_type}/. Sources: harbinger/src/harbinger/job_templates/router.py:65-72

# harbinger/src/harbinger/job_templates/router.py
@router.get(
    "/{c2_type}/",
    response_model=TemplateList,
    tags=["proxy_jobs", "crud"],
)
async def job_templates(
    c2_type: schemas.C2Type,
    user: models.User = Depends(current_active_user),
    db: AsyncSession = Depends(get_db),
):
    if c2_type == schemas.C2Type.c2:
        return dict(templates=[key for key in C2_JOB_BASE_MAP.keys()])
    if c2_type == schemas.C2Type.proxy:
        return dict(templates=[key for key in PROXY_JOB_BASE_MAP.keys()])

Sources: harbinger/src/harbinger/job_templates/router.py:65-72

C2 Job Building

Go-based C2 connectors, like the Apollo Mythic connector, are responsible for translating generic Harbinger job arguments into specific commands and arguments for their respective C2 frameworks. For example, the BuildApolloTask function maps Harbinger arguments to Apollo-specific task parameters for commands like ps, ls, download, upload, runassembly, and runbof. Sources: go/cmd/mythic_go/apollo.go:30-36, go/cmd/mythic_go/apollo.go:42-106

// go/cmd/mythic_go/apollo.go
func BuildApolloTask(job base_worker.RunJob, file_ids []base_worker.InputFile) ([]base_worker.Task, error) {
	input_arguments := base_worker.HarbingerArguments{}
	tasks := []base_worker.Task{}
	err := json.Unmarshal([]byte(job.C2Job.Arguments), &input_arguments)
	if err != nil {
		return tasks, err
	}
	if len(file_ids) > 0 {
		input_arguments.File = file_ids[0].Id
	}
	arguments := ""
	command := job.C2Job.Command
	switch job.C2Job.Command {
	case "ps":
	case "ls":
		arguments = input_arguments.Path
	case "download":
		arguments = fmt.Sprintf("-Path %s", input_arguments.Path)
	case "sleep":
		arguments = fmt.Sprintf("%d %d", *input_arguments.Sleep, *input_arguments.Jitter)
	case "rm":
		arguments = input_arguments.Path
	case "upload":
		arguments_json := ApolloMythicArguments{
			File:       input_arguments.File,
			RemotePath: input_arguments.Remotename,
			Host:       input_arguments.Host,
		}
		arguments_bytes, err := json.Marshal(arguments_json)
		if err != nil {
			return tasks, err
		}
		arguments = string(arguments_bytes)
	case "runassembly":
		if len(file_ids) == 0 {
			return tasks, fmt.Errorf("no files provided to run task")
		}
		arguments_json := ApolloMythicArguments{
			File: input_arguments.File,
		}
		arguments_bytes, err := json.Marshal(arguments_json)
		if err != nil {
			return tasks, err
		}
		tasks = append(tasks, base_worker.Task{Command: "register_assembly", Arguments: string(arguments_bytes)})
		command = "execute_assembly"
		arguments_json = ApolloMythicArguments{
			AssemblyName:      file_ids[0].Name,
			AssemblyArguments: input_arguments.Arguments,
		}
		arguments_bytes, err = json.Marshal(arguments_json)
		if err != nil {
			return tasks, err
		}
		arguments = string(arguments_bytes)
	case "runbof":
		if len(file_ids) ==

Sources: go/cmd/mythic_go/apollo.go:42-106

File Management and Analysis

Harbinger provides robust capabilities for managing and analyzing files collected during operations. This includes file upload/download, and various parsers for extracting relevant information from different file types.

File Operations

Files are uploaded to and downloaded from an S3-compatible storage backend. The gRPC server handles the streaming of file data during UploadFile (client streaming) and DownloadFile (server streaming) operations. Sources: harbinger/src/harbinger/rpc/server.py:100, harbinger/src/harbinger/connectors/base.py:76, go/proto/v1/messages_grpc.pb.go:43-44

The FileUploader and download_file utilities interact with the file storage. Sources: harbinger/src/harbinger/rpc/server.py:38, harbinger/src/harbinger/worker/activities.py:46

File Parsing

After files are uploaded, Temporal activities can trigger parsing workflows (ParseFile) to extract and process their content. Harbinger uses a system of OutputParser and BaseFileParser classes to handle different data types and formats. Sources: harbinger/src/harbinger/worker/output.py:41-43, harbinger/src/harbinger/worker/files/parsers.py:34-40, harbinger/src/harbinger/database/router.py:32

The OutputParser abstract base class defines the interface for matching and parsing text output, while BaseFileParser handles the processing of file content. Sources: harbinger/src/harbinger/worker/output.py:41-55, harbinger/src/harbinger/worker/files/parsers.py:34-40

# harbinger/src/harbinger/worker/output.py
class OutputParser(abc.ABC):
    needle: list[str] = []
    labels: list[str] = []

    def __init__(self, db: AsyncSession) -> None:
        self.db = db

    @abc.abstractmethod
    async def match(self, text: str) -> bool:
        raise NotImplementedError("This should be implemented")

    @abc.abstractmethod
    async def parse(
        self,
        text: str,
        c2_implant_id: str | UUID4 | None = None,
        c2_output_id: str | UUID4 | None = None,
        file_id: str | UUID4 | None = None,
    ) -> None:
        raise NotImplementedError("This should be implemented")

Sources: harbinger/src/harbinger/worker/output.py:41-55

Specialized parsers, such as those for Certipy JSON output, inherit from BaseFileParser and implement their specific parsing logic to extract structured data. Sources: harbinger/src/harbinger/worker/files/parsers.py:27-28

Conclusion

Harbinger is designed as a modular and extensible platform for red team operations, combining a user-friendly web interface with powerful backend processing capabilities. Its reliance on gRPC for C2 communication, Temporal for workflow orchestration, and a structured approach to data modeling and file analysis enables efficient and scalable adversary emulation. The architecture supports integration with various C2 frameworks and facilitates automated data collection and analysis, enhancing operational effectiveness.

Harbinger Wiki

Home

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduction to Harbinger