Algorithm Analyzer Tool User Guide【免费下载链接】hcommHCOMMHuawei Communication是HCCL的通信基础库提供通信域以及通信资源的管理能力。项目地址: https://gitcode.com/cann/hcommIntroductionThe HCCL algorithm analyzer is used to simulate the execution of HCCL algorithms in an offline environment, verifying algorithm logic and memory operations. The HCCL algorithm analyzer provides efficient and fast batch execution of test tasks to meet developer needs.PrinciplesKey points:The algorithm analyzer stubs the platform and framework layers to obtain the task sequence for all ranks during algorithm execution.The task information of all ranks is organized into aDirected Acyclic Graph (DAG).Validations are performed based ongraph algorithms, such as memory read/write conflict detection and semantic validation. 1) Memory conflict detection analyzes whether there are potential read/write conflicts based on synchronization within the graph. 2) Semantic validation simulates the execution of the task graph, recordsdata movement information, and checks whether thedata movement informationin the UserOutput memory meets the operator requirements after simulation.Environment SetupRefer to the environment setup and source code download in Source Build to prepare the prerequisites for compiling the algorithm analyzer.Test Case WritingLLT Test Case OverviewAn algorithm checker test case consists of 5 steps, as shown in the 5 boxes in the figure below. The following sections describe how to write each step to accommodate different operator requirements, as well as how to use the checker tool for issue location when problems arise.LLT Test Case Step DetailsTopology GenerationTopoMeta structure introductionchecker: TopoMeta is used to represent a topology structure. TopoMeta is a three-layer vector structure.PhyDeviceId: Represents the physical ID of an NPU.ServerMeta: Composed of PhyDeviceId, representing the number of cards on a server and their corresponding PhyDeviceIds.SuperPodMeta: Composed of ServerMeta, representing the server composition of a super node.TopoMeta: Represents the overall cluster topology.TopoMeta generation methodsThere are two ways to generate TopoMeta:Specify the number of super nodes, the number of servers, and the number of cards per server, then use theGenTopoMetafunction provided by theRankTable_For_LLTclass to generate it. Suitable for symmetric topology scenarios.Fully customize super nodes, servers, and card counts for asymmetric topology scenarios. As shown below, the TopoMeta has one super node with two servers inside — one server has 2 cards and the other has 3 cards.Rank table generationOnce TopoMeta is available, use theGenRankTablefunction provided by theRankTable_For_LLTclass to generate the rank table.Environment Variable ConfigurationSetting environment variablesEnvironment variables affect the logic flow in the code. Use thesetenvfunction to configure the required conditions before test case execution.Cleaning environment variablesSince environment variables are process-level and an LLT task runs in the same process, environment variable usage may affect other test cases. To clean up environment variables, theTearDownfunction in the test suite currently calls the environment variable cleanup function.The current cleanup function handles the following environment variables. If new environment variables are added in the future, they need to be added to this function.Log Level ConfigurationChecker log level (default: ERROR)Call thesetCheckerLogWarn()interface to set the log levelSet the checker log level to WARNING:Set the log level via environment variableEnable WARNING level logging:export CHECK_LOG_LEVEL2Enable ERROR level logging:export CHECK_LOG_LEVEL3HCCL log level (default: ERROR)The current environment variable method is deprecated. Use the following approach to set the log level and print logs.Enable DEBUG level logging:export ASCEND_LOG_LEVEL0Enable INFO level logging:export ASCEND_LOG_LEVEL1Enable WARNING level logging:export ASCEND_LOG_LEVEL2Enable ERROR level logging:export ASCEND_LOG_LEVEL3Operator Parameter ConfigurationTestOpParamis used to configure test parameters. The table below describes the main parameters used.ParameterRequired or OptionalDescriptionRemarksopTypeRequiredSpecifies the operator type to be testedbatchsendrecv is currently under developmenttagRequiredThe tag of the operator execution entryCan be set arbitrarilyalgNameOptionalSpecifies the algorithm name to executeIf specified, skips algorithm selection; otherwise auto-selectsopModeRequiredSpecifies single operator mode or graph modereduceTypeOptionalRequired when a reduce type is involveddevtypeRequiredThe hardware type for executionSupports 310P3 V / 310P3 Duo / 910A / 910B / 910Cis310P3VOptionalMust be set to true when running on 310P3 V hardwarecountRequiredNumber of data elementsdataTypeRequiredData typeRunning the CheckerPass theTestOpParam,rankTable,TopoMeta, and other parameters generated in the previous steps to theCheckfunction of theCheckerobject for execution.Verifying Check ResultsCheck that the return value ofCheckisHcclResult::HCCL_SUCCESS.LLT Test Case FilteringWhen there are many test cases and you only need to execute a specific one, modify the test case name inmain.cc.Test ExecutionRun the following commands from the source code root directory to compile and execute algorithm analyzer test cases:# Compile all test suite cases and execute automatically bash build.sh --st # Compile individual test suite cases and execute automatically bash build.sh --open_hccl_test bash build.sh --executor_hccl_test bash build.sh --executor_reduce_hccl_test bash build.sh --executor_pipeline_hccl_test # Manually execute test cases ./build/test/st/algorithm/testcase/testcase/open_hccl_test ./build/test/st/algorithm/testcase/testcase/executor_hccl_test ./build/test/st/algorithm/testcase/testcase/executor_reduce_hccl_test ./build/test/st/algorithm/testcase/testcase/executor_pipeline_hccl_testResult ExamplesResult AnalysisThe test case execution results are shown below:The meaning of each field is as follows:[run]: Indicates the test case being executed[OK]: Indicates successful execution and verification passed[FAIL]: Indicates execution failure. Analyze the specific cause based on the printed logs.Issue LocationMemory Conflict DetectionSymptomsWhen a region of memory between two synchronization signals is concurrently written by multiple tasks, or is read while being written, a memory conflict occurs. In real runtime environments, this typically manifests as random accuracy issues.Under the current Mesh structure, false positives may occur when Reduce operators are present. This is because, in a Mesh structure, a memory block may be simultaneously written by other cards within one synchronization interval. In this case, the hardware can ensure the atomicity of the Reduce operation and no accuracy issues occur in actual execution. However, from the checkers perspective, multiple reads and writes to the same memory occur between two synchronizations, so it is flagged as an error.Except for the scenario above, the following error output indicates a risk of memory conflicts in the task scheduling:[1]there is memory use conflict in two SliceMemoryStatus [2]one is startAddr is 0, size is 3200, status is WRITE. [3]another is startAddr is 0, size is 3200, status is WRITE. [4]failed to check memory BufferType::OUTPUT_CCL [5]memory conflict between node [rankId:1, queueId:0, index:1] and node [rankId:2, queueId:0, index:1] [6]check rank memory conflict failed for rank 0Lines 2 and 3 indicate the start address, size, and read/write status of the two conflicting memory blocks.The status can be READ or WRITE. READ means the memory block is being read; WRITE means it is being written. Being read and being written are abstract memory operation semantics, not limited to write tasks and read tasks.Memory blocks that may have READ status include the src of a localcopy task, the src of a read task, and the src of a write task. Memory blocks that may have WRITE status include the dst of a localcopy task, the dst of a read task, and the dst of a write task.Line 4 indicates the type of the conflicting memory block.Line 5 indicates which two tasks caused the memory conflict.Line 6 indicates the rank ID where the memory conflict occurred.The above error log indicates that two tasks are simultaneously performing write operations in the range 0 to 3200 of theOUTPUT_CCLtype.Debugging MethodEnable task printing before calling theCheckfunction.checker.EnableTaskPrint();Based on the error log, locate the two tasks that caused the memory conflict and check the synchronization arrangement before and after these two tasks.The error log in the Symptoms section indicates that two tasks are simultaneously performing write operations in the range 0 to 3200 of theOUTPUT_CCLtype.Semantic Validation FailureBasic ConceptsThe algorithm analyzer uses relative addresses to represent memory, consisting of three fields: memory type, offset address, and size, represented by theDataSlicestructure:class DataSlice { public: // Some method functions private: BufferType type; u64 offset; u64 size; }The supported memory types include Input, Output, CCL_Input, CCL_Output, Scratch, etc.During collective communication algorithm execution, complex data movement and reduction operations are involved. The algorithm analyzer usesBufferSemanticto recorddata movement relationships, which includes a destination memory expression and multiple source memory expressions. The destination memory is represented by the member variablesstartAddrandSize. The source memory is represented by theSrcBufDesstructure, which is defined as follows:struct BufferSemantic { u64 startAddr; mutable u64 size; // Size, shared between source and destination memory mutable bool isReduce; // Whether a reduce operation was performed; when multiple srcBufs exist, it must be a reduce scenario mutable HcclReduce0p reduceType; // Type of reduce operation mutable std::setSrcBufDes srcBufs; // The rank or ranks from which this data originates }; struct SrcBufDes { RankId rankId; // Rank ID of the data source BufferType bufType; // Memory type of the data source mutable u64 srcAddr; // Offset address relative to the source memory type };Semantic Calculation ExampleThe following example illustrates what semantic calculation is.Initial state: There are two ranks, Rank0 and Rank1, with two memory types: Input and Output.Action of state 1: Copy the data block from rank0s Input at offset address 20, size 30, to rank0s Output at offset address 35. Result: A semantic block is generated on rank0s Output, recording the movement information.Action of state 2: Copy the data block from rank1s Input at offset address 70, size 15, to rank0s Output at offset address 50. Result: The destination memory overlaps with an existing semantic block, so the existing semantic block needs to be split, producing two semantic blocks.Result ValidationDuring semantic analysis execution, many semantic blocks are generated (recording many data movement relationships). After execution, the semantic blocks in the Output memory are checked against expectations.The following uses a 2-rank AllGather example to illustrate the normal and abnormal scenarios for the semantic blocks in Rank0s Output memory. Assume the input data size is 100 bytes.Correct scenario:Incorrect scenario:Debugging ApproachThe semantic validation phase can detect two types of errors:Missing data.Incorrect data source.For reduction scenarios, similar issues can occur, such as missing participating ranks or different data offset addresses among participating ranks. Typically, the semantic check provides hints when reporting errors. Analysis should be performed using the hints along with the task sequence printed by the algorithm analyzer.【免费下载链接】hcommHCOMMHuawei Communication是HCCL的通信基础库提供通信域以及通信资源的管理能力。项目地址: https://gitcode.com/cann/hcomm创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考
