小亿 发布的文章

N100

CPU

openvino@5ca032a8cb7e:/opt/intel/openvino_2023.0.0.10926$ ./samples/cpp/samples_bin/benchmark_app -m /tmp/resnet50-binary-0001.xml -d CPU
[Step 1/11] Parsing and validating input arguments
[ INFO ] Parsing input parameters
[Step 2/11] Loading OpenVINO Runtime
[ INFO ] OpenVINO:
[ INFO ] Build ................................. 2023.0.0-10926-b4452d56304-releases/2023/0
[ INFO ] 
[ INFO ] Device info:
[ INFO ] CPU
[ INFO ] Build ................................. 2023.0.0-10926-b4452d56304-releases/2023/0
[ INFO ] 
[ INFO ] 
[Step 3/11] Setting device configuration
[ WARNING ] Performance hint was not explicitly specified in command line. Device(CPU) performance hint will be set to THROUGHPUT.
[Step 4/11] Reading model files
[ INFO ] Loading model files
[ INFO ] Read model took 23.37 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Network inputs:
[ INFO ]     0 (node: 0) : f32 / [N,C,H,W] / [1,3,224,224]
[ INFO ] Network outputs:
[ INFO ]     1463 (node: 1463) : f32 / [...] / [1,1000]
[Step 5/11] Resizing model to match image sizes and given batch
[Step 6/11] Configuring input of the model
[ INFO ] Model batch size: 1
[ INFO ] Network inputs:
[ INFO ]     0 (node: 0) : u8 / [N,C,H,W] / [1,3,224,224]
[ INFO ] Network outputs:
[ INFO ]     1463 (node: 1463) : f32 / [...] / [1,1000]
[Step 7/11] Loading the model to the device
[ INFO ] Compile model took 183.24 ms
[Step 8/11] Querying optimal runtime parameters
[ INFO ] Model:
[ INFO ]   NETWORK_NAME: torch-jit-export
[ INFO ]   OPTIMAL_NUMBER_OF_INFER_REQUESTS: 2
[ INFO ]   NUM_STREAMS: 2
[ INFO ]   AFFINITY: CORE
[ INFO ]   INFERENCE_NUM_THREADS: 4
[ INFO ]   PERF_COUNT: NO
[ INFO ]   INFERENCE_PRECISION_HINT: f32
[ INFO ]   PERFORMANCE_HINT: THROUGHPUT
[ INFO ]   EXECUTION_MODE_HINT: PERFORMANCE
[ INFO ]   PERFORMANCE_HINT_NUM_REQUESTS: 0
[ INFO ]   ENABLE_CPU_PINNING: YES
[ INFO ]   SCHEDULING_CORE_TYPE: ANY_CORE
[ INFO ]   ENABLE_HYPER_THREADING: YES
[ INFO ]   EXECUTION_DEVICES: CPU
[Step 9/11] Creating infer requests and preparing input tensors
[ WARNING ] No input files were given: all inputs will be filled with random values!
[ INFO ] Test Config 0
[ INFO ] 0  ([N,C,H,W], u8, [1,3,224,224], static): random (image/numpy array is expected)
[Step 10/11] Measuring performance (Start inference asynchronously, 2 inference requests, limits: 60000 ms duration)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
[ INFO ] First inference took 34.82 ms
[Step 11/11] Dumping statistics report
[ INFO ] Execution Devices: [ CPU ]
[ INFO ] Count:               3136 iterations
[ INFO ] Duration:            60049.55 ms
[ INFO ] Latency:
[ INFO ]    Median:           37.45 ms
[ INFO ]    Average:          38.28 ms
[ INFO ]    Min:              34.27 ms
[ INFO ]    Max:              81.46 ms
[ INFO ] Throughput:          52.22 FPS

GPU

openvino@5ca032a8cb7e:/opt/intel/openvino_2023.0.0.10926$ ./samples/cpp/samples_bin/benchmark_app -m /tmp/resnet50-binary-0001.xml -d GPU
[Step 1/11] Parsing and validating input arguments
[ INFO ] Parsing input parameters
[Step 2/11] Loading OpenVINO Runtime
[ INFO ] OpenVINO:
[ INFO ] Build ................................. 2023.0.0-10926-b4452d56304-releases/2023/0
[ INFO ] 
[ INFO ] Device info:
[ INFO ] GPU
[ INFO ] Build ................................. 2023.0.0-10926-b4452d56304-releases/2023/0
[ INFO ] 
[ INFO ] 
[Step 3/11] Setting device configuration
[ WARNING ] Performance hint was not explicitly specified in command line. Device(GPU) performance hint will be set to THROUGHPUT.
[Step 4/11] Reading model files
[ INFO ] Loading model files
[ INFO ] Read model took 23.11 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Network inputs:
[ INFO ]     0 (node: 0) : f32 / [N,C,H,W] / [1,3,224,224]
[ INFO ] Network outputs:
[ INFO ]     1463 (node: 1463) : f32 / [...] / [1,1000]
[Step 5/11] Resizing model to match image sizes and given batch
[Step 6/11] Configuring input of the model
[ INFO ] Model batch size: 1
[ INFO ] Network inputs:
[ INFO ]     0 (node: 0) : u8 / [N,C,H,W] / [1,3,224,224]
[ INFO ] Network outputs:
[ INFO ]     1463 (node: 1463) : f32 / [...] / [1,1000]
[Step 7/11] Loading the model to the device
[ INFO ] Compile model took 4854.81 ms
[Step 8/11] Querying optimal runtime parameters
[ INFO ] Model:
[ INFO ]   OPTIMAL_NUMBER_OF_INFER_REQUESTS: 32
[ INFO ]   NETWORK_NAME: torch-jit-export
[ INFO ]   EXECUTION_DEVICES: GPU.0
[ INFO ]   AUTO_BATCH_TIMEOUT: 1000
[ INFO ]   LOADED_FROM_CACHE: NO
[Step 9/11] Creating infer requests and preparing input tensors
[ WARNING ] No input files were given: all inputs will be filled with random values!
[ INFO ] Test Config 0
[ INFO ] 0  ([N,C,H,W], u8, [1,3,224,224], static): random (image/numpy array is expected)
[Step 10/11] Measuring performance (Start inference asynchronously, 32 inference requests, limits: 60000 ms duration)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
[ INFO ] First inference took 979.00 ms
[Step 11/11] Dumping statistics report
[ INFO ] Execution Devices: [ GPU.0 ]
[ INFO ] Count:               6016 iterations
[ INFO ] Duration:            60524.62 ms
[ INFO ] Latency:
[ INFO ]    Median:           319.04 ms
[ INFO ]    Average:          321.27 ms
[ INFO ]    Min:              85.62 ms
[ INFO ]    Max:              331.83 ms
[ INFO ] Throughput:          99.40 FPS

MULTI

openvino@5ca032a8cb7e:/opt/intel/openvino_2023.0.0.10926$ ./samples/cpp/samples_bin/benchmark_app -m /tmp/resnet50-binary-0001.xml -d MULTI:GPU,CPU
[Step 1/11] Parsing and validating input arguments
[ INFO ] Parsing input parameters
[Step 2/11] Loading OpenVINO Runtime
[ INFO ] OpenVINO:
[ INFO ] Build ................................. 2023.0.0-10926-b4452d56304-releases/2023/0
[ INFO ] 
[ INFO ] Device info:
[ INFO ] CPU
[ INFO ] Build ................................. 2023.0.0-10926-b4452d56304-releases/2023/0
[ INFO ] 
[ INFO ] GPU
[ INFO ] Build ................................. 2023.0.0-10926-b4452d56304-releases/2023/0
[ INFO ] 
[ INFO ] MULTI
[ INFO ] Build ................................. 2023.0.0-10926-b4452d56304-releases/2023/0
[ INFO ] 
[ INFO ] 
[Step 3/11] Setting device configuration
[ WARNING ] Performance hint was not explicitly specified in command line. Device(MULTI) performance hint will be set to THROUGHPUT.
[Step 4/11] Reading model files
[ INFO ] Loading model files
[ INFO ] Read model took 22.80 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Network inputs:
[ INFO ]     0 (node: 0) : f32 / [N,C,H,W] / [1,3,224,224]
[ INFO ] Network outputs:
[ INFO ]     1463 (node: 1463) : f32 / [...] / [1,1000]
[Step 5/11] Resizing model to match image sizes and given batch
[Step 6/11] Configuring input of the model
[ INFO ] Model batch size: 1
[ INFO ] Network inputs:
[ INFO ]     0 (node: 0) : u8 / [N,C,H,W] / [1,3,224,224]
[ INFO ] Network outputs:
[ INFO ]     1463 (node: 1463) : f32 / [...] / [1,1000]
[Step 7/11] Loading the model to the device
[ INFO ] Compile model took 5027.79 ms
[Step 8/11] Querying optimal runtime parameters
[ INFO ] Model:
[ INFO ]   PERFORMANCE_HINT: THROUGHPUT
[ INFO ]   NETWORK_NAME: torch-jit-export
[ INFO ]   OPTIMAL_NUMBER_OF_INFER_REQUESTS: 34
[ INFO ]   MODEL_PRIORITY: MEDIUM
[ INFO ]   MULTI_DEVICE_PRIORITIES: GPU,CPU
[ INFO ]   CPU: 
[ INFO ]     CPU_BIND_THREAD: YES
[ INFO ]     CPU_THREADS_NUM: 0
[ INFO ]     CPU_THROUGHPUT_STREAMS: 2
[ INFO ]     DEVICE_ID: 
[ INFO ]     DUMP_EXEC_GRAPH_AS_DOT: 
[ INFO ]     DYN_BATCH_ENABLED: NO
[ INFO ]     DYN_BATCH_LIMIT: 0
[ INFO ]     ENFORCE_BF16: NO
[ INFO ]     EXCLUSIVE_ASYNC_REQUESTS: NO
[ INFO ]     NETWORK_NAME: torch-jit-export
[ INFO ]     OPTIMAL_NUMBER_OF_INFER_REQUESTS: 2
[ INFO ]     PERFORMANCE_HINT: THROUGHPUT
[ INFO ]     PERFORMANCE_HINT_NUM_REQUESTS: 0
[ INFO ]     PERF_COUNT: NO
[ INFO ]   GPU: 
[ INFO ]     AUTO_BATCH_TIMEOUT: 1000
[ INFO ]     EXECUTION_DEVICES: GPU.0
[ INFO ]     NETWORK_NAME: torch-jit-export
[ INFO ]     OPTIMAL_NUMBER_OF_INFER_REQUESTS: 32
[ INFO ]   EXECUTION_DEVICES: GPU CPU
[Step 9/11] Creating infer requests and preparing input tensors
[ WARNING ] No input files were given: all inputs will be filled with random values!
[ INFO ] Test Config 0
[ INFO ] 0  ([N,C,H,W], u8, [1,3,224,224], static): random (image/numpy array is expected)
[Step 10/11] Measuring performance (Start inference asynchronously, 34 inference requests, limits: 60000 ms duration)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
[ INFO ] First inference took 783.98 ms
[Step 11/11] Dumping statistics report
[ INFO ] Execution Devices: [ GPU CPU ]
[ INFO ] Count:               7820 iterations
[ INFO ] Duration:            61118.62 ms
[ INFO ] Throughput:          127.95 FPS

i7 1165G7

CPU

openvino@9594bb13b1f6:/opt/intel/openvino_2023.0.0.10926$ ./samples/cpp/samples_bin/benchmark_app -m /tmp/resnet50-binary-0001.xml -d CPU
[Step 1/11] Parsing and validating input arguments
[ INFO ] Parsing input parameters
[Step 2/11] Loading OpenVINO Runtime
[ INFO ] OpenVINO:
[ INFO ] Build ................................. 2023.0.0-10926-b4452d56304-releases/2023/0
[ INFO ] 
[ INFO ] Device info:
[ INFO ] CPU
[ INFO ] Build ................................. 2023.0.0-10926-b4452d56304-releases/2023/0
[ INFO ] 
[ INFO ] 
[Step 3/11] Setting device configuration
[ WARNING ] Performance hint was not explicitly specified in command line. Device(CPU) performance hint will be set to THROUGHPUT.
[Step 4/11] Reading model files
[ INFO ] Loading model files
[ INFO ] Read model took 14.82 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Network inputs:
[ INFO ]     0 (node: 0) : f32 / [N,C,H,W] / [1,3,224,224]
[ INFO ] Network outputs:
[ INFO ]     1463 (node: 1463) : f32 / [...] / [1,1000]
[Step 5/11] Resizing model to match image sizes and given batch
[Step 6/11] Configuring input of the model
[ INFO ] Model batch size: 1
[ INFO ] Network inputs:
[ INFO ]     0 (node: 0) : u8 / [N,C,H,W] / [1,3,224,224]
[ INFO ] Network outputs:
[ INFO ]     1463 (node: 1463) : f32 / [...] / [1,1000]
[Step 7/11] Loading the model to the device
[ INFO ] Compile model took 145.95 ms
[Step 8/11] Querying optimal runtime parameters
[ INFO ] Model:
[ INFO ]   NETWORK_NAME: torch-jit-export
[ INFO ]   OPTIMAL_NUMBER_OF_INFER_REQUESTS: 4
[ INFO ]   NUM_STREAMS: 4
[ INFO ]   AFFINITY: CORE
[ INFO ]   INFERENCE_NUM_THREADS: 8
[ INFO ]   PERF_COUNT: NO
[ INFO ]   INFERENCE_PRECISION_HINT: f32
[ INFO ]   PERFORMANCE_HINT: THROUGHPUT
[ INFO ]   EXECUTION_MODE_HINT: PERFORMANCE
[ INFO ]   PERFORMANCE_HINT_NUM_REQUESTS: 0
[ INFO ]   ENABLE_CPU_PINNING: YES
[ INFO ]   SCHEDULING_CORE_TYPE: ANY_CORE
[ INFO ]   ENABLE_HYPER_THREADING: YES
[ INFO ]   EXECUTION_DEVICES: CPU
[Step 9/11] Creating infer requests and preparing input tensors
[ WARNING ] No input files were given: all inputs will be filled with random values!
[ INFO ] Test Config 0
[ INFO ] 0  ([N,C,H,W], u8, [1,3,224,224], static): random (image/numpy array is expected)
[Step 10/11] Measuring performance (Start inference asynchronously, 4 inference requests, limits: 60000 ms duration)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
[ INFO ] First inference took 8.27 ms
[Step 11/11] Dumping statistics report
[ INFO ] Execution Devices: [ CPU ]
[ INFO ] Count:               16604 iterations
[ INFO ] Duration:            60015.06 ms
[ INFO ] Latency:
[ INFO ]    Median:           14.59 ms
[ INFO ]    Average:          14.44 ms
[ INFO ]    Min:              8.19 ms
[ INFO ]    Max:              35.32 ms
[ INFO ] Throughput:          276.66 FPS

GPU

openvino@9594bb13b1f6:/opt/intel/openvino_2023.0.0.10926$ ./samples/cpp/samples_bin/benchmark_app -m /tmp/resnet50-binary-0001.xml -d GPU
[Step 1/11] Parsing and validating input arguments
[ INFO ] Parsing input parameters
[Step 2/11] Loading OpenVINO Runtime
[ INFO ] OpenVINO:
[ INFO ] Build ................................. 2023.0.0-10926-b4452d56304-releases/2023/0
[ INFO ] 
[ INFO ] Device info:
[ INFO ] GPU
[ INFO ] Build ................................. 2023.0.0-10926-b4452d56304-releases/2023/0
[ INFO ] 
[ INFO ] 
[Step 3/11] Setting device configuration
[ WARNING ] Performance hint was not explicitly specified in command line. Device(GPU) performance hint will be set to THROUGHPUT.
[Step 4/11] Reading model files
[ INFO ] Loading model files
[ INFO ] Read model took 13.30 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Network inputs:
[ INFO ]     0 (node: 0) : f32 / [N,C,H,W] / [1,3,224,224]
[ INFO ] Network outputs:
[ INFO ]     1463 (node: 1463) : f32 / [...] / [1,1000]
[Step 5/11] Resizing model to match image sizes and given batch
[Step 6/11] Configuring input of the model
[ INFO ] Model batch size: 1
[ INFO ] Network inputs:
[ INFO ]     0 (node: 0) : u8 / [N,C,H,W] / [1,3,224,224]
[ INFO ] Network outputs:
[ INFO ]     1463 (node: 1463) : f32 / [...] / [1,1000]
[Step 7/11] Loading the model to the device
[ INFO ] Compile model took 2835.35 ms
[Step 8/11] Querying optimal runtime parameters
[ INFO ] Model:
[ INFO ]   OPTIMAL_NUMBER_OF_INFER_REQUESTS: 64
[ INFO ]   NETWORK_NAME: torch-jit-export
[ INFO ]   EXECUTION_DEVICES: GPU.0
[ INFO ]   AUTO_BATCH_TIMEOUT: 1000
[ INFO ]   LOADED_FROM_CACHE: NO
[Step 9/11] Creating infer requests and preparing input tensors
[ WARNING ] No input files were given: all inputs will be filled with random values!
[ INFO ] Test Config 0
[ INFO ] 0  ([N,C,H,W], u8, [1,3,224,224], static): random (image/numpy array is expected)
[Step 10/11] Measuring performance (Start inference asynchronously, 64 inference requests, limits: 60000 ms duration)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
[ INFO ] First inference took 985.15 ms
[Step 11/11] Dumping statistics report
[ INFO ] Execution Devices: [ GPU.0 ]
[ INFO ] Count:               24832 iterations
[ INFO ] Duration:            60188.07 ms
[ INFO ] Latency:
[ INFO ]    Median:           154.87 ms
[ INFO ]    Average:          154.95 ms
[ INFO ]    Min:              58.05 ms
[ INFO ]    Max:              158.39 ms
[ INFO ] Throughput:          412.57 FPS

MULTI

openvino@9594bb13b1f6:/opt/intel/openvino_2023.0.0.10926$ ./samples/cpp/samples_bin/benchmark_app -m /tmp/resnet50-binary-0001.xml -d MULTI:GPU,CPU
[Step 1/11] Parsing and validating input arguments
[ INFO ] Parsing input parameters
[Step 2/11] Loading OpenVINO Runtime
[ INFO ] OpenVINO:
[ INFO ] Build ................................. 2023.0.0-10926-b4452d56304-releases/2023/0
[ INFO ] 
[ INFO ] Device info:
[ INFO ] CPU
[ INFO ] Build ................................. 2023.0.0-10926-b4452d56304-releases/2023/0
[ INFO ] 
[ INFO ] GPU
[ INFO ] Build ................................. 2023.0.0-10926-b4452d56304-releases/2023/0
[ INFO ] 
[ INFO ] MULTI
[ INFO ] Build ................................. 2023.0.0-10926-b4452d56304-releases/2023/0
[ INFO ] 
[ INFO ] 
[Step 3/11] Setting device configuration
[ WARNING ] Performance hint was not explicitly specified in command line. Device(MULTI) performance hint will be set to THROUGHPUT.
[Step 4/11] Reading model files
[ INFO ] Loading model files
[ INFO ] Read model took 13.32 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Network inputs:
[ INFO ]     0 (node: 0) : f32 / [N,C,H,W] / [1,3,224,224]
[ INFO ] Network outputs:
[ INFO ]     1463 (node: 1463) : f32 / [...] / [1,1000]
[Step 5/11] Resizing model to match image sizes and given batch
[Step 6/11] Configuring input of the model
[ INFO ] Model batch size: 1
[ INFO ] Network inputs:
[ INFO ]     0 (node: 0) : u8 / [N,C,H,W] / [1,3,224,224]
[ INFO ] Network outputs:
[ INFO ]     1463 (node: 1463) : f32 / [...] / [1,1000]
[Step 7/11] Loading the model to the device
[ INFO ] Compile model took 2946.17 ms
[Step 8/11] Querying optimal runtime parameters
[ INFO ] Model:
[ INFO ]   PERFORMANCE_HINT: THROUGHPUT
[ INFO ]   NETWORK_NAME: torch-jit-export
[ INFO ]   OPTIMAL_NUMBER_OF_INFER_REQUESTS: 68
[ INFO ]   MODEL_PRIORITY: MEDIUM
[ INFO ]   MULTI_DEVICE_PRIORITIES: GPU,CPU
[ INFO ]   CPU: 
[ INFO ]     CPU_BIND_THREAD: YES
[ INFO ]     CPU_THREADS_NUM: 0
[ INFO ]     CPU_THROUGHPUT_STREAMS: 4
[ INFO ]     DEVICE_ID: 
[ INFO ]     DUMP_EXEC_GRAPH_AS_DOT: 
[ INFO ]     DYN_BATCH_ENABLED: NO
[ INFO ]     DYN_BATCH_LIMIT: 0
[ INFO ]     ENFORCE_BF16: NO
[ INFO ]     EXCLUSIVE_ASYNC_REQUESTS: NO
[ INFO ]     NETWORK_NAME: torch-jit-export
[ INFO ]     OPTIMAL_NUMBER_OF_INFER_REQUESTS: 4
[ INFO ]     PERFORMANCE_HINT: THROUGHPUT
[ INFO ]     PERFORMANCE_HINT_NUM_REQUESTS: 0
[ INFO ]     PERF_COUNT: NO
[ INFO ]   GPU: 
[ INFO ]     AUTO_BATCH_TIMEOUT: 1000
[ INFO ]     EXECUTION_DEVICES: GPU.0
[ INFO ]     NETWORK_NAME: torch-jit-export
[ INFO ]     OPTIMAL_NUMBER_OF_INFER_REQUESTS: 64
[ INFO ]   EXECUTION_DEVICES: GPU CPU
[Step 9/11] Creating infer requests and preparing input tensors
[ WARNING ] No input files were given: all inputs will be filled with random values!
[ INFO ] Test Config 0
[ INFO ] 0  ([N,C,H,W], u8, [1,3,224,224], static): random (image/numpy array is expected)
[Step 10/11] Measuring performance (Start inference asynchronously, 68 inference requests, limits: 60000 ms duration)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
[ INFO ] First inference took 831.04 ms
[Step 11/11] Dumping statistics report
[ INFO ] Execution Devices: [ GPU CPU ]
[ INFO ] Count:               32096 iterations
[ INFO ] Duration:            61114.52 ms
[ INFO ] Throughput:          525.18 FPS

i7 10700K + A380

CPU

openvino@0bf0cca613c1:/opt/intel/openvino_2023.3.0.13775$ ./samples/cpp/samples_bin/samples_bin/benchmark_app -m /tmp/resnet50-binary-0001.xml -d CPU          
[Step 1/11] Parsing and validating input arguments
[ INFO ] Parsing input parameters
[Step 2/11] Loading OpenVINO Runtime
[ INFO ] OpenVINO:
[ INFO ] Build ................................. 2023.3.0-13775-ceeafaf64f3-releases/2023/3
[ INFO ] 
[ INFO ] Device info:
[ INFO ] CPU
[ INFO ] Build ................................. 2023.3.0-13775-ceeafaf64f3-releases/2023/3
[ INFO ] 
[ INFO ] 
[Step 3/11] Setting device configuration
[ WARNING ] Performance hint was not explicitly specified in command line. Device(CPU) performance hint will be set to THROUGHPUT.
[Step 4/11] Reading model files
[ INFO ] Loading model files
[ INFO ] Read model took 16.88 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Network inputs:
[ INFO ]     0 (node: 0) : f32 / [N,C,H,W] / [1,3,224,224]
[ INFO ] Network outputs:
[ INFO ]     1463 (node: 1463) : f32 / [...] / [1,1000]
[Step 5/11] Resizing model to match image sizes and given batch
[Step 6/11] Configuring input of the model
[ INFO ] Model batch size: 1
[ INFO ] Network inputs:
[ INFO ]     0 (node: 0) : u8 / [N,C,H,W] / [1,3,224,224]
[ INFO ] Network outputs:
[ INFO ]     1463 (node: 1463) : f32 / [...] / [1,1000]
[Step 7/11] Loading the model to the device
[ INFO ] Compile model took 150.30 ms
[Step 8/11] Querying optimal runtime parameters
[ INFO ] Model:
[ INFO ]   NETWORK_NAME: torch-jit-export
[ INFO ]   OPTIMAL_NUMBER_OF_INFER_REQUESTS: 4
[ INFO ]   NUM_STREAMS: 4
[ INFO ]   AFFINITY: CORE
[ INFO ]   INFERENCE_NUM_THREADS: 16
[ INFO ]   PERF_COUNT: NO
[ INFO ]   INFERENCE_PRECISION_HINT: f32
[ INFO ]   PERFORMANCE_HINT: THROUGHPUT
[ INFO ]   EXECUTION_MODE_HINT: PERFORMANCE
[ INFO ]   PERFORMANCE_HINT_NUM_REQUESTS: 0
[ INFO ]   ENABLE_CPU_PINNING: YES
[ INFO ]   SCHEDULING_CORE_TYPE: ANY_CORE
[ INFO ]   ENABLE_HYPER_THREADING: YES
[ INFO ]   EXECUTION_DEVICES: CPU
[ INFO ]   CPU_DENORMALS_OPTIMIZATION: NO
[ INFO ]   CPU_SPARSE_WEIGHTS_DECOMPRESSION_RATE: 1
[Step 9/11] Creating infer requests and preparing input tensors
[ WARNING ] No input files were given: all inputs will be filled with random values!
[ INFO ] Test Config 0
[ INFO ] 0  ([N,C,H,W], u8, [1,3,224,224], static): random (image/numpy array is expected)
[Step 10/11] Measuring performance (Start inference asynchronously, 4 inference requests, limits: 60000 ms duration)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
[ INFO ] First inference took 8.26 ms
[Step 11/11] Dumping statistics report
[ INFO ] Execution Devices: [ CPU ]
[ INFO ] Count:               12204 iterations
[ INFO ] Duration:            60034.62 ms
[ INFO ] Latency:
[ INFO ]    Median:           19.72 ms
[ INFO ]    Average:          19.66 ms
[ INFO ]    Min:              15.66 ms
[ INFO ]    Max:              31.56 ms
[ INFO ] Throughput:          203.28 FPS

GPU: iGPU

openvino@e25a1984b7fd:/opt/intel/openvino_2023.3.0.13775$ ./samples/cpp/samples_bin/samples_bin/benchmark_app -m /tmp/resnet50-binary-0001.xml -d GPU
[Step 1/11] Parsing and validating input arguments
[ INFO ] Parsing input parameters
[Step 2/11] Loading OpenVINO Runtime
[ INFO ] OpenVINO:
[ INFO ] Build ................................. 2023.3.0-13775-ceeafaf64f3-releases/2023/3
[ INFO ] 
[ INFO ] Device info:
[ INFO ] GPU
[ INFO ] Build ................................. 2023.3.0-13775-ceeafaf64f3-releases/2023/3
[ INFO ] 
[ INFO ] 
[Step 3/11] Setting device configuration
[ WARNING ] Performance hint was not explicitly specified in command line. Device(GPU) performance hint will be set to THROUGHPUT.
[Step 4/11] Reading model files
[ INFO ] Loading model files
[ INFO ] Read model took 51.93 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Network inputs:
[ INFO ]     0 (node: 0) : f32 / [N,C,H,W] / [1,3,224,224]
[ INFO ] Network outputs:
[ INFO ]     1463 (node: 1463) : f32 / [...] / [1,1000]
[Step 5/11] Resizing model to match image sizes and given batch
[Step 6/11] Configuring input of the model
[ INFO ] Model batch size: 1
[ INFO ] Network inputs:
[ INFO ]     0 (node: 0) : u8 / [N,C,H,W] / [1,3,224,224]
[ INFO ] Network outputs:
[ INFO ]     1463 (node: 1463) : f32 / [...] / [1,1000]
[Step 7/11] Loading the model to the device
[ INFO ] Compile model took 5838.75 ms
[Step 8/11] Querying optimal runtime parameters
[ INFO ] Model:
[ INFO ]   OPTIMAL_NUMBER_OF_INFER_REQUESTS: 16
[ INFO ]   SUPPORTED_METRICS: OPTIMAL_NUMBER_OF_INFER_REQUESTS SUPPORTED_METRICS NETWORK_NAME SUPPORTED_CONFIG_KEYS EXECUTION_DEVICES
[ INFO ]   NETWORK_NAME: torch-jit-export
[ INFO ]   SUPPORTED_CONFIG_KEYS: AUTO_BATCH_TIMEOUT
[ INFO ]   EXECUTION_DEVICES: OCL_GPU.0
[ INFO ]   AUTO_BATCH_TIMEOUT: 1000
[Step 9/11] Creating infer requests and preparing input tensors
[ WARNING ] No input files were given: all inputs will be filled with random values!
[ INFO ] Test Config 0
[ INFO ] 0  ([N,C,H,W], u8, [1,3,224,224], static): random (image/numpy array is expected)
[Step 10/11] Measuring performance (Start inference asynchronously, 16 inference requests, limits: 60000 ms duration)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
[ INFO ] First inference took 1152.58 ms
[Step 11/11] Dumping statistics report
[ INFO ] Execution Devices: [ OCL_GPU.0 ]
[ INFO ] Count:               2352 iterations
[ INFO ] Duration:            60578.62 ms
[ INFO ] Latency:
[ INFO ]    Median:           414.39 ms
[ INFO ]    Average:          411.05 ms
[ INFO ]    Min:              212.83 ms
[ INFO ]    Max:              420.91 ms
[ INFO ] Throughput:          38.83 FPS

GPU: A380

openvino@a649c849386e:/opt/intel/openvino_2023.0.0.10926$ ./samples/cpp/samples_bin/benchmark_app -m /tmp/resnet50-binary-0001.xml -d GPU
[Step 1/11] Parsing and validating input arguments
[ INFO ] Parsing input parameters
[Step 2/11] Loading OpenVINO Runtime
[ INFO ] OpenVINO:
[ INFO ] Build ................................. 2023.0.0-10926-b4452d56304-releases/2023/0
[ INFO ] 
[ INFO ] Device info:
[ INFO ] GPU
[ INFO ] Build ................................. 2023.0.0-10926-b4452d56304-releases/2023/0
[ INFO ] 
[ INFO ] 
[Step 3/11] Setting device configuration
[ WARNING ] Performance hint was not explicitly specified in command line. Device(GPU) performance hint will be set to THROUGHPUT.
[Step 4/11] Reading model files
[ INFO ] Loading model files
[ INFO ] Read model took 17.53 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Network inputs:
[ INFO ]     0 (node: 0) : f32 / [N,C,H,W] / [1,3,224,224]
[ INFO ] Network outputs:
[ INFO ]     1463 (node: 1463) : f32 / [...] / [1,1000]
[Step 5/11] Resizing model to match image sizes and given batch
[Step 6/11] Configuring input of the model
[ INFO ] Model batch size: 1
[ INFO ] Network inputs:
[ INFO ]     0 (node: 0) : u8 / [N,C,H,W] / [1,3,224,224]
[ INFO ] Network outputs:
[ INFO ]     1463 (node: 1463) : f32 / [...] / [1,1000]
[Step 7/11] Loading the model to the device
[ INFO ] Compile model took 3670.05 ms
[Step 8/11] Querying optimal runtime parameters
[ INFO ] Model:
[ INFO ]   OPTIMAL_NUMBER_OF_INFER_REQUESTS: 4
[ INFO ]   NETWORK_NAME: torch-jit-export
[ INFO ]   EXECUTION_DEVICES: GPU.0
[ INFO ]   AUTO_BATCH_TIMEOUT: 1000
[ INFO ]   LOADED_FROM_CACHE: NO
[Step 9/11] Creating infer requests and preparing input tensors
[ WARNING ] No input files were given: all inputs will be filled with random values!
[ INFO ] Test Config 0
[ INFO ] 0  ([N,C,H,W], u8, [1,3,224,224], static): random (image/numpy array is expected)
[Step 10/11] Measuring performance (Start inference asynchronously, 4 inference requests, limits: 60000 ms duration)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
[ INFO ] First inference took 7.45 ms
[Step 11/11] Dumping statistics report
[ INFO ] Execution Devices: [ GPU.0 ]
[ INFO ] Count:               25344 iterations
[ INFO ] Duration:            60017.17 ms
[ INFO ] Latency:
[ INFO ]    Median:           9.47 ms
[ INFO ]    Average:          9.47 ms
[ INFO ]    Min:              4.31 ms
[ INFO ]    Max:              14.91 ms
[ INFO ] Throughput:          422.28 FPS

MULTI: CPU + A380

openvino@0bf0cca613c1:/opt/intel/openvino_2023.3.0.13775$ ./samples/cpp/samples_bin/samples_bin/benchmark_app -m /tmp/resnet50-binary-0001.xml -d MULTI:GPU,CPU
[Step 1/11] Parsing and validating input arguments
[ INFO ] Parsing input parameters
[Step 2/11] Loading OpenVINO Runtime
[ INFO ] OpenVINO:
[ INFO ] Build ................................. 2023.3.0-13775-ceeafaf64f3-releases/2023/3
[ INFO ] 
[ INFO ] Device info:
[ INFO ] CPU
[ INFO ] Build ................................. 2023.3.0-13775-ceeafaf64f3-releases/2023/3
[ INFO ] 
[ INFO ] GPU
[ INFO ] Build ................................. 2023.3.0-13775-ceeafaf64f3-releases/2023/3
[ INFO ] 
[ INFO ] MULTI
[ INFO ] Build ................................. 2023.3.0-13775-ceeafaf64f3-releases/2023/3
[ INFO ] 
[ INFO ] 
[Step 3/11] Setting device configuration
[ WARNING ] Performance hint was not explicitly specified in command line. Device(MULTI) performance hint will be set to THROUGHPUT.
[Step 4/11] Reading model files
[ INFO ] Loading model files
[ INFO ] Read model took 35.56 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Network inputs:
[ INFO ]     0 (node: 0) : f32 / [N,C,H,W] / [1,3,224,224]
[ INFO ] Network outputs:
[ INFO ]     1463 (node: 1463) : f32 / [...] / [1,1000]
[Step 5/11] Resizing model to match image sizes and given batch
[Step 6/11] Configuring input of the model
[ INFO ] Model batch size: 1
[ INFO ] Network inputs:
[ INFO ]     0 (node: 0) : u8 / [N,C,H,W] / [1,3,224,224]
[ INFO ] Network outputs:
[ INFO ]     1463 (node: 1463) : f32 / [...] / [1,1000]
[Step 7/11] Loading the model to the device
[ INFO ] Compile model took 10122.19 ms
[Step 8/11] Querying optimal runtime parameters
[ INFO ] Model:
[ INFO ]   NETWORK_NAME: torch-jit-export
[ INFO ]   EXECUTION_DEVICES: GPU CPU
[ INFO ]   PERFORMANCE_HINT: THROUGHPUT
[ INFO ]   OPTIMAL_NUMBER_OF_INFER_REQUESTS: 132
[ INFO ]   CPU: 
[ INFO ]     AFFINITY: CORE
[ INFO ]     CPU_DENORMALS_OPTIMIZATION: NO
[ INFO ]     CPU_SPARSE_WEIGHTS_DECOMPRESSION_RATE: 1
[ INFO ]     ENABLE_CPU_PINNING: YES
[ INFO ]     ENABLE_HYPER_THREADING: YES
[ INFO ]     EXECUTION_DEVICES: CPU
[ INFO ]     EXECUTION_MODE_HINT: PERFORMANCE
[ INFO ]     INFERENCE_NUM_THREADS: 16
[ INFO ]     INFERENCE_PRECISION_HINT: f32
[ INFO ]     NETWORK_NAME: torch-jit-export
[ INFO ]     NUM_STREAMS: 4
[ INFO ]     OPTIMAL_NUMBER_OF_INFER_REQUESTS: 4
[ INFO ]     PERFORMANCE_HINT: THROUGHPUT
[ INFO ]     PERFORMANCE_HINT_NUM_REQUESTS: 0
[ INFO ]     PERF_COUNT: NO
[ INFO ]     SCHEDULING_CORE_TYPE: ANY_CORE
[ INFO ]   GPU: 
[ INFO ]     AUTO_BATCH_TIMEOUT: 1000
[ INFO ]     EXECUTION_DEVICES: OCL_GPU.0
[ INFO ]     NETWORK_NAME: torch-jit-export
[ INFO ]     OPTIMAL_NUMBER_OF_INFER_REQUESTS: 128
[ INFO ]     SUPPORTED_CONFIG_KEYS: AUTO_BATCH_TIMEOUT
[ INFO ]   MODEL_PRIORITY: MEDIUM
[ INFO ]   LOADED_FROM_CACHE: NO
[ INFO ]   SCHEDULE_POLICY: DEVICE_PRIORITY
[ INFO ]   MULTI_DEVICE_PRIORITIES: GPU,CPU
[Step 9/11] Creating infer requests and preparing input tensors
[ WARNING ] No input files were given: all inputs will be filled with random values!
[ INFO ] Test Config 0
[ INFO ] 0  ([N,C,H,W], u8, [1,3,224,224], static): random (image/numpy array is expected)
[Step 10/11] Measuring performance (Start inference asynchronously, 132 inference requests, limits: 60000 ms duration)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
[ INFO ] First inference took 763.51 ms
[Step 11/11] Dumping statistics report
[ INFO ] Execution Devices: [ GPU CPU ]
[ INFO ] Count:               38412 iterations
[ INFO ] Duration:            61324.81 ms
[ INFO ] Throughput:          626.37 FPS