13
Measurement accuracy
Measurement accuracy is calculated based on the following equation:
Measurement accuracy = Basic accuracy × C × D × E × F × G
[E: Measurement cable length coefficient]
[C: Level coefficient] V: Setting value (corresponds to when V mode) [V]
Measurement level
1 V
1
0 m
1
1 m
1.5
2 m
2
4 m
3
Coefficient
Coefficient (DC resistance measurement)
Settable range for frequency
0 m: Up to 8 MHz, 1 m: 8 MHz, 2 m: Up to 2 MHz, 4 m: Up to 1MHz
Measurement level
0.010 V to 0.999 V
1 V
1
1.001 V to 5 V
Coefficient (AC measurement)
1+0.2/V
1+0.2/V
[F: DC bias coefficient]
[D: Measurement speed coefficient]
DC bias coefficient
Coefficient
OFF
ON
2
FAST MED SLOW SLOW2
Measurement speed
1
Coefficient
DC resistance measurement
AC measurement
4
8
3
4
2
2
1
1
[G: Temperature coefficient]
Operating temperature
Coefficient
t [°C
]
1+0.1×|t-23|
Basic accuracy
Accuracy is calculated based on coefficients A and B from the basic ac-
When the operating temperature (t) is 23°C±5°C, use a coefficient of 1.
curacy chart shown below.
Free software for calculating
accuracy
(LCR application disc)
1 kΩ range or higher
100 Ω range or lower
10×Zx
Range
Range
Zx
Basic accuracy= ± A+B×
(
−1
Basic accuracy= ± A+B×
(
−1
)
)
Automatically calculate measurement
accuracy based on user-entered
measurement conditions and measurement
results. Free download from the Hioki
website.
Zx : Impedance of the measurement conductor
A: Noted in basic accuracy chart. (Upper value: Z accuracy [% rdg.]; lower value: θ accuracy [°])
B: Noted in basic accuracy chart. (Upper value: Z accuracy [% rdg.]; lower value: θ accuracy [°])
A is the accuracy of R when DC (± % rdg.)
Temperature and humidity ranges: 23℃ ± 5℃, 80% RH or less (no condensation),
at least 60 minutes after power ON, after performing open and short compensation
Conditions
B is the coefficient for the resistance of the sample
Basic accuracy
Guaranteed ac-
Range
100MΩ
10MΩ
1MΩ
DC
4Hz to 99.99Hz 100Hz to 999.99Hz
1kHz to 10kHz
10.001kHz to 100kHz 100.01kHz to 1MHz 1.0001MHz to 8MHz
curacy rang
A=1
B=1
A=6
A=5
B=5
B=3
A=3
A=2
B=2
B=2
A=3
A=2
B=2
B=2
8MΩ to 200MΩ
A=0.5 B=0.3
A=0.2 B=0.1
A=0.8 B=1
A=0.8 B=0.5
A=0.5 B=0.3
A=0.4 B=0.2
A=0.5 B=0.3
A=0.4 B=0.2
A=0.3 B=0.05
A=0.2 B=0.02
A=0.2 B=0.03
A=0.1 B=0.02
A=0.05 B=0.02
A=0.03 B=0.02
A=2
A=2
B=1
B=1
800kΩ to 10MΩ
80kΩ to 1MΩ
8kΩ to 100kΩ
800Ω to 10kΩ
80Ω to 1kΩ
A=0.4 B=0.08
A=0.3 B=0.08
A=0.3 B=0.05
A=0.2 B=0.02
A=0.5 B=0.1
A=0.6 B=0.1
A=3
A=3
B=0.5
B=0.5
A=0.1 B=0.01 A=0.3 B=0.03
A=0.2 B=0.02
A=0.1 B=0.01 A=0.3 B=0.03
A=0.3 B=0.01
A=0.1 B=0.01 A=0.3 B=0.02
A=0.2 B=0.02
A=0.2 B=0.03
A=0.1 B=0.02
A=0.2 B=0.02
A=0.1 B=0.02
A=0.2 B=0.02
A=0.1 B=0.02
A=0.25 B=0.04
A=0.2 B=0.02
A=1
A=1
B=0.3
B=0.3
A=2
A=2
B=0.5
B=0.3
100kΩ
10kΩ
1kΩ
A=0.3 B=0.02
A=0.2 B=0.02
A=0.5 B=0.05
A=0.5 B=0.05
A=2
A=1.5
B=0.5
B=0.3
A=0.2 B=0.02
A=0.1 B=0.02
A=0.2 B=0.02
A=0.15 B=0.02
A=0.4 B=0.02
A=0.4 B=0.02
A=1.5
A=1.5
B=0.2
B=0.2
A=0.1 B=0.02 A=0.3 B=0.02
A=0.2 B=0.01
A=0.2 B=0.15 A=0.5 B=0.1
A=0.3 B=0.1
A=0.2 B=0.02
A=0.15 B=0.01
A=0.4 B=0.05
A=0.3 B=0.03
A=0.2 B=0.02
A=0.1 B=0.01
A=0.4 B=0.05
A=0.3 B=0.03
A=0.2 B=0.02
A=0.15 B=0.02
A=0.4 B=0.05
A=0.3 B=0.03
A=0.5 B=0.03
A=0.5 B=0.03
A=0.8 B=0.1
A=0.5 B=0.05
A=1.5 B=1
A=0.7 B=0.5
A=1.5
A=1.5
A=2
A=2
A=3
A=3
B=0.2
B=0.2
B=1.5
B=1
B=3
B=2
8Ω to 100Ω
100Ω
10Ω
800mΩ to 10Ω
80mΩ to 1Ω
A=0.3 B=0.3
A=1.5 B=1
A=0.8 B=0.5
A=1
B=0.3
A=1
B=0.3
A=1
B=0.3
1Ω
A=0.5 B=0.2
A=0.5 B=0.2
A=0.5 B=0.2
A=1 B=1
A=8
A=5
B=8
B=4
A=5
A=3
B=4
B=2
A=3
A=2
B=2
B=1.5
A=2
A=2
B=2
B=1.5
A=4
A=3
B=3
B=4
1mΩ to 100mΩ
100mΩ
● Method of determining basic accuracy
● Calculation example
Impedance Zx of sample: 500 Ω (actual measurement value)
• Calculate the basic accuracy from the sample impedance, measurement
range, and measurement frequency and the corresponding basic accuracy
A and coefficient B from the table above.
Measurement conditions: When frequency 10 kHz and range 1 kΩ
Insert coefficient A = 0.2 and coefficient B = 0.02 for the Z basic accuracy
from the table above into the expression.
•
The calculation expression to use differs for each of the 1 kΩ range and
above and 100 Ω range and below.
10 × 500
-
•
For C and L, obtain basic accuracy A and coefficient B by determining the
measurement range from the actual measurement value of impedance or the
approximate impedance value calculated with the following expression.
1 = 0.28 (± %rdg.
)
Z basic accuracy = 0.2 + 0.02
×
103
Similarly, insert coefficient A = 0.1 and coefficient B = 0.02 for the θ
basic accuracy, as follows:
Zx (Ω) ωL (H) (θ 90º)
1
10 × 500
-
(θ
-
90º)
(ω: 2 x π x Measurement frequency [Hz])
Guaranteed accuracy measurement level range
1 = 0.18 (± deg.
)
θ basic accuracy = 0.1 + 0.02
×
ωC (F)
103
R (Ω) (θ 0º)
The range of measurement levels for which accuracy is guaranteed varies with the setting conditions.
Range Sample’s impedance DC 4 Hz to 99.99 Hz 100 Hz to 999.99 Hz 1 kHz to 10 kHz 10.001 Hz to 100 kHz
100.01 kHz to 1 MHz 1.0001 MHz to 5 MHz 5.0001 MHz to 8 MHz
100 MΩ
8 MΩ to 200 MΩ
10 MΩ to 100 MΩ
800 kΩ to 10 MΩ
1 MΩ to 10 MΩ
80 kΩ to 1 MΩ
100 kΩ to 1 MΩ
8 kΩ to 100 kΩ
10 kΩ to 100 kΩ
800 Ω to 10 kΩ
1 kΩ to 10 kΩ
10 MΩ
0.101 V to 5 V
0.050 V to 5 V
0.501 V to 5 V
0.101 V to 5 V
1 MΩ
100 kΩ
10 kΩ
1 kΩ
0.501 V to 5 V
0.101 V to 1 V
0.050 V to 5 V
0.010 V to 5 V
0.050 V to 1 V
0.101 V to 1 V
80 Ω to 1 kΩ
100 Ω
10 Ω
1 Ω
8 Ω to 100 Ω
800 mΩ to 10 Ω
80 mΩ to 1 Ω
0.101 V to 1 V
0.501 V to 1 V
0.050 V to 5 V
0.101 V to 5 V
0.101 V to 5 V
0.501 V to 5 V
1 mΩ to 100 mΩ
100 mΩ
The guaranteed accuracy range during DC bias operation is 10 mΩ or greater. The accuracy for DC resistance (Rdc) measurement is guaranteed only when offset values are acquired. The guaranteed accuracy range varies
with the sample’s impedance.
ASM-SENSOR [ ASM GMBH ]
