1. ESPHome 2025.2 removed support for dbuezas’ CC1101 custom component. We need a way forward.

2. IoT controls generally assume you can specifically set a dimmer/speed setting for things (such as “30 percent”), but how do you map that to a fan with only up/down buttons?

3. Using Google Home (or Alexa, etc.) is more convenient, but can we expose these fans to them without an additional hop to the cloud?

4. What if the fan is switched off at the wall or via its original remote? How can Home Assistant learn its new state?

Caveat: The solutions below fit my hardware and use-case; YMMV.

And with that said, here is how I solved each of the shortcomings!

1. Using the newer CC1101 implementation

Since the previous CC1101 implementation is no longer supported as of February 2025, I switched to using an unmerged PR (lol). But the configuration syntax is very different, and the documentation does not explain how to broadcast a timing array (the format that I have for my signals). Here is the syntax that works for me.

(Optional) Add a sub-device:

This is a new feature added in ESPHome 2025.7.

  devices:
    - id: radio_control_device
      name: "Radio Control"

Use the PR’s external_component:

external_components:
  - source: github://pr#6300
    components: [ cc1101 ]

Set up with the GPIO pins that were used

spi:
  clk_pin: GPIO18
  miso_pin: GPIO19
  mosi_pin: GPIO23

cc1101:
  id: transceiver
  cs_pin: GPIO5

remote_transmitter:
  id: cc1101_transmitter
  pin: GPIO33 # This is GDO0
  carrier_duty_percent: 100%
  on_transmit:
    then:
      - cc1101.begin_tx: transceiver
  on_complete:
    then:
      - cc1101.end_tx: transceiver

Allow setting the broadcast frequency per broadcast

I need to broadcast at both 433.92MHz and 350MHz, so I can’t let it default to 433.92MHz. The way you can set the frequency on-demand is to create a number mapping to a field defined by the CC1101 implementation:

number:
  - platform: cc1101
    tuner:
      frequency:
        id: tuner_frequency
        name: "Tuner Frequency (kHz)"

I then create a wrapper script to handle all command broadcasts:

script:
  - id: radio_tx
    mode: queued  # Enforce that transmissions don't overlap.
    parameters:
      freq_khz: int  # The frequency to broadcast at.
      code: int[]  # The signal.
      trailing_delay: bool  # Whether to have a gap before broadcasting the next signal.
    then:
      - number.set:  # Set the broadcast frequency
          id: tuner_frequency
          value: !lambda 'return freq_khz;'
      - delay: 3ms
      - lambda: |-
          esphome::remote_base::RawTimings timings;
          timings.reserve(code.size());
          for (auto v : code)
            timings.push_back(static_cast<int32_t>(v));
          auto call = id(cc1101_transmitter).transmit();
          call.get_data()->set_data(timings);
          call.set_send_times(2);
          call.set_send_wait(10);
          call.perform();
      - delay: !lambda 'return trailing_delay ? 700 : 0;'

The mode: queued means I can send any number of commands and the script will make sure each signal is sent in the correct order, with an adequate trailing delay for the receiver to not mix it in with the next signal.

Unfortunately, the script mechanism only supports sending bool, int, float, string, and array variations of each. The passed in code is not technically the data type that set_data() expects, so we need to first cast the entire array into a new array.

Individual signal commands

The rest is to just define the actual signals that the fan supports. For example:

  - platform: template
    id: office_light
    icon: "mdi:ceiling-fan-light"
    name: Office Light
    on_press:
      - lambda: |-
          std::vector<int> timings{461,-356,425,-356,469,-352,441,-354,441,-350,441,-354,445,-354,445,-356,443,-356,443,-352,447,-356,445,-5176,843,-346,425,-788,417,-780,837,-364,835,-346,447,-742,869,-324,843,-386,419,-752,477,-744,415,-788,829,-384,401,-766,859,-348,439,-758,855,-346,835,-378,429,-754,441,-778,431,-746,427,-776,437,-746,441,-766,449,-750,841,-374,831,-348,841,-386,817,-350,467,-744,443,-750,849,-350,439,-774,837,-350,795,-454,341,-852,385,-812,377,-820,383,-826,343,-832,387,-824,377,-798,421,-782,409,-808,819,-380,793,-384,425,-778,411,-804,415,-760,415,-802,413,-776,433,-778,419,-746,441,-778,849,-324,847,-384,425,-772,443,-740,839,-362,849,-352,847,-354,851,-352,823,-380,837,-352,823,-386,835,-354,429,-25634,429,-352,463,-350,439,-350,409,-420,407,-390,375,-424,373,-422,409,-392,373,-424,409,-390,411,-390,409,-5184,805,-380,423,-792,411,-798,821,-380,797,-362,453,-776,801,-386,847,-352,457,-752,431,-754,425,-774,831,-382,429,-774,823,-380,415,-780,815,-372,839,-364,413,-784,443,-744,439,-778,411,-782,445,-744,451,-752,445,-740,873,-332,855,-354,853,-326,841,-386,421,-780,441,-744,843,-358,449,-750,841,-366,839,-344,427,-768,441,-780,431,-746,449,-752,459,-744,449,-752,425,-782,427,-776,431,-772,815,-356,839,-384,423,-752,475,-744,417,-782,445,-748,415,-786,447,-750,451,-754,433,-746,853,-356,859,-346,445,-746,439,-774,839,-356,849,-352,825,-358,861,-352,823,-362,845,-384,823,-364,805,-378,449};
          id(radio_tx).execute(433920, timings, true);
    device_id: radio_control_device

Note: The vector of positive and negative numbers denotes the amount of time it should broadcast nothing (negative values) and the amount of time it should broadcast a pulse (positive values).

2. Create a wrapper around the up/down buttons

Most trivially, you could create virtual buttons for Levels 1/2/3 (let’s assume you have 3 levels, but you can expand this to N levels) by just sending the relevant number of “up” signals. But the main problem is not knowing what the current fan level is.

Entering a known state

We need to first tell the fan to turn off before we can send the appropriate number of “up” commands. At least for my fan, if you spam “down”, it should lower the fan to the off state. But for a 3-level fan, that means you need to send three “down” signals just to turn the fan off. We can reduce this down to 2 commands by taking advantage of the fan on/off “toggle” button as well, which will turn the fan off if it was on, or on if it was off. It kind of has the same issue where we don’t know if the fan was on or not, but sending an “up” will always turn the fan on. So, we can create a virtual “off” button by making it send “up” and then “toggle”. This is acceptable because the toggle turns the fan off before the fan can really react to the extraneous “up” command.

ESPHome YAML

Here are the virtual buttons for setting my office fan to off and to speeds 1-3. Note for the 3rd/highest level, you can skip the “off” procedure and just spam “up”.

button:
  - platform: template
    id: office_fan_off
    name: Office Fan Off
    icon: "mdi:fan-off"
    on_press:
      - button.press: office_fan_up # We don't know the state. Turn it on beforehand.
      - button.press: office_fan    # Now toggle it off.
    device_id: radio_control_device
  - platform: template
    id: office_fan_speed_1
    icon: "mdi:fan-speed-1"
    name: Office Fan Speed 1
    on_press:
      - button.press: office_fan_off
      - delay: 10ms # This yields to the system to reduce perceived runtime according to "Component web_server took a long time for an operation" warnings.
      - button.press: office_fan_up
    device_id: radio_control_device
  - platform: template
    id: office_fan_speed_2
    name: Office Fan Speed 2
    icon: "mdi:fan-speed-2"
    on_press:
      - button.press: office_fan_off
      - delay: 10ms
      - button.press: office_fan_up
      - delay: 10ms
      - button.press: office_fan_up
    device_id: radio_control_device
  - platform: template
    id: office_fan_speed_3
    name: Office Fan Speed 3
    icon: "mdi:fan-speed-3"
    on_press:
      - button.press: office_fan_up
      - delay: 10ms
      - button.press: office_fan_up
      - delay: 10ms
      - button.press: office_fan_up

3. Expose Home Assistant entities as Matter devices

We’ll use tobst4r’s Matter Hub to expose the fans as Matter devices to services like Google Home and Alexa. But first, we need to create a HA-native fan entity out of the signals that we can send.

Fan Configuration

This goes into Home Assistant’s configuration.yaml:

fan:
  - platform: template
    fans:
      office_fan:
        unique_id: 42febbf9-da5d-4e2b-8001-9a10bf2e12b5
        friendly_name: "Office Fan"
        turn_on:
          service: script.turn_on # Does not block on the script returning
          target:
            entity_id: script.set_office_fan_speed
          data:
            variables:
              percentage: 50
        turn_off:
          action: button.press
          target:
            entity_id: button.office_fan_off
        set_percentage:
          service: script.turn_on
          target:
            entity_id: script.set_office_fan_speed
          data:
            variables:
              percentage: "{{ percentage }}"

This maps the fan’s “turn on” function to setting the fan to a 50% fan level, which maps to level 2 in the script below. The “turn off” function will call the office_fan_off wrapper that I defined in part 1.

Note that both turn_on and set_percentage will call script.set_office_fan_speed in a non-blocking manner, which is necessary because of some debouncing we have to do. More on that later.

And then this script goes into scripts.yaml:

set_office_fan_speed:
  alias: Office Fan Speed Setting
  mode: restart
  sequence:
    - delay:
        milliseconds: 500
    - service: button.press
      target:
        entity_id: >
          {% if percentage <= 5 %}
            button.office_fan_off
          {% elif percentage <= 40 %}
            button.office_fan_speed_1
          {% elif percentage <= 70 %}
            button.office_fan_speed_2
          {% elif percentage <= 100 %}
            button.office_fan_speed_3
          {% endif %}

The script must include the mode: restart and the delay because there are two quirks we need to handle:

1. Google Home (don’t know about others) lets you drag a slider to set the fan percentage, but it will incessantly send dozens of intermediate percentages while you’re actively sliding. Each of these events will cause the script to be called and signals to be sent.

2. Home Assistant will first call turn_on and then set_percentage immediately after. This also results in duplicate calls to the script.

In both cases, we are sending too many signals to the fan. We need to somehow ignore all the previous calls and just send out the signal for the very last call. So, mode: restart says that the script will throw out its current run if another call comes in before it finishes. We then add a 500ms delay to the script execution so that it takes long enough for the extraneous calls to come in during the execution. The result is the script gets restarted a bunch of times before finally executing only the last call.

The reason the fan configuration uses service: script.turn_on syntax is because this syntax makes the calls non-blocking. Otherwise, in the second case, turn_on's call to the script will wait until the script returns (including the delay) before it calls set_percentage. And this means the fan will first be set to 50% before being set to the actual level you wanted.

Exposing to Matter

Go to the Home-Assistant-Matter-Hub web UI and edit your Matter bridge to include fans.office_fan like so:

It will then be exposed through Matter as a Fan type:

Provided you paired the Matter Hub “device” to your smart home provider of choice (Google, Alexa, etc.), the fan should show up on their platform as well.

4. Knowing the fan state at all times

If you control the fan outside of Home Assistant/Matter, then HA doesn’t know the fan’s current state. It could be that the fan was turned off but we still thought it was set to level 2. This isn’t a huge issue, but it means that if you want to turn the fan back on, you first have to tell HA to turn the fan “off” (from “level 2”) and then back on. What if we could add something to let us glean the fan’s state?

Enter the Shelly PM Mini. It sits between the fan and the house wiring and measures the wattage of whatever it’s hooked up to. This can be installed in the base (the part that touches the ceiling) of the ceiling fan, and has excellent integration with Home Assistant.

In this case, we have to be lucky that each possible fan state has a unique power usage. For me, there are 7 (8 including Off) states that need to not overlap:

Fortunately, my fan does not have any overlapping power usage between the possible states. However, the fan supports dimming the light, which will ruin our ability to make sense of anything purely from wattage. I have never bothered to use the light in a dimmed state though, so this does not matter to me, but YMMV.

All we need now is to put some logic into Home Assistant to set the fan and light states using the wattage reading from the power meter. Here are the additional config to add to the configuration.yaml:

Fan entity

fan:
  - platform: template
    fans:
      office_fan:

        ... the config we added earlier ...

        availability_template: >-
          {% set s = states('sensor.shellypm_282c_power') %}
          {{ s not in ['unavailable', 'unknown', 'none'] }}

        value_template: >
          {% set p = (states('input_number.stable_shellypm_282c_power') | float) %}
          {{ (p > 0 and (p < 20 or p > 22)) }}

        percentage_template: >
          {% set p = states('input_number.stable_shellypm_282c_power') | float %}
          {% if (p > 9 and p < 13) or (p > 31 and p < 34) %}
            33
          {% elif (p > 28 and p <= 31) or (p > 48 and p < 52) %}
            66
          {% elif (p > 68 and p < 72) or (p > 88 and p < 100) %}
            100
          {% elif p <= 9 %}
            0
          {% else %}
            100
          {% endif %}

Regardless of the currently-set percentage, we will detect fan levels 1-3 and map them to specific percentages. In this way, the reported percentage will eventually settle at 0, 33, 66, or 100%.

You can see that I added +/- 2W (where possible) to account for variance in the measured usage, and that each case is handling the expected wattage range with the light being either off or on.

availability_template is there to notice when the power meter is offline. Since the power meter is wired in-line with the fan, it is safe to assume that if the power meter is offline, so is the fan.

Light entity

The fan has a light, so let’s also create a light entity as well.

light:
  - platform: template
    lights:
      office_fan_light:
        unique_id: 621b19dc-2927-4978-a322-f18753139cf2
        friendly_name: "Office Fan Light"
        icon_template: mdi:ceiling-fan-light
        availability_template: >-
          {% set s = states('sensor.shellypm_282c_power') %}
          {{ s not in ['unavailable', 'unknown', 'none'] }}
        turn_on:
          action: button.press
          target:
            entity_id: button.office_light
        turn_off:
          action: button.press
          target:
            entity_id: button.office_light
        value_template: >
          {% set p = (states('input_number.stable_shellypm_282c_power') | float) %}
          {{ (p > 20 and p < 22) or
             (p > 31 and p < 34) or
             (p > 48 and p < 52) or
             (p > 88 and p < 100) }}

Ignoring intermediate readings

You’ll notice that the configurations are reading the wattage value from input_number.stable_shellypm_282c_power, note the stable in the name.

This is because of a quirk that creates finicky behavior where the power meter will report intermediary wattages before settling into the new wattage. For example, if I turn the fan from level 1 (11W) to level 2 (30W), the meter will momentarily read something like 25W before settling into 30W. The intermediate readings should be thrown out, which is the role of this automation I put into automations.yaml:

- id: "1752985898000"
  alias: "Check office fan wattage"
  trigger:
    - platform: state
      entity_id: sensor.shellypm_282c_power
      for:
        seconds: 2
  action:
    - service: input_number.set_value
      target:
        entity_id: input_number.stable_shellypm_282c_power
      data:
        value: "{{ states('sensor.shellypm_282c_power')|float }}"

It takes the raw readings from the power meter, and requires that the reading stay the same for 2 seconds before saving that value to input_number.stable_shellypm_282c_power. That’s a variable I declare in configuration.yaml:

input_number:
  stable_shellypm_282c_power:
    name: Established stable power reading from the Shelly PM 282C
    min: 0
    max: 2000
    step: 0.1
    initial: 0

And with all that, you will not see finicky behavior where the wattage reading momentarily falls outside of the if condition ranges.

Enjoy your controllable fan!

In Home Assistant:

… and in Google Home!

And it’ll still update to the real state of the fan even if you use the original remote, or turn off the switch on the wall!

Hear me out

It’s physically impossible for any (self-contained) space heater to maintain a target temperature with any respectable precision. And I do mean maintain. As in, if I want 70° F, I don’t want 69° or 71° F. In practice, my house thermostat is able to maintain the room temperature between 69.5 and 70.0° F. Space heaters cannot dream of achieving this precise range and I observe them varying by more than 10 degrees.

Why is that? Simple. The thermometer is inside the space heater. That means the thermometer is measuring a temperature that’s higher than the room temperature due to being influenced by the heater itself.

Designers can apply some tricks to alleviate this issue, such as placing the thermometer as far from the heating element as possible, but space heaters are only so big and the thermometer can only be so far. Real thermostats sit on walls, tens of feet away from any vents/registers, for a reason.

Since we know that the thermometer would be reading an elevated temperature when the heater is on, let’s logic out what would happen if we actually tried to maintain 70° F. Let’s be naive and make the heater turn off once the thermometer reads 70.5° (a reasonable cut-off point you can have with a wall thermostat). The heater heats itself up and almost immediately reaches 70.5° F. Now the heater turns off while the room temperature barely budged.

Obviously not a functional approach, so let’s be less naive: Give the heater some wiggle room. Wait until the temperature reading is far higher than the target temperature before turning off. Can there be any precision with this approach? No, and that is just an unavoidable fact of physical reality.

What can be done?

The thermometer has to be detached from the space heater. Perhaps there are high end models with remote sensors, but I have not found them and they likely cost more than it’s worth. Instead, I’ve set out to solve this with my own thermometer, a smart plug, and Home Assistant automations.

Thermometer

Get anything that can integrate with Home Assistant. Ideally with frequent reports and precise readings. I have seen some that only report at 0.5° F increments and that’s simply too wide for my tastes, but you can always make do with accepting a slightly looser control of the room’s temperature range.

I personally use the SHT40-series thermometer and ESP8266/ESP32 microcontrollers running ESPHome to have good precision and frequent readings.

Specifically, in ESPHome I have the temperature polled every second and the temperature is averaged out like so:

    sliding_window_moving_average:
        window_size: 60
        send_every: 30

This applies a moving average of the last minute’s worth of readings, and reports the temperature to Home Assistant every 30 seconds.

Of course, you should put the sensor a good distance away from the space heater.

Smart Plug

Optimistically, you can use anything that integrates with Home Assistant. But there are two things to consider:

Don’t burn your house down

Space heaters can use basically the entire amperage of the circuit and can lead to melting/burning cheap smart plugs that exaggerate the amperage they support. Do your research and find a trustworthy brand/model. Did you know that your usual wall outlet is rated for 15A but your usual cheap smart plugs (and extension cables, for that matter) are only rated 13A? And that’s still just their marketed claim — they could have secretly cheapened out even further. Crazy…

But in any case, monitor the temperature of the smart plug at the beginning until you’re sure it handles the load. I’ve also gone a step farther and only use the medium power (900W or 8A) setting on the space heater.

What happens if the network or Home Assistant goes down?

If the smart plug was on and Home Assistant can no longer instruct it to turn off, will the heater turn the room into a sauna? How can we implement safety cut-offs?

You can use the heater itself where you heat the room to the max temperature you’d want it to reach and then slowly lower the target temperature until it switches off, but that depends on a trust that no person or pet would ever accidentally change that setting.

Maybe some smart plugs have timers where it can automatically turn off after a set time, but is that on-device or cloud-dependent?

I went with the timer option, and to make sure the timeout runs on-device, I opted to use ESPHome on a Sonoff S31. This enables me to put a 1 hour timeout straight into the smart plug’s firmware so it is for sure running on-device. At the same time, I make sure that it doesn’t turn back on after a power outage. The YAML config is pretty simple:

switch:
  - platform: gpio
    name: "${name} Relay"
    pin: GPIO12
    id: relay
    # Stay off after a power outage.
    restore_mode: RESTORE_DEFAULT_OFF
    # For use with the space heater:
    on_turn_on:
      - script.stop: turn_off_after_one_hour
      - script.execute: turn_off_after_one_hour
    on_turn_off:
      - script.stop: turn_off_after_one_hour

# For use with the space heater:
script:
  - id: turn_off_after_one_hour
    then:
      - delay: 60min
      - switch.turn_off: relay

Automations in Home Assistant

Let’s set up automations to use a space heater to keep a room between 71° and 71.5° F.

There is a gotcha I’ve learned the hard way and I’ll explain it now and help you avoid it. First, consider this setup:

alias: Corner Bedroom Heater - Turn off if above 71.5F
description: ""
triggers:
  - entity_id:
      - sensor.corner_bedroom_temperature
    above: 71.5
    trigger: numeric_state
actions:
  - type: turn_off
    device_id: <your device id>
    entity_id: <your entity id>
    domain: switch
mode: single

This is how one would naively configure to have the smart plug turn off if the temperature is above 71.5° F. But its major flaw is that Home Assistant has to observe the moment when the number goes from <=71.5 to >71.5. If there is any sort of lapse in communication, or Home Assistant was updating/restarting, it is possible for it to miss the moment that the value changed to >71.5. When that happens, this automation will not trigger and it will not instruct the smart plug to turn off.

Instead, you have to poll the value. The best you can do with the available automation syntax is a poll every minute, so let’s do that where we poll on the 0th second of every minute:

alias: Corner Bedroom Heater - Turn off if above 71.5F
description: ""
triggers:
  - seconds: "0"
    trigger: time_pattern
conditions:
  - condition: numeric_state
    entity_id: sensor.corner_bedroom_temperature
    above: 71.5
actions:
  - type: turn_off
    device_id: <your device id>
    entity_id: <your entity id>
    domain: switch
mode: single

This way, things that could cause the previous setup to fail would be rectified within a minute. I suppose you could create a duplicate automation entry that triggers on the 30th second of every minute to bring the latency down to 30 seconds too. Up to you.

Here would be the corresponding “turn on” automation:

alias: Corner Bedroom Heater - Turn on if below 71F
description: ""
triggers:
  - seconds: "0"
    trigger: time_pattern
conditions:
  - condition: numeric_state
    entity_id: sensor.corner_bedroom_temperature
    below: 71
actions:
  - type: turn_on
    device_id: <your device id>
    entity_id: <your entity id>
    domain: switch
mode: single

And that’s the bare minimum automation setup. You can of course add any other conditions you’d like, such as only triggering during the evening:

  - condition: time
    after: "17:00:00"
    before: "22:00:00"

And in that case, you probably want a final shut-off automation or else the smart plug might run the heater for up to an hour longer.

alias: Corner Bedroom Heater - Turn off at the end
description: ""
mode: single
triggers:
  - at: "22:01:00"
    trigger: time
conditions: []
actions:
  - type: turn_off
    device_id: <your device id>
    entity_id: <your entity id>
    domain: switch

That’s all there is to it! Guess which room used a space heater?

Signal Gathering

These are my remotes:

Both of these are radio-based and use 434 and 350MHz signals respectively. Let's first see what kind of signals they emit and will have to hope it's the same signal every time (no rolling codes). Since I have a Flipper Zero, I'll use that:

From the look of the signal timeline graph, it did detect that I had pressed a button on the remote. And when I press Send to try broadcasting it again, it works! Rinse and repeat for every other button on the remote and we now have a recording of every signal we can know of. Now to look into how we can make a permanent device to broadcast these signals via Home Assistant controls.

Software

On the Home Assistant side, I'll still need some sort of radio device to broadcast these signals, so let's see what ESPHome would let us do. Looks like remote_transmitter is what I want, but it's strange... I don't think it's the right solution for a couple reasons.

1. The docs specifically say it is for 433MHz with no configurability, but I also need to broadcast 350MHz.

2. It appears to just use a GPIO pin directly as the transmitter. As neat as that is, further reading reveals that this will create interference at many other frequencies unless we use a low-pass filter.

Is there an alternative? Further searching reveals a popular radio chip, the TI CC1101. It's actually the same radio used in the Flipper Zero! With this, we can do standard SPI communication with the CC1101 and let it deal with the details of actually transmitting my commands. With dbuezas' CC1101 driver for ESPHome, the software side of things is covered.

Hardware

A CC1101 board with antenna can be had for about $6 with shipping. I picked the "CC1101 433MHZ SMA" variant to be specific. For the microcontroller, I went with an ESP32 D1 Mini which is also about $6 with shipping, specifically I picked a microUSB model.

Next is to make a case for it. The D1 Mini is likely pretty standardized in its dimensions, but there are a lot of variations of the CC1101 boards. I modeled something for my particular boards, so YMMV.

Some soldering later...

The two shells can be rubber banded together, or you can print as many of the "Case Wrap" model as you want.

Processing the commands

I used my Flipper Zero to read the commands sent by each button on my remotes, but it's not a requirement. Dbuezas' CC1101 driver includes code to log the signals that it detects straight into ESPHome's log output.

But this is just a raw recording which implies some issues:

1. There can be noise before and after the desired signal. If not trimmed out, the noise in the beginning will still be broadcasted each time you want to send the command - adding latency.

2. The remotes like to send the command multiple times in quick succession, but this is often unnecessary and adds significantly more characters into the YAML. I reduce it down to only sending the command once, and if you would prefer to continue sending it a few times then I think it's best to repeat the signal in code rather than with a longer recorded signal.

3. It's hard to see just from the raw numbers what constitutes as the "command".

To help us identify the specific command from the raw reading, dbueza created a corresponding tool to visualize the signal, which is incredibly helpful! And the only incompatibility it had with my Flipper recordings was it expected comma delimiters instead of spaces. No big deal, find and replace spaces with commas and you can input the Flipper's readings into the visualizer:

It seems to send the signal a couple times, but in reality you can just take a single segment like this:

Hovering the cursor to the left of the left-most rectangle, you can see which number it underlines & highlights in the CSV. In this case, a particular 453. Remember where this specific 453 is.

Do the same for the last rectangle, where it ends with 411:

We will just need to select the entire string between the 453 and the 411 and put that code into the code: [] line as seen in the sample YAML file. It should look like this:

button:
  - platform: template
    name: Office Light
    on_press:
      - lambda: get_cc1101(transceiver).setFreq(433.92);
      - lambda: get_cc1101(transceiver).beginTransmission();
      - remote_transmitter.transmit_raw:
          code: [453,-344,459,-352,403,-414,371,-422,409,-390,375,-424,375,-424,407,-388,409,-388,377,-422,409,-392,411,-5172,803,-414,435,-746,413,-800,837,-356,821,-380,429,-774,825,-380,837,-354,435,-744,443,-772,447,-750,815,-398,413,-756,843,-368,451,-736,847,-368,821,-382,421,-784,415,-778,415,-770,443,-776,429,-784,415,-752,457,-746,837,-354,841,-384,821,-354,841,-382,409,-776,441,-776,811,-358,445,-754,817,-400,841,-342,425,-786,449,-754,411,-770,441,-774,431,-778,391,-808,409,-778,419,-772,445,-754,449,-756,431,-784,389,-806,409,-776,819,-386,447,-752,413,-768,451,-760,837,-366,833,-362,839,-382,831,-326,847,-382,819,-364,839,-382,819,-364,443,-770,805,-386,839,-352,817,-390,445,-748,411,-800,411,-1000000]
      - lambda: get_cc1101(transceiver).endTransmission();

Note, I made two changes compared to the sample:

1. The lambda: get_cc1101(transceiver).setFreq(433.92); line is added because my two ceiling fans use different frequencies. I simply always set the relevant frequency prior to each transmission.

2. I added a -1000000 to the end of the code list to ensure it waits for 1 second before moving on to any further broadcasts. This is because there's a cooldown period where the fan ignores follow-up commands if they happen too soon after the first command.

And that's all! Do the same process for each button on the remote(s) and those buttons can now appear in Home Assistant like so:

And like everything else in Home Assistant, it is ripe for automations! 🤩

Bonus Round - Light Dimming

My office light has a light dimming function where you hold on the light button (on the remote) and the light will progressively dim and brighten. You then let go of the button when it reaches the brightness you want. The signal it sends for this is different from the usual light on/off command and has to be handled separately. The problem is finding a clean way to make ESPHome continuously send the command until an arbitrary moment.

My first approach was to make a button that would send the command for an entire second, then I would keep clicking on it approximately every second until it reached the desired brightness. However, it depends on my own timing for each sequential click because each command is queued up to run right after the previous command. Clicking too quickly actually causes it to send for longer than intended as the queue has to work through a backlog of commands.

It's a terrible UX. Can we do better?

What about a toggle switch?

It should be possible to use a toggle switch that sets a corresponding boolean variable, then have a loop that continuously sends the dimming command while the variable is true.

The problem is this doesn't work:

switch:
  - platform: template
    name: Office Light Dimming Hold
    id: office_dimming
    turn_on_action:
      - script.execute: office_dimmer_script

The switch always immediately turned back off. I couldn't figure out why it was behaving differently from all the other toggle switches I had on other devices. Turns out there is what I'd consider to be a bug with the platform: template switches. Michele184 on the Home Assistant forum posted a workaround here.

So the working configuration is:

switch:
  - platform: template
    name: Office Light Dimming Hold
    id: office_dimming
    turn_on_action:
      - switch.template.publish: 
          id: office_dimming
          state: ON
      - script.execute: office_dimmer_script
    turn_off_action:
      - switch.template.publish: 
          id: office_dimming
          state: OFF

script:
  - id: office_dimmer_script
    then:
      - while:
          condition:
            switch.is_on: office_dimming
          then:
            - lambda: |-
                ESP_LOGD("Office Light", "Dim triggered");
                get_cc1101(transceiver).setFreq(433.92);
                get_cc1101(transceiver).beginTransmission();
            - remote_transmitter.transmit_raw:
                code: [449,-386,391,-376,423,-382,391,-414,417,-380,415,-360,417,-392,427,-386,433,-352,405,-388,439,-354,441,-5178,819,-384,421,-752,451,-780,811,-366,853,-354,421,-782,835,-342,841,-362,445,-756,447,-768,439,-748,837,-380,431,-740,825,-410,415,-774,813,-372,857,-348,421,-788,413,-780,443,-734,441,-774,433,-780,391,-776,475,-740,835,-358,851,-344,839,-368,857,-346,423,-768,437,-768,815,-408,401,-770,849,-344,853,-364,441,-736,441,-772,449,-744,411,-804,449,-746,415,-768,439,-778,431,-748,449,-750,459,-744,835,-354,447,-780,411,-786,417,-782,415,-772,831,-382,423,-748,485,-722,427,-790,823,-354,863,-350,851,-348,441,-744,843,-364,841,-386,815,-362,839,-344,443,-770,839,-350,873,-352,817,-350,445,-1]
            - lambda: |-
                get_cc1101(transceiver).endTransmission();
            - delay: 25ms

Now I can simulate holding the dimming function with a toggle switch!

That sucks, but should be solvable right? Let's go on a journey!

What are the elements we care about?

Let's start with scoping out the actual elements at play here: the ESP32 chip, the MGM111 chip, and the utility power meter.

The ESP32 chip is the main microcontroller in the Vue device and communicates with the MGM111 chip over UART.

The MGM111 chip speaks Zigbee which is how it communicates with the power meter.

When installing the ESPHome firmware, we are only modifying the ESP32 chip and need our firmware to transmit the same UART commands to the MGM111 chip that the stock firmware would send. We would then need to make sense of the responses.

Is the payload itself an error message?

The payload was expected to be 152 bytes, but mine is 44 bytes. It's significantly smaller, so maybe it's just an error response? How to try to confirm this?

Let's try to induce an error. As seen in the code, a meter-join request is sent before requesting meter readings. Logically, if we try to request a reading without requesting to join the meter first, that would return an error, right?

Turns out the read request returns a 44-byte payload like before. Dang, so it is an error. It also looks almost identical to the previous payload.

Is the meter join request even successful?

Let's try to induce an error at this stage.

We can try preventing the Zigbee communication with the meter from happening, but we also need the ESP32 to still be able to connect to my wifi and show its logs. How to do this... Let's stick it in a metal bottle!

I can just point the mouth of the bottle near a wifi AP and away from the utility meter. So what did that do?

The meter-read request no longer gets a response, so we've successfully broken the Zigbee communication, but the meter-join request still returns a 1. That is suspicious because 1 happens to be the same value it returns even when there is a Zigbee signal. This has proven to be kind of a dead end. What now?

What if we can confirm the stock firmware gets the same responses?

The ESP32 and MGM111 communicate over a UART bridge, so in theory we should be able to sniff the traffic. Let's flash the original firmware back on to see how it behaves.

Fortunately, jrouvier already documented the pins of interest on the board, which for us would be the P5 group of pins. Time to sniff!

Here is the communication for requesting a meter reading:

request : 24720D
response: 2401722C182F010000002563A5150900000100002515EE960000000103002201000002030022E803000004002A5500000D

Looking at the docs again, if we take out the header and footer bytes we can see that it is the same 44-byte payload I saw while running ESPHome, so that payload is expected and valid! Fantastic, now to analyze the bytes.

Identify the bytes of interest

Let's compare three readings to see which bytes change between them:

There are only 3 (plus an incrementing byte) groups of bytes that ever change, so what are they?

Let's call the incrementing byte "H0", and the three groups shown above can be H1, H2, and H3. Let's also swap their endianness. We can also take the reported watts from the Emporia app to know what value we should be expecting.

Hex Group 3

H3 is clearly related to the wattage reading because all the negative wattages start with F's, but what's up with 33.7 and 45.3? According to H3 they're supposed to be negative... and the other calculated wattages are close-but-not-equal to the reported wattage. What's going on here?

The data above were hand-picked, but I think we need to look at many in a row to help detect patterns. So, I created a quick and dirty tool. See the pattern below?

The quoted kW doesn't line up with the calculated W value, but it is offset (for the most part)! In this case, it is offset by a minute. So either my system's clock or Emporia's clock is off. Couldn't be me — Emporia's clock must be off by a minute. 😇

This also explains the 33.7 and 45.3 values having negative hex values. Given they were so relatively close to 0, the adjacent minute's reading must have been negative.

Now what about the other hex groups?

Hex Groups 1 & 2

Let's sort chronologically instead:

What pattern do you see with H1 and H2?

Both columns have sections where they don't change (see the top rows of H2 and bottom rows of H1), but does that correlate with anything? That's right, H2 doesn't change when the wattage is positive, while H1 doesn't change when the wattage is negative. They also seem to increase in value over time (with rollover).

These must be counting up the cumulative watt-hour usage. But is it Wh or kWh? Let's convert some to decimal and check:

In this table, each row is one minute after the previous row. If we take the difference between two rows (a watt-minute) and multiply it by 60, we can convert it to instantaneous watts for that minute. This should match the actual wattage reading at the time, and it does! So I can confirm that H1 is Watt-hours consumed while H2 is Watt-hours produced.

UPDATE (2024-01-11)

Upon seeing more data, I've concluded that H1 and H2 are actually four bytes each, not two. I will keep the original writing above but have updated the remainder of the article below.

Why didn't jrouvier's code work?

Their work was based on the returned payload from the MGM111 firmware version 2, but mine shipped with version 7. I've documented the difference in the payload here.

TL;DR, this is the V7 payload format:

Show me the code

Here you go: https://github.com/nekorevend/esphome-emporia-vue-utility

And it works!

With the combination of Home Assistant / InfluxDB / Grafana, I can record and visualize historical data:

If you like #datahoarding, this is great fun.. right?

Side note: I'm focusing on humidity readings for this post because calibrating humidity has been a rabbit hole while calibrating the temperature has been more straightforward. That said, the tool I provide will calibrate temperature as well.

Descent into Madness

There's an adage called Segal's law that states:

A man with a watch knows what time it is. A man with two watches is never sure.

Well, I have way more than one sensor and they certainly don't agree with each other. Here are several of them installed very closely together on a single breadboard to ensure they are measuring the same environment.

Here's their data on a timeline:

They've formed two groups of measurements, but which group is correct?

... Is either group correct?

Not a great situation. There's no point in hoarding data that isn't even accurate. So what can be done? Follow me on my journey!

Step 1: How do others calibrate?

When searching about humidity calibration methodologies, I encountered an unexpected ally: cigar enthusiasts! Cigars must be stored in a particular humidity range (65-75%), so they have an interest in having a properly calibrated humidity sensor.

According to discussion boards and in videos, the technique is this:

1. Use table salt (NaCl) and make it damp - like wet snow.

2. Seal it in a container along with your humidity sensor.

3. Wait several hours for the salt to regulate the humidity in the box to about 75%.

4. Observe the humidity reported by the sensor.

5. Tweak the sensor up or down by the appropriate amount to make it say 75%.

6. Done!

I couldn't use a lid with my Tupperware container (like in the video) because I need to run a power cord, so this is my setup:

<Tangent>

The photo above shows that I sealed the sensors and salt in a Ziploc bag. I want to take a quick aside to mention that it took me weeks of container iterations to settle on that method of having a well-sealed container. The bag's only hole is a corner I cut to run a power cord through, but that corner was subsequently taped back shut.

I also tried using a Qi wireless receiver to avoid even needing to cut a hole for the power cord at all, but the heat generated by wireless charging proved to be too much.

All of my prior container "designs" had (bad enough) leaks that led to inconsistent readings, such as with my earliest approach of holding a plastic wrap down with a rubber band:

Don't do that. Try using a Ziploc bag instead.

Anyway, back to talking about the salt calibration method!

</Tangent>

This method works fine for cigars because table salt happens to regulate a humidity (75%) that fits the ideal range for cigars (65-75%). So, this single-point calibration is good enough.

What's the issue?

I am looking for accurate measurements across the entire spectrum that I'd encounter at home. The single-point calibration method has a major weakness: you don't know if the sensor's inaccuracy is uniform across the spectrum.

In other words, if at 75% the sensor reads 70%, then that's a simple +5 to calibrate it. But would that sensor report 30% when the reality is 35%? Spoiler: no.

Step 2: Can I calibrate with more than one point?

Different salts maintain different humidity levels. Let's add a second point of reference. I chose to use Magnesium Chloride (MgCl₂) because at 33% it allows covering a reasonable spectrum alongside NaCl's 75%. Compared to other low-humidity salt options, MgCl₂ is also easier to get. I could find it on Amazon while others were special enough to only be sold by dedicated science supply stores.

Using more than two points of reference would be nice. But for practical purposes, I am going to limit to only using NaCl and MgCl₂... at least for now.

So how does the data look?

We can see that all of the sensors tend to read higher on the low end and lower on the high end. The "F" and "G" sensors happen to be too high (almost 40% instead of 33%) while being spot on at 75%, while the other sensors are spot on at 33% but deviate to about 71% when they should report 75%.

ESPHome provides a few calibration functions, and a basic one we can use here is calibrate_linear. So I can just add something like this to the .yaml config:

    filters:
      - calibrate_linear:
          - 39.386 -> 33.613
          - 75.231 -> 75.500

Easy, right? But wait, that wasn't so bad... was the rabbit hole just learning how to seal the container? Unfortunately not. 🙃

Relative Humidity

The humidity percentages I've been talking about are a "relative humidity" value. They are relative to the current temperature. So if anything, I've only defined the calibration for humidity specifically at my room temperature (77.5°F / 25.278°C). If I were to seriously calibrate the humidity readings, I would have to use a formula that considers both temperature and humidity.

Step 3: Gather humidity values at multiple temperatures

Fortunately, calibrating using salts still works here. As I linked earlier, there exist tables that define the temperature-to-humidity relationship for each salt.

I went with this table because it gives more precision (to the hundredths place).

While the humidity level changes with temperature, it changes by very little.

As for what kind of temperatures I could realistically produce with any sense of long-term stability without laboratory equipment... I opted for room temperature and the refrigerator, which are 25 and 4°C, respectively. So how do the sensors vary depending on temperature?

Most of their slopes don't match the Expected slope, so we need to apply a temperature-dependent adjustment.

Step 4: Interpolations

We need to make a few interpolations to be able to identify the adjustments needed (for each sensor) to derive the correct humidity at any given temperature. To do so, we consider:

1. For any given temperature, what are the expected humidities for NaCl and MgCl₂?

2. For any given temperature, what are the incorrect humidities reported by the sensor for NaCl and MgCl₂?

3. With those two figured out, what is the necessary correction slope to address humidities that aren't 33 or 75%?

1. Reference Humidities at any Arbitrary Temperature

The original humidity table I used only provides humidities at 5° intervals. I let Google Sheets derive polynomial trendlines to match this data.

For a given temperature x, the humidity with NaCl would be calculated with this:

$$33.697 - (7.98 \times 10^{-3})x - (1.09 \times 10^{-3})x^2 - (9.71 \times 10^{-9})x^3$$

For MgCl₂, it'd be this:

$$75.51 + 0.0396x - (2.65 \times 10^{-3})x^2 + (2.84 \times 10^{-5})x^3$$

2. Reported Humidities at any Arbitrary Temperature

As mentioned before, I will only be trying two temperature points: 4 and 25°C. We will just have a linear relationship between the two measurements.

Unfortunately, it'd be awkward to try to use ESPHome's calibrated_linear because it isn't available as a function call. Instead, we can use a segmented_linear implementation I had written in the past. You can read about it here!

Yes, it was intended for inputting three or more points, but it will behave identically to calibrated_linear when given two points and is available as a simple function call.

3. Corrected Humidity outside of 33 or 75%.

Let this just be a linear relationship based on the offsets that need to be done to each sensor at 33 and 75% humidity. We will use segmented_linear here as well.

Step 5: Implementation

The calibration options built into ESPHome are inadequate for this task, but they allow calling C++ code in a lambda. So let's make a custom function! What are our inputs?

• Temperature.

  • Correct temperature, post-calibration.

• Humidity.

  • Original reading, pre-calibration.

• Function to calculate the expected humidity for a given temperature.

  • Need at least two of these at different humidity levels to have an adjustment slope/curve on the spectrum.

  • For my needs, I will just implement support for two points.

• Function to calculate the reported humidity for a given temperature.

  • Needs to be a matching set to coincide with the same respective expected humidities.

So we need two floats and four lambdas.

float calibrated_humidity(
        float temp,
        float hum,
        const std::function<float(float)> &expected1,
        const std::function<float(float)> &expected2,
        const std::function<float(float)> &measured1,
        const std::function<float(float)> &measured2) {
    // ...
}

We'll also need helper functions to derive the linear fit line:

float correlation_coefficient(float x1, float x2, float y1, float y2) {
    float avg_x = (x1 + x2) / 2;
    float avg_y = (y1 + y2) / 2;
    float numerator = (x1 - avg_x) * (y1 - avg_y) + (x2 - avg_x) * (y2 - avg_y);
    float denominator = sqrt(
                            pow(x1 - avg_x, 2) +
                            pow(x2 - avg_x, 2)
                            ) * 
                        sqrt(
                            pow(y1 - avg_y, 2) +
                            pow(y2 - avg_y, 2)
                        );
    return numerator / denominator;
}

std::pair<float, float> linear_fit(float x1, float y1, float x2, float y2) {
    float correlation_coefficient_value = correlation_coefficient(x1, y1, x2, y2);
    float slope = correlation_coefficient_value * (y2 - y1) / (x2 - x1);
    float intercept = y1 - slope * x1;
    return std::make_pair(slope, intercept);
}

So all together, our calibration call would be...

float calibrated_humidity(
        float temp,
        float hum,
        const std::function<float(float)> &expected1,
        const std::function<float(float)> &expected2,
        const std::function<float(float)> &measured1,
        const std::function<float(float)> &measured2) {
    std::pair<float, float> pair = linear_fit(
            measured1(temp), 
            measured2(temp),
            expected1(temp),
            expected2(temp)
        );
    float slope = pair.first;
    float intercept = pair.second;
    return (slope * hum) + intercept;
}

Let's put all of that into a calibration.h file. We can then use it in an ESPHome YAML configuration like so:

esphome:
  name: my_sensor
  includes:
    - calibration.h
# ...
- &hum_calibrate
    lambda: |-
      static auto expected1 = [](float x) -> float {
          return 33.67 - ((7.98 * pow(10, -3)) * x) - ((1.09 * pow(10, -3)) *
              pow(x, 2)) - ((9.71 * pow(10, -9)) * pow(x, 3));
      };
      static auto expected2 = [](float x) -> float {
          return 75.51 + (0.0396 * x) - ((2.65 * pow(10, -3)) * pow(x, 2)) +
              ((2.84 * pow(10, -5)) * pow(x, 3));
      };
      static auto measured1 = [](float x) -> float {
          return -0.22*x + 42.4;
      };
      static auto measured2 = [](float x) -> float {
          return -0.453*x + 71.4;
      };
      return calibrated_humidity(
          id(my_calibrated_temperature).state,
          x, expected1, expected2, measured1, measured2
      );
# ...

Step 6: Calibrating other sensors

I'll admit, the salt test calibration process is tedious and annoying and I don't want to do it again. But I do want to calibrate my other sensors. I can take the salt-calibrated sensors as my source of truth and place the additional sensors near them so they measure the same environment. From there, it'd be a matter of calibrating the other sensors according to what the "true" sensors are reading.

But how can I confidently do this? Going back to how I use both the temperature and humidity readings to properly calibrate the humidity, I'd need at least four data points:

1. Low humidity @ Low temperature

2. High humidity @ Low temperature

3. Low humidity @ High temperature

4. High humidity @ High temperature

The problem is I don't have control over the humidity anymore due to not wanting to redo the salt test. It's a matter of luck to see what the weather gives me, and enough time for the weather to vary enough. Ultimately, I'm not in a hurry and the setup would be set-and-forget, so I can allow for weeks of data to be gathered.

Procedure

Let's gather lots of data, and I will use my garage environment since it would have the widest swings (of at least the temperature). The data we care about are the calibrated temperatures and humidities from the calibrated sensors and raw readings from the uncalibrated sensors. The data will have a wave pattern on a daily cadence. Here are the calibrated sensors:

You can see they pretty much agree with each other. The humidity graph has a wider delta between lines due to the sensors having a rated humidity margin of error of ±1.0-1.8% depending on model.

Now let's add a couple of uncalibrated humidity readings that we want to calibrate:

The two new lines don't agree with the calibrated lines. We have our work set out for us.

We will want to:

1. Choose two humidity points (from the calibrated sensors only), a low and a high.

   • You might be tempted to find the widest difference between low and high, but by its nature, it is an outlier. An outlier will mean we only find one temperature for that humidity at the one time it occurs.

     • We are looking for examples of low and high temperatures existing with the same humidity level, so it is a necessity to choose humidity levels that occur more than once.

   • I think a reasonable method is to calculate the standard deviation and use the -1 and +1 stddev humidity values.

2. Find all timestamps that have the chosen humidity points.

3. Get the lowest and highest observed temperatures among those timestamps, for each humidity point.

4. Once we have the timestamps for those temperatures, we need to get the respective humidity values at those timestamps from the sensors we want to calibrate.

5. Now we know both the expected and observed humidity levels for two different temperature levels at two different expected humidities. This is enough data to feed into my calibrated_humidity() function mentioned above.

   • One difference is the expected lambdas were created to support the salt test, where the expected humidity changes depending on the temperature. But in this approach, we are matching based on the humidity. Naturally, the humidity level becomes a constant. Therefore, the two expected lambdas can just return a static value (either the low or high humidity, respectively).

Enter those values into the YAML like so:

lambda: |-
  static auto expected1 = [](float x) -> float {
      return 38.002;
  };
  static auto expected2 = [](float x) -> float {
      return 48.856;
  };
  static auto measured1 = [](float x) -> float {
      static std::vector<std::vector<float>> mapping = {
        // {Temperature, Humidity}
        {27.068, 40.548},
        {34.536, 40.816},
      };
      return segmented_linear(mapping, x);
  };
  static auto measured2 = [](float x) -> float {
      static std::vector<std::vector<float>> mapping = {
        // {Temperature, Humidity}
        {20.859, 49.383},
        {30.523, 48.517},
      };
      return segmented_linear(mapping, x);
  };
  return calibrated_humidity(
      id(temperature).state,
      x, expected1, expected2, measured1, measured2
  );

Now the newly-calibrated sensors fit in with the rest!

Step 7: But that's so much data to process...

I agree! No one should be doing that by hand, especially if you intend to calibrate multiple sensors. That's why I wrote a tool to do it for me (and you!).

I've written two ways of using the script:

1. Let it query your InfluxDB directly (from_influx.py).

2. Give it some CSV files with the same data we need (from_csv.py).

From CSV

The CSV way is the easier place for us to start understanding the tool.

There are four collections of data we need to pass in:

1. Reference Temperatures

2. Reference Humidities

3. Uncalibrated Temperatures

4. Uncalibrated Humidities

The command would look like this:

python from_csv.py 
    --reference_temperature_csv ref_temp.csv
    --reference_humidity_csv ref_hum.csv
    --uncalibrated_temperature_csv uncal_temp.csv
    --uncalibrated_humidity_csv uncal_hum.csv

All of the CSV files must be in the format SENSOR_NAME,TIMESTAMP,VALUE. The value unit you use here doesn't matter as long as you understand that the output will be in the same units (which might not be the unit you need in your ESPHome config). The timestamp must be parsable by:

datetime.strptime(input, '%Y-%m-%dT%H:%M:%SZ')

To be explicit, the tool is expecting lots of values. In my case, I'm using a 30-second interval. For a hypothetical week of data gathered, it would amount to 20,160 data points per file, per sensor. The script will interpolate between the gaps, but it's always better to have smaller gaps.

From InfluxDB

The tool can query the database directly to grab the same data as we'd have passed into the CSV. But naturally, in this case, you'd have to specify the entity IDs of the sensors as well as the date range you want to sample.

python from_influx.py
    --start_time "2023-08-31 07:00:00" --end_time "2023-09-03 07:00:00"
    --reference_temperature_sensors sensor_a_temp,sensor_b_temp
    --reference_humidity_sensors sensor_a_hum,sensor_b_hum
    --uncalibrated_temperature_sensors sensor_c_temp,sensor_d_temp
    --uncalibrated_humidity_sensors sensor_c_hum,sensor_d_hum

The start and end times expect UTC.

--output_to_csv exists as well to simply output the CSV files that you could pass into from_csv.py.

You also need to create an influx_config.py file to contain the details needed for the tool to connect to your database instance.

INFLUX_ORG='your_org'
INFLUX_BUCKET='your_bucket'  # probably "homeassistant"
INFLUX_TOKEN='your_token'
INFLUX_URL='http://your_db_ip:8086'

Example output

Here's what you can expect to see for a given uncalibrated sensor. You can then copy and paste the calibrate_linear and lambda sections into the appropriate spots in your sensor's ESPHome YAML configuration.

Sensor: sensor_c_temp

========== Temperature Calibration ==========
calibrate_linear:
  method: exact
  datapoints:
    - 21.985 -> 20.100
    - 23.958 -> 22.200
    - 25.942 -> 24.300
    - 27.808 -> 26.400
    - 29.633 -> 28.500
    - 31.490 -> 30.600
    - 33.600 -> 32.700
    - 35.722 -> 34.800

=========== Humidity Calibration ============
lambda: |-
  static auto expected1 = [](float x) -> float {
    return 38.065;
  };
  static auto expected2 = [](float x) -> float {
    return 48.819;
  };
  static auto measured1 = [](float x) -> float {
    static std::vector<std::vector<float>> mapping = {
      {23.116, 36.839}, {31.777, 38.114}
    };
    return segmented_linear(mapping, x);
  };
  static auto measured2 = [](float x) -> float {
    static std::vector<std::vector<float>> mapping = {
      {20.678, 44.844}, {28.367, 46.836}
    };
    return segmented_linear(mapping, x);
  };
  return calibrated_humidity(
    id(temperature).state,
    x, expected1, expected2, measured1, measured2
  );

Assumptions made

To avoid complicating the logic, I've made some assumptions about what's being provided to the tool.

1. The set of sensors given for uncalibrated_temperature_sensors and uncalibrated_humidity_sensors are identical. Same length and contains the same sensors.

   • This means that your sensors are expected to be temperature+humidity combo sensors. This is true for me so I did not dive in further. If you are trying to calibrate sensors that are only one of these types... try to fake some dummy data. I would duplicate the data from another sensor and just change the name in the CSV, then use the from_csv.py route.

     • Make sure the dummy name matches the format of the sensor you want to calibrate, see the next assumption below.

   • Note that this restriction does not apply to the reference sensors. The tool will just average them together before using the data.

2. Your sensor names are consistent between temperature and humidity. To match up the temperature and humidity entities for each sensor, I am assuming that the list of temperature entities and list of humidity entities would alpha-sort into the same order.

   • The respective sensors at each index of both lists should end up referring to the same actual sensor.

Requirements

Setting some ground rules for what I care about.

• Must be able to tell if the door is opened or closed.

• Can open/close the door.

• Long-lasting set-and-forget.

  • No batteries.

  • No moving parts.

  • Impervious to the dirty environment of a garage.

  • No dependence on some company's cloud service staying online.

• No subscription model.

Note: The way the myQ solution can tell if the door is open is by having you attach a sensor to the door itself, but that sensor is battery-powered. No good!

Design

Control

I will need to be able to tell my garage door to open/close. But how?

The Button Side

The dead simple way is to understand that all a button does is short two wires when pressed. If we disassemble the button on the wall, we can connect two more wires to that button and use our microcontroller to simulate a button press by shorting those wires.

But there's a catch. My wall button is not just a button like this:

Mine has additional functions including light control and a motion sensor:

Note: The main button cap was already removed when I took this photo.

So, I couldn't simply connect two wires to the two "W" and "R" wires coming from the wall since they are not the correct wires we need to short:

Instead, I will need to solder my wires to the button itself. Specifically these pins:

Solder from the back:

Route the new wires out the side, and we're done with modding the wall control!

The Microcontroller Side

For some reason, the wall button is powered by something close to 20 volts AC. The microcontroller (a Raspberry Pi Pico W) operates at 3.3 volts DC on its GPIO, so we can't connect the wires directly to the microcontroller. Instead, we need to use a relay. The microcontroller tells the relay when to short the wires while being isolated from the 20VAC power thanks to the relay.

Let's 3D print a case for the microcontroller:

The relay is on the bottom-left and the microcontroller is on the right, but let's get to what the thing at the top is next!

Sensing the Door

My go-to for this would be an accelerometer sensor that can detect when it's vertical (closed) or horizontal (opened). Such a sensor would need to attach to the door itself. Since the door moves, we wouldn't be able to (easily) power a door-mounted sensor via wire. The logical answer is to use a battery-powered sensor like the myQ does, but I refuse to be beholden to dealing with low batteries. How can we sense that the door is open without needing to attach a sensor to the door?

How about putting a sensor on the wall near the door? It could work, but how would we sense the door? The options are...

1. Use an Ultrasonic Distance Sensor to sense if something (the door) is nearby. But that doesn't have long-term reliability. The sensing element can be blocked by dirt over time or a spider could move into just the wrong place.

2. Use light-based sensors to do fundamentally the same thing as the ultrasonic sensor. It carries all the same issues plus one more: a bright sunny day could overload the sensor's ability to see its own light beacon.

3. A mechanical switch could be triggered by the door being in position or not. But anything with moving parts will wear down so that won't last forever.

4. Lastly, magnets! I can just place a magnet on the door and have a sensor on the wall to detect if a magnet is nearby. It is impervious to dirt, bugs, and light. The clear choice!

Let's go with this magnet sensor and magnet. But that sensor is tiny and bare, how can we mount it? How would we wire it?

As you can see, the sensor has 3 pins, so we would need to run a cable with at least 3 wires, and the cable would have to be relatively long to reach the microcontroller that I will install near the garage door wall button. Does anything fit the bill? How about a regular 3.5mm audio cable!

Let's also get this 3.5mm jack breakout board. We can then solder the sensor to it along with a pullup resistor like so:

3D print a case for it to position the sensor at the right spot:

Oh yeah, remember that third thing at the top in the microcontroller box? It's the jack for the other end of the audio cable!

There's a hiccup though, this angle turns out to be bad:

Not a big deal, just needed to use a wedge shape to give it an angle:

Final Hardware

Now all we need to do is mount everything else and wire it all up!

Closeup of my crude wiring:

The box on the wall:

You can see the two wires from the wall button going to the relay in the box, the audio cable plugging into the top, a microUSB cable to power the microcontroller, and hey, what's that bunch of wires coming out of the bottom of the box?

Not going to get into it for this article, but I figure I might as well have a temperature and humidity sensor for the garage. It just hangs out of the box to guarantee that the heat from the microcontroller will not impact the thermometer.

Before you say it, I didn't care to organize anything since the garage is already a mess. Sorry, not sorry. 🙂

Software

As I said at the beginning, I will use ESPHome and Home Assistant. ESPHome is a microcontroller firmware that abstracts away boilerplate logic for most IoT needs and allows you to easily configure something like my garage controller using just a YAML config.

Here is the full basic YAML (scroll to the bottom to skip the boilerplate):

esphome:
  name: garage-controller

rp2040:
  board: rpipicow
  framework:
    # Required until https://github.com/platformio/platform-raspberrypi/pull/36 is merged
    platform_version: https://github.com/maxgerhardt/platform-raspberrypi.git

# Enable logging
logger:

# Enable Home Assistant API
api:
  encryption:
    key: "<your key>"

ota:
  password: "<your password>"

wifi:
  ssid: !secret wifi_ssid
  password: !secret wifi_password

substitutions:
  name: "Garage"
  garage: "Garage Door"

i2c:
  - id: i2c_a
    sda: GPIO20
    scl: GPIO21

sensor:
  - platform: sht3xd
    temperature:
      name: "${name} Temperature"
      accuracy_decimals: 2
    humidity:
      name: "${name} Humidity"
      accuracy_decimals: 2
    address: 0x44
    update_interval: 1s

switch:
  - platform: gpio
    pin: 
      number: GPIO22
    id: garage_door_relay
    name: "${garage} Remote"
    icon: "mdi:garage"
    on_turn_on:
    - delay: 200ms
    - switch.turn_off: garage_door_relay

binary_sensor:
  - platform: gpio
    pin:
      number: GPIO8
    id: garage_door_status
    name: "${garage} Status"
    device_class: garage_door
    filters:
      - delayed_on: 200ms

The switch: and binary_sensor: sections are all we needed to add to support toggling the garage door and know if the door is open. Simple!

Here they are in the Home Assistant UI:

And they're fully available for automation, such as automatically closing the door if it's been open for too long.

I can also see a timeline of when the door's been open!

Perfect, all I've ever wanted for my garage!

"Eh, the files look fine to me..."

Unfortunately, it's hard to notice that it's happening unless you happen to be watching some footage and notice that the next file doesn't start where the previous file left off.

Some file managers display the "date modified" timestamp without seconds, so it's impossible to tell at a glance. Even if it did have seconds, scanning through the list and judging what the expected seconds would be for each file sounds tedious at best.

"I checked a few by hand, it seems like my camera doesn't have the issue."

This can be a "sometimes" problem. I have personally experienced missing footage and discovered that it correlates with high-activity scenes. When a lot is happening on camera, the bitrate of the recording is higher. In this situation, the dashcam and/or memory card don't always keep up well enough to start the next recording promptly. And the joke's on us: an emergency event probably qualifies as a high-activity scene...

Let's make a tool to automatically flag these gaps in recordings. That way we can troubleshoot and attempt to fix it before we regret missing footage when it matters most!

Approach

We don't need to parse much data from each video to figure this out. By using the video's timestamp as well as the video duration, we know when the next video's timestamp is expected to be.

Known Limitation

Since the timestamp only has a precision down to the second, we won't be able to detect gaps smaller than a second. We have to accept this limitation. This also means the precision at which we can calculate the gap is limited. A recording can start at, say, 12:00:00.678 but have a timestamp of 12:00:00. Our calculated gap would not be able to factor in the 678ms.

But that's fine since the real goal is to know that gaps have occurred rather than know the exact gap size down to the millisecond.

Let's write it!

But first a quick note, I am assuming that the camera has named the files in some sequential order that follows an alphanumeric sort. So, the tool will simply alpha-sort (using a priority queue because it's faster) the file paths and parse them in that order. os.walk() might already return the paths in the order we want, but I will still sort it to be safe.

Each file is then compared to the previous file and the gap between them is calculated.

Here it is on GitHub!

And this is the tool's output:

$ python3 detect.py -d "/tank1/cameras/A119S"
Detected 6408ms gap between ".../2017_0114_141750_008.MP4" and ".../2017_0114_141805_009.MP4"
Detected 3520ms gap between ".../2017_0114_141805_009.MP4" and ".../2017_0114_141816_010.MP4"
Detected 1000ms gap between ".../2017_0116_121056_017.MP4" and ".../2017_0116_121557_018.MP4"
Detected 1000ms gap between ".../2017_0205_153459_307.MP4" and ".../2017_0205_153959_308.MP4"
Detected 1000ms gap between ".../2017_0226_175522_708.MP4" and ".../2017_0226_180023_709.MP4"
Detected 1000ms gap between ".../2017_0402_204918_285.MP4" and ".../2017_0402_205419_286.MP4"
Detected 9528ms gap between ".../2017_0708_104906_511.MP4" and ".../2017_0708_104917_512.MP4"
Detected 1000ms gap between ".../2017_0708_110435_516.MP4" and ".../2017_0708_110935_517.MP4"
Detected 1000ms gap between ".../2017_0716_184607_697.MP4" and ".../2017_0716_185108_698.MP4"
Detected 1000ms gap between ".../2017_0803_201801_997.MP4" and ".../2017_0803_202302_998.MP4"

Looks like it works! We have gaps though. What can be done?

Fixing your dashcam

Just some general tips.

Lower your recording quality

In other words, use a lower bitrate. It might help to lower your recording quality so in a high-activity scene the raised bitrate is still within the capabilities of your device/storage.

Use a constant bitrate (CBR)

This means high-activity scenes will not cause the bitrate to increase.

Reformat your memory card

It's possible for prolonged use to bog down the filesystem on the card, where its metadata/access tables are not as efficiently organized (or something. I'm not an expert). A reformat would give the filesystem a fresh start. I know some dashcams even have an automated reminder to reformat the card periodically.

Get a faster memory card

This is assuming the bottleneck is the card and not the dashcam.

Get a better dashcam

If all else fails. 🙂

Background

I run a Network-Attached Storage (NAS) computer at home using ZFS on Linux.

ZFS supports quick and easy snapshotting thanks to its copy-on-write design. These snapshots allow outputting diffs between any two arbitrary snapshots. This is how the incremental nature of my backups work!

Requirements

I have goals for these backups that leads me to conclude that I must roll my own solution. Here are some of the goals:

Self-Encryption

I have no intent to believe any claims like...

Your data is private and encrypted by us and we promise we will never look at it!

or even...

We encrypt it with your encryption key and only you will ever know what the key is. Therefore, your data is private!

Aside from completely impractical wishful thinking, I will never be able to fully vet (and continue vetting every update!) that their software isn't saving my key or doing any other nefarious thing. So, my approach is designed in a way that I'd be comfortable sending my backups straight to <insert your adversarial organization of choice here>!

Fault-Tolerant

Let's assume the cloud provider can have bit-rot that they can't/don't fix. We must handle this ourselves.

Compatible

Perhaps a legacy mindset, but you never know if a cloud provider will decide to have a max file size limit that is low enough to be a problem. So to be safe, I am following the lead of FAT32's max file size and am ensuring my backups are split into 4GiB files.

Also, none of the tools used to create/read the backup should be proprietary to any closed-source platform.

Considerations

I have about 10TB of data to back up and that is sure to grow.

Internet

Your internet connection can be a bottleneck, both in speed and whether you have a data cap. I am fortunate enough to have symmetric fiber internet with no cap, but if you have limited internet access then this article isn't suitable. Some providers allow you to physically ship hard drives with your data, which you should consider.

Storage (local)

In my design, we're creating an entire backup of the data right on the system we're backing up. This requires the system to have as much free space available as the size of the dataset that we are backing up. If you're backing up everything at once, this means your system’s capacity needs to be double the size of your data.

However, you can plug in an external drive(s) as a staging ground for the backup files. This could even be considered your onsite backup!

I also suggest separating your data into logical sets like "cameras", "media", etc. This allows you to snapshot individual datasets and only be limited to needing enough free space to match your largest dataset.

Storage (cloud)

Let's spend as little as possible (of course!). Generally, the cheapest tier is called cold storage and is cheaper because they remove the drives (maybe even tape?) and shelve them until you request the data. The drawback is that you have to wait for your data to become available, which is usually counted in hours. But is that a problem?

Ideally, I will never need to download this backup. And if something catastrophic happens to my NAS, I probably have a day(s) of lead time to wait for the delivery of replacement hardware. Likewise, cloud cold storage tiers would take several hours of lead time for them to plug the drives back in for me to begin downloading. I just need to coordinate those two lead times and the access delay with cold storage becomes a non-issue.

Here's the comparison of the main contenders.

AWS and Azure are both the cheapest for storage, but the download pricing gives AWS the win.

"But you only compared four!"

Yes, I was not exhaustive in my search, but I have checked other providers like IDrive and Carbonite. Their pricing models are more of a flat rate with a storage cap. IDrive for example wants $99.50/yr for 10TB ($9.95/TB/yr) or $199.50/yr for 20TB ($9.98/TB/yr).

This comes out to be cheaper than AWS or Azure if you want to use exactly their limit. But there is no mention of prorated pricing! Needing to pay for 20TB of storage for, say, 10.1TB of data does not sit well with me.

Overview

Self-Encryption

Let's encrypt the data ourselves. We can use OpenSSL to encrypt it using a command like:

openssl enc -aes-256-cbc -md sha512 -pbkdf2 -iter 250000 -pass pass:my_password

Fault-Tolerant

Par2 is a tool to generate parity files. Let's generate them for our data files. It'd use a command like:

par2create -r5 -n1 -m3000 -q my_filename

Which would generate two files:

my_filename.par2
my_filename.vol000+100.par2

Compatible

The backups should be split into 4GiB files. We can simply use the Linux split command:

split -b 4G --suffix-length=6 - my_filename_prefix_

Which would generate files like:

my_filename_prefix_aaaaaa
my_filename_prefix_aaaaab
my_filename_prefix_aaaaac
...

Uploading

Finally, we need a command to upload the files to AWS, which can look like:

s3 sync --exclude * \
        --include my_filename_prefix_* \
        my_local_directory \
        s3://path/to/incremental/directory \
        --storage-class DEEP_ARCHIVE

Design

Naming and file structure

Let's have sensible names/prefixes to make this work. The date format shall go from broad-to-specific, ie. year, then month, then day. The only order that should exist.

For the ZFS snapshots, I settled on the name format offsite-YYYYMMDD.

The incremental backup would need to specify the snapshot date range that it pertains to, so for a given dataset my_dataset, the prefix format is my_dataset-FYYYYMMDD-TYYYYMMDD-part-. Where F is for "from" and T is for "to".

On AWS, there are two distinct directories to have.

Incremental backups need a baseline backup to increment on:

my_aws_bucket/my_dataset/baseline-YYYYMMDD/

Then the increments themselves would go into

my_aws_bucket/my_dataset/incrementals/FYYYYMMDD-TYYYYMMDD/

Altogether, here's a realistic example with actual dates:

ZFS Snapshots

$ zfs list -t all | grep "cameras.*offsite"
tank1/cameras@offsite-20201123
tank1/cameras@offsite-20220610
tank1/cameras@offsite-20220620
...
tank1/cameras@offsite-20230726

Incremental backup files and their parity files

$ ls -alh | grep cameras
-rw-r--r--  1 root root 4.0G Jul 26 15:01 cameras-F20230719-T20230726-part-aaaaaa
-rw-r--r--  1 root root  40K Jul 26 15:02 cameras-F20230719-T20230726-part-aaaaaa.par2
-rw-r--r--  1 root root 206M Jul 26 15:02 cameras-F20230719-T20230726-part-aaaaaa.vol000+100.par2
-rw-r--r--  1 root root 3.9G Jul 26 15:01 cameras-F20230719-T20230726-part-aaaaab
-rw-r--r--  1 root root  40K Jul 26 15:03 cameras-F20230719-T20230726-part-aaaaab.par2
-rw-r--r--  1 root root 199M Jul 26 15:03 cameras-F20230719-T20230726-part-aaaaab.vol000+100.par2

On AWS S3

Baseline files:

$ s3 ls my_bucket/cameras/baseline-20201123/
2020-11-25 23:25:10          0
2020-11-27 20:09:56 4294967296 cameras-T20201123-part-aaaaaa
2020-11-27 20:09:56      40428 cameras-T20201123-part-aaaaaa.par2
2020-11-27 20:09:56  215037572 cameras-T20201123-part-aaaaaa.vol000+100.par2
2020-11-27 20:09:58 4294967296 cameras-T20201123-part-aaaaab
2020-11-27 20:09:56      40428 cameras-T20201123-part-aaaaab.par2
2020-11-27 20:09:56  215037572 cameras-T20201123-part-aaaaab.vol000+100.par2
2020-11-27 20:09:56 4294967296 cameras-T20201123-part-aaaaac
2020-11-27 20:10:16      40428 cameras-T20201123-part-aaaaac.par2
2020-11-27 20:10:18  215037572 cameras-T20201123-part-aaaaac.vol000+100.par2
...

Incremental files:

$ s3 ls my_bucket/cameras/incrementals/F20230719-T20230726/
2023-07-26 22:03:55 4294967296 cameras-F20230719-T20230726-part-aaaaaa
2023-07-26 22:03:55      40436 cameras-F20230719-T20230726-part-aaaaaa.par2
2023-07-26 22:03:55  215037628 cameras-F20230719-T20230726-part-aaaaaa.vol000+100.par2
2023-07-26 22:03:55 4152776288 cameras-F20230719-T20230726-part-aaaaab
2023-07-26 22:03:55      40436 cameras-F20230719-T20230726-part-aaaaab.par2
2023-07-26 22:03:55  207928428 cameras-F20230719-T20230726-part-aaaaab.vol000+100.par2

Script Procedure

It is a relatively straightforward operation:

1. Find the latest backup that is on AWS (date A).

2. Create a new snapshot (date B).

3. Export the incremental snapshot from date A to date B.

   1. Pipe to openssl for encryption.

   2. Pipe to split to become 4GiB files.

4. Run par2create on each split file.

5. Upload all of it to AWS.

Automating

We'll just use crontab. I set mine to run weekly:

0 15 * * 3 bash -lc "/path/to/backup.sh"

Storing the encryption keys

We don't want to put our password directly into the command. You need to create a JSON file to contain the passwords and pass it in with the --config flag.

The JSON format is:

{
    "my_dataset_A": {
        "pass": "my_password"
    },
    "my_dataset_B": {
        "pass": "my_password2"
    },
}

WARNING: Put special care into escaping the special characters in your password!

And that's all for now! Visit the GitHub project to explore the script yourself!

To be fair, iOS does provide filtering by type, specifically these types:

Notice something missing? Photos! There is no filter available to only see your photos. The closest thing you can do is to filter to Live Photos, but that does not include any non-Live photos. Non-Live happens more often than you think, such as for long-exposure night photos and photos you take during videos (or you simply turned off Live).

Photos aren't the only issue with the provided filters. The "Videos" filter includes camera videos and screen recordings even though "Screen Recordings" is its own filter. So there is also no way to view only your camera videos without screen recordings mixed in! Let's try to solve all of this.

So how can you filter to only seeing your photos, without the distraction of screenshots, videos, etc. mixed in?

I don't have a solution on the iPhone itself (Apple, please fix this!), but I do export all of my media to a NAS (network-attached storage) and have written a tool to help separate them there. Once the tool finishes processing your files, it outputs a simple report like this:

Copying /media/iphone/all_media/IMG_1551.HEIC to /cameras/iPhone 13/IMG_1551.HEIC
Copying /media/iphone/all_media/IMG_1552.HEIC to /cameras/iPhone 13/IMG_1552.HEIC
Copying /media/iphone/all_media/IMG_1553.HEIC to /cameras/iPhone 13/IMG_1553.HEIC
Copying /media/iphone/all_media/IMG_1554.HEIC to /cameras/iPhone 13/IMG_1554.HEIC
Copying /media/iphone/all_media/IMG_1555.HEIC to /cameras/iPhone 13/IMG_1555.HEIC
Copying /media/iphone/all_media/IMG_1551.MOV to /cameras/iPhone 13/IMG_1551.MOV
Copying /media/iphone/all_media/IMG_1547.MOV to /cameras/iPhone 13 Videos/IMG_1547.MOV
Copying /media/iphone/all_media/IMG_1548.MOV to /cameras/iPhone 13 Videos/IMG_1548.MOV
Copying /media/iphone/all_media/IMG_1550.PNG to /media/iphone/screenshots/IMG_1550.PNG
Copying /media/iphone/all_media/IMG_1549.PNG to /media/iphone/screenshots/IMG_1549.PNG
Done!

Handled:
5 new camera photos.
1 new videos from live photos.
2 new camera videos.
2 new screenshots.

NOTE: As of 2023-07-06, the tool supports separating Photos/Live Photos, Videos, Screen Recordings, and Screenshots, but it cannot differentiate between your photos and those that were sent to you (via AirDrop or otherwise). Below I detail how this tool works today, and include sections about how I plan to support AirDrop in the future.

How does it work?

With the help of ExifTool and MediaInfo, the script can find hints in the metadata to make educated guesses about the type of each file. Below I will outline my logic for each type.

Is it an iPhone photo?

Not every JPEG or HEIC file is a photo taken by your device — it could have just been a funny meme you downloaded or a drawing you saved. It could have also been AirDropped to you.

Let's first address how to determine if a photo was taken by the iPhone's stock camera app. Note: I am considering third-party camera apps as out-of-scope.

All photos from the camera app have metadata and provide two key data points:

1. The maker of the lens. In this case, we want EXIF:LensMake or EXIF:Make to say "Apple".

2. The model of the lens. In this case, we want EXIF:LensModel or EXIF:Model to include "iPhone" or "iPad".

Easy. Any picture that lacks this metadata will not be included.

What about photos from AirDrop?

This gets complicated. iOS does not provide any mechanism to distinguish between your own photos and photos from others. There is metadata about the model of iPhone that took the photo, but it's too easy for a friend to have the same model. That said, there are two methodologies I found and will leverage both for the tool.

1) Boot Time

There is a combination of fields that can help us identify a specific device:

1. Composite:RunTimeSincePowerUp contains the device's uptime in seconds.

2. EXIF:DateTimeOriginal gives the time the photo was taken.

3. EXIF:OffsetTimeOriginal defines the timezone for the given time.

Not sure why Apple opted to include the device's uptime in each photo taken, but we'll use it! With the uptime and date info available, we can calculate when the device last booted up. We can then group up the photos by their shared boot times.

... Just don't turn your phone on at the exact same second as a friend! But even if you do, it shouldn't matter:

2) Filename Numbers

iOS increments a single counter when numbering all types. Here's an example list:

If we depend on iOS's file-naming behavior (and all files have their original filenames), and combine that knowledge with the Date Taken/Created, we can pretty accurately identify one device from another even if they have overlapping file numbers.

Unfortunately, there needs to be some interaction to make this work. The tool can use the two methodologies to group up files to their respective devices, but it is unable to identify which group is your device.

We could make assumptions like "the device with the most photos must be the owner's", but it's possible that a peer is a far more prolific photographer. Or maybe the device with the most screenshots would be the owner's since it's unlikely that people are AirDropping screenshots... but is that always going to be true? Some owners might not take screenshots at all!

I don't think there are any 100% reliable assumptions we can make, so the best course of action is to prompt the user to check on a photo(s) from each group and answer whether that photo is their own.

Is it an iPhone video?

Just like with photos, not every MOV file is a video taken by your device. And here's the kicker: the video portion of Live Photos is not embedded into the image file (like is the case on some Android phones) and ends up saved as a separate MOV file. I am in the school of thought that these "Live Photo videos" should be kept alongside their corresponding photos and, therefore, should be treated differently from regular video recordings.

Fortunately, with the help of MediaInfo, there are some metadata clues we can use:

1. Video recordings have a comapplequicktimemake field that says "Apple".

2. Live Photo videos have a livephoto field.

AirDrop?

While photos have a field to help group by boot time, videos don't have that data. But the filename methodology is still viable.

Is it a screenshot?

We can use ExifTool to parse two fields:

1. The ICC_Profile:DeviceManufacturer or ICC_Profile:PrimaryPlatform field should say "APPL".

2. The EXIF:UserComment field should say "Screenshot".

AirDrop?

Same with videos, we are limited to using the filename methodology.

Is it a screen recording?

For whatever reason, these files are MP4 instead of MOV, so that is already one differentiator. And of course, to differentiate from any other MP4, we can use metadata.

The performer field should say "ReplayKitRecording".

AirDrop?

We are limited to the filename methodology.

In an ideal world, we shouldn't need this tool.

I just detailed a lot of thought and effort that went into working around a limitation of iOS. I want to make it clear that I wish I didn't need to do this at all. Credit where credit is due, we must compare with the competition. Android allows you to navigate the folder structure and avoids the problem:

A human can navigate to what they want on their own without the need for some tool(s) people put online. Isn't that better?

As things are now though, enjoy my tool! Hope you find it as useful as I do.

To be continued...

As mentioned before, I still need to design and implement support for differentiating between your photos and those shared to you. I will save that for a followup post.

This article is no longer necessary as of Mat931's PR!

Now you can have the behavior of what I called "segmented linear" by specifying the "exact" method:

filters:
  - calibrate_linear:
      method: exact
      datapoints:
          - 10 -> 12
          - 55 -> 50
          - 100 -> 105

Horray!

The original article below is kept for posterity.

ORIGINAL

ESPHome provides a few functions to help calibrate the measurements being reported by the sensors you set up. But I am finding them to be inadequate.

Inadequate how?

The two most relevant functions are calibrate_linear and calibrate_polynomial.

calibrate_linear is pretty straightforward. You give it at least two pairs of numbers and it generates a best-fit straight line for what adjustment it should apply. Let's use an example:

If a watt meter is reporting 10W when it should be 12W, and 100W when it's supposed to be 105W, then you can use this syntax to adjust for the difference:

    filters:
      - calibrate_linear:
          - 10 -> 12
          - 100 -> 105

But what if the sensor's inaccuracy isn't a linear relationship? In the above example, the sensor is consistently measuring a bit too low. But what if it reported 55W when it should be 50W? That relationship is no longer linear, and the output of this calibration will be incorrect at all known points. There was a bug filed for this but was closed... but not thanks to a fix. They simply closed the bug after updating the documentation to say what's wrong.

calibrate_polynomial isn't necessarily suitable either. There is no guarantee that the polynomial equation would be a perfect fit for the known data points. It would be more correct than a linear calibration, but I think that the least a calibration can do is to exactly match the known points.

As stated in the bug, a "piecewise linear fit" would be preferred, and I agree. At least, I agree with my interpretation of it: do calibrate_linear's behavior in between each pair of data points.

Let's implement that!

ESPHome uses a Python-to-C++ mix of code that I don't have the time to get into right now. In the interest of expedience for my own needs, I am just writing a pure C++ function. I also foresee a point of ambiguity that will create some bikeshedding when I do eventually create a PR for ESPHome. The ambiguity is this:

How to handle values outside your known measurements?

Say I have data points at 12, 50, and 105 watts. What should be done for values outside of that range? Like say, 5 or 500 watts.

I think it's fine to take the nearest pair and extend out its linear slope to handle this. But if you have several points, there can appear to be a non-linear relationship. To better match that, it can be valid to say that you want to use the nearest n pairs and get an average slope from that (much like how calibrate_linear works right now), or even switch to calibrate_polynomial.

Let's just implement the simplest one: extending the linear slope of the nearest pair.

float calibrate_segmented_linear(std::vector<std::vector<float>> mapping, float x) {
    float res = x;
    if (x < mapping[0][0]) {
        // Less than mapping
        // Use the left-most pair.
        float before_a = mapping[0][0];
        float after_a = mapping[0][1];
        float before_b = mapping[1][0];
        float after_b = mapping[1][1];
        float before_diff = before_b - before_a;
        float after_diff = after_b - after_a;
        float diff = before_a - x;
        float ratio = diff / before_diff;
        res = after_a - (ratio * after_diff);
    } else if (x > mapping[mapping.size() - 1][0]) {
        // More than mapping
        // Use the right-most pair.
        int i = mapping.size() - 1;
        float before_a = mapping[i-1][0];
        float after_a = mapping[i-1][1];
        float before_b = mapping[i][0];
        float after_b = mapping[i][1];
        float before_diff = before_b - before_a;
        float after_diff = after_b - after_a;
        float diff = x - before_b;
        float ratio = diff / before_diff;
        res = after_b + (ratio * after_diff);
    } else {
        // Within mapping
        // Find and use the pair that x sits between.
        for (int i = 1; i < mapping.size(); i++) {
            float before_a = mapping[i-1][0];
            float after_a = mapping[i-1][1];
            float before_b = mapping[i][0];
            float after_b = mapping[i][1];
            if (x <= before_b) {
                float before_diff = before_b - before_a;
                float after_diff = after_b - after_a;
                float diff = x - before_a;
                float ratio = diff / before_diff;
                res = after_a + (ratio * after_diff);
                break;
            }
        }
    }
    return res;
}

That's it! Just create a header file like calibration.h in ESPHome and import it like this:

esphome:
  name: my_device
  includes:
    - calibration.h

Which can be used in a filter like so:

sensor:
  - platform: a_supported_watt_meter
    update_interval: 1s
    power:
      name: "My Power"
      accuracy_decimals: 1
      filters:
        - lambda: |-
            static std::vector<std::vector<float>> mapping = {
                {10.0, 12.0},
                {55.0, 50.0},
                {100.0, 105.0},
            };
            return calibrate_segmented_linear(mapping, x);

This is not the cleanest code, but it is functional. Would be nice to still be able to use the - 10.0 -> 12.0 syntax, but it gets the job done.

Optimizations can be made such as doing a binary search when x is within the mapping values, but the longest mapping list I have is only 11 items. I don't think binary search is worthwhile when n is so small.

A real-world use-case

This calculation method happens to be the way that the EPA defines their AQI. AQI is calculated based on the measured µg/m³ weight concentration and we can use calibrate_segmented_linear to calculate PM2.5 AQI like so:

lambda: |-
    static std::vector<std::vector<float>> mapping = {
        {0, 0},
        {12.1, 50},
        {35.5, 100},
        {55.5, 150},
        {150.5, 200},
        {250.5, 300},
        {350.5, 400},
        {500.5, 500},
    };
    return calibrate_segmented_linear(mapping, id(pm_2_5).state);

Now that's a whole lot cleaner than this manual if-else logic!