Skip to content

Cannot serialize small floats / denormalized numbers? #333

@SmooreInsight

Description

@SmooreInsight

Disclaimer: I am new to the wilderness of backend and most of the terminology I am using was learned from using Google Gemini as a tutor. If I don't make sense, that's why. I am not clear on what I'm doing, but I do want to be. Suggested learning resources for a deeper dive are welcome.

I have a float64 numpy array obtained from a bin file with the following in python:
file = bin_of_npfloat64
dt = np.dtype('float64')
arr = np.fromfile(
file,
dtype = dt
)
arr = np.ascontiguousarray(arr)
serialized_data = msgpack.packb(arr)

At first I had errors like this from trying to serialize with msgpack to later create a protobuf, but realized that my python version was 3.1 and might need updating for a lot of things.
raise TypeError("Cannot serialize %r" % (obj,))
TypeError: Cannot serialize array([9.88131292e-324, 2.76676762e-322, 7.50979782e-322,...

Now, after the update, I get this error (edited for simplicity).
File ~/anaconda3/lib/python3.11/site-packages/msgpack/init.py:36, in packb(o, **kwargs)
30 def packb(o, **kwargs):
31 """
32 Pack object o and return packed bytes
33
34 See :class:Packer for options.
35 """
---> 36 return Packer(**kwargs).pack(o)
File ... packer.pyx:294, in msgpack._cmsgpack.Packer.pack()
File ... packer.pyx:300, in msgpack._cmsgpack.Packer.pack()
File ... packer.pyx:297, in msgpack._cmsgpack.Packer.pack()
File ... packer.pyx:291, in msgpack._cmsgpack.Packer.pack()
...
TypeError: can not serialize 'numpy.ndarray' object'

Gemini suggests it's a limitation of msgpack being able to handle small values. The small values seemed so extremely small that I first asked my coworker if they were actually our data and not corruption or due to approximations that are unnecessary, but the more experienced coworker insisted that nothing should have changed in how the float64 is handled.

Is this a limitation of what msgpack can handle?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions