issue_comments

Frustrates me when package authors create a "drop in" replacement with the same import name...this kind of thing has bitten me more than once! Would've been nicer I think for bpylist2 to do "import bpylist2 as bpylist"

Yes, I just recreated my virtual environment from scratch and the error went away.

The problem occurred when I ran pip install datasette-bplist in the same virtual environment - https://github.com/simonw/datasette-bplist/blob/master/setup.py depends on bpylist which is incompatible with bpylist2.

Did removing old bpylist solve the original problem or do you still have a photo that throws circular reference?

@simonw does Photos show valid reverse geolocation info? Are you sure you're using bpylist2 and not bpylist? They're both unfortunately imported as "bpylist" so if you somehow got the wrong (original bpylist) version installed, it could be the issue.

Aha! It looks like I accidentally installed the old bplist into the same environment: $ pip freeze | grep bpylist bpylist==0.1.4 bpylist2==3.0.0

@RhetTbull I tried that workaround and it turns out I'm getting this error on ALL of my photos now!

It's weird: a few day ago this wasn't happening. Now it's happening to everything. I'm not sure what I might have changed.

Ugh....Yeah, I think easiest is to catch the exception and return no place as you suggest. This particular bit of code involves un-archiving a serialized NSKeyedArchiver which uses an object table and it is certainly possible to create a circular reference that way. Because this is happening in the decode, the circular reference must be in the original data. Does Photos show valid reverse geolocation info for the photo in question? If so, Photos may be doing something beyond a simple decode of the binary plist. For now, I'll push a patch to catch the exception.

https://github.com/Marketcircle/bpylist/pull/2 looks relevant here.

I'm going to set it up so that Python decimal.Decimal is treated in a FLOAT column, until someone convinces me otherwise!

sqlite-utils currently treats the SQLite NUMERIC concept as a float:

https://github.com/simonw/sqlite-utils/blob/daf2a245aa4e0b0cf62a94c1232cfb858821803b/tests/test_column_affinity.py#L28-L30

From https://www.sqlite.org/datatype3.html#type_affinity :

A column with NUMERIC affinity may contain values using all five storage classes. When text data is inserted into a NUMERIC column, the storage class of the text is converted to INTEGER or REAL (in order of preference) if the text is a well-formed integer or real literal, respectively. If the TEXT value is a well-formed integer literal that is too large to fit in a 64-bit signed integer, it is converted to REAL. For conversions between TEXT and REAL storage classes, only the first 15 significant decimal digits of the number are preserved. If the TEXT value is not a well-formed integer or real literal, then the value is stored as TEXT.

https://www.sqlite.org/datatype3.html#affinity_name_examples suggests that DECIMAL(10,5) should be mapped to the SQLite affinity of NUMERIC - which I've not worked with before.

It definitely makes sense to me that this library should support decimal.Decimal.

Here are the current supported types:

https://github.com/simonw/sqlite-utils/blob/396bee92364fc3a88f6c76969366dd1c4c9c944d/sqlite_utils/db.py#L35-L55

Should decimal.Decimal be stored as a text or float in SQLite I wonder?

So it appears it's possible for photo.place to raise that exception. A workaround could be to catch that and treat those photos as not having a place.

More from the debugger: ```

/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/osxphotos/photoinfo.py(614)place() -> self._place = PlaceInfo5(self._info["reverse_geolocation"]) And: /Users/simon/Dropbox/Development/photos-to-sqlite/photos_to_sqlite/utils.py(91)osxphoto_to_row() -> place = photo.place ```

This has now been resolved and is not present in current version of datasette.

Sample query @simonw mentioned now returns as expected.

https://aggreg8streams.tinyviking.ie/simplestreams?sql=select+*+from+cloudimage+where+%22content_id%22+%3D+%22com.ubuntu.cloud%3Areleased%3Adownload%22+order+by+id+limit+10