Codecov Report

Merging #330 (fc3de90) into main (718a8f6) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #330   +/-   ##
=======================================
  Coverage   96.59%   96.59%           
=======================================
  Files           5        5           
  Lines        2230     2230           
=======================================
  Hits         2154     2154           
  Misses         76       76           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 718a8f6...fc3de90. Read the comment docs.

Codecov Report

Merging #1463 (2dd94c5) into main (0d5cc20) will decrease coverage by 0.00%.
The diff coverage is n/a.

:exclamation: Current head 2dd94c5 differs from pull request most recent head 481212c. Consider uploading reports for the commit 481212c to get more accurate results

@@            Coverage Diff             @@
##             main    #1463      +/-   ##
==========================================
- Coverage   91.83%   91.83%   -0.01%     
==========================================
  Files          34       34              
  Lines        4422     4421       -1     
==========================================
- Hits         4061     4060       -1     
  Misses        361      361              

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e1012e7...481212c. Read the comment docs.

Codecov Report

Merging #1471 (01b05ed) into main (0d5cc20) will decrease coverage by 0.00%.
The diff coverage is n/a.

:exclamation: Current head 01b05ed differs from pull request most recent head 847238a. Consider uploading reports for the commit 847238a to get more accurate results

@@            Coverage Diff             @@
##             main    #1471      +/-   ##
==========================================
- Coverage   91.83%   91.83%   -0.01%     
==========================================
  Files          34       34              
  Lines        4422     4421       -1     
==========================================
- Hits         4061     4060       -1     
  Misses        361      361              

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a673a93...847238a. Read the comment docs.

Nevermind, had to remove the branch argument in the workflow to make vercel publish work

I am running into the same problem. Is there any workaround?

Any workaround for making it work with datasette-publish-vercel. Currently getting below error

module initialization error: init() got an unexpected keyword argument 'config'
module initialization error
init() got an unexpected keyword argument 'config'

Codecov Report

Merging #1485 (be9aed9) into main (0d5cc20) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1485   +/-   ##
=======================================
  Coverage   91.83%   91.83%           
=======================================
  Files          34       34           
  Lines        4422     4422           
=======================================
  Hits         4061     4061           
  Misses        361      361           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0d5cc20...be9aed9. Read the comment docs.

Codecov Report

Merging #1481 (18a4907) into main (6388617) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1481   +/-   ##
=======================================
  Coverage   91.83%   91.83%           
=======================================
  Files          34       34           
  Lines        4421     4421           
=======================================
  Hits         4060     4060           
  Misses        361      361           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6388617...18a4907. Read the comment docs.

Maybe the exif-loader from the SpatiaLite project could be useful as a reference even it is written in C and it also saves images as blobs https://www.gaia-gis.it/fossil/spatialite-tools/file?name=exif_loader.c&ci=tip. The tool is also integrated into the spatialite-gui application.

I found some user documentation from the web archive http://web.archive.org/web/20180629041238/https://www.gaia-gis.it/spatialite-2.3.1/spatialite-exif-2.3.1.html.

Thanks, but this one has a different error type. Unfortunately, still not working.

My search yielded these four entries: https://github.com/simonw/datasette/issues?q=PermissionError%3A+%5BWinError+32%5D+

Maybe this is the closet hit?
https://github.com/simonw/datasette/issues/744

@simonw any feedback/thoughts?

Hi @simonw - wondering if you might be able to shed some light here. I've seemed to reproduce this issue.
Here's the stacktrace:

...     db["potholes"].insert(pothole, pk='id', alter=True, replace=True)
... 

Traceback (most recent call last):
  File "<stdin>", line 3, in <module>
  File "/Users/patricktrainer/.pyenv/versions/3.9.0/lib/python3.9/site-packages/sqlite_utils/db.py", line 2481, in insert
    return self.insert_all(
  File "/Users/patricktrainer/.pyenv/versions/3.9.0/lib/python3.9/site-packages/sqlite_utils/db.py", line 2596, in insert_all
    self.insert_chunk(
  File "/Users/patricktrainer/.pyenv/versions/3.9.0/lib/python3.9/site-packages/sqlite_utils/db.py", line 2424, in insert_chunk
    row = list(self.rows_where("rowid = ?", [self.last_rowid]))[0]
IndexError: list index out of range

Interesting enough, I found that omitting the pk param does not throw the error.

Let me know how I can help out!

Similar trouble with ageinfo using 0.22. Here's what my ageinfo.js file looks like:

window.YTD.ageinfo.part0 = [
  {
    "ageMeta" : { }
  }
]

Commenting out the registration for ageinfo in archive.py gets my archive to import.

Wow, that was fast! Thank you!

Codecov Report

Merging #1476 (2eb01ff) into main (b28b6cd) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1476   +/-   ##
=======================================
  Coverage   91.83%   91.83%           
=======================================
  Files          34       34           
  Lines        4421     4421           
=======================================
  Hits         4060     4060           
  Misses        361      361           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b28b6cd...2eb01ff. Read the comment docs.

:wave: I just put together a small example using the lambda container image support: https://github.com/sethvincent/datasette-aws-lambda-example

It uses mangum and AWS's python runtime interface client to handle the lambda event stuff.

I'd be happy to help with a publish plugin for AWS lambda as I plan to use this for upcoming projects.

The example uses the serverless cli for deployment but there might be a more suitable deployment approach for the plugin. It would be cool if users didn't have to install anything additional other than the aws cli and its associated config/credentials setup.

Codecov Report

Merging #1385 (db78094) into main (ea627ba) will not change coverage.
The diff coverage is n/a.

:exclamation: Current head db78094 differs from pull request most recent head 8d78c8c. Consider uploading reports for the commit 8d78c8c to get more accurate results

@@           Coverage Diff           @@
##             main    #1385   +/-   ##
=======================================
  Coverage   91.70%   91.70%           
=======================================
  Files          34       34           
  Lines        4364     4364           
=======================================
  Hits         4002     4002           
  Misses        362      362           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ea627ba...8d78c8c. Read the comment docs.

dupe of #293?

Codecov Report

Merging #326 (078a087) into main (77c240d) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #326   +/-   ##
=======================================
  Coverage   96.58%   96.58%           
=======================================
  Files           5        5           
  Lines        2223     2223           
=======================================
  Hits         2147     2147           
  Misses         76       76           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 77c240d...078a087. Read the comment docs.

Additionally, assuming the line numbers match up with the provided enenx file, the mentioned line plus one before and after is as follows:

<![CDATA[>]]>
</span></div>
<div style="padding: 0px; font-family: Arial, sans-serif; font-size: 12px; line-height: 16px; white-space: pre-wrap;"><br style=" padding: 0px;"/>

Codecov Report

Merging #1453 (4f492a7) into main (67cbf0a) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1453   +/-   ##
=======================================
  Coverage   91.83%   91.83%           
=======================================
  Files          34       34           
  Lines        4421     4421           
=======================================
  Hits         4060     4060           
  Misses        361      361           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 67cbf0a...4f492a7. Read the comment docs.

Codecov Report

Merging #1448 (a211747) into main (a1a33bb) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1448   +/-   ##
=======================================
  Coverage   91.82%   91.82%           
=======================================
  Files          34       34           
  Lines        4418     4418           
=======================================
  Hits         4057     4057           
  Misses        361      361           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a1a33bb...a211747. Read the comment docs.

Perceptual hashes might be what you're after : http://phash.org

Codecov Report

Merging #321 (db2dd6d) into main (9258f4b) will increase coverage by 0.16%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main     #321      +/-   ##
==========================================
+ Coverage   96.41%   96.58%   +0.16%     
==========================================
  Files           5        5              
  Lines        2206     2223      +17     
==========================================
+ Hits         2127     2147      +20     
+ Misses         79       76       -3     

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9258f4b...db2dd6d. Read the comment docs.

I'm happy with this functionality left the way you describe. In my case the data is homogeneous but other cases would work just by being consistent on the encoding. Thanks a lot, Simon!

Oh, I misread. Yes some files will not be valid UTF-8, I'd throw a warning and continue (not adding that file) but if you want to get more elaborate you could allow to define a policy on what to do. Not adding the file, index binary content or use a conversion policy like the ones available on Python's decode.

From https://stackoverflow.com/questions/24616678/unicodedecodeerror-in-python-when-reading-a-file-how-to-ignore-the-error-and-ju :
- 'ignore' ignores errors. Note that ignoring encoding errors can lead to data loss.
- 'replace' causes a replacement marker (such as '?') to be inserted where there is malformed data.
- 'surrogateescape' will represent any incorrect bytes as code points in the Unicode Private Use Area ranging from U+DC80 to U+DCFF. These private code points will then be turned back into the same bytes when the surrogateescape error handler is used when writing data. This is useful for processing files in an unknown encoding.
- 'xmlcharrefreplace' is only supported when writing to a file. Characters not supported by the encoding are replaced with the appropriate XML character reference &#nnn;.
- 'backslashreplace' (also only supported when writing) replaces unsupported characters with Python’s backslashed escape sequences.

I was thinking that an approach could be making FILE_COLUMNS a generator (_get_file_columns(mode)) or you can just have a different set of columns (is there something else that makes sense to be changed on the text scenario?).

About UTF-8 I was referring to the encoding to use when reading files. This can be difficult to auto-detect but I believe that UTF-8 is pretty much the standard for text files.

This would fix #21 and make #22 obsolete.

Did not see that #13 fixes the same issue in a similar way. You can decide which one to merge ;)

I think the issue is that I have records like these:

 <Record type="HKQuantityTypeIdentifierDietaryCholesterol" sourceName="MyFitnessPal" sourceVersion="35120" unit="mg" creationDate="2021-07-04 20:55:27 +0200" startDate="2021-07-04 20:55:00 +0200" endDate="2021-07-04 20:55:00 +0200" value="124">
  <MetadataEntry key="meal" value="Dinner"/>
  <MetadataEntry key="Meal" value="Dinner"/>
 </Record>

And if sqlite is case insensitive, then metadata_meal and metadata_Meal result in the same column.

Codecov Report

Merging #247 (af989af) into main (1fe73c8) will decrease coverage by 0.03%.
The diff coverage is 85.71%.

@@            Coverage Diff             @@
##             main     #247      +/-   ##
==========================================
- Coverage   96.28%   96.24%   -0.04%     
==========================================
  Files           5        5              
  Lines        2179     2186       +7     
==========================================
+ Hits         2098     2104       +6     
- Misses         81       82       +1     

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1fe73c8...af989af. Read the comment docs.

Codecov Report

Merging #1433 (ddba6cc) into main (2883098) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1433   +/-   ##
=======================================
  Coverage   91.82%   91.82%           
=======================================
  Files          34       34           
  Lines        4418     4418           
=======================================
  Hits         4057     4057           
  Misses        361      361           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2883098...ddba6cc. Read the comment docs.

Codecov Report

Merging #1430 (9419947) into main (b1fed48) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##             main    #1430   +/-   ##
=======================================
  Coverage   91.71%   91.71%           
=======================================
  Files          34       34           
  Lines        4417     4418    +1     
=======================================
+ Hits         4051     4052    +1     
  Misses        366      366           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b1fed48...9419947. Read the comment docs.

This looks awesome. We'll definitely make extensive use of this feature!

On Thu, Aug 12, 2021 at 5:52 PM Simon Willison @.***>
wrote:

I like this. Need to solve for mobile though where the cog menu isn't
visible - I think I'll do that with a definition list at the top of the
page.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/simonw/datasette/issues/942#issuecomment-898022235,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAERSNYEF6QRZO2HRJGRIWDT4RGDFANCNFSM4QEC6ATA
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email
.

--
Zane A. Selvans, PhD
Chief Data Wrangler
Catalyst Cooperative
https://catalyst.coop
@.***
Signal/WhatsApp/SMS: +1 720 443 1363
Twitter: @ZaneSelvans https://twitter.com/ZaneSelvans
PGP https://www.gnupg.org/: 0x64F7B56F3A127B04

Actually, I forgot to include the bib_pub_year in the extract ...

But also, I tried again with empty string values instead of NULL values and it seems to place the foreign key properly / correctly...

sql = """\
INSERT INTO "circulation_info" ("item_id", "bib_title", "bib_creator", "bib_format", "bib_pub_year", "checkout_date")
VALUES
(1, "title one", "creator one", "Book", 2018, "2021-08-12 00:01"),
(2, "title two", "creator one", "Book", 2019, "2021-08-12 00:02"),
(3, "title three", "", "DVD", 2020, "2021-08-12 00:03"),
(4, "title four", "", "DVD", "", "2021-08-12 00:04"),
(5, "title five", "", "DVD", "", "2021-08-12 00:05")
"""

with sqlite3.connect('test_bib_2.db') as con:
    con.execute(sql)

db["circulation_info"].extract(
    [
        "bib_title",
        "bib_creator",
        "bib_format",
        "bib_pub_year"
    ],
    table="bib_info", 
    fk_column="bib_info_id"
)

{'id': 1, 'item_id': 1, 'bib_info_id': 1, 'bib_pub_year': 2018, 'checkout_date': '2021-08-12 00:01'}
{'id': 2, 'item_id': 2, 'bib_info_id': 2, 'bib_pub_year': 2019, 'checkout_date': '2021-08-12 00:02'}
{'id': 3, 'item_id': 3, 'bib_info_id': 3, 'bib_pub_year': 2020, 'checkout_date': '2021-08-12 00:03'}
{'id': 4, 'item_id': 4, 'bib_info_id': 4, 'bib_pub_year': '', 'checkout_date': '2021-08-12 00:04'}
{'id': 5, 'item_id': 5, 'bib_info_id': 5, 'bib_pub_year': '', 'checkout_date': '2021-08-12 00:05'}

---

{'id': 1, 'bib_title': 'title one', 'bib_creator': 'creator one', 'bib_format': 'Book'}
{'id': 2, 'bib_title': 'title two', 'bib_creator': 'creator one', 'bib_format': 'Book'}
{'id': 3, 'bib_title': 'title three', 'bib_creator': '', 'bib_format': 'DVD'}
{'id': 4, 'bib_title': 'title four', 'bib_creator': '', 'bib_format': 'DVD'}
{'id': 5, 'bib_title': 'title five', 'bib_creator': '', 'bib_format': 'DVD'}

I think I ran into an issue that's perhaps related with extract()

I have a case where I want to create a lookup table for all the related title data where there are possibly multiple null values in the related columns ....

sql = """\
INSERT INTO "circulation_info" ("item_id", "bib_title", "bib_creator", "bib_format", "bib_pub_year", "checkout_date")
VALUES
(1, "title one", "creator one", "Book", 2018, "2021-08-12 00:01"),
(2, "title two", "creator one", "Book", 2019, "2021-08-12 00:02"),
(3, "title three", NULL, "DVD", 2020, "2021-08-12 00:03"),
(4, "title four", NULL, "DVD", NULL, "2021-08-12 00:04"),
(5, "title five", NULL, "DVD", NULL, "2021-08-12 00:05")
"""

with sqlite3.connect('test_bib.db') as con:
    con.execute(sql)

when I run the extract() method ...

db["circulation_info"].extract(
    [
        "bib_title",
        "bib_creator",
        "bib_format"        
    ],
    table="bib_info", 
    fk_column="bib_info_id"
)

db = sqlite_utils.Database("test_bib.db")

for row in db["circulation_info"].rows:
    print(row)

print("\n---\n")

for row in db["bib_info"].rows:
    print(row)

results in this ..

{'id': 1, 'item_id': 1, 'bib_info_id': 1, 'bib_pub_year': 2018, 'checkout_date': '2021-08-12 00:01'}
{'id': 2, 'item_id': 2, 'bib_info_id': 2, 'bib_pub_year': 2019, 'checkout_date': '2021-08-12 00:02'}
{'id': 3, 'item_id': 3, 'bib_info_id': None, 'bib_pub_year': 2020, 'checkout_date': '2021-08-12 00:03'}
{'id': 4, 'item_id': 4, 'bib_info_id': None, 'bib_pub_year': None, 'checkout_date': '2021-08-12 00:04'}
{'id': 5, 'item_id': 5, 'bib_info_id': None, 'bib_pub_year': None, 'checkout_date': '2021-08-12 00:05'}

---

{'id': 1, 'bib_title': 'title one', 'bib_creator': 'creator one', 'bib_format': 'Book'}
{'id': 2, 'bib_title': 'title two', 'bib_creator': 'creator one', 'bib_format': 'Book'}
{'id': 3, 'bib_title': 'title three', 'bib_creator': None, 'bib_format': 'DVD'}
{'id': 4, 'bib_title': 'title four', 'bib_creator': None, 'bib_format': 'DVD'}
{'id': 5, 'bib_title': 'title five', 'bib_creator': None, 'bib_format': 'DVD'}

Seems like it's correctly generating the row data for those lookups, but it's not correctly updating the foreign key back to the primary table? Looks like it just results in a NULL value in that original table.

Any ideas on why? Thanks again!

I added parsing of text/html emails using BeautifulSoup.

Around half of the emails in my archive don't include a text/plain payload so adding html parsing makes a good chunk of them searchable.

Codecov Report

Merging #312 (43bc064) into main (ee469e3) will decrease coverage by 0.02%.
The diff coverage is 96.84%.

@@            Coverage Diff             @@
##             main     #312      +/-   ##
==========================================
- Coverage   96.30%   96.28%   -0.03%     
==========================================
  Files           5        5              
  Lines        2168     2179      +11     
==========================================
+ Hits         2088     2098      +10     
- Misses         80       81       +1     

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ee469e3...43bc064. Read the comment docs.

Just added two more fixes:

• Added parsing for rfc 2047 encoded unicode headers
• Body is now stored as TEXT rather than a BLOB regardless of what order the messages are parsed in.

I was able to run this on my Takeout export and everything seems to work fine. @simonw let me know if this looks good to merge.

there's a lot of complexity here, that's probably not worth addressing. i got what i needed by patching the dockerfile that cloudrun uses to install a newer version of sqlite.

yes, filter clause on aggregate queries were added to sqlite3 in 3.30

Codecov Report

Merging #1418 (532170f) into main (54b6e96) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1418   +/-   ##
=======================================
  Coverage   91.64%   91.64%           
=======================================
  Files          34       34           
  Lines        4382     4382           
=======================================
  Hits         4016     4016           
  Misses        366      366           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 54b6e96...532170f. Read the comment docs.

@simonw Awesome; thanks so much!

Codecov Report

Merging #303 (4c3bf97) into main (c7e8d72) will increase coverage by 0.24%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main     #303      +/-   ##
==========================================
+ Coverage   96.04%   96.28%   +0.24%     
==========================================
  Files           4        5       +1     
  Lines        1998     2129     +131     
==========================================
+ Hits         1919     2050     +131     
  Misses         79       79              

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c7e8d72...4c3bf97. Read the comment docs.

Does the library support this now?

I'm not sure why but my most recent import, when displayed in Datasette, looks like this:

https://user-images.githubusercontent.com/9599/109985836-0ab00080-7cba-11eb-97d5-0631a0835b61.png">

I did some investigation into this issue and made a fix here. The problem was that some messages (like gchat logs) don't have a Message-Id and we need to use X-GM-THRID as the pkey instead.

@simonw While looking into this I found something unexpected about how sqlite_utils handles upserts if the pkey column is None. When the pkey is NULL I'd expect the function to either use rowid or throw an exception. Instead, it seems upsert_all creates a row where all columns are NULL instead of using the values provided as parameters.

@simonw is this PR of interest to you?

Hi @maxhawkins , I'm sorry, I haven't had any time to work on this. I'll have some time tomorrow to test your commits. I think they look great. I'm great with your commits superseding my initial attempt here.

I added a follow-up commit that deals with emails that don't have a Date header: https://github.com/maxhawkins/google-takeout-to-sqlite/commit/4bc70103582c10802c85a523ef1e99a8a2154aa9

One thing I noticed is this importer doesn't save attachments along with the body of the emails. It would be nice if those got stored as blobs in a separate attachments table so attachments can be included while fetching search results.

How does this commit look? https://github.com/maxhawkins/google-takeout-to-sqlite/commit/72802a83fee282eb5d02d388567731ba4301050d

It seems that Takeout's mbox format is pretty simple, so we can get away with just splitting the file on lines begining with From. My commit just splits the file every time a line starts with From and uses email.message_from_bytes to parse each chunk.

I was able to load a 12GB takeout mbox without the program using more than a couple hundred MB of memory during the import process. It does make us lose the progress bar, but maybe I can add that back in a later commit.

ordered lists didn't work either, btw

The solution here is to upload an image to the bucket first. It is caused because it does not properly handle the case when there are no images in the bucket.

Codecov Report

Merging #1296 (527a056) into main (c73af5d) will decrease coverage by 0.11%.
The diff coverage is n/a.

:exclamation: Current head 527a056 differs from pull request most recent head 8f00c31. Consider uploading reports for the commit 8f00c31 to get more accurate results

@@            Coverage Diff             @@
##             main    #1296      +/-   ##
==========================================
- Coverage   91.62%   91.51%   -0.12%     
==========================================
  Files          34       34              
  Lines        4371     4255     -116     
==========================================
- Hits         4005     3894     -111     
+ Misses        366      361       -5     

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c73af5d...8f00c31. Read the comment docs.

Codecov Report

Merging #1400 (e95c685) into main (c73af5d) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1400   +/-   ##
=======================================
  Coverage   91.62%   91.62%           
=======================================
  Files          34       34           
  Lines        4371     4371           
=======================================
  Hits         4005     4005           
  Misses        366      366           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c73af5d...e95c685. Read the comment docs.

I also love the idea for this feature and wonder if it could work without having to download the whole database into memory at once if it's a rather large db. Obviously this could be slower but could have many use cases.

My comment is partially inspired by this post about streaming sqlite dbs from github pages or such
https://news.ycombinator.com/item?id=27016630

Same here for US West (N. California) us-west-1. Running on Catalina.

Hi Simon

Thank you for adding the escape_function, but it does not work on my datasette-installation (0.33). I've added the following file to my datasette-dir: /plugins/sql_functions.py:

from datasette import hookimpl

def escape_fts_query(query):
    bits = query.split()
    return ' '.join('"{}"'.format(bit.replace('"', '')) for bit in bits)

@hookimpl
def prepare_connection(conn):
    conn.create_function("escape_fts_query", 1, escape_fts_query)`

It has no effect on the standard queries to the tables though, as they still produce errors when including any characters like '-', '/', '+' or '?'

Does the function only work when using costum queries, where I can include the escape_fts-function explicitly in the sql-query?

PS. I'm calling datasette with --plugins=plugins, and my other plugins work just fine.
PPS. The fts5 virtual table is created with 'sqlite3' like so:

CREATE VIRTUAL TABLE "cases_fts" USING FTS5( title, subtitle, resume, suggestion, presentation, detail = full, content_rowid = 'id', content = 'cases', tokenize='unicode61', 'remove_diacritics 2', 'tokenchars "-_"' );

Thanks!

P.s. wondering if you have explored using the spatialite functionality with the location data in workouts?

Hi Simon -- just experimenting with your excellent software!

Up to this point in time I have been using the (paid) HealthFit App to export my workouts from my Apple Watch, one walk at the time into either .GPX or .FIT format and then using another library to suck it into Python and eventually here to my "Emmaus Walking" app:

https://share.streamlit.io/mjboothaus/emmaus_walking/emmaus_walking/app.py

I just used healthkit-to-sqlite to convert my export.zip file and it all "just worked".

I did notice the issue with various numeric fields being stored in the SQLite db as TEXT for now and just thought I'd flag it - but you're already self-reported this issue.

Keep up the great work!

I was curious if you have any thoughts about periodically exporting "export.zip" and how to just update the SQLite file instead of re-creating it each time. Hopefully Apple will give some thought to managing this data in a more sensible fashion as it grows over time. Ideally one could pull it from iCloud (where it is allegedly being backed up).

Codecov Report

Merging #868 (b452fcb) into master (0005281) will increase coverage by 0.49%.
The diff coverage is 96.19%.

:exclamation: Current head b452fcb differs from pull request most recent head c99caba. Consider uploading reports for the commit c99caba to get more accurate results

@@            Coverage Diff             @@
##           master     #868      +/-   ##
==========================================
+ Coverage   82.91%   83.40%   +0.49%     
==========================================
  Files          26       27       +1     
  Lines        3547     3634      +87     
==========================================
+ Hits         2941     3031      +90     
+ Misses        606      603       -3     

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 180c7a5...c99caba. Read the comment docs.

I do like the idea of there being a option for turning that on by default so that you could use those terms in the default "Search" bar presented when you browse to a table where FTS has been enabled. Maybe even a small inline pop up with a short bit explaining the FTS feature and the keywords (e.g. case matters). What are the side-effects of turning that on in the query string, or even by default as you suggested? I see that you stated in the docs... "to ensure they do not cause any confusion for users who are not aware of them", but I'm not sure what those could be.

Isn't it the case that those keywords are only picked up by sqlite in where you're using the MATCH clause?

Seems like a really powerful feature (even though there are a lot of hurdles around setting it up in the sqlite db ... sqlite-utils makes that so simple by the way!)

I had setup a full text search on my instance of Datasette for title data for our public library, and was noticing that some of the features of the SQLite FTS weren't working as expected ... and maybe the issue is in the escape_fts() function

vs removing the function...

Also, on the issue of sorting by rank by default .. perhaps something like this could work for the baked-in default SQL query for Datasette?

link to the above search in my instance of Datasette

Thanks Simon for nailing that one down! It does seem a little confusing that the ProxyPreservehost option is set to Off By default, but this config totally did the trick and fixed the issue

<Location /collection-analysis/>
   ProxyPass http://127.0.0.1:8010/collection-analysis/
   ProxyPreservehost On
</Location>

Codecov Report

Merging #1386 (e974ed1) into main (ea627ba) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1386   +/-   ##
=======================================
  Coverage   91.70%   91.70%           
=======================================
  Files          34       34           
  Lines        4364     4364           
=======================================
  Hits         4002     4002           
  Misses        362      362           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ea627ba...e974ed1. Read the comment docs.

Codecov Report

Merging #293 (ae0f46a) into main (747be60) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #293   +/-   ##
=======================================
  Coverage   96.03%   96.03%           
=======================================
  Files           4        4           
  Lines        1994     1994           
=======================================
  Hits         1915     1915           
  Misses         79       79           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 747be60...ae0f46a. Read the comment docs.

Great!

Codecov Report

Merging #1378 (0c132d1) into main (83e9c8b) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1378   +/-   ##
=======================================
  Coverage   91.68%   91.68%           
=======================================
  Files          34       34           
  Lines        4340     4340           
=======================================
  Hits         3979     3979           
  Misses        361      361           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 83e9c8b...0c132d1. Read the comment docs.

Codecov Report

Merging #277 (abbd324) into main (a19ce1a) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##             main     #277   +/-   ##
=======================================
  Coverage   96.06%   96.06%           
=======================================
  Files           4        4           
  Lines        1828     1829    +1     
=======================================
+ Hits         1756     1757    +1     
  Misses         72       72           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a19ce1a...abbd324. Read the comment docs.

Codecov Report

:exclamation: No coverage uploaded for pull request base (main@78aebb6). Click here to learn what that means.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #273   +/-   ##
=======================================
  Coverage        ?   96.10%           
=======================================
  Files           ?        4           
  Lines           ?     1873           
  Branches        ?        0           
=======================================
  Hits            ?     1800           
  Misses          ?       73           
  Partials        ?        0           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 78aebb6...df7a37b. Read the comment docs.

Sorry, I got confused because SQLite has a JSON column type, even if it is treated as TEXT, and I though automatic facets were available for JSON arrays stored as JSON only :)

So, I do things like this a lot, too. I like the idea of piping in from stdin. Something like this would be nice to do in a makefile:

cat file.csv | sqlite-utils --csv --table data - 'SELECT * FROM data WHERE col="whatever"' > filtered.csv

If you assumed that you're always piping out the same format you're piping in, the option names don't have to change. Depends how much you want to change formats.

@simonw -- i've created an omega-query that fetched most of what was interesting to me for a single user.

found by poking around in the "Explorer" tab in https://docs.github.com/en/graphql/overview/explorer

note: pagination is still required via first and last but it seems to allow unlimited history.

query MyQuery {
  __typename
  user(login: "<user>") {
    id
    pinnedItems(first: 100) {
      edges {
        node
      }
    }
    pullRequests(first: 100) {
      nodes {
        body
        title
        state
        createdAt
      }
    }
    createdAt
    issues(first: 100) {
      pageInfo {
        endCursor
        startCursor
      }
      nodes {
        title
        url
        createdAt
        body
      }
    }
    issueComments(first: 100) {
      edges {
        node {
          id
          updatedAt
          url
          body
        }
      }
    }
    repositories(first: 100) {
      nodes {
        createdAt
        description
        parent {
          name
        }
        pinnedIssues(first: 100) {
          edges {
            node {
              id
            }
          }
        }
        pinnedDiscussions(first: 100) {
          edges {
            node {
              id
            }
          }
        }
      }
    }
    starredRepositories(first: 100) {
      edges {
        node {
          id
        }
      }
    }
  }
}

it looks like the v4 GraphQL API is the only way to get data beyond 90 days from GitHub.

this is significant change, but may be worth considering in the future.

i have a basic working version at https://github.com/khimaros/github-to-sqlite

this can be tested with github-to-sqlite events.db khimaros/events

caveat: the GitHub API doesn't seem to provide a complete history of events.

There is a feature for this at the moment, but it's a little bit hidden: you can use ?_json=col to tell
Datasette that you would like a specific column to be exported as nested JSON: https://docs.datasette.io/en/stable/json_api.html#special-json-arguments

Thanks :)

I considered trying to make this automatic - so it detects columns that appear to contain valid JSON and outputs them as nested objects - but the problem with that is that it can lead to inconsistent results - you might hit the API and find that not every column contains valid JSON (compared to the previous day) resulting in the API retuning string instead of the expected dictionary and breaking your code.

If a developer is not sure if the JSON fields are valid, but then retrieves and parse them, it should handle errors too. Handling inconsistent data is necessary due to the nature of SQLite. A global or dataset option to render the data as they have been defined (JSON, boolean, etc.) when requesting JSON could allow the user to download a regular JSON from the browser without having to rely on APIs. I would guess someone could just make a custom template with an extra JSON-parsed download button otherwise :)

Codecov Report

Merging #1373 (d117ba7) into main (03418ee) will increase coverage by 7.65%.
The diff coverage is 93.29%.

:exclamation: Current head d117ba7 differs from pull request most recent head 51ff366. Consider uploading reports for the commit 51ff366 to get more accurate results

@@            Coverage Diff             @@
##             main    #1373      +/-   ##
==========================================
+ Coverage   84.02%   91.68%   +7.65%     
==========================================
  Files          28       34       +6     
  Lines        3774     4340     +566     
==========================================
+ Hits         3171     3979     +808     
+ Misses        603      361     -242     

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e797565...51ff366. Read the comment docs.

I'm having the same issue (same versions of python and twitter-to-sqlite). It's the user-timeline command. Other commands are working.

@mroswell That is a very nice solution. I wonder if custom classes, like col-columnName-value could be automatically added to cells when facets on such column are enabled, to allow custom styling without having to modify templates or add custom JavaScript code.

sqlite-utils transform does not allow setting the column type to boolean:

Error: Invalid value for '--type': 'bool' is not one of 'INTEGER', 'TEXT', 'FLOAT', 'BLOB'.

Sure, I am sorry if my message hasn't been clear enough. I am also a new user :)

At the beginning, I just call sqlite-utils insert "$db" "$table" "$jsonfile" to create the database. sqlite-utils convert JSON values into TEXT, when it tries to determine the schema automatically. I then try to transform the table to set JSON as type:

sqlite-utils transform species.sqlite species --type criteria json
Usage: sqlite-utils transform [OPTIONS] PATH TABLE
Try 'sqlite-utils transform --help' for help.

Error: Invalid value for '--type': 'json' is not one of 'INTEGER', 'TEXT', 'FLOAT', 'BLOB'.

true and false json values are cast to integer, which is not optimal.

Codecov Report

Merging #1374 (0ef0dd5) into main (cd7678f) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1374   +/-   ##
=======================================
  Coverage   91.68%   91.68%           
=======================================
  Files          34       34           
  Lines        4340     4340           
=======================================
  Hits         3979     3979           
  Misses        361      361           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cd7678f...0ef0dd5. Read the comment docs.

any fixes for that recursive issue with temp file? I get it using both heroku and cloudrun, although it seems to still publish and deploy fine

shell=True is added to line 56 (I guess it used to be 54) of heroku.py as detailed in the original issue. (for posterity)

Presumably this would also require adding Content-Security-Policy to the Vary header though, which will have a nasty effect on Cloudflare and Fastly and such like.

No, because Vary header is about request headers that cause the response to vary, not response headers.

Codecov Report

Merging #1291 (8789157) into main (0a7621f) will not change coverage.
The diff coverage is n/a.

:exclamation: Current head 8789157 differs from pull request most recent head 9bf089f. Consider uploading reports for the commit 9bf089f to get more accurate results

@@           Coverage Diff           @@
##             main    #1291   +/-   ##
=======================================
  Coverage   91.51%   91.51%           
=======================================
  Files          34       34           
  Lines        4255     4255           
=======================================
  Hits         3894     3894           
  Misses        361      361           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0a7621f...9bf089f. Read the comment docs.

Codecov Report

Merging #1357 (1b27643) into main (6e9b07b) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##             main    #1357   +/-   ##
=======================================
  Coverage   91.71%   91.72%           
=======================================
  Files          34       34           
  Lines        4333     4336    +3     
=======================================
+ Hits         3974     3977    +3     
  Misses        359      359           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6e9b07b...1b27643. Read the comment docs.

Codecov Report

Merging #1346 (3bffc35) into main (7b106e1) will increase coverage by 0.14%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main    #1346      +/-   ##
==========================================
+ Coverage   91.56%   91.71%   +0.14%     
==========================================
  Files          34       34              
  Lines        4282     4332      +50     
==========================================
+ Hits         3921     3973      +52     
+ Misses        361      359       -2     

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7b106e1...3bffc35. Read the comment docs.

Codecov Report

Merging #1348 (56cba8f) into main (7b106e1) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1348   +/-   ##
=======================================
  Coverage   91.56%   91.56%           
=======================================
  Files          34       34           
  Lines        4282     4282           
=======================================
  Hits         3921     3921           
  Misses        361      361           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7b106e1...56cba8f. Read the comment docs.

Any updates?

Codecov Report

Merging #1319 (cfca570) into main (1b69753) will increase coverage by 0.02%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main    #1319      +/-   ##
==========================================
+ Coverage   91.51%   91.53%   +0.02%     
==========================================
  Files          34       34              
  Lines        4255     4255              
==========================================
+ Hits         3894     3895       +1     
+ Misses        361      360       -1     

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1b69753...cfca570. Read the comment docs.

Codecov Report

Merging #1339 (3d8598b) into main (a443dba) will increase coverage by 0.01%.
The diff coverage is 91.66%.

:exclamation: Current head 3d8598b differs from pull request most recent head 387c837. Consider uploading reports for the commit 387c837 to get more accurate results

@@            Coverage Diff             @@
##             main    #1339      +/-   ##
==========================================
+ Coverage   91.55%   91.56%   +0.01%     
==========================================
  Files          34       34              
  Lines        4262     4280      +18     
==========================================
+ Hits         3902     3919      +17     
- Misses        360      361       +1     

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a443dba...387c837. Read the comment docs.

I've also been investigating serving postgresql databases over postgrest. I like the idea of hosting some static html + js on github, but having it backed by datasets I can update and control on the database server. I started from SQLite + datasette but would like to host larger datasets (with smaller materialized views exposed publicly). I think the postgrest model where all the authorization and ownership is defined in database role grants is really powerful. But I really miss being able to define an ad-hoc query in sql, then instantly link to a json representation of it like datasette does.

P.S.: I've been sort of following along as you pop up in hacker news here and there. It's been great! Thanks for doing this all out in the open!

wow, thanks a lot @simonw , problem is solved. I converted my current json file into utf-8 format with Python script. It's working now. I'm using with Window 10.

would be nice to have

I'm stuck also because datasette wants itsdangerous~=1.1 instead of allowing itsdangerous-2.0.0

Thanks Simon this is working very well.

Any chance you could push a new datasette release with the updated dependencies in the setup.py, @simonw?