issues

Table actions

7 rows where comments = 1 and repo = 197882382 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: user, updated_at, closed_at, author_association, created_at (date), updated_at (date), closed_at (date)

type 2

state 2

repo 1

  • healthkit-to-sqlite · 7 ✖
id node_id number title user state locked assignee milestone comments created_at updated_at ▲ closed_at author_association pull_request body repo type active_lock_reason performed_via_github_app reactions draft state_reason
743071410 MDExOlB1bGxSZXF1ZXN0NTIxMDU0NjEy 13 SQLite does not have case sensitive columns tomaskrehlik 1689944 open 0     1 2020-11-14T20:12:32Z 2021-08-24T13:28:26Z   FIRST_TIME_CONTRIBUTOR dogsheep/healthkit-to-sqlite/pulls/13

This solves a weird issue when there is record with metadata key that is only different in letter cases.

See the test for details.

 healthkit-to-sqlite 197882382 pull     
{
    "url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/13/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 0  
978086284 MDExOlB1bGxSZXF1ZXN0NzE4NzM0MTkx 22 Make sure that case-insensitive column names are unique FabianHertwig 32016596 open 0     1 2021-08-24T13:13:38Z 2021-08-24T13:26:20Z   FIRST_TIME_CONTRIBUTOR dogsheep/healthkit-to-sqlite/pulls/22

This closes #21.

When there are metadata entries with the same case insensitive string, then there is an error when trying to create a new column for that metadata entry in the database table, because a column with that case insensitive name already exists.

xml <Record type="HKQuantityTypeIdentifierDietaryCholesterol" sourceName="MyFitnessPal" sourceVersion="35120" unit="mg" creationDate="2021-07-04 20:55:27 +0200" startDate="2021-07-04 20:55:00 +0200" endDate="2021-07-04 20:55:00 +0200" value="124"> <MetadataEntry key="meal" value="Dinner"/> <MetadataEntry key="Meal" value="Dinner"/> </Record>

The code added in this PR checks if a key already exists in a record and if so adds a number at its end. The resulting column names look like the example below then. Interestingly, the column names viewed with Datasette are not case insensitive.

text startDate, endDate, value, unit, sourceName, sourceVersion, creationDate, metadata_meal, metadata_Meal_2, metadata_Mahlzeit

 healthkit-to-sqlite 197882382 pull     
{
    "url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/22/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 0  
977128935 MDU6SXNzdWU5NzcxMjg5MzU= 21 Duplicate Column FabianHertwig 32016596 open 0     1 2021-08-23T15:00:44Z 2021-08-23T17:00:59Z   NONE  

Hey, thank you for this repo!

When I try to convert my export, I get a multiple column error. Here is the stack trace:

sh (.venv) (base) computer:bodyweight_app user$ healthkit-to-sqlite ./data/Health_export.zip ./data/healthkit.db Importing from HealthKit [###############################-----] 87% 00:00:22 Traceback (most recent call last): File "/MyProject/.venv/bin/healthkit-to-sqlite", line 10, in <module> sys.exit(cli()) File "/MyProject/.venv/lib/python3.7/site-packages/click/core.py", line 829, in __call__ return self.main(*args, **kwargs) File "/MyProject/.venv/lib/python3.7/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/MyProject/.venv/lib/python3.7/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File "/MyProject/.venv/lib/python3.7/site-packages/click/core.py", line 610, in invoke return callback(*args, **kwargs) File "/MyProject/.venv/lib/python3.7/site-packages/healthkit_to_sqlite/cli.py", line 57, in cli convert_xml_to_sqlite(fp, db, progress_callback=bar.update, zipfile=zf) File "/MyProject/.venv/lib/python3.7/site-packages/healthkit_to_sqlite/utils.py", line 41, in convert_xml_to_sqlite write_records(records, db) File "/MyProject/.venv/lib/python3.7/site-packages/healthkit_to_sqlite/utils.py", line 146, in write_records batch_size=50, File "/MyProject/.venv/lib/python3.7/site-packages/sqlite_utils/db.py", line 2579, in insert_all extracts=extracts, File "/MyProject/.venv/lib/python3.7/site-packages/sqlite_utils/db.py", line 1246, in create extracts=extracts, File "/MyProject/.venv/lib/python3.7/site-packages/sqlite_utils/db.py", line 767, in create_table self.execute(sql) File "/MyProject/.venv/lib/python3.7/site-packages/sqlite_utils/db.py", line 421, in execute return self.conn.execute(sql) sqlite3.OperationalError: duplicate column name: metadata_Meal

 healthkit-to-sqlite 197882382 issue     
{
    "url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/21/reactions",
    "total_count": 5,
    "+1": 5,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
470856782 MDU6SXNzdWU0NzA4NTY3ODI= 6 Break up records into different tables for each type simonw 9599 closed 0     1 2019-07-22T01:54:59Z 2019-07-22T03:28:55Z 2019-07-22T03:28:50Z MEMBER  

I don't think there's much benefit to having all of the different record types stored in the same enormous table. Here's what I get when I use _facet=type:

I'm going to try splitting these up into separate tables - so HKQuantityTypeIdentifierBodyMassIndex becomes a table called rBodyMassIndex - and see if that's nicer to work with.

 healthkit-to-sqlite 197882382 issue     
{
    "url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/6/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
470637152 MDU6SXNzdWU0NzA2MzcxNTI= 2 Import workouts simonw 9599 closed 0     1 2019-07-20T05:20:21Z 2019-07-20T06:21:41Z 2019-07-20T06:21:41Z MEMBER  

From #1

 healthkit-to-sqlite 197882382 issue     
{
    "url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/2/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
470640505 MDU6SXNzdWU0NzA2NDA1MDU= 4 Import Records simonw 9599 closed 0     1 2019-07-20T06:11:20Z 2019-07-20T06:21:41Z 2019-07-20T06:21:41Z MEMBER  

From #1: python 'Record': {'attr_counts': {'creationDate': 2672233, 'device': 2665111, 'endDate': 2672233, 'sourceName': 2672233, 'sourceVersion': 2671779, 'startDate': 2672233, 'type': 2672233, 'unit': 2650012, 'value': 2672232}, 'child_counts': {'HeartRateVariabilityMetadataList': 2318, 'MetadataEntry': 287974}, 'count': 2672233, 'parent_counts': {'Correlation': 2, 'HealthData': 2672231}},

 healthkit-to-sqlite 197882382 issue     
{
    "url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/4/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
470637068 MDU6SXNzdWU0NzA2MzcwNjg= 1 Use XML Analyser to figure out the structure of the export XML simonw 9599 closed 0     1 2019-07-20T05:19:02Z 2019-07-20T05:20:09Z 2019-07-20T05:20:09Z MEMBER   healthkit-to-sqlite 197882382 issue     
{
    "url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/1/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issues] (
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [number] INTEGER,
   [title] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [state] TEXT,
   [locked] INTEGER,
   [assignee] INTEGER REFERENCES [users]([id]),
   [milestone] INTEGER REFERENCES [milestones]([id]),
   [comments] INTEGER,
   [created_at] TEXT,
   [updated_at] TEXT,
   [closed_at] TEXT,
   [author_association] TEXT,
   [pull_request] TEXT,
   [body] TEXT,
   [repo] INTEGER REFERENCES [repos]([id]),
   [type] TEXT
, [active_lock_reason] TEXT, [performed_via_github_app] TEXT, [reactions] TEXT, [draft] INTEGER, [state_reason] TEXT);
CREATE INDEX [idx_issues_repo]
                ON [issues] ([repo]);
CREATE INDEX [idx_issues_milestone]
                ON [issues] ([milestone]);
CREATE INDEX [idx_issues_assignee]
                ON [issues] ([assignee]);
CREATE INDEX [idx_issues_user]
                ON [issues] ([user]);