- 10687 Views
- 1 replies
- 0 kudos
Resolved! 'Unity Catalog Volumes is not enabled on this instance' error
Hi all,tl;dr I ran the following on a docker-backed personal compute instance (running 13.3-LTS)```%sqlUSE CATALOG hail;USE SCHEMA volumes_testing;CREATE VOLUME 1kg COMMENT 'Testing 1000 Genomes volume';```But this gives```ParseException: [UC_VOLU...
- 10687 Views
- 1 replies
- 0 kudos
- 0 kudos
Resolved with the setting "spark.databricks.unityCatalog.volumes.enabled" = "true"
- 0 kudos
- 3341 Views
- 1 replies
- 0 kudos
Databricks learning festival, but my trial is over
I just emailed the onboarding-help email account to ask for an extension for 2 weeks as I want to complete the Data Engineer course to prepare for my new position. I have 2 accounts where the trial expired, one community account which cannot be used ...
- 3341 Views
- 1 replies
- 0 kudos
- 0 kudos
is what happened when trying to sign up with another email.
- 0 kudos
- 6032 Views
- 1 replies
- 0 kudos
Resolved! Convert multiple string fields to int or long during streaming
Source data looks like: { "IntegrityLevel": "16384", "ParentProcessId": "10972929104936", "SourceProcessId": "10972929104936", "SHA256Hash": "a26a1ffb81a61281ffa55cb7778cc3fb0ff981704de49f75f51f18b283fba7a2", "ImageFileName": "\\Device\\Harddisk...
- 6032 Views
- 1 replies
- 0 kudos
- 0 kudos
Thanks for confirming that the readStream.withColumn() approach is the best available option. Unfortunately, this will force me to maintain a separate notebook for each of the event types, but it does work. I was hoping to create just one paramet...
- 0 kudos
- 5763 Views
- 1 replies
- 0 kudos
unpersist doesn't clear
from pyspark.sql import SparkSessionfrom pyspark import SparkContext, SparkConffrom pyspark.storagelevel import StorageLevelspark = SparkSession.builder.appName('TEST').config('spark.ui.port','4098').enableHiveSupport().getOrCreate()df4 = spark.sql('...
- 5763 Views
- 1 replies
- 0 kudos
- 0 kudos
Thank you so much for taking time and explaining the concepts
- 0 kudos
- 4591 Views
- 2 replies
- 2 kudos
Bug report: the delimiter option does not work when run on DLT
I have a semicolon separated file in an ADLS container that's been added to Unity Catalog as an External location.When I run the following code on an all-purpose cluster, it runs ok and displays the schema.import dlt @dlt.table def test_data_csv(): ...
- 4591 Views
- 2 replies
- 2 kudos
- 2 kudos
@Retired_mod can you confirm that .option("delimiter", ";") is ignored when run in a DLT pipeline? (please see the post above) My colleage confirmed the behavior.
- 2 kudos
- 1223 Views
- 0 replies
- 0 kudos
Question for exam project
For my exam i have to do a small project for the company im interning at. I am creating a datawarehouse where i will have to transfer data from another database, and then transforming it to a star schema. would databricks be good for this, or is it t...
- 1223 Views
- 0 replies
- 0 kudos
- 6472 Views
- 1 replies
- 2 kudos
Get exceptionTraceId details
I'm getting the following error: module.consumer_stage_catalog.databricks_external_location.catalog: Creating... ╷ │ Error: cannot create external location: AWS IAM role does not have READ permissions on url s3://[bucket name]/catalogs. Please conta...
- 6472 Views
- 1 replies
- 2 kudos
- 6103 Views
- 1 replies
- 0 kudos
Resolved! Can we pass parameters thru SQL UDF's?
Is it possible to pass a parameter to a SQL UDF to another SQL UDF that is called by the first SQL UDF?Below is an example where I would like to call tbl_filter() from tbl_func() by passing the tbl_func.a_val parameter to tbl_filter(). Obviously, I c...
- 6103 Views
- 1 replies
- 0 kudos
- 2667 Views
- 0 replies
- 0 kudos
company profile
At Inspired Elements, we redefine living spaces in London, offering bespoke fitted wardrobes and fitted kitchens that seamlessly blend functionality with exquisite design. Our commitment to innovation and quality ensures every piece is a work of art,...
- 2667 Views
- 0 replies
- 0 kudos
- 8744 Views
- 1 replies
- 1 kudos
Resolved! Αdd columns delta table
Hello. Do you know if you can add columns at a specific position (before / after a column) by altering a delta table ?
- 8744 Views
- 1 replies
- 1 kudos
- 1 kudos
yes, using the FIRST or AFTER parameter.https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-alter-table-manage-column.html#add-column
- 1 kudos
- 10167 Views
- 1 replies
- 0 kudos
Pivot on multiple columns
I want to pass multiple column as argument to pivot a dataframe in pyspark pivot likemydf.groupBy("id").pivot("day","city").agg(F.sum("price").alias("price"),F.sum("units").alias("units")).show(). One way I found is to create multiple df with differ...
- 10167 Views
- 1 replies
- 0 kudos
- 9641 Views
- 4 replies
- 4 kudos
Unity catalog - external table lastUpdateversion
We are currently upgrading our Lakehouse to use the Unity Catalog benefits. We will mostly use external tables because alle our DETLA tables are already stored in Azure Storage. I try to figure out how to update the table property "delta.lastUpdateve...
- 9641 Views
- 4 replies
- 4 kudos
- 4 kudos
I am in the same boat.That is the reason I opted to use managed tables instead. OK; it means migrating tables and changing notebooks but besides not having to struggle with external tables, you also get something in return (liquid clustering f.e.).
- 4 kudos
- 1660 Views
- 2 replies
- 0 kudos
checklist for : process to move and deploy in the prod
Hi Team,Could you please help me with best practices to move and deploy (code, workspace, notebooks, etc) in the prod?Regards,Phanindra
- 1660 Views
- 2 replies
- 0 kudos
- 0 kudos
the most important is to use Repos!Link your workspace with git and use feature branches and pull requests to promote code/notebooks.Check the databricks docs on Repos. If you have further questions; shoot.
- 0 kudos
- 6154 Views
- 2 replies
- 0 kudos
Databricks setup/deployment checklist/best practices
Hi Team, could you please share or guide us on any checklist/best practices for Databricks setup/deployment?
- 6154 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Phani1 , here are some best practices https://github.com/Azure/AzureDatabricksBestPractices/tree/master and you could take these points as your "checklist".Choose the right Databricks Workspace:Decide on the appropriate Azure region for your Data...
- 0 kudos
- 3859 Views
- 1 replies
- 0 kudos
Resolved! Read file with Delta Live Tables from external location (Unity Catalog)
As far as I understand, Delta Live Tables should now support reading data from an external location, but I can’t get it to work. I’ve added an ADLS container to Unity Catalog as an external location. There’s a folder in the container containing an ex...
- 3859 Views
- 1 replies
- 0 kudos
- 0 kudos
I misspelled the folder name; I got it working now The error message could have been more informative
- 0 kudos
